Skip to content

Troubleshoot the failing boundary

Start with the failed decision, its exact input, and the authority that made it. “Deployment failed” can mean a build never published an image, a signature identity mismatched, admission denied a Pod, or Argo CD could not reconcile. Those failures require different repairs. Preserve the original error and timestamp before changing configuration.

Diagnose the failed decision
  • Identify the boundary

    Is failure in publishing, verification, admission or runtime?

  • Authentication failure

    Check OIDC audience, federation, IAM and both registry stores.

  • Missing or mismatched claim

    Check exact digest, issuer/subject and statement fields.

  • Policy or reconciliation

    Check image match, namespace exclusions, webhook and Argo desired state.

  • Runtime observation

    Check Falco sensor, rule match and separate optional routing.

  • Identify the boundary Authentication failureaccess denied
  • Identify the boundary Missing or mismatched claimverification fails
  • Identify the boundary Policy or reconciliationPod or sync fails
  • Identify the boundary Runtime observationexpected event absent
Resolve the failed boundary without weakening trust checks. An admission outage is not equivalent to an ordinary policy denial.
Text equivalent and limits
  1. Is failure in publishing, verification, admission or runtime?
  2. Check OIDC audience, federation, IAM and both registry stores.
  3. Check exact digest, issuer/subject and statement fields.
  4. Check image match, namespace exclusions, webhook and Argo desired state.
  5. Check Falco sensor, rule match and separate optional routing.

Read the labeled relationships as the stated contract. Optional connections are dashed and explicitly labeled; comparisons do not imply a sequential handoff. This figure explains inspected code or recorded observations, not current cloud execution.

Cloud authentication or registry access fails

Section titled “Cloud authentication or registry access fails”

Confirm the job event/ref and repository identity, then provider name, CI service-account email, and repository-specific writer grants. The provider’s repository condition is distinct from Deploy’s main-only job condition. For Kyverno, confirm both the KSA-to-GSA binding and the annotation on the actual admission-controller account. A writer can publish an artifact while a reader still cannot fetch its metadata.

Check application and metadata repository addresses independently. A 401/403 fetching metadata is an access failure; it does not prove the image lacks a signature. Repair the narrow grant or binding after review, then recheck with the intended reader. Do not substitute a JSON key to bypass diagnosis.

Compare the actual verified certificate subject with the consumer’s expected sign-attest.yml@refs/heads/main identity. deploy.yml invokes the reusable signer; it is not the signing workflow’s certificate subject. A fork, tag, or workflow rename intentionally changes the contract. Run the local identity consistency script, then inspect the actual certificate and provenance without exposing OIDC token material.

The baseline JMESPath conditions evaluate inside the predicate body. Expressions start at invocation.configSource and builder, not at predicate.invocation. The expected entry point is the signer. The source-backed Python regression test evaluates these exact shipped expressions against a captured predicate. Passing it does not establish that a current registry statement has the same contents.

Use the verification table to identify whether the failure concerns builder, workflow entry point, source URI, commit, materials, or digest. CI validates commit and material bindings explicitly; admission’s listed conditions are narrower.

The checked-in Kyverno Helm value is config.maxContextSize: 8Mi. Historical troubleshooting describes duplicate registry patterns and a real SBOM exceeding earlier settings. Inspect the actual installed values and downloaded attestation size before raising the budget. A context error is a verifier-processing failure, not a demonstrated untrusted artifact.

Prefer a reviewed Helm value change over an untracked ConfigMap patch that a later upgrade may overwrite. Do not copy older Docker Hub addresses or historical 5Mi examples into the final GCP configuration: the pinned policy uses one GAR scope and the checked-in values use 8 MiB.

Read the Application’s actual repository, target revision, chart path, diff, and resource error. main at the historical moment and main today are different sources. A denial reported during sync is Kyverno’s decision; Argo CD does not verify signatures. A manual live image edit can be reverted by self-heal. Repair the reviewed desired state rather than repeatedly patching a Deployment.

The baseline Falco values set rule_matching: all because an earlier broad rule could consume an event first. Check mounted rule files, enabled rule status, kernel driver health, event collection, namespace exclusions, and the actual process name. The custom rule excludes kube-system and falco-system; its name does not itself check signatures.

Inspect deployed webhook failurePolicy, selectors, endpoints, and the installed engine/chart version. validationFailureAction: Enforce describes policy violations; it does not alone define outage handling. The source runbook records older Gatekeeper deadlocks and suggests webhook deletion, but Gatekeeper is inactive in this final baseline. Treat bypassing any active webhook as an exceptional, reviewed recovery step with documented exposure and restoration checks. The owner capture set does not include an outage/recovery experiment.

Historical troubleshooting notes, final Kyverno values, and final Falco configuration.