Troubleshoot the failing boundary

Start with the failed decision, its exact input, and the authority that made it. “Deployment failed” can mean a build never published an image, a signature identity mismatched, admission denied a Pod, or Argo CD could not reconcile. Those failures require different repairs. Preserve the original error and timestamp before changing configuration.

Cloud authentication or registry access fails

Confirm the job event/ref and repository identity, then provider name, CI service-account email, and repository-specific writer grants. The provider’s repository condition is distinct from Deploy’s main-only job condition. For Kyverno, confirm both the KSA-to-GSA binding and the annotation on the actual admission-controller account. A writer can publish an artifact while a reader still cannot fetch its metadata.

Check application and metadata repository addresses independently. A 401/403 fetching metadata is an access failure; it does not prove the image lacks a signature. Repair the narrow grant or binding after review, then recheck with the intended reader. Do not substitute a JSON key to bypass diagnosis.

“No expected identity matched”

Compare the actual verified certificate subject with the consumer’s expected sign-attest.yml@refs/heads/main identity. deploy.yml invokes the reusable signer; it is not the signing workflow’s certificate subject. A fork, tag, or workflow rename intentionally changes the contract. Run the local identity consistency script, then inspect the actual certificate and provenance without exposing OIDC token material.

Provenance rejects a trusted image

The baseline JMESPath conditions evaluate inside the predicate body. Expressions start at invocation.configSource and builder, not at predicate.invocation. The expected entry point is the signer. The source-backed Python regression test evaluates these exact shipped expressions against a captured predicate. Passing it does not establish that a current registry statement has the same contents.

Use the verification table to identify whether the failure concerns builder, workflow entry point, source URI, commit, materials, or digest. CI validates commit and material bindings explicitly; admission’s listed conditions are narrower.

Admission reports a context-size error

The checked-in Kyverno Helm value is config.maxContextSize: 8Mi. Historical troubleshooting describes duplicate registry patterns and a real SBOM exceeding earlier settings. Inspect the actual installed values and downloaded attestation size before raising the budget. A context error is a verifier-processing failure, not a demonstrated untrusted artifact.

Prefer a reviewed Helm value change over an untracked ConfigMap patch that a later upgrade may overwrite. Do not copy older Docker Hub addresses or historical 5Mi examples into the final GCP configuration: the pinned policy uses one GAR scope and the checked-in values use 8 MiB.

Argo CD is out of sync or unhealthy

Read the Application’s actual repository, target revision, chart path, diff, and resource error. main at the historical moment and main today are different sources. A denial reported during sync is Kyverno’s decision; Argo CD does not verify signatures. A manual live image edit can be reverted by self-heal. Repair the reviewed desired state rather than repeatedly patching a Deployment.

A loaded runtime rule produces no event

The baseline Falco values set rule_matching: all because an earlier broad rule could consume an event first. Check mounted rule files, enabled rule status, kernel driver health, event collection, namespace exclusions, and the actual process name. The custom rule excludes kube-system and falco-system; its name does not itself check signatures.

A webhook is unavailable

Inspect deployed webhook failurePolicy, selectors, endpoints, and the installed engine/chart version. validationFailureAction: Enforce describes policy violations; it does not alone define outage handling. The source runbook records older Gatekeeper deadlocks and suggests webhook deletion, but Gatekeeper is inactive in this final baseline. Treat bypassing any active webhook as an exceptional, reviewed recovery step with documented exposure and restoration checks. No outage/recovery test was performed for this handbook.

Source anchors

Historical troubleshooting notes, final Kyverno values, and final Falco configuration.