Skip to main content
View Markdown ↗

Copy this page

Select and copy the Markdown below, then paste it into your LLM.

On this page

Keep the result of a recovery rehearsal

Record both a successful database restore and a rehearsal that misses writes made after the backup.

Jump to the written guide ↓

Follow along

Goal: record a successful recovery rehearsal and a failed one as signed evidence. A signed failed result remains a failed result.

Before you start

  • Install CI/lock and connect it to your platform using the approved human or agent enrollment flow.
  • Work in a disposable test repository with a committed recovery rehearsal at scripts/restore-rehearsal.sh. It must return zero only when your recovery checks pass, and nonzero when they fail.
  • The rehearsal must write its findings to out/restore-report.json on every run, passing or failing. The recovery policy binds to that file's bytes, so include a run identifier or timestamp in it: two runs that write identical bytes cannot be told apart.
  • Prepare disposable test data and an isolated database environment. The commands below wrap your own reviewed rehearsal, not a built-in CI/lock recovery test.
  • The second command uses the fixture's REHEARSAL_LATE_WRITES switch. Your script must implement this switch, or you must use its documented failure injection instead. Merely setting an environment variable does not create a recovery gap.

1. Confirm the rehearsal and identity

cilock version
cilock agent status
git rev-parse HEAD
test -x ./scripts/restore-rehearsal.sh

Expected: the script exists and is executable, and the intended enrolled agent has a usable signing identity. If running as a person, follow signing and identity instead of assuming an agent session is a human session.

2. Record the successful rehearsal

Use the platform endpoint configured by your approved enrollment instructions. CI/lock captures command output and file observations; keep secrets out of the rehearsal output.

mkdir -p ../recovery-evidence
rm -rf out
cilock run --step recovery-rehearsal \
  -o rehearsal.json -- ./scripts/restore-rehearsal.sh
cp out/restore-report.json ../recovery-evidence/rehearsal-report.json

Expected: the restore runs against the disposable environment, its checks pass, and rehearsal.json is written. It records the report as a product of the step. Each run starts without out/, so its report is new output. Keep the copy outside the repository: CI/lock records the files a run starts with as its inputs, so a copy kept beside the script would be an input of every later run. Choose checks that test your recovery requirements. Comparing restored row counts and fingerprints is narrower than proving complete application recovery.

3. Record a known recovery gap

For a rehearsal implementing late-write injection with the REHEARSAL_LATE_WRITES input:

rm -rf out
REHEARSAL_LATE_WRITES=37 cilock run \
  --step recovery-rehearsal --ignore-command-exit-code \
  -o gap.json -- ./scripts/restore-rehearsal.sh
cp out/restore-report.json ../recovery-evidence/gap-report.json

Expected: the rehearsal identifies missing data and exits nonzero, while CI/lock writes the evidence. The flag keeps evidence collection running after the wrapped command fails. It must not be used as a reason to treat the rehearsal as successful.

4. Inspect both records

With jq installed:

jq -r .payload rehearsal.json | base64 -d | jq '.predicate'
jq -r .payload gap.json | base64 -d | jq '.predicate'

Then compare the report each record lists as its product with the copy you kept:

jq -r .payload gap.json | base64 -d | jq -c '.predicate.attestations[]
  | select(.type | contains("/product/")) | .attestation.leaves[] | {path, fileDigest}'
shasum -a 256 ../recovery-evidence/gap-report.json

Expected: the records identify the same rehearsal step and their respective command results. Each lists out/restore-report.json with the same digest shasum prints for its copy. Decoding a payload is inspection, not signature or policy verification. If platform storage is configured, also inspect the stored records and their subjects in the platform.

Troubleshooting and cleanup

A signing or upload error is different from a failed restore. Preserve the complete stderr and exit status while diagnosing it. If the injected run succeeds, check that your script actually implements the failure switch. Clean up only the disposable resources your rehearsal created, using that script's documented teardown.

Next: evaluate both results against a policy. Configure technical-control mappings separately; completing this exercise does not establish compliance with a framework or control.

Browse the other walkthroughs