# Production Certification

Phase 14 turns release readiness into repeatable evidence. It does not treat a
local SQLite test run as proof of production behavior and it does not allow a
backup timestamp to stand in for a restore drill.

## Certification contract

Every candidate has an immutable `RELEASE_ID` using 7-100 letters, numbers,
dots, underscores, or hyphens. Build the frontend before generating the release
manifest so it covers all three locked inputs:

```bash
pnpm install --frozen-lockfile
pnpm build
php artisan ops:release-manifest
php artisan ops:release-manifest --verify
```

The manifest stores SHA-256 checksums for `composer.lock`, `pnpm-lock.yaml`, and
the Vite build manifest. It is written to
`bootstrap/cache/production-release.json` and must travel with the immutable
release artifact.

Run the fail-closed preflight after migrations and Laravel caches are built:

```bash
php artisan ops:certify
php artisan ops:certify --json > production-certification.json
```

The command returns non-zero unless all of these pass:

- `APP_ENV=production` and the production configuration assertion;
- live MySQL 8+ connectivity with no pending migrations;
- a shared cache store capable of acquiring an atomic lock;
- every critical monitoring/publication/outbox schedule is registered;
- a real restore drill timestamp is within `BACKUP_MAX_AGE_HOURS`;
- the release identity and every manifest checksum match.

Failures contain bounded operational messages, not credentials, connection
strings, SQL, or provider responses. Successful responses expose `X-Release-ID`
so load balancers and incident responders can correlate traffic with the
certified artifact.

## Recovery drill

Perform this in an isolated account/network using copies of production backup
artifacts and separately retrieved recovery-critical secrets.

1. Record the selected database backup identifier and private-media snapshot.
2. Restore MySQL into an empty MySQL 8 instance; never restore over the source.
3. Restore private catalog/CV media and audit archives outside the public root.
4. Supply the active `APP_KEY`, retained previous encryption keys, auth/analytics
   HMAC keys, and provider credentials from the recovery secret store.
5. Run migrations only if the candidate release is forward-compatible with the
   restored schema.
6. Run `ops:release-manifest --verify`, `ops:certify --json`, readiness, login,
   queued notification, scheduler, public booking, and checksum sampling.
7. Confirm audit archive hashes and sample media SHA-256 values against their
   database records.
8. Record achieved RPO/RTO, evidence location, operator/reviewer, and the UTC
   completion time. Only this completion time may become
   `BACKUP_LAST_VERIFIED_AT`.
9. Destroy the isolated recovery environment under the approved data handling
   procedure.

## Load and security certification

The manual `Production certification` GitHub workflow requires both a release
ID and the latest successful restore-drill timestamp. It provisions MySQL 8.4
and Redis, migrates and caches a production-configured candidate, verifies the
release manifest/preflight, and executes `.github/load/production-smoke.js`.

The bounded smoke ramps to ten virtual users and fails when HTTP errors reach
1%, p95 duration reaches 750 ms, or checks fall to 99% or below. Every sampled
route must return a request ID, anti-sniffing header, matching release identity,
and no debug exception page. These defaults are initial certification budgets,
not a forecast of production capacity; update them only from an approved SLO
and rerun against production-like infrastructure and data volume.

Store the uploaded JSON report and release manifest with the change record for
90 days. Preserve external MySQL query plans, resource graphs, vulnerability
scan output, and restore drill evidence in the same record because they cannot
be proven by the application repository alone.

## Deployment rehearsal

1. Prove ordinary CI and the manual production-certification workflow are green.
2. Verify the signed/immutable artifact and its release manifest in staging.
3. Confirm an encrypted backup and the latest restore drill satisfy the approved
   RPO/RTO before any schema change.
4. Enable maintenance/draining if the change requires it, run backward-compatible
   migrations with the migration principal, and switch the release atomically.
5. Rebuild configuration/route/view caches and restart queue workers so no worker
   retains old code.
6. Verify `/up`, `/health/ready`, release headers, login, queue delivery,
   scheduler execution, log ingestion, and one critical public flow.
7. Observe 5xx, p95/p99 latency, database saturation, queue age/failures,
   scheduler failures, and outbox backlog for the approved rollback window.

## Rollback decision

Rollback the application release when health, error/latency, or critical-flow
budgets breach and the previous application is schema-compatible. Stop and use
the recovery procedure instead when a migration is destructive, old code cannot
read the new schema, or data integrity is uncertain. Never run migration
rollback automatically in production. Record the decision time, release IDs,
symptoms, schema compatibility review, operator, and outcome.

## Remaining external evidence

Phase 14 is locally implemented when the command/tests/static/build/audit gates
pass. Production certification remains pending until the MySQL 8.4 workflow,
restore drill, production-like load run, vulnerability review, and deployment /
rollback rehearsal have produced retained evidence for the exact release.