Observe Metrics Freshness

Keep backend telemetry fresh without changing health thresholds.

Visual overview

Primary diagram

System map

Reviewed changes improve telemetry freshness and log coverage. Encrypted backups are restored in a private resource-limited environment, with recovery credit only after real application checks pass.

Render the repository-owned system map when you need it.

Diagram is not rendered yet.

View as text

Observe monitoring and isolated recovery

Reviewed changes improve telemetry freshness and log coverage. Encrypted backups are restored in a private resource-limited environment, with recovery credit only after real application checks pass.

flowchart TD
  accTitle: Observe monitoring and isolated recovery
  accDescr: Reviewed changes improve telemetry freshness and log coverage. Encrypted backups are restored in a private resource-limited environment, with recovery credit only after real application checks pass.
  review["Reviewed change"] --> check["Protected preflight"]
  check --> timer["One-minute metrics timer"]
  timer --> verify["Fresh timestamps and healthy applications"]
  logs["Recent logs from all three production shards"] --> sanitize["Publish only scope and timestamps"]
  backup["Deployed app, configuration and actual data"] --> encrypted["Encrypted backup cloud round trip"]
  encrypted --> private["Sequential private restore: 2 CPUs, 8 GiB"]
  private --> behavior["Scheduler leases and messages; translation inference and replay"]
  behavior --> outcome["Record genuine pass or failure with revision and expiry"]
  verify --> rollback["Protected rollback if needed"]

Keep backend monitoring up to date

The backend previously refreshed monitoring data every five minutes. Observe warns after three minutes, so healthy monitoring could appear stale. Refreshing every minute fixes that mismatch without changing the score rules.

The protected deployment checks the change first, then restarts only the monitoring timer. Production apps, databases and shadow workers stay running in their existing state. Verify that monitoring timestamps stay fresh and public pages remain healthy. If necessary, remove the timer change through the protected pipeline.

Observe also checks that each of the three production news-processing workers has recent logs. The dashboard receives only timestamps, not private log contents. A worker can have working logs and still fail to process news; those failures remain visible.

Recovery testing copies the scheduler and translation worker into a private disposable environment. Their live containers stay unchanged. The test must use restored data and show real isolated behavior before Observe credits recovery. It does not prove production speed or activate either worker.