Skip to content
NutsNews chestnut logoNutsNews Wiki

Technical guide · Overview

Page status: Active

Observability

NutsNews is built with observability from the start.

Visual overview

Primary diagram

System map

NutsNews is built with observability from the start.

Render the repository-owned system map when you need it.

Diagram is not rendered yet.

View as text
Observability

NutsNews is built with observability from the start.

flowchart TB
  accTitle: Observability
  accDescr {
    NutsNews is built with observability from the start.
  }
  A["Observability"] --> B["OBSERVABILITY.md"]
  B --> C["Auto-generated placeholder diagram"]

Observability

Fullscreen diagram view.

Observability

Section titled “Observability”

NutsNews is built with observability from the start.

A fully automated system needs visibility. If no human manually publishes every article, the platform needs to explain what it is doing.

Observability Layers

Section titled “Observability Layers”

NutsNews uses multiple observability layers:

LayerPurpose
Admin PortalInternal operational dashboards
AI Usage DashboardOpenAI usage and cost visibility
Worker Shard DashboardWorker health and failed run visibility
Feed Health DashboardRSS source quality visibility
Better Stack UptimeExternal availability monitoring
UptimeRobotAdditional external availability, keyword, API, and public page monitoring
Lighthouse CIGitHub Actions quality checks for public web performance, accessibility, SEO, best practices, and Core Web Vitals-style regressions
Better Stack LogsExisting app/Worker structured log searches until those repos migrate
Grafana CloudVPS infrastructure metrics, centralized Loki logs, Explore queries, dashboards, quota alerts, and backup alerts
Backend Health ReportDaily read-only backend host report from GitHub Actions with JSON artifact and optional SMTP delivery
SentryApplication error monitoring
CloudflareCDN and Worker visibility
Cloudflare Cache ObservabilityExpected-vs-actual cache header dashboard, scheduled checks, and GitHub Actions alerting
VercelDeployment and frontend runtime visibility
SupabaseDatabase state and operational tables

Centralized Logging

Section titled “Centralized Logging”

NutsNews infrastructure logs are centralized in Grafana Cloud Logs through Grafana Alloy on the VPS. The infra repo owns the host-side pipeline for systemd journal logs, auth/security logs, Caddy JSON access/error logs, Docker/Compose logs for NutsNews runtime containers, backup/reporting logs, and Ops Portal logs.

Use low-cardinality Loki labels such as:

env
host
service
unit
container
compose_project
source
level

Keep high-cardinality data such as request IDs, raw IP addresses, user IDs, full dynamic paths, and arbitrary error strings out of labels. Query those as parsed fields or structured metadata.

The older app/Worker Better Stack searches remain useful until those application repositories explicitly migrate their runtime logging. Do not treat an infra-only PR as an app logging migration.

Existing app/Worker service names:

nutsnews-web
nutsnews-worker
nutsnews-controller

Important log fields:

FieldPurpose
levelSeverity such as info, warn, or error
serviceWhich part of the platform created the log
eventWhat happened
messageHuman-readable summary
environmentProduction, preview, or local
shardIndexWhich Worker shard produced the log
durationMsHow long an operation took
statusRequest or operation status
acceptedCountNumber of accepted articles
rejectedCountNumber of rejected articles

Better Stack Uptime

Section titled “Better Stack Uptime”

Better Stack Uptime checks whether the public site is reachable from outside the platform.

Use it to answer:

Is the site reachable?

UptimeRobot

Section titled “UptimeRobot”

UptimeRobot is an additional external monitoring layer for simple public uptime and content checks.

Recommended first monitors:

Monitor nameTypeURL
NutsNews WebsiteHTTP(s)https://www.nutsnews.com
NutsNews Homepage ContentKeywordhttps://www.nutsnews.com
NutsNews Articles APIHTTP(s) or API GEThttps://www.nutsnews.com/api/articles?limit=1
NutsNews Privacy PageHTTP(s)https://www.nutsnews.com/privacy
NutsNews Contact PageHTTP(s)https://www.nutsnews.com/contact

The homepage keyword monitor should alert when this keyword is missing:

NutsNews

Important safety rule: do not monitor refresh-triggering Worker or controller URLs such as /?limit=1 or /?shard=0. Those URLs can run production ingestion work. Only add Worker/controller monitors after safe read-only /health routes exist.

Detailed setup lives in:

docs/UPTIMEROBOT_ONBOARDING.md

Lighthouse CI

Section titled “Lighthouse CI”

Lighthouse CI is the automated web quality gate for NutsNews public pages.

It runs in GitHub Actions and checks:

  • Performance
  • Accessibility
  • Best practices
  • SEO
  • Largest Contentful Paint
  • Cumulative Layout Shift
  • Total Blocking Time

Repository-layout rule:

GitHub Actions workflow: .github/workflows/lighthouse-ci.yml
Lighthouse config: web/lighthouserc.js
NPM install/build/start commands: run from web/

Recommended first audited URLs:

http://localhost:3000/
http://localhost:3000/privacy
http://localhost:3000/contact

Do not audit Worker refresh URLs, controller trigger URLs, admin pages, OAuth routes, or any route that can perform ingestion, AI review, translation, refresh work, database writes, or authenticated work.

Detailed setup lives in:

docs/LIGHTHOUSE_CI_ONBOARDING.md

axe Playwright Accessibility CI

Section titled “axe Playwright Accessibility CI”

axe Playwright Accessibility CI answers:

Did a public NutsNews page introduce a serious or critical automated accessibility regression?

It runs from GitHub Actions using Playwright and @axe-core/playwright.

Checked pages:

http://127.0.0.1:3100/
http://127.0.0.1:3100/about
http://127.0.0.1:3100/privacy
http://127.0.0.1:3100/contact

Repository-layout rule:

GitHub Actions workflow: .github/workflows/accessibility-ci.yml
Playwright config: web/playwright.config.ts
axe test file: web/tests/accessibility.spec.ts
NPM install/build/test commands: run from web/

The first threshold fails on only serious and critical axe violations. This keeps CI practical while still catching the issues most likely to hurt launch quality, App Store review polish, and reader usability.

Do not audit Worker refresh URLs, controller trigger URLs, admin pages, OAuth routes, or routes that can perform ingestion, AI review, translation, refresh work, database writes, or authenticated work.

Detailed setup lives in:

docs/AXE_PLAYWRIGHT_ACCESSIBILITY_CI.md

Manual WAVE browser-extension checks should still be run before App Store review and major public launches.

Better Stack Logs

Section titled “Better Stack Logs”

Better Stack Logs answer:

What happened inside the app, Worker, or controller while those repos still use Better Stack?

Useful searches:

service:nutsnews-web
service:nutsnews-worker
service:nutsnews-controller
level:error
shardIndex:0
event:api.log_test.completed

Grafana Cloud

Section titled “Grafana Cloud”

Grafana Cloud is used for Prometheus-style metrics, Loki logs, dashboards, alert rules, and low-frequency Synthetic Monitoring where time-series visibility is more useful than raw logs.

Worker-uplift telemetry is governed by NutsNews Worker-Uplift Telemetry Scope. RabbitMQ metrics, worker service metrics, and structured logs are required; full traces and exemplars are deferred; article/model payload telemetry is forbidden.

The ramideltoro/nutsnews-infra Grafana Cloud dashboards are managed by OpenTofu. Dashboard variables that feed regex label matchers must keep their All value as .*; otherwise PromQL such as deployment_environment=~"$environment" and instance=~"$instance" can render as =~"" and hide every real non-empty label value. Node-exporter panels must match the labels Grafana Cloud actually receives from the integration, currently job=~"integrations/node_exporter" and instance=~"$instance", not service_namespace="nutsnews". The NutsNews CPU Load Processes dashboard also uses Grafana’s $__rate_interval for CPU rate windows and distinct 1m, 5m, and 15m load-average targets so legends stay clear.

Dashboard fixes should go through the infra repository PR flow first, then the protected Grafana Cloud apply workflow after merge.

Current home-server backup monitoring uses this Prometheus data source:

grafanacloud-kindcantaloupe2036-prom

Confirmed backup metric query:

home_server_backup_last_success{instance="chingadera", job="integrations/unix"}

Confirmed value meaning:

1 = last backup succeeded
0 = last backup failed

The VPS observability layer is managed from ramideltoro/nutsnews-infra:

  • Ansible installs and configures Grafana Alloy on the VPS when explicitly enabled.
  • Alloy ships host/systemd/log/textfile telemetry and Docker/Compose logs for NutsNews runtime containers; cAdvisor/container metrics remain disabled by default until that separate privilege boundary is reviewed.
  • OpenTofu manages the Grafana Cloud folder, dashboards, quota guardrail alerts, log-pipeline alerts, and optional Synthetic Monitoring checks.
  • Grafana Cloud telemetry write credentials and Grafana automation credentials are separate.
  • Real Grafana URLs, usernames, tokens, tenant IDs, Synthetic Monitoring targets, and backend config stay out of Git.

Detailed setup lives in:

NUTSNEWS_GRAFANA_CLOUD_OBSERVABILITY.md

Detailed Grafana Explore queries, dashboard setup, and alert ideas live in:

docs/GRAFANA_BACKUP_MONITORING.md

Sentry

Section titled “Sentry”

Sentry tracks frontend, runtime, and Worker errors.

Use it to answer:

What errors happened in production?

Important env vars:

NEXT_PUBLIC_SENTRY_DSN
SENTRY_ORG
SENTRY_PROJECT
SENTRY_AUTH_TOKEN
SENTRY_DSN

Admin Dashboards

Section titled “Admin Dashboards”

/admin/ai-usage

Section titled “/admin/ai-usage”

Answers:

  • Is OpenAI usage controlled?
  • What is the estimated cost?
  • Which shards are using AI?
  • Did cost protection trigger?

/admin/shards

Section titled “/admin/shards”

Answers:

  • Are Worker shards healthy?
  • Which shards are stale?
  • Which runs failed?
  • What was the latest error?

/admin/feed-health

Section titled “/admin/feed-health”

Answers:

  • Which feeds fail often?
  • Which feeds lack thumbnails?
  • Which feeds produce accepted articles?

/admin/feeds

Section titled “/admin/feeds”

Answers:

  • Which feeds are active?
  • Which feeds are disabled?
  • Which feeds should be enabled or disabled?

Grafana Cloud: Home Server Backups

Section titled “Grafana Cloud: Home Server Backups”

Answers:

  • Did the last backup succeed?
  • When was the last successful backup?
  • How old is the newest successful backup?
  • How many backups are available now?
  • When is the next backup expected, if exported by the backup script?

Health Check Commands

Section titled “Health Check Commands”

Public site

Section titled “Public site”
Terminal window
curl -I "https://www.nutsnews.com/"

Article API

Section titled “Article API”
Terminal window
curl -s "https://www.nutsnews.com/api/articles?page=0"

UptimeRobot validation set

Section titled “UptimeRobot validation set”
Terminal window
curl -I "https://www.nutsnews.com/"
curl -s "https://www.nutsnews.com/" | grep -i "NutsNews"
curl -s "https://www.nutsnews.com/api/articles?limit=1"
curl -I "https://www.nutsnews.com/privacy"
curl -I "https://www.nutsnews.com/contact"

Cache

Section titled “Cache”

Quick HIT-rate spot check:

Terminal window
./scripts/validate_cloudflare_cache_hit_rate.sh https://www.nutsnews.com

Full expected-vs-actual cache policy report:

Terminal window
cd web
npm run audit:cache -- --url https://www.nutsnews.com

Protected dashboard:

/admin/cache

Detailed guide:

docs/CLOUDFLARE_CACHE_OBSERVABILITY.md

Worker

Section titled “Worker”
Terminal window
curl "https://nutsnews-worker-0.nutsnews.workers.dev/?limit=1"

Controller

Section titled “Controller”
Terminal window
curl "https://nutsnews-controller.nutsnews.workers.dev/?shard=0"

Better Stack web log test

Section titled “Better Stack web log test”
Terminal window
curl "https://www.nutsnews.com/api/log-test"

Grafana backup metric discovery

Section titled “Grafana backup metric discovery”

Run this in Grafana Cloud Explore, not in the terminal:

{__name__=~"home_server_backup_.*", instance="chingadera", job="integrations/unix"}

Grafana last backup success

Section titled “Grafana last backup success”

Run this in Grafana Cloud Explore:

home_server_backup_last_success{instance="chingadera", job="integrations/unix"}

Controller and Shard Debugging

Section titled “Controller and Shard Debugging”

Detailed manual commands live in:

docs/CONTROLLER_AND_SHARDS.md

Useful Better Stack searches:

service:nutsnews-controller
service:nutsnews-controller level:warn
service:nutsnews-controller level:error
service:nutsnews-worker shardIndex:0
service:nutsnews-worker level:error

Useful Wrangler tail commands:

Terminal window
cd controller && npx wrangler tail nutsnews-controller
cd worker && npx wrangler tail --config generated-wrangler/wrangler.shard0.jsonc