Skip to content
NutsNews chestnut logoNutsNews Wiki

Technical guide · Overview

Page status: HistoricalHistory group: Updates

Backfill repeat fix

This update fixes two translation backfill problems:

Visual overview

Primary diagram

System map

This update fixes two translation backfill problems:

Render the repository-owned system map when you need it.

Diagram is not rendered yet.

View as text
Backfill repeat fix

This update fixes two translation backfill problems:

flowchart TB
  accTitle: Backfill repeat fix
  accDescr {
    This update fixes two translation backfill problems:
  }
  A["Backfill repeat fix"] --> B["updates/README_BACKFILL_REPEAT_FIX.md"]
  B --> C["Auto-generated placeholder diagram"]

Backfill repeat fix

Fullscreen diagram view.

Backfill repeat fix

Section titled “Backfill repeat fix”

This update fixes two translation backfill problems:

  1. The backfill script could translate the same articles repeatedly because it loaded a broad capped list of existing summaries, then missed rows it had just saved. It now checks existing article_summaries only for the current candidate article URLs, in small URL batches.
  2. Failed Ollama rows could be retried every batch. The script now supports a local failed-row cache. Set FAILED_TRANSLATION_CACHE=/tmp/nutsnews-translation-failures.json to skip rows that failed earlier in the same long backfill run. Set RETRY_FAILED=1 when you want to try those rows again.

For all-article backfills, use:

Terminal window
LANGUAGE_CODES="de-CH,de,el" \
BACKFILL_SOURCE=articles \
SCAN_ALL_CANDIDATES=1 \
CANDIDATE_LIMIT=5000 \
CANDIDATE_PAGE_SIZE=1000 \
BACKFILL_LIMIT=60 \
PUBLISH_READY=0 \
FAILED_TRANSLATION_CACHE=/tmp/nutsnews-de-ch-de-el-failed-translations.json \
node scripts/backfill_article_summaries.mjs