# Bokmål–Nynorsk reranker: complete research plan

This plan uses the specified dataset, requires **no manual review**, and targets **NbAiLab/nb-asr-qwen06b-bokmal-nynorsk-reranker**. It covers data construction, controlled experiments, automatic validation, and the final Hugging Face package. Implementation and training benchmarks started on 12 October 2026. Updated 13 October 2026: seed 42 and the evaluation/early-stopping protocol remain fixed; the development regressions and annotation repairs below are recorded explicitly. The V11 recipe has passed development qualification and the single seed-42 full-data campaign has been submitted. Final weights have not yet been validated or released.

**Acceptance criterion:** Query and Document must express the same meaning across Bokmål and Nynorsk. Correct differences in words, grammatical forms, and mention order are expected. A swap is negative only when it changes meaning; meaning-preserving reorderings are positive examples.

## 01 · Freeze the dataset and define the text columns

The verified dataset counts are:

| Supplied split | Paragraph pairs |
|---|---:|
| Train | 807,164 |
| Validation | 1,000 |
| Test | 1,000 |
| Total | 809,164 |

Pin dataset revision `619e36b85e4943f23cd85b66edcde331995e4872`.

The dataset contains original texts in `nb` and `nn`, and machine rewrites in `bokmaal_normform` and `nynorsk_normform`. Start with the **normalized columns as the primary training pair**, and retain the originals for automatic consistency checks and robustness evaluation. The dataset card identifies the subset as CC0.

[Dataset card](https://huggingface.co/datasets/NbAiLab/merged_npk_ndla_parallel_paragraphs_normform_cc0/blob/main/README.md)

The machine rewrites must not be treated as independently verified translations. Their preservation of numbers, names, and pronouns is something we will check automatically.

## 02 · Protect validation and test data before augmentation

Keep the supplied validation and test splits intact. From the supplied training split, reserve approximately:

| Purpose | Planned original pairs |
|---|---:|
| Training | 782,164 |
| Internal model-selection set | 10,000 |
| Internal threshold-calibration set | 10,000 |
| Additional blind challenge set | 5,000 |

These are allocation targets **before filtering and grouping**. Final counts will change.

Group paragraphs from the same document where document identity is recoverable. Also group exact and near duplicates across original and normalized fields. Keep every original paragraph and all its derived variants in one partition.

If overlaps with supplied validation/test are found, remove the corresponding training examples. Report any overlap between the two supplied evaluation splits.

Never move reserved examples back into training after using them for evaluation.

## 03 · Audit the corpus automatically

Produce counts for:

- Empty fields, duplicate pairs, and language-identification disagreements.
- Paragraph lengths in tokens.
- Number-containing paragraphs and the types of numeric expressions.
- Paragraphs containing eligible pronoun pairs or entity pairs.
- Differences between each original text and its normalized version involving numbers, names, or pronouns.
- Cross-language alignment anomalies.

Canonicalize numeric formatting only when interpretation is unambiguous. Preserve signs, decimals, dates, units, and identifier structure.

Quarantine uncertain training pairs rather than automatically labeling them negative. On validation/test, report results on both the supplied population and the automatically screened subset, with the exclusion reasons and counts.

This prevents filtering from silently making the benchmark easier.

## 04 · Construct several distinct augmentation families

Preserve the untouched pair as its positive counterpart.

| Family | Construction | Intended difficulty |
|---|---|---|
| Number-value edits | Change one digit, magnitude, sign, decimal value, or date component | Detect a small factual difference |
| Number swaps | Exchange two different compatible numeric spans | Detect changed assignments despite identical numbers |
| Pronoun swaps | Exchange two different pronouns with compatible grammatical forms | Detect changed participants or references |
| Entity swaps | Exchange two different complete entity spans of the same type | Detect changed participants, locations, or organizations |
| General semantic negatives | Controlled negation, modality, omission, or entity replacement | Maintain broader fidelity checking |
| Equivalent positives | Unambiguous formatting changes and permitted language/style variants | Prevent rejection based on superficial differences |

For example:

```text
Original: Ola fekk 20 kroner, Kari fekk 30.
Negative: Ola fekk 30 kroner, Kari fekk 20.
```

Number swaps are a separate experiment from number-value edits: they test whether the model understands associations rather than merely comparing numeric inventories.

Use Humit for morphology and a Norwegian NER model for entity spans. Humit supports both written standards; it does not itself provide cross-language alignment or pronoun-reference resolution.

[Humit documentation](https://huggingface.co/Humit-Oslo/humit-tagger-large)

Start by constructing swaps on the **Bokmål side**, where we want to anchor the annotations. Then evaluate Nynorsk-side construction using target annotations or confident alignment. The model always receives the same interface: Bokmål as `Query`, Nynorsk as `Document`, regardless of which side was modified.

We must explicitly test target-side errors; good results on corrupted Bokmål alone would be insufficient.

## 05 · Apply automatic acceptance rules to generated examples

Every transformation must:

- Change the intended span and leave unrelated text intact.
- Avoid exchanging identical values or identical spans.
- Preserve complete multiword entities.
- Preserve compatible grammatical forms for pronouns.
- Handle capitalization and punctuation consistently.
- Avoid overlapping spans and ambiguous alignments.
- Retain the edited content within the model’s input.
- Produce no exact input pair carrying conflicting labels.

Pronoun swaps use conservative, explicit rules. **Never change han, hun, ho, ham or henne in generated negatives.** Nynorsk han/ho can refer to inanimate nouns according to grammatical gender; Bokmål den can therefore legitimately correspond to han/ho. Keep such valid translations positive. A mismatch in the inventory of these forms is not, by itself, a reason to remove a parallel positive. Skip cases whose interpretation cannot be established automatically.

After this exclusion, the original cached natural edits supply no eligible pronoun negatives. A later training-only repair uses the cached normalized Humit tags to generate conservative single-person substitutions in real paragraphs. Require exactly one aligned unambiguous subject identity next to a finite verb in each language; replace that subject on only one side by a different person of the same number. Exclude quoted/metalinguistic text, protected gendered forms and honorific De. The explicit inventory avoids assuming that Humit uses identical singular tags (ent/eint) in both standards. Preparation yielded 127,219 new negatives from 71,600 of the 704,173 training pairs, with input/code/output hashes. The fixed 100k pilot actually samples 3,884 of these negatives. These are automatic training labels, not human-verified errors. The reserved pronoun evaluation still uses controlled, unambiguous first-/second-person and plural participant challenges; adding natural training contexts does not establish natural-error accuracy. Report that limitation explicitly. Add an independent small regression fixture for valid noun-gender anaphora, unchanged gendered participants, and clear person changes. Count excluded ambiguous negatives separately from token-length exclusions.

Store the operation, edited spans, original values, corruption side, annotation versions, and eligibility checks with each example.

Swaps are candidate augmentations, not automatically negative examples. Label a swap `no` only when a conservative automatic rule establishes changed meaning. Label established meaning-preserving reorderings `yes`; skip ambiguous cases. For example, “Ola hjelpte Kari” → “Kari hjelpte Ola” is negative, while “Ola og Kari kom” → “Kari og Ola kom” is positive. Keep these categories separate during evaluation.

Generate a bounded candidate pool—initially up to two negatives per eligible paragraph—and sample categories deliberately. Paragraphs without eligible spans remain useful positives. Do not force an augmentation for every row.

## 06 · Freeze this exact prompt

Use an English system message, Norwegian task instruction, and English interface labels, following the disfluency reranker’s interface. This is the proposed exact prompt; only the two text placeholders change:

```text
<|im_start|>system
Judge whether Document is a faithful translation of Query under Instruct. Answer only "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: Avgjør om Query og Document uttrykker samme mening på bokmål og nynorsk. Svar yes hvis informasjonen er bevart. Svar no hvis informasjon er endret, lagt til eller fjernet. Vær særlig oppmerksom på tallverdier, datoer, mengder, personlige pronomen og hvem som gjør hva. Tall og pronomen skal vise til de samme størrelsene og deltakerne i begge tekstene.
<Query>: {bokmal_text}
<Document>: {nynorsk_text}<|im_end|>
<|im_start|>assistant
<think>

</think>

```

Preserve the final suffix exactly as `"<think>\n\n</think>\n\n"`.

The training target is separately stored as `yes` or `no`. **Do not feed the gold label into the position used to score it.**

Train cross-entropy over the two next-token logits:

```text
score_yes = softmax([logit(no), logit(yes)])[yes]
```

Use identical rendering during training, validation, and inference. This retains the upstream reranker’s two-token scoring method.

[Qwen documentation](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)

This prompt judges semantic equivalence. It does not require matching word forms or mention order. Preserve participant roles, numeric values, and their associations; accept grammatical restructuring that preserves meaning.

## 07 · Run controlled pilot experiments before scaling

Use the same 100,000 training paragraphs, evaluation sets, prompt, optimizer settings, and example budget.

| Experiment | Training data |
|---|---|
| B0 | Upstream Qwen3-Reranker-0.6B, without new training |
| B1 | Existing disfluency reranker, without new training |
| E0 | Positive pairs plus general semantic negatives |
| E1 | E0 plus number-value edits |
| E2 | E0 plus number-assignment swaps |
| E3 | E0 plus pronoun swaps |
| E4 | E0 plus entity swaps |
| E5 | Combined augmentation recipe |

Maintain approximately 50% positive and 50% negative training exposure. Adding an augmentation replaces part of the negative sampling budget; it does not increase the run’s total training exposure.

Keep equivalent numeric-format positives common across these experiments. Otherwise, an apparent gain could come from teaching different acceptance rules.

Follow with focused comparisons on the strongest recipe:

- Remove numeric augmentation.
- Remove pronoun augmentation.
- Compare Bokmål-only corruption with corruption on both sides.
- Compare normalized-only training with a controlled mixture of original and normalized pairs.
- Compare training with and without established meaning-preserving reorderings as positive examples.

Earlier recipe trials are exploratory. Their pronoun policy was superseded, so their results must not be presented as comparable ablations of the final model. For full-data training, use **one predetermined seed: 42**. Do not train several seeds and select the luckiest result. The selection partition qualifies the fixed-seed model against the requirements; it does not choose among random seeds.

The ordinary-reference development diagnostic exposed a template-generalization failure in the earlier corrected pilot: all 36 clear person changes in fronted-time sentences were accepted at step 500. Its automatic full-data launch was cancelled. The revised matched-budget pilot assigns **15% of anchors to plain synthetic person-reference examples**, varying word order, negation and singular/plural subjects without a hypothetical prefix. Remaining anchors use the prior natural/structured/transfer mix; the total exposure budget and 50/50 positive/negative balance stay fixed. Diagnostic sentences are not training examples. Plain-reference controls are included in full selection, and the separate ordinary-reference and grammatical-gender regressions must pass before scaling. These are development diagnostics, not untouched corpus test results. Plural forms occur in training; only selected lexical/surface templates remain held out.


Development repairs and outcomes (all pilots use seed 42; these are different recipes, not competing random seeds):

| Recipe | Change and outcome |
|---|---|
| V5 plain reference | Added 15% ordinary person-reference anchors. Full 10k selection passes category gates, but the expanded independent grid accepts 11/96 clear person changes. Full-data launch cancelled. |
| V6 object reference | Half of the plain-reference branch uses twelve varied object-bearing predicates, excluding diagnostic reading/eating/buying/writing predicates. Same budget on four GPUs; completed in about 29 minutes. Selected step 1,000 has development loss 0.0007045, but misses 3/96 person changes, rejects two valid gendered translations, and fails strict single/batch numerical consistency. Full-data launch cancelled. |
| V7 natural reference | Added training-only natural subject-person edits. The selected checkpoint passes all 146 semantic/language development fixtures, but full selection accepts 2/154 changed entity-role examples (1.30%, above the 1% limit). Strict batch numerical consistency also fails. Full-data launch refused. |
| V8 entity weight | Doubled the conditional entity-frame weight from 1:1:1 to 1:1:2 at fixed overall structured rate. Final checkpoint accepts 1/96 clear person changes and rejects the valid “Hun hjelper ham.” / “Ho hjelper han.” translation. Full-data launch cancelled. Completed full selection passes category gates (99.91% supplied/original positive acceptance; 1/154 entity-role false acceptance), which does not override the independent failures. |
| V9 human reference | Added ordinary valid human third-person translations with digit-only negatives at about 5.1% of anchors, preserving participant pronouns. Completed in 29.5 minutes; best final step loss 0.00572846. All six semantic, twelve gender, numerical consistency and 308 entity-role checks pass, but 3/96 person-change negatives are accepted. Full-data launch cancelled. Completed full selection passes category gates (99.89% supplied positive acceptance, zero entity-role false acceptance); the independent person errors still disqualify it. |
| V10 person rate | Increased plain-reference sampling from 15% to 30% at fixed budget. Best final development loss 0.00030054, but accepts 5/96 independent person changes and fails numerical consistency (maximum score difference 0.00002319). Semantic/gender and entity-role gates pass. Full-data launch cancelled; increased sampling alone did not fix the coverage gap. |
| V11 person lexicon | Expanded training plain object predicates from 12 to 36, keeping reading/eating/buying/writing excluded and all other rates, seed and budget fixed. Completed in 30.9 minutes; best final step loss 0.02313729. All 128 person, six semantic and twelve gender checks pass, and entity-role replay passes its gate. The original numerical scorer fails; the verified execution-profile repair below passes all 146 checks. Full 10k selection passes: 99.91% supplied-positive acceptance, 99.7669% minimum positive-control acceptance, and worst protected false acceptance 1/154 (0.64935%, entity/role). The single seed-42 full-data campaign has been submitted. |
| V12 three spelling views | Authorized parallel pilot on a separate four-GPU node. Same 100,000 source IDs, seed 42, 1,562 steps and synthetic components as V11; natural corpus anchors sample one distinct original/original, original/normalized or normalized/normalized view. Training and evaluation are pending; this does not replace the running full-data baseline. |

The independent person-reference grid expanded from 48 to **128 cases** after object-bearing reading/eating examples exposed a coverage gap: 32 valid translations and 96 participant changes. Keep the original cases and all discovered failures. This is development data; it is not an untouched corpus-test estimate. The numerical interface check retains an FP32 absolute score tolerance of **0.00001** and writes failures before exiting. Reload agreement, batch agreement and training-interface agreement are separate checks. Do not relax a failed gate or discard an error to qualify a candidate.

After V11, an isolated numerical probe preserved all 146 fixtures and the same tolerance. The baseline scorer differs by 0.00002076 between single and batched inference; eager attention alone and padding alignment alone still fail. Combining eager attention with right padding to multiples of 128 reduces the difference to 0.00000201 and passes every check. This versioned **eager-pad128-v1** execution profile is implemented in both evaluation and exported inference. Gradient updates retain SDPA; development evaluation restores the training backend and mode afterward. The prompt, labels, thresholds and tolerances are unchanged.

The actual production code also passes a separate development-only numerical test on **734 corpus and concatenation-stress inputs, spanning 163–2,038 tokens**, with batch sizes 6, 16 and 32. No decisions change; the largest stress difference is 0.00000149 and training-interface difference is zero. A fresh four-GPU smoke stops synchronously at step 3/6. These establish tested numerical behavior, not held-out accuracy or a guarantee for all hardware. The full-data launcher requires these checks and matching numerical source hashes. Reports, exported configuration and release freeze must identify the same profile; packaging rechecks frozen artifact hashes before publication. Earlier campaigns and their failed reports remain unchanged.


The V6 generator broadens development templates, so aggregate losses are not directly comparable with V5. V6, V7 and V8 share evaluation inputs. V9 adds separately gated valid human-reference and digit-change controls; its aggregate development loss is therefore not directly comparable with V8. V9, V10 and V11 share all evaluation inputs. V10 changes sampling only; V11 changes only the opted-in training predicate inventory, leaving the default evaluation generator unchanged. Existing challenge inputs remain unchanged. Human-reference training and held-out evaluation use separate predicate inventories. Synthetic examples can repeat across source groups: report distinct input counts, and do not interpret source-group uncertainty as evidence of unseen-template performance. The final blind/test partitions remain unused for recipe tuning.

## 08 · Use conservative training settings and measure the actual cost

The completed V11 selection suite contains **312,304 challenges from 10,000 paragraphs**, with **32 classification errors (99.98975% pooled accuracy)** at threshold 0.5. Repeated synthetic patterns contribute to this result; it is not final blind-test or human-validated natural translation accuracy. Number-value changes are rejected in 5,382/5,383 cases; controlled pronoun changes have zero false acceptances. The historical pilot selection report retains its original scorer, while the new numerical profile was verified separately and is frozen for full training and release.

Full-data preparation job **2509418** completed, and four-GPU training **2509419** is running, followed by launcher **2509420**. The first development check at step 1,000 scored 30,570 challenges with 3 errors (99.99019% pooled accuracy); step 2,000 has 5 errors (99.98364%) but lower balanced category loss, 0.00128581, and is the current best by the fixed loss criterion. These are checkpoint-selection results, not final held-out accuracy. One seed (42), 704,173 retained paragraphs, two epochs maximum, batch 256, evaluation every 1,000 steps, and patience-three early stopping remain fixed. Estimated training time is **3–4 hours after preparation and queueing**, with a six-hour allocation. The full model must pass fresh selection, calibration, held-out and inference checks before publication.

**Parallel three-view experiment:** The original and normalized columns provide three candidate positive views: `nb–nn`, `nb–nynorsk_normform`, and `bokmaal_normform–nynorsk_normform`. Across the same 704,173 retained rows, these yield 982,404 unique exact pairs, 278,264 more than normalized-only (+39.5%). Deleted quality rows remain excluded, and all variants inherit their source group and split. All three strings coincide on 449,322 rows, so blindly tripling the data would mostly repeat examples.

V12 uses the same 100,000 hash-selected pilot paragraphs as V11. It assigns one distinct view per paragraph by seed, **fixed across epochs**, and applies it only to natural corpus anchors. Of the assigned source pairs, 18,994 differ from normalized-only. Actual token-cache counts are 200,000 examples and 42,094,753 tokens, with zero conflicting-label removals. All 25 synthetic component/family counts remain identical. Natural pronoun-negative count changes from 3,025 to 2,840 because cached edits are accepted only when the exact edited-side text matches the chosen spelling; this coverage difference must be considered when interpreting results. Cached offsets are never transferred across changed spellings. All 982,404 known eligible training-positive pair hashes are protected from negative collisions.

The V12 cache job **2510069** completed and four-GPU training **2510070** is running in parallel with the baseline. After training, selection job **2510071** and diagnostics **2510072** run independently on single GPUs. Diagnostics compare V11/V12 spelling-view acceptance on selection data and run the unchanged 146 inference regressions. The evaluator passed a separate GPU engineering smoke on 64 selection paragraphs (job 2510073); this is not a model-quality result. V12 has no automatic full-data or final-held-out launch. Compare recipes on development data before any decision to scale.

We do **not** label `nb–nb` negative merely because it differs from the two Nynorsk references: shared text can be valid in both standards and retains its meaning. Target-standard conformity would require a separate task and evidence.

Frozen full-data protocol (updated 13 October 2026; V11 qualified and the seed-42 campaign was submitted):

| Setting | Choice |
|---|---|
| Base | Qwen/Qwen3-Reranker-0.6B, revision e61197ed45024b0ed8a2d74b80b4d909f1255473 |
| Seed | 42, fixed before full-data training |
| Method | Full fine-tuning, existing causal-LM head |
| Learning rate / optimizer | 2e-6 / AdamW |
| Effective batch / per-GPU microbatch | 256 examples / 32 examples, with token-limited internal chunks |
| Hardware | One node with four GPUs; no multi-node training |
| Precision | BF16 training; FP32 selection, calibration and final inference |
| Schedule / gradient clipping | Cosine, 3% warmup / maximum norm 0.1 |
| Context limit | 2,048 tokens including the complete prompt; no silent truncation |
| Training ceiling | Two epochs; early stopping may finish sooner |
| Periodic evaluation | Every 1,000 optimizer steps and at the training ceiling |
| Development subset | Fixed first 1,000 paragraphs of the group-isolated selection partition; never blind/test |
| Stopping metric | Balanced category log loss, including natural and controlled semantic challenges |
| Early stopping | Three consecutive evaluations without an absolute loss improvement greater than 0.001 |
| Checkpoint retained | Lowest observed selection-loss checkpoint, even if a smaller improvement did not reset patience |
| Recovery | Optimizer, scheduler, random states, source/input hashes and early-stopping state are checkpointed |

After the corrected audit, **704,173 training pairs** remain. This includes **9,625 pairs restored** because a person-pronoun inventory mismatch alone is insufficient evidence of an error. All other integrity quarantine rules remain. With one positive and one negative per training anchor, two epochs give at most about **2.82 million examples**, or approximately **11,000 optimizer steps** at batch 256, before overlength exclusions and batch rounding. The pilot uses 100,000 anchors and 1,562 steps.

Build the token cache before starting distributed training so other ranks do not time out while rank zero tokenizes the corpus. Verify synchronized stopping in a short four-GPU smoke test first. A completed early-stopped run is successful training, not an interrupted or failed job.

After training, evaluate the best checkpoint on the complete 10,000-pair selection partition. If it fails the declared requirements, do not release it or silently relax the gates. Calibrate the threshold only after the checkpoint is fixed. Freeze weights, threshold, prompt and evaluation inputs before scoring blind/test data. A failed final gate is reported as a failed gate; blind/test results are not a tuning loop.

Do not silently truncate paragraph pairs containing the relevant evidence. Longer examples need a larger context budget or an explicitly evaluated aligned-chunk strategy.

## 09 · Validate with automatic challenge sets and operational metrics

Build separate challenge suites from held-out paragraphs:

- Number-value changes.
- Number-assignment swaps.
- Pronoun swaps and substitutions.
- Entity swaps.
- Meaning-preserving reorderings, labeled positive.
- General semantic errors.
- Equivalent formatting and legitimate language variation.

Reserve some transformation templates and replacement patterns exclusively for evaluation. Where possible, obtain naturally occurring disagreements from the original/normalized variants, but describe their labels as uncertain unless an automatic rule establishes the difference.

Report these metrics separately:

| Metric | Purpose |
|---|---|
| Acceptance of supplied correct pairs | Measure false rejection |
| False acceptance of numeric errors | Primary sensitivity metric |
| False acceptance of pronoun errors | Primary sensitivity metric |
| False acceptance of entity/role swaps | Measure relational understanding |
| Acceptance of meaning-preserving reorderings | Detect inappropriate reliance on mention order |
| Acceptance of equivalent formats | Detect superficial shortcuts |
| Correct target ranked above corrupted target | Measure contrastive discrimination |

Also report performance by paragraph length, corruption side, source domain where recoverable, and original versus normalized text.

Choose checkpoints on the internal model-selection set. Choose the decision threshold on the calibration set, using the supplied validation set as an additional distribution check. Run the blind challenge set and supplied test set after the recipe and threshold are frozen.

Initial engineering targets would be:

- At least **98% acceptance** of supplied/original positive pairs and every populated valid-positive control family, including equivalent number formats, valid reordering and controlled semantic-equivalence positives.
- At most **1% false acceptance** in each protected number, pronoun, entity/role and meaning-transfer bucket; report individual families separately.
- No material regression on broader semantic errors.

These are targets, not predicted results. If they cannot be met together, report the tradeoff rather than hiding it behind aggregate F1.

Report confidence intervals grouped by original paragraph or document. Ten variants from one paragraph are not ten independent observations. The supplied 1,000-row test split will have fewer eligible cases for each error category; publish those denominators.

This category-specific evaluation follows the motivation of translation challenge sets such as ACES, which expose weaknesses hidden by aggregate scores.

[ACES study](https://aclanthology.org/2025.cl-1.4/)

**Without manual review, all reported accuracy remains conditional on the supplied positive labels and our automatic corruption rules.** We will not call it human-validated translation accuracy.

## 10 · Package the selected model with complete instructions

The final destination will be:

```text
NbAiLab/nb-asr-qwen06b-bokmal-nynorsk-reranker
```

The package should contain the selected weights, tokenizer, scoring template, and a runnable scoring example supporting single pairs, JSONL batches, and candidate ranking.

**Required model-card attribution:** “This model was created by Thea Tollersrud and Per Egil Kummervold in the NB-ASR project.”

Its complete model card must document:

- Authorship: Thea Tollersrud and Per Egil Kummervold, NB-ASR project.
- The semantic-equivalence task and the distinction between meaning-changing swaps and valid reorderings.
- Base model and selected checkpoint.
- Dataset revision, columns, splits, and final retained counts.
- Every augmentation family and automatic filtering rule.
- The exact prompt above.
- Training settings and reproducibility details.
- Validation results, ablations, thresholds, and confidence intervals.
- Installation and runnable scoring examples.
- Score interpretation and long-input handling.
- The absence of manual review and the resulting evaluation limits.
- The difference between rejecting a changed translation and generating a corrected translation.

Before release, verify that reloaded weights reproduce saved evaluation scores within the documented numerical tolerance, that batch and individual scoring agree, and that the example script works against the exported package. Preserve the selected checkpoint and dataset manifests so the published model can be traced back to its training run.


Reusing this protocol for another language pair:

- Create a new experiment directory and model repository; keep this run and all reserved splits immutable.
- Record source/target language codes, dataset repository/revision, original and normalized columns, licensing and direction. Map rows into the common source/target fields and document that mapping; existing code uses nb and nn internally and currently assumes Bokmål first.
- Pin the base model, tokenizer, annotation models, dependencies and runtime. Choose and record the seed before training.
- Rewrite and freeze the task instruction for the new pair. Use byte-identical prompt rendering during training, evaluation and inference; verify the yes/no answer tokens.
- Review every language-specific rule: personal-pronoun paradigms, grammatical gender, case, honorifics, reference resolution, asymmetric predicates, negation, dates, decimals, thousands separators, units, entity transfer and capitalization. Do not copy Norwegian rules without validating their meaning in the new pair.
- Rebuild grouped splits before augmentation. Keep calibration, blind and supplied-test partitions out of gradient updates and checkpoint selection. Give every generated example source IDs, group IDs, edited spans, labels, operation provenance and split.
- Build positive controls as carefully as negatives, including acceptable paraphrases and grammatical restructuring. Create independent regression fixtures for known cross-language ambiguities.
- Run baselines and matched-budget pilot ablations; then freeze the selected recipe, full-data seed, maximum budget, evaluation frequency, stopping metric, patience and minimum improvement before full training.
- Run the four-GPU initialization, scoring, checkpoint and early-stopping smoke checks on the destination hardware. Rebuild caches before DDP; never assume another cluster has Olivia’s architecture or GPU transport settings.
- Preserve actual retained counts, example-family counts, exclusions, configs, prompt, hashes, metrics, best-checkpoint step, completion reason, calibration, held-out reports and inference verification. Publish the exact training and inference source, dependency versions, data manifests, adaptation instructions and limitations with the model card.

The reusable implementation guide and machine-readable run configuration accompany the final Hugging Face release. Language-specific augmentation rules require adaptation; this is not a claim that the existing code is a zero-change multilingual trainer.

The [reproduction and language-pair adaptation guide](https://bokmal-nynorsk-reranker-plan.peregilk.chatgpt.site/training-guide.md) records artifact schemas, commands, stopping behavior, portability checks, an operator-specification checklist and a copyable YAML handover record. Every operator must document eligibility, exact edits, language-specific inventories, positive controls, conditional versus overall rates, exclusions, provenance and automatic-label limitations. The handover must be sufficient to reproduce and adapt the workflow without relying on this conversation. Preserve reasons for changes and failed pilots alongside the successful run.