# Reproducing and adapting the equivalence reranker This is a binary semantic-equivalence scorer, initialized directly from Qwen/Qwen3-Reranker-0.6B. It does not generate translations. Authors: Thea Tollersrud and Per Egil Kummervold, NB-ASR project. All validation is automatic. ## Frozen decisions for this run - Dataset: NbAiLab/merged_npk_ndla_parallel_paragraphs_normform_cc0, revision 619e36b85e4943f23cd85b66edcde331995e4872. - Base revision: e61197ed45024b0ed8a2d74b80b4d909f1255473. - Primary columns: bokmaal_normform and nynorsk_normform; original nb/nn are retained for robustness evaluation. Internal nb means source, nn means target. - Training: 704,173 retained source pairs; 9,625 person-inventory-only exclusions restored. Other integrity exclusions remain. See data_manifest.json. - Reserved source pairs: selection 10,000, calibration 10,000, blind 5,000; supplied validation/test 1,000 each. Descendants inherit their source group. - Seed 42 is fixed before full training. There is no competition among seeds. - The revised recipe assigns 15% of anchors to plain, unprefixed person-reference pairs with varied word order, negation and singular/plural subjects. Remaining anchors follow the prior natural/structured/transfer mix. The earlier recipe's step-500 development diagnostic accepted all 36 person changes in fronted-time sentences, despite excellent hypothetical-template scores. This motivated the correction. Diagnostic strings are not training examples. Full selection now includes plain-reference controls, and the independent person-reference development grid plus grammatical-gender regressions must pass before full training. The grid originally had 48 examples. A step-1000 follow-up expanded it to 128 after object-bearing reading and eating constructions exposed more errors: 32/32 positives passed, but 9/96 negatives were accepted. This expanded grid remains development data, not a final-test generalization estimate. - V5 completed at step 1,562 with development loss 0.0322193, but its expanded development grid accepted 11/96 person changes (all 32 positives passed). V6 keeps the same seed, base, 100k-anchor pilot budget, global batch, learning rate and 15% plain-reference rate, adding twelve object-bearing predicates to half of that branch. Reading/eating/buying/writing diagnostic predicates are excluded from the new training branch. V6 uses four GPUs on one node for speed; it is a recipe repair, not another seed. Selection's plain-reference templates broaden with this generator, so aggregate development loss is not directly comparable across V5/V6. Compare the unchanged independent regressions and shared natural challenge families separately. Final held-out data is untouched. - V6 completed in about 29 minutes on four GPUs; its best step was 1,000 with development loss 0.0007045. It still missed three eating/person changes, rejected the home and human gendered-participant positives, and exceeded the strict FP32 single/batch tolerance. Full-data launch was cancelled. V7 changes only the training annotation file, adding 127,219 same-number person negatives from 71,600 real training pairs. Pilot budget, seed, generator rates and the V6 evaluation inputs remain fixed. The training annotation manifest records all source/code/output hashes. This pilot tests natural-context supervision; it is not an additional full-data seed. - V7 completed 1,562 steps in 1,715 seconds; its best step was 1,500 with development loss 0.0265206. The completed-best independent diagnostic passed all 128 person cases, 12 grammatical-gender cases and six semantic examples. Numerical consistency failed: maximum single/batch score difference was 0.0000135452 versus the unchanged FP32 limit of 0.00001, on a correctly rejected eating/person change. Decisions, reload and training-interface checks agreed. Preserve `runs/natural-reference-v7-complete` as development evidence. Pilot semantic qualification does not certify numerical consistency of a future full-data model; the final exported model must pass all release checks. A follow-up in `runs/natural-reference-v7-numerical-analysis` isolated the discrepancy: computing the output projection in FP64 still left a 0.0000134043 single/batch difference. Different transformer hidden states dominate it; output-head rounding alone is not the remedy. No production inference change or tolerance relaxation was made. - V7 full selection (10,000 source pairs; 292,304 challenges) failed the entity gate: two of 154 structured entity reversals were accepted (1.2987%, limit 1%). The worst other protected-family rates were number 0.01858%, pronoun 0.00667% and meaning transfer 0.00402%; the lowest positive-control acceptance was 99.7669%. Expanded person and grammatical-gender regressions passed. Full-data training was withheld. `runs/selection-full10k-focus-v7-natural-reference/` records qualification and exact reconstructed error inputs. The failing extracted entity strings include function-word prefixes and short lowercase names. They remain in selection; do not remove them or copy them into training to satisfy the gate. The next development hypothesis is increased structured entity-role sampling at a fixed total structured rate and pilot budget. - V8 tests that sampling hypothesis with `structured_entity_weight=2`, giving conditional activity:quantity:entity weights 1:1:2 instead of 1:1:1. The total structured branch rate remains one third after plain-reference and transfer branch selection; expected entity anchors rise from about 7.08% to 10.625% of all anchors. The actual mutation family is still sampled within each frame, so entity-frame counts differ from entity-negative counts. Seed 42, direct upstream base, 100k anchors, 1,562 steps, four GPUs, learning rate, natural annotation file, language generators and all evaluation inputs remain fixed. Full-data seed policy and gates remain unchanged. The preflight record in `runs/entity-weight-preflight-v1` confirms exact legacy output at default weight and unchanged total branch membership on 10k actual training rows; the cache records actual tokenizer/filtering counts. This is a development recipe change, not another full-data seed or a held-out result. V8's step-500 development loss was 0.0316768. Its snapshot passed 128/128 person cases, 12/12 grammatical-gender cases and numerical consistency (maximum FP32 single/batch difference 0.0000043083), but rejected the valid human-participant pair "Hun hjelper ham." / "Ho hjelper han.". A separate replay of the entire existing 154-example selection entity-role family rejected all reversals and accepted 153/154 matched positives. These are early development results, not qualification of the completed checkpoint. The replay's exact inputs and hashes live in `runs/entity-role-selection-diagnostic-v1`; model-specific reports live in `runs/entity-weight-step500-diagnostic`. No selection errors were removed. V8 completed 1,562 steps in 1,740 seconds; final step 1,562 was best with development loss 0.00036594. Completed-best checks rejected the valid human participant pair again and accepted one `eat_food:others->addressee` negative (127/128 person cases correct). All 12 noun-gender checks and numerical consistency passed (maximum difference 0.0000066757). Conditional full-data qualification/launch jobs were cancelled; full selection continues for the experiment record. The output-head and protected-entity-slot audits do not establish a cause of these semantic errors. The next development hypothesis is a small matched-pair branch of valid third-person human translations with digit-only negatives: teach legitimate gender/case forms while never changing han/hun/ho/ham/henne in a negative. Diagnostic sentences remain excluded from training, and all existing evaluation gates remain fixed. - V9 tests `human_reference_rate=0.06` conditional on not taking the 15% plain person branch: about 5.1% of all anchors receive valid third-person human translations with a digit-only negative. Female-agent/male-patient and male-agent/female-patient pairs use hun/ho, ham/han and henne/henne without changing those forms in a negative. Five ordinary transfer predicates and varied fronted word order avoid copying the independent diagnostic sentences. Each positive/negative pair differs in exactly one digit value on one side; typed participant identities remain fixed. The existing plain branch retains identical membership, while this new branch replaces a small fraction of the remaining natural/transfer/structured anchors. Entity weight stays 2, and the seed, base, 100k-anchor budget, 1,562 steps and other rates remain fixed. Selection/development adds a separate human-positive and number-negative family (two examples per source row); final held-out controls use different predicates and time phrases. Existing generated challenge objects remain unchanged, but aggregate development loss now includes the new families and cannot be compared directly with V8. Compare shared families and unchanged independent regressions. `runs/human-reference-preflight-v1` records legacy zero-rate compatibility, digit-only/protected-form invariants and evaluation compatibility. This branch teaches preservation of legitimate human pronouns; it does not add gendered pronoun substitutions or train on development diagnostic inputs. - BF16 full-model training, LR 2e-6, AdamW, cosine with 3% warmup, norm clipping 0.1, global batch 256, per-GPU microbatch 32 on four GPUs on one node. - Maximum 2,048 prompt-inclusive tokens. No silent truncation. Internal chunks respect max_micro_tokens=8192 while preserving the effective global batch. - Maximum two epochs. Every 1,000 steps and at the training ceiling, evaluate a fixed 1,000-paragraph selection subset in FP32, including controlled probes. - Stop after three checks without a loss decrease greater than 0.001 from the last meaningful-improvement reference. This is absolute balanced category log loss. Smaller decreases can accumulate to a meaningful improvement. - Save the checkpoint with the lowest observed loss independently of patience. Post-training evaluation uses best/, not final/. Preserve both stopping and best-checkpoint metadata. The ceiling's cosine schedule is not shortened after stopping. Do not select an epoch or seed using blind/test results. ## Artifact map ### Numerical execution profile introduced after the V11 pilot V11's completed best checkpoint passes all 128 person-reference cases, all six semantic examples and twelve grammatical-gender checks. Its entity-role replay has 100% positive acceptance and 1/154 negative false acceptance, within the fixed gate. The original SDPA/unrounded-padding scorer fails the FP32 batch consistency check: maximum difference 0.0000207573, with both decisions correct. An isolated four-profile development probe retained the unchanged 146 fixtures and 0.00001 tolerance. Eager attention alone and padding alignment alone fail; their combination passes with maximum difference 0.00000201166. A further selection-only check passes on 670 inputs spanning 163–581 tokens at batch sizes 6, 16 and 32. These are numerical engineering checks, not held-out accuracy. Reports and scripts are in `runs/person-lexicon-v11-numerical-profiles`. `scoring_profile.py` defines `eager-pad128-v1`: eager attention and right padding to multiples of 128 tokens. The production implementation is checked separately in `runs/scoring-profile-preflight-v1`: all 146 unchanged regressions pass, as do 734 corpus/stress inputs spanning 163–2,038 tokens across batch sizes 6/16/32. The maximum stress difference is 0.00000149012 and training-interface difference is zero; backend/mode restoration, including exceptions, passes. CPU fixture checks separately verify profile metadata and package inclusion; they are not model-quality evidence. The fresh four-GPU stopping smoke, job 2509361, completed successfully at step 3/6 with synchronized early stopping and the new profile recorded in its evaluation reports (`runs/scoring-profile-smoke-v1`). Full-data launch requires this smoke, passing production preflight reports and matching numerical source hashes. Gradient updates retain SDPA; `evaluate.score` temporarily switches attention for evaluation and restores it even if evaluation raises an exception. This changes numerical execution, not the prompt, labels, threshold or tolerances. Earlier frozen campaigns retain their original source and reports; future full-data development reports identify the new scoring profile, so their floating-point losses need not match older reports exactly. The exported scorer, evaluation reports, inference verification, frozen release snapshot and `reranker_config.json` record the same profile. Release checks reject missing or mismatched profiles. Package `scoring_profile.py` beside `reranker.py` and `prompt.py`. The release freeze hashes it and `human_reference.py` as well as the other runtime generators. Adapting to another machine or numerical profile requires fresh consistency, length, interface and final-release verification; passing these development samples does not guarantee numerical identity for all possible inputs or hardware. The full-data four-GPU array requests six hours of wall time, based on V11's 30.9-minute pilot and approximately 11,002 maximum full-data optimizer steps before early stopping. The working estimate is 3–4 hours, with a safety margin; queue time and actual throughput remain uncertain. This allocation limit does not replace the fixed two-epoch budget or early-stopping rule. If an allocation ends before completion, resume its optimizer-complete checkpoint rather than starting another seed. Preserve `runs/full-data-resource-estimate-v1.json` as the evidence and assumptions behind this resource choice. V8 full selection subsequently completed: supplied/original positive acceptance 99.91%, minimum populated positive-control acceptance 98.6014%, worst protected false acceptance 1/154 (0.6494%, entity/role). These category-only gates pass, but the independent 1/96 person false acceptance and semantic failure still disqualify V8. See `runs/selection-full10k-focus-v8-entity-weight/qualification-findings.json`. V9 completed its fixed 1,562-step pilot with best step 1,562 and development loss 0.005728459359047678. All six semantic examples, twelve grammatical-gender regressions, strict numerical consistency and the 308-case selection-only entity-role replay pass. However, 3/96 independent person-change negatives are accepted, so its conditional full-data launch was cancelled. Full selection is retained as development evidence; final held-out partitions remain unused. The completed V9 full selection report has supplied/original positive acceptance 99.89%/99.88%, minimum populated positive-control acceptance 99.7669%, zero entity/role false acceptance and worst protected false acceptance 0.037154% (numbers). These category gates pass, but the independent 3/96 person errors still disqualify it. See `runs/selection-full10k-focus-v9-human-reference/qualification-findings.json`. V10 is a matched-budget sampling ablation: increase `plain_reference_rate` from 0.15 to 0.30, keeping seed 42, source subset, base, generators, evaluation inputs, learning rate and 1,562-step budget fixed. Half of this branch remains object-bearing, raising its expected overall share from 7.5% to 15%. This tests exposure rather than copying diagnostic sentences into training. With the unchanged conditional human rate 0.06, transfer rate 0.25 and structured rate one third, expected anchor shares are plain 30%, human 4.2%, transfer 16.45%, structured 16.45% and natural 32.9%. Report actual cache counts as well. No qualification claim follows from changing these rates; all gates remain fixed. V10 completed at best step 1,562, development loss 0.0003005414420563073, but accepts 5/96 independent person-change negatives and fails strict numerical consistency (maximum score difference 0.0000231862). Semantic/gender checks and the entity-role replay gate pass. Its conditional full-data launch was cancelled. Increasing plain-reference exposure alone did not fix the predicate coverage gap. V11 tests `plain_reference_extended=true` at the same 30% plain-reference rate, seed and pilot budget. Only the training object's predicate pool expands from 12 to 36, adding varied predicates including short Nynorsk forms. Reading, eating, buying and writing remain excluded. Half the plain branch still uses object-bearing predicates; the other half and all evaluation inputs are unchanged. The false default preserves earlier generation, and combining extended mode with held-out templates is rejected. Preserve the inventory and preflight output when adapting this option to another pair. This is a development repair, not an independent test result or a promise of cross-predicate generalization. V11 completed full selection qualification on 10,000 selection paragraphs: 312,304 generated challenges, 32 classification errors (99.98975% pooled accuracy at threshold 0.5). Supplied/original positive acceptance is 99.91%/99.90%; the lowest populated positive-control acceptance is 99.7669%. Worst protected false acceptance is 1/154 entity-role challenges (0.64935%); number-value false acceptance is 1/5,383 and controlled person/pronoun changes have zero false acceptances. Repeated synthetic patterns contribute to the pooled score; it is not independent natural-error or final blind-test accuracy. Preserve `runs/selection-full10k-focus-v11-person-lexicon/qualification-findings.json`. The single seed-42 full-data campaign was submitted as `campaigns/final-full-v2-pronoun-safe`: cache job 2509418, four-GPU training 2509419, post-training launcher 2509420. The frozen configuration uses direct Qwen base weights, all 704,173 retained training paragraphs, a two-epoch ceiling, evaluation every 1,000 steps and patience-three early stopping. Source hashes and launch settings were checked in `runs/full-launch-audit-v11.json`. Its own selection, calibration, held-out and inference gates remain required; pilot qualification does not establish final-model quality. The release's training/config.json contains the actual arguments, including paths, rates, budgets and stopping settings. training/prompt.txt is the exact rendered input. prompt.py is the rendering implementation. The classifier uses the next-token no/yes logits; check both answer strings tokenize to one token. - prepare.py: pinned download, column mapping and initial splits. - audit_data.py / prepare_pronoun_safe_data.py: integrity flags and corrected training retention. The latter removes only person-inventory quarantine reasons. - annotate.py / annotation_loader.py / merge_annotations.py: pinned Humit and NER, conservative span edits, merge and provenance. Assets are not bundled; their identifiers/revisions and hashes are recorded in annotation_asset_manifest. - augment.py: numeric value/assignment changes, equivalent number formats and conservative semantic edits. - structured.py / extra_probes.py / build_structured_slots.py: controlled propositions and role, pronoun, quantity, modality and omission challenges. - plain_reference.py: ordinary synthetic statements with explicit person changes, without a hypothetical prefix. Training and blind/test use separate predicates and fronted-time phrases. Pronoun forms recur across splits; this is not a holdout of the entire plural paradigm. check_person_reference.py supplies a separate development diagnostic, not a corpus-level generalization estimate. `plain_reference_extended` enables the broader training-only object inventory; it must propagate through cache preparation, training and full-data config. - human_reference.py: ordinary valid third-person human correspondences paired with a single changed digit value. `human_reference_rate` is conditional after the plain-person branch and before transfer/structured/natural selection; its default of zero preserves prior training generation. The added evaluation families are separately gated: positive acceptance at least 98%, number false acceptance at most 1%. Language adaptation must replace both participant forms and predicates, including the separate held-out inventory. - pronoun_policy.py: excludes mutations involving han/hun/ho/ham/henne. These exclusions are specific to this language pair. Valid noun-gender anaphora remains positive. No natural pronoun negatives survive in the original cached edits after this exclusion; controlled accuracy is not a natural-error estimate. `natural_reference.py` and `prepare_natural_reference.py` prepare a later training-only addition from cached normalized Humit tags: require one aligned explicit subject person next to a finite verb in each language, and change one side to a different person of the same number. Quoted/metalinguistic text, gendered forms and honorific De are excluded. This is a single-person substitution, not a paired role swap. The explicit inventory avoids assuming that Humit's Bokmål `ent` and Nynorsk `eint` tag spellings are identical. Labels inherit automatic tags and the corpus positive-alignment assumption. Training coverage does not establish held-out natural-pronoun accuracy. - prepare_ddp_cache.py: build examples.pt, config.json, prompt.txt and data_stats before distributed startup. Do not start DDP with a missing cache. - train.py / checkpointing.py / early_stopping.py: full fine-tuning, exact optimizer-state recovery and synchronized early stopping. `structured_entity_weight` is a positive integer configuring conditional activity:quantity:entity frame weights as 1:1:WEIGHT. Its default of 1 preserves the earlier generator. Cache preparation, training and full-data launch must use the same value; changing it requires a fresh cache and experiment. - evaluate.py / statistics_tools.py: category, side, domain, length and operator reports, source-group uncertainty, distinct scored input counts and explicit exclusions. Synthetic sentences can repeat across source groups; the reported source-group intervals are not evidence of unseen-template generalization. - select_recipe.py: selection qualification; calibrate.py: threshold on reserved calibration positives; freeze_release.py: immutable model/input hashes. - check_inference.py / check_grammatical_gender.py: scoring consistency and language-specific regression fixtures. - release_gate.py / render_model_card.py / package_release.py: held-out gates, documented package and upload. Never upload a failed candidate as validated. runtime_manifest.json records the tested CUDA, GPU, PyTorch, Transformers, NumPy and safetensors versions. The training runtime is NVIDIA's ARM64 PyTorch 25.06 container on Olivia; inference requirements alone do not recreate that runtime. asset_manifest.json records the corpus and base downloads, while annotation_asset_manifest.json records the Humit, NorBERT and NER assets. training_annotation_manifest.json records the actual training edit file's preparation, counts and hashes when available. New natural-reference edits must be generated from the training split only, with a fresh output directory: ```bash python prepare_natural_reference.py --data /path/to/train.jsonl \ --raw-dir /path/to/normalized-humit-cache \ --existing-edits /path/to/existing-edits.jsonl \ --output /path/to/new-training-annotations ``` Use the resulting edits.jsonl as the training config's annotations path. This does not modify the reserved evaluation annotations. Preserve the generated manifest and generator copies; adapt the linguistic inventory for a new pair. Training manifests record counts and hashes rather than bundling the corpus. Reconstruct data and annotation assets from the pinned sources; old absolute cluster paths are provenance, not portable download locations. Source manifests cover the frozen Python files. Slurm scripts describe Olivia, including its ARM64 runtime, four-GPU node limit and tested NCCL setting; they require adaptation on another machine. Historical pilot launchers are experiment records, not the recommended entry point for a new pair. ## Running a new full-data experiment 1. Create a fresh experiment directory. Copy the released Python source into it together with prompt.py. Install the recorded training runtime/dependencies and download the pinned base and annotation assets into local directories. Never overwrite this release, its reserved data or its checkpoints. 2. Recreate split JSONL, annotation JSONL and structured-slot JSONL. Review train.load_rows, evaluate.build_challenges and the preparation scripts for the exact schema. Rows carry id, group, nb, nn, screened and domain; preserve original_nb/original_nn when available. The group is the split-isolation unit. 3. Write a fresh config.json using this release's training/config.json as a template. Replace every data, model, annotation, dev and output path. Freeze the seed, prompt, sampling rates, budgets and stopping rule before training. 4. In the same environment that will train, run: ```bash python prepare_ddp_cache.py --config /absolute/path/to/config.json ``` Pass the same configuration to train.py as CLI flags: underscores become hyphens, true booleans become flags, false/null entries are omitted, other values become a flag/value pair. slurm/final_train_array.sh illustrates this conversion. Do not substitute different sampling arguments after caching. 5. Run distributed training on one node using torchrun with four processes and those arguments. Match global_batch to world_size times micro_batch times accumulation. Verify the hardware-specific NCCL configuration with a short all-reduce and early-stopping smoke test before committing the full run. 6. Check complete.json, best_checkpoint.json, family counts and exclusions. Early stopping is successful completion. SIGTERM/stop_after is interruption and requires resuming the optimizer-complete checkpoint. Resume must preserve world size, arguments, code, cached examples and hashed data. 7. Evaluate best/ on full selection; qualify it against predeclared targets. Require at least 98% acceptance in supplied/original positives and each populated valid-positive control family (including equivalent number formats), and at most 1% false acceptance in every protected semantic-risk bucket. Calibrate on the reserved calibration partition. Freeze the chosen weights, prompt, threshold, code and final challenge inputs. Score validation, blind and test and run exported inference verification. Publish only if the gate passes; retain all reports and document failures. Reusing a blind/test result to tune another candidate makes it development data for future claims. ## Adapting to another language pair Record a new dataset/base revision and license, source/target codes, direction, column mapping, normalization policy and split construction. Rewrite the exact task instruction; keep identical rendering for training and inference. Avoid assuming that the Norwegian instruction is appropriate for another pair. Review and replace language-specific rules in annotate.py, pronoun_policy.py, augment.py, structured.py, extra_probes.py, plain_reference.py, human_reference.py and natural_reference.py. Review personal pronouns, grammatical gender, case, honorifics, reflexives, ambiguity, predicates, negation, number/date formats, units, inflections and entity alignment. Configure suitable tagger/NER assets. Use matched positive controls to prove that legitimate differences are accepted. Replace the Norwegian grammatical-gender fixture with independent regressions for the new pair. These decisions require linguistic adaptation; the current implementation is not a zero-change multilingual pipeline. Build new grouped splits before augmentation. Run upstream baselines and matched-budget pilots, then freeze the full-data recipe and seed. Publish the actual training distribution, natural versus controlled challenge coverage, failure and exclusion counts, stopping/best steps, thresholds, uncertainty and automatic-label limitations. Include runnable inference examples, the exact prompt, training config, code, dependency/runtime versions, hashes and manifests. ### Required record for each new language pair Keep the following record beside the experiment config and fill it before full training. Record decisions and their reasons, including failed pilots; this prevents a future reader from having to reconstruct the protocol from job logs. | Record | What to preserve | | --- | --- | | Task and ownership | Language pair, scoring direction, intended use, authors, license and output model repository. | | Inputs | Dataset and base-model revisions, column mapping, normalization, data counts before/after each exclusion, asset versions and hashes. | | Split isolation | Group definition, deduplication policy, exact reserved IDs and split hashes. Check repeated or overlapping paragraphs across groups before augmentation; all derived examples stay with their parent. | | Language decisions | Pronoun/entity alignment rules, ambiguous forms left untouched, numeric formats considered equivalent, and examples explaining each choice. | | Challenge construction | Each operator's eligibility rules, changed side, positive controls, rejection reasons, provenance and natural/synthetic counts. Distinguish paired swaps from single substitutions. | | Experiments | Hypothesis, baseline, omitted/added operators, fixed budget, seed, config/source hashes, job IDs, results and reason to retain or reject each recipe. | | Final training | Frozen recipe, cache hash, hardware/runtime, actual throughput, completion/stopping reason and selected checkpoint. | | Validation | Development versus held-out roles, predeclared gates, threshold procedure, family denominators, distinct inputs, uncertainty, exclusions and remaining failure cases. | | Release | Exact prompt, inference example, scoring direction/threshold, numerical verification, model card and immutable release manifest. | For an ablation, keep the source pairs, seed, evaluation inputs and training budget fixed, change the specified operator or rate, and record the resulting sampling distribution. Use selection/development results to decide the recipe. Do not repeatedly consult the final held-out reports to choose an ablation. When automatic labels are used without manual review, state that explicitly and retain uncertainty about ambiguous real translations. ### Copyable handover record Start the next language pair with this record in its own experiment directory. Replace every placeholder; do not inherit Norwegian linguistic assumptions or reuse this run's reserved IDs for a different corpus. ```yaml task: source_language: REQUIRED target_language: REQUIRED scoring_direction: REQUIRED authors: REQUIRED output_repository: REQUIRED inputs: dataset_id_and_revision: REQUIRED dataset_license: REQUIRED source_and_target_columns: REQUIRED base_model_id_and_revision: REQUIRED tagger_and_ner_assets_and_revisions: REQUIRED normalization_and_exclusion_policy: REQUIRED split_ids_groups_counts_and_hashes: REQUIRED language_rules: exact_prompt_file: REQUIRED safe_and_excluded_pronoun_changes: REQUIRED numeric_equivalences_and_value_changes: REQUIRED entity_alignment_and_role_swap_rules: REQUIRED positive_controls_and_independent_regressions: REQUIRED experiments: baseline_and_ablation_records: REQUIRED frozen_full_training_config_and_seed: REQUIRED code_cache_and_annotation_manifests: REQUIRED runtime_hardware_and_scheduler_commands: REQUIRED validation: selection_calibration_and_heldout_roles: REQUIRED predeclared_gates_and_stopping_rule: REQUIRED automatic_label_limitations: REQUIRED handover: reproduce_data_command_and_outputs: REQUIRED build_cache_command_and_outputs: REQUIRED train_and_resume_commands: REQUIRED evaluate_calibrate_freeze_and_package_commands: REQUIRED actual_best_checkpoint_and_completion_reason: REQUIRED reports_model_card_and_release_manifest: REQUIRED ``` Each command record must include its working directory, environment, arguments, input paths and expected output paths. Resolve local paths from the recorded asset revisions and manifests, rather than treating historical cluster paths as portable inputs. Keep failed experiments and their explanations alongside the successful recipe. A handover is complete when another researcher can rebuild the artifacts and identify every language-specific decision from these files without relying on this conversation. ### Documenting an augmentation for reuse For every operator, keep a short specification beside its implementation with: - The semantic fact being changed, and the eligibility conditions under which the change makes the translation incorrect. - The source/target forms, tag mappings, span alignment and ambiguity exclusions. Separate language-independent logic from each language's inventories. - A complete source/target positive pair and its corresponding negative pair, identifying the changed side and exact edited spans. - Positive controls for legitimate translation differences; train versus evaluation predicate inventories and any vocabulary shared between them. - Sampling probability, branch order and conditional versus overall rate, plus actual generated counts, rejected cases and provenance fields. - Automatic invariants and their saved results. State which assumptions those checks cannot verify, especially corpus alignment and tagger correctness. The digit-change human-reference operator illustrates the distinction: its language-independent invariant is that exactly one side changes only digits, while its participant forms, word order, predicates and units are specific to Bokmål/Nynorsk. Preserving a participant string alone does not establish that a different language's grammatical construction preserves its referent. Keep a fresh pair's filled handover record, operator specifications, commands and reports in the experiment directory before freezing its source manifest. Keep frozen historical campaigns unchanged; improvements to this guide apply to the next frozen campaign and final release. Record unavailable checks explicitly rather than presenting the handover template as completed evidence. ### Portability checks before submitting jobs The `nb` and `nn` field names are internal schema conventions. They can carry a new source/target pair, but the prompt, linguistic generators and documentation must all describe that pair consistently. A language-name substitution alone does not adapt the generators. Inspect `prepare.py` for dataset/column/split assumptions; `download_assets.py` and the annotation scripts for model identifiers; `prompt.py` for the task instruction; and the language-specific modules listed above. Also replace Norwegian assumptions in `plain_reference.py`, `human_reference.py`, `natural_reference.py`, `prepare_natural_reference.py`, `check_person_reference.py`, `check_inference.py` and the model-card examples. Rebuild every annotation, structured-slot and training cache after these changes. The campaign launchers, Slurm scripts, status publishing and `package_release.py` contain project paths or repository names. Review them before reuse and set a fresh output repository rather than invoking the current campaign unchanged. Update `model_card.md.in` and `render_model_card.py` for the new dataset, language names, authors and actual counts. Follow the explicit config/cache/training procedure above when using a different scheduler. Before a full run, verify one positive and one unambiguous negative from every operator, the ambiguous-form exclusions, group isolation, prompt/token limits, single-versus-batched scoring, checkpoint recovery and synchronized stopping. These are automatic consistency checks, not a substitute for measuring generalization. Preserve their outputs with the experiment record. ## Parallel spelling-view ablation (separate from the frozen baseline) The user identified three meaning-preserving candidate views: original NB to original NN; original NB to normalized NN; normalized NB to normalized NN. In the prepared schema these are `(original_nb, original_nn)`, `(original_nb, nn)` and `(nb, nn)`. Current natural positives use only the last view. Keep the running final-full-v2-pronoun-safe experiment frozen; the user subsequently authorized a parallel matched-budget pilot. This does not replace the already-running full-data baseline or launch another full seed. A read-only exact-string count on the current 704,173 retained training IDs finds 2,112,519 candidate views, 982,437 distinct pairs within source rows and 982,404 globally distinct pairs. Normalized-only has 704,140 unique pairs; the proposal adds 278,264 (+39.5%). All three views coincide on 449,322 rows. These counts precede additional view eligibility checks. Preserve `runs/three-view-feasibility-counts.json` and its source hash. Expand only the current retained training-ID allowlist. Existing deleted rows stay deleted; every descendant inherits its original paragraph/document group and partition. Deduplicate exact pairs while retaining all source/view provenance. Do not independently split views or canonicalize away the spelling differences being studied. Existing `original_eligible=False` in pronoun-safe-v1 is a blanket normalized-only campaign policy, not a per-row original-view quality verdict; assess candidate-view eligibility explicitly. Preserve numeric and entity integrity checks and the protected gender-pronoun policy. For a matched-exposure pilot, sample one eligible distinct positive view per paragraph per epoch rather than tripling every paragraph. Maintain the same number of anchors, optimizer budget, seed and component rates. Construct each negative from the actual chosen view with matching side-specific annotations; transferring offsets between text versions is unsafe. Build positive-conflict checks across all eligible views before accepting negatives. Measure actual view/family counts, excluded views and distinct input counts. Reusing an immutable cache across epochs will not implement per-epoch resampling: either materialize a deterministic epoch-specific cache or clearly document a fixed view assignment for the matched pilot. Compare normalized-only and mixed-view recipes on the same development inputs: valid spelling-view acceptance and worst populated view, protected corruption false acceptance, calibration and ordinary person/role regressions. Use only selection data for this ablation; preserve untouched final blind/test partitions for the chosen model. Any expansion of final evaluation must be declared before consulting its outputs. Full-model release checks remain required regardless of pilot performance. The parallel V12 pilot has been submitted; see the concrete run record below. The authorized V12 campaign is `campaigns/focus-v12-three-view`, run `runs/focus-v12-three-view-s42`. Preparation command: ```bash python3 bokmal_nynorsk/prepare_three_views.py --output /cluster/work/projects/nn30001k/nb-asr/nb-asr-emb/bokmal_nynorsk/data/three-view-pilot-v1 --limit 100000 python3 bokmal_nynorsk/check_three_views.py bokmal_nynorsk/data/three-view-pilot-v1 bokmal_nynorsk/runs/three-view-preflight-v1/report.json python3 bokmal_nynorsk/submit_three_view_pilot.py ``` These commands create immutable outputs and refuse to overwrite initialized experiments; inspect submitted jobs before any retry. One preparation attempt failed before submission because of an incorrect requirements filename; its unused snapshot is retained as `focus-v12-three-view-initialization-incomplete`. The corrected campaign submitted cache 2510069, four-GPU training 2510070, full selection 2510071 and independent diagnostics 2510072. The last two depend only on training and can run in parallel. No automatic full-data V12 or held-out launch is attached. V11 full-data job 2509419 continues independently. Preparation retains the exact previous hash-selected 100,000 pilot source IDs. All three candidate views pass inherited numeric alignment. Selected pairs are globally unique within the pilot; 18,994 differ from normalized-only. A view's canonical name is the first spelling combination producing that string pair; its aliases retain the other combinations, so name counts alone must not be interpreted as distinct-spelling exposure. There are 982,404 eligible positive hashes across all 704,173 retained training rows. All protect against generated negative collisions, even for paragraphs outside the pilot subset. Cached annotation edits are reconstructed from their exact edited-side source surface. That surface must equal the selected spelling; only the untouched opposite side is substituted. Of the eligible annotations scanned, 2,516 are excluded because their edited-side surface differs. No offsets are transferred between different spellings and no new original-surface Humit tagging is claimed. Prepared annotations include 15,298 pronoun edits, 24 entity edits and 24 valid reorderings. Their actual sampled counts come from the token-cache statistics. The CPU preflight checks all prepared annotations and source IDs, 4,000 default example outputs against frozen V11, and 4,000 mixed-view outputs for deterministic behavior, balanced labels, unchanged synthetic branches and zero positive-hash conflicts. Token lengths and model quality are verified separately. `evaluate_three_views.py` compares completed V11 and V12 pilot checkpoints on all three selection views using the same verified FP32 numerical profile. Duplicate pairs share one inference score; per-view totals and overlength exclusions remain explicit. Full selection and unchanged 146 semantic/language/ numerical regressions run separately. Identity pairs such as NB-to-identical-NB are not manufactured negatives: a different reference string does not establish that a shared string is invalid, nor that its meaning differs. Target-standard conformity, if desired, requires a separate task and evidence. The actual GPU evaluator smoke (job 2510073) completed successfully on 64 selection paragraphs; two loads of the immutable V11 checkpoint produced identical prediction files and per-view summaries. Preserve `runs/three-view-evaluator-preflight-v1/findings.json`. This verifies the evaluator workflow, not the full V12 result.