Config-Driven Experiments, or How I Stopped Editing Code to Change a Learning Rate

I used to change hyperparameters by editing Python code. Line 42 of train_teacher.py:

lr = 2e-5
batch_size = 16
epochs = 30

When I needed a different config, I'd change those lines, save, and re-run. Then change them back. Except sometimes I'd forget. Or I'd change lr but not the filename prefix, so the CSV would say lr2e-05 when I'd actually run lr 3e-5. Or I'd change batch size in the training script but not the evaluation script.

This is how I ended up reporting two different F1 values for the same distilled model in the same paper.

The Combinatorial Trap

Nine configs per model (3 learning rates × 3 batch sizes). Teachers and students had different epoch budgets. Distillation had fixed hyperparameters (α = 0.7, T = 4.0) that were design decisions, not search variables. Quantisation had its own backend config (GPU for FP16 and INT4, CPU for INT8).

Each decision was trivial. But they interact, and they interact differently per pipeline stage.

Consider the distillation grid. Twelve teacher-student pairs, each inheriting training config from standalone fine-tuning. "Inherits" means I copied values from one script into another. Manually. Twelve times.

When I needed to add a fourth learning rate to the search, that meant editing four files: teacher script, student script, distillation script (which inherits from both), and whatever comparison script reads results. Miss one, and you get a mismatch between what you trained and what you report.

I missed one.

The Moment I Stopped

I was running the twelve-pair distillation grid. The notebook had a loop:

for teacher_name in ['bangla_bert_base', 'mbert', 'xlm_roberta']:
    for student_name in ['banglabert_small', 'sahajbert', 'xtremedistil', 'mixed_distil']:
        teacher_ckpt = f"checkpoints/{teacher_name}_best.pt"
        # ... 30 lines of distillation code ...
        # ... hardcoded lr, batch_size, alpha, temperature ...

The problem wasn't the loop. It was everything inside that was hardcoded: learning rate, batch size, epoch budget, alpha, temperature, sequence length, warmup ratio, weight decay, dropout, early stopping patience, threshold search range. None were in a config.

When I needed to test a different temperature, I had to find it inside the loop, change it, and hope I'd find it again when reverting.

What Config-Driven Actually Means

A YAML file capturing the full experimental state:

# config/distillation.yaml
distillation:
  alpha: 0.7
  temperature: 4.0
  max_epochs: 15

teacher_configs:
  - teacher_id: "bangla_bert_base"
    best_config: "lr_3e-5_batch_64_epochs_30"
  - teacher_id: "xlm_roberta"
    best_config: "lr_2e-5_batch_32_epochs_30"

students:
  - id: "sahajbert"
    checkpoint: "neuropark/sahajBERT"

The distillation config references the teacher's best config. Provenance is in the config itself.

The Unexpected Benefits

The methodology section writes itself. When we wrote Section 3 of the paper, the config files were the draft. "Learning rates were searched over {1e-5, 2e-5, 3e-5}" came directly from search_space. "Distillation hyperparameters were fixed at α = 0.7 and T = 4.0" was verifiable because the value was in the config, not buried in a script.

The [MISSING] tags become actionable. Our paper is peppered with markers: "preprocessing pipeline undocumented," "batch size not supplied," "confirm whether pos_weight was applied." Half would be impossible if configs existed from the start. The [MISSING] tags are symptoms of config-free development.

Reproducibility becomes mechanical.

python run.py --config config/distillation.yaml \
              --teacher bangla_bert_base \
              --student sahajbert

The config specifies everything. The script reads the config. If it's wrong, the error is in the config, not scattered across four scripts.

The Failure Modes

Config drift. The config says [0.3, 0.7] for threshold search, but I'd expanded it to [0.2, 0.8] in the script and forgotten to update it. The paper reported the config value, not the actual value.

Implicit defaults. Our quantisation config didn't specify the execution device. I assumed FP16 and INT4 run on GPU, INT8 on CPU. When a new format was added, the script defaulted to GPU. Wrong.

Config is not code. Same config (α = 0.7, T = 4.0), different results, because the baseline implementation used a softmax head while ours used a single-logit formulation. The config was identical. The code was not.

What I'd Tell My Past Self

Start with the config, not the code. Before writing a training loop, write the config that describes what it will do. The config forces you to enumerate every decision before you've written any Python.

Make the config the single source of truth. If the paper says α = 0.7, the config should say α = 0.7, and the script should read alpha from the config. If any of the three disagree, you have a bug that might surface as a 0.0046 F1 discrepancy between two result sets.

Accept that the config will be wrong sometimes. The goal isn't perfection. It's making config-driven the default, so that breaking the pattern is the exception.

The config files for this project are maybe 200 lines of YAML. The code that reads them is maybe 500 lines of Python. The methodology section is maybe 3,000 words. The config is the cheapest part and the most valuable.

---

This post is part of a series on research engineering lessons from building a distillation and quantisation pipeline for Bangla hate speech detection. The full paper is titled "When Smaller Teachers Win."