Experiment Tracking Before You Have 200 Runs You Can't Tell Apart

A post-mortem on the most embarrassing problem in my distillation and quantisation research.

I was writing the error analysis section of our paper when someone asked: "What's the hyperparameter configuration of the model that produced these predictions?"

I looked at the CSV filename:

bangla_bert_base_lr2e-05_batch16_epochs10_sahajBERT_lr2e-05_batch16_epochs10_...

And I felt that sinking feeling. The error analysis was from a teacher trained at lr 2e-5, batch 16, 10 epochs. The student (SahajBERT) was the same config. But our best result? Teacher at batch 64, lr 3e-5, 30 epochs. Student at batch 32, lr 2e-5, 20 epochs.

Two different models. The error analysis described the wrong one. And the 0.8626 headline against the 0.8336 evaluation? Not a validation and test distinction. A configuration mismatch.

The Scale

Seven models. Nine hyperparameter configs each. Five folds. That's 315 baseline runs before distillation even starts. Plus 12 distillation pairs, 32 quantisation configurations, latency measurements, threshold sweeps.

I lost count somewhere north of 400. And I couldn't tell you which CSV corresponded to which run for most of them.

Three pipeline stages side by side: fine-tuning with 315 runs, distillation with 12 pairs, and quantisation with 32 configs. The arrows between stages are marked with question marks, showing that no system recorded which run produced the checkpoint each next stage consumed. — Figure 1. Artifacts crossed three pipeline stages; run-level logging never followed them across.

Where Each Tool Broke

The notebook. Started clean. By the time I was running the distillation grid, re-running "cell 47" would silently overwrite a CSV that cell 83 needed. I spent an afternoon debugging F1 scores that dropped by 0.02, only to find I'd re-run a cell that exported predictions with the wrong threshold.

MLflow. Helped with logging. Didn't help with provenance. When the distillation stage needed frozen teacher logits from fine-tuning, there was no mechanism saying "this distillation run consumed checkpoint X from run Y." The dependency was in my head.

The manual spreadsheet. Worked until I needed to reconcile the distillation results (SahajBERT at 0.8626) against the quantisation results (same model at 0.858). Different checkpoints. Different exports. Never reconciled.

The Three Lessons

1. The problem isn't logging, it's provenance

When artifact A is consumed by stage B, which run of A produced it? MLflow tracks runs. It doesn't track artifacts across pipeline stages. I'd have a file called sahajbert_fold3_pred.csv and no way to know, three weeks later, what produced it.

2. Fold-level variance is the story

Our fold SDs ranged from 0.0057 to 0.0088. The gap between Bangla-BERT-Base and mBERT on fold means is 0.0049, smaller than a single fold's SD. Best-fold values overstate performance by margins comparable to model differences. If I'd only tracked "best F1," I'd have missed this entirely.

Horizontal comparison on a shared macro F1 scale. The model-to-model gap between Bangla-BERT-Base and mBERT measures 0.0049. The fold-to-fold spread within a single model runs from 0.0057 to 0.0088, so the model gap ends before the spread even begins. — Figure 2. Between-model gap against within-model fold spread, macro F1.

3. The reconciliation problem is the real killer

The quantisation compression ratios implied each student had been quantised from a different teacher. Was that intentional? No. Bookkeeping error. No tracking system caught it, because none of them had any notion of "these two result sets should agree with each other."

The Uncomfortable Truth

Experiment tracking in research is a different problem than in production. In production, you track a single model through CI/CD. In research, you track a space of models, and the interesting results live at the intersections.

The tool that would solve this doesn't exist. It would need to: track artifacts across pipeline stages, assert invariants across result sets, enforce provenance chains, and surface gaps in your tracking (you logged 9 configs for XLM-RoBERTa but only 3 for SahajBERT, and SahajBERT is your best model).

Until then: name everything by run ID, not config string. Build a reconciliation script. And when you report a number anywhere in your paper, validate that it matches every other place the same model appears.

I learned this during paper revision, when a co-author asked a question I should have answered in thirty seconds. It took three days.

This post is part of a series on research engineering lessons from building a distillation and quantisation pipeline for Bangla hate speech detection. The full paper is titled "When Smaller Teachers Win."