Writing Research Code That Isn't a Single 800-Line Notebook

Let me describe a file I once maintained. A Jupyter notebook that started clean for fine-tuning BanglaBERT on hate speech classification. By project end, it had grown to 800 lines containing: data loading, preprocessing, model initialization, training loop, validation loop, metric computation, MLflow logging, threshold calibration, confusion matrix plot, per-class F1 bar chart, and, in a cell I'd added at 2 AM, a manual section where I was copy-pasting F1 scores into markdown.

Cell 1 loaded data. Cell 12 trained the model. Cell 34 plotted results. Cell 41 exported predictions. Cell 53 was a scratch cell I never deleted. Cell 58 was error analysis. Cell 62 was a duplicate of cell 34 with different formatting.

I called this file main.ipynb. It worked until I needed to run the same procedure for seven models, twelve distillation pairs, and thirty-two quantisation configurations. I did not want to run 315 cells 315 times.

The Pipeline Has a Shape

Our paper's methodology is a five-stage pipeline:

1. Corpus preparation
2. Teacher fine-tuning (3 models × 9 configs × 5 folds)
3. Student fine-tuning (4 models × 9 configs × 5 folds)
4. Distillation grid (3 teachers × 4 students)
5. Quantisation sweep (4 students × 4 formats × 2 conditions)
6. Evaluation and error analysis

Each stage produces artifacts consumed by the next. This is a directed acyclic graph. The code should reflect that.

The mistake I made, and that I see in almost every research codebase, was treating this graph as a linear script. But the pipeline isn't linear. It branches (three teachers, four students), merges (distillation pairs), and fans out again (quantisation formats). A linear notebook can't represent this without becoming a tangled mess of global variables and cell-order dependencies.

What Ended Up Where

Scripts for pipeline stages. Training, distillation, quantisation, and evaluation went into Python scripts. Not because I like scripts more than notebooks, but because these stages need to be run many times with different configs, reproduced without human intervention, and handle failure gracefully. When XLM-RoBERTa at batch 64 ran out of memory on our T4 GPU, the script caught the exception, logged the failure, and moved to the next config. A notebook would have died and lost all its cell state.

Notebooks for analysis. Error analysis lived in a notebook because it required human judgment. Looking at a Bangla comment and deciding if it's hate speech, implicit hostility, or casual banter requires cultural knowledge no script can provide. The notebook also needed to be iterative: categorize 50 errors, realize the categories are wrong, redefine, re-categorize.

But the notebook's inputs were clean artifacts produced by scripts. It loaded a CSV from the evaluation script, which loaded predictions from the quantisation script, which loaded a checkpoint from the distillation script, which used a teacher from fine-tuning, which used a config from YAML. The notebook was the end of the pipeline, not the middle.

The Specific Decisions

Directory structure mirrors the pipeline.

project/
├── config/          # YAML files
├── scripts/         # train.py, distill.py, quantise.py, evaluate.py, latency.py
├── notebooks/       # error_analysis.ipynb, threshold_calibration.ipynb
├── checkpoints/     # teachers/, students/, distilled/
├── results/         # baselines/, distillation/, quantisation/, latency/
└── mlruns/

Shared utilities extracted early. All seven models needed the same metric computation. In the notebook, I'd copy the metric cell and re-run it for each model. In the refactored codebase, one compute_metrics() function. 15 lines. Saved hours. A bug fix (we initially used average='weighted' instead of average='macro') fixed it everywhere.

Latency as a separate script. Latency measurement has a different protocol: fixed 128-token sequence, 10 warm-up plus 100 timed iterations, different backends per format. Making it separate forced explicit documentation and made measurement errors discoverable. When identical architectures showed 2x to 4x latency differences between distilled and non-distilled conditions, re-running just the measurements was trivial. Resource contention turned out to be the cause.

The Honest Admission

Research code is messier than production code. That's a property of the work, not a failure. Research involves exploration, dead ends, and iterative refinement without clear specs.

But research also involves stages that need to be reproducible, run hundreds of times, handle failure, and produce artifacts for other stages. Those are script work.

The practical minimum:

• If a stage runs more than 10 times, it's a script.
• If a stage produces an artifact consumed by another stage, the artifact has provenance.
• If a stage requires human judgment, it's a notebook.
• If your paper has a pipeline figure, your code should have a matching directory structure.
• Write the methodology section first, then organize code to match it.

The paper's methodology is about 3,000 words. The code is five scripts, three notebooks, and five config files. None of it is elegant. None would pass a software code review. But it's traceable: I can look at any number in the paper and trace it back to the exact script, config, checkpoint, and data that produced it.

That's the standard research code should be held to: not beauty, but provenance.

---

This post is part of a series on research engineering lessons from building a distillation and quantisation pipeline for Bangla hate speech detection. The full paper is titled "When Smaller Teachers Win."