Great on Training Data, Useless Everywhere Else

A machine learning model can look excellent during development and still fail in real use. Consider an image classifier that reports 99 percent training accuracy, then reaches only 62 percent on images from another camera. The same problem can affect regression, forecasting, text, medical, and tabular models.

Nothing crashed. The model loaded correctly. The prediction code ran. The model simply succeeded at one task and failed at the task you actually cared about.

People often call every gap between training and real use "overfitting." Sometimes that diagnosis is correct. Sometimes the new data comes from a different population. Sometimes information leaked into evaluation. Sometimes the test set answered an easier question than deployment will ask.

These failures look similar but require different remedies. This article separates them from first principles and ends with a reusable workflow.

1. Fitting is not generalizing

A supervised machine learning dataset contains inputs and correct outputs. We normally write an input as x and its correct target as y. A model uses parameters, represented by θ, to produce a prediction:

ŷ = f_θ(x)

For a spam filter, x may be an email, y may be spam or legitimate, and the model predicts one of those classes.

During training, a loss function measures disagreement between predictions and correct answers. The learning algorithm adjusts the model parameters to reduce average training loss:

L_train(θ) = 1/n ⋅ ∑_i=1ⁿ l(f_θ(x_i), y_i)

Reducing this value is called fitting. It shows that the model can adapt to the examples it receives.

Generalization asks a harder question: does the learned rule remain useful for relevant examples that did not participate in fitting?

Imagine a student who memorizes 100 practice answers. Repeated questions produce a perfect score, while unfamiliar questions based on the same principles may cause failure. Memorization and understanding look identical on the practice sheet.

Machine learning evaluation exists to separate those possibilities.

Training performance measures fit to observed examples. Generalization performance measures usefulness on relevant unseen examples.

The word "relevant" matters. A model designed for several hospitals should not be judged only on new patients from the hospital that supplied its training data. The intended users, locations, devices, and time periods form the target population. Evaluation data must represent that population closely enough for the intended decision.

Most introductory evaluation methods assume that training and evaluation examples are independent observations from the same target distribution. This is the independent and identically distributed, or IID, assumption. One example should not reveal another, and both sets should represent the same population.

Real data does not always satisfy this assumption. Measurements from one patient can be related, nearby video frames can look almost identical, and future transactions may differ from older transactions. In these cases, generalization is still meaningful, but the split must respect the dependency, group, location, or time structure instead of pretending every row is interchangeable.

2. Training, validation, and test data

The usual development process assigns data three different roles.

Suppose you train 30 models and keep the one with the highest validation score. Your selection depended on that validation set, even if the models did not directly fit its labels. The test set provides a final estimate after those choices are complete.

A common division is 70 percent training, 15 percent validation, and 15 percent testing. These values are conventions, not laws. Time dependent data may require chronological splitting. Repeated observations may require group based splitting.

A basic split in Python

from sklearn.model_selection import train_test_split

# X contains input features; y contains labels

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.30, stratify=y, random_state=42
)

X_validation, X_test, y_validation, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50,
    stratify=y_temp, random_state=42,
)

stratify=y preserves approximately the same class proportions. random_state=42 makes the split reproducible.

The code is easy. Deciding what may cross the split is harder.

3. Overfitting

Overfitting occurs when a model becomes highly adapted to its training sample but performs substantially worse on relevant unseen data.

At first, both scores improve. Later, training accuracy continues rising while validation accuracy falls. The model is becoming better at the training sample and worse at transferring beyond it.

The difference is often called the generalization gap:

G = S_train - S_held-out

Here the gap is 20 percentage points. This definition assumes that a larger score is better, as with accuracy or F1. For a loss, compare the values in the opposite direction. Gap size also depends on dataset size, label noise, task difficulty, and uncertainty, so it is evidence rather than a universal verdict.

Why it happens

• Too much flexibility. A flexible model can represent useful patterns as well as coincidences and noise.
• Narrow coverage. Many images from one camera may contain less useful variation than fewer images from several devices.
• Noisy labels. A powerful model may eventually memorize incorrect or ambiguous labels.
• Training for too long. Early training often captures broad patterns. Later training may fit smaller irregularities that do not transfer.
• Shortcut features. A shortcut predicts labels inside a dataset without representing the intended concept. Suppose pneumonia images mostly came from Hospital A and healthy images from Hospital B. If their corner markers differ, the model may learn the marker instead of lung pathology.

The same marker relationship can produce a strong internal score, then disappear at Hospital C. Geirhos and colleagues describe such shortcuts as a recurring cause of failure under altered conditions [1].

Underfitting is different

If training and validation performance are both poor, stronger regularization may worsen the problem. The model may need better features, cleaner labels, greater capacity, longer training, or corrected code.

A useful first check is the tiny sample test, adapted from Andrej Karpathy's neural network debugging recipe [2]. Train on eight or sixteen examples with augmentation and regularization disabled. A flexible model should usually reduce its loss substantially on that tiny sample. It may not reach zero when labels are noisy, the model has strong built in constraints, or some regularization remains active. Failure still suggests checking the pipeline before making broad claims about generalization.

4. Distribution shift

A model can avoid obvious overfitting and still fail after release. Standard evaluation often assumes that development data and future data arise from the same statistical process:

P_train(X, Y) = P_target(X, Y)

Real systems frequently violate this assumption. A review classifier may receive short social posts, and a fraud detector may face behavior that did not exist last year.

Distribution shift means the conditions changed between development and use.

Three common forms

Covariate shift means the input distribution changes while the relationship between input and target remains approximately stable. Daytime and nighttime road images look different, but the meaning of a pedestrian does not.

P_train(X) ≠ P_target(X)    and    P_train(Y | X) = P_target(Y | X)

Label shift means class frequencies change. A disease may appear in 20 percent of a deliberately balanced research sample but only 2 percent of the deployment population. This change affects the practical meaning of a positive prediction.

P_train(Y) ≠ P_target(Y)    and    P_train(X | Y) = P_target(X | Y)

Concept shift means the relationship between inputs and targets changes. A phrase associated with spam two years ago may now appear in ordinary messages. The old rule has become less valid.

These categories follow established frameworks presented by Moreno Torres and colleagues [3] and Quiñonero Candela and colleagues [4]. Different changes require different responses.

Detecting and responding to shift

No single test proves that a deployed model remains reliable. Combine data checks with performance measurements when recent labels become available. Rabanser and colleagues compare several shift detection methods and find that no single detector is reliable on its own [5].

For numerical features, a Kolmogorov Smirnov test can compare two distributions. Suppose the training cameras produced a mean brightness value for each image, and a new camera produces systematically darker images. The short example below tests whether those two brightness distributions differ.

from scipy.stats import ks_2samp

# One brightness value per image
train_brightness = [0.62, 0.58, 0.65, 0.60, 0.63]
new_camera_brightness = [0.39, 0.42, 0.37, 0.44, 0.40]

statistic, p_value = ks_2samp(train_brightness, new_camera_brightness)
print(f"KS statistic: {statistic:.2f}, p value: {p_value:.4f}")

A small p value indicates that the observed brightness difference would be unusual if both samples arose from the same distribution. It therefore provides evidence of a possible input change, but it does not establish that predictive performance has declined. This example is intentionally simplified because it evaluates only mean image brightness. Images are high dimensional, and changes in texture, color, composition, background, or relationships among features may remain undetected by a single univariate test. Shift assessment should therefore combine several diagnostic checks with evaluation on recent labeled examples whenever such data are available.

Importance weighting can emphasize training examples that resemble the target population. Domain adaptation can learn features that transfer across sources. Both require evaluation on target data rather than trust as automatic repairs.

Overfitting asks whether the learned rule was too specific to the training sample. Distribution shift asks whether evaluation and use occur under different conditions. A model can suffer from both.

5. Leakage and weak evaluation

Data leakage occurs when information unavailable during genuine prediction influences model development or evaluation. It often produces unrealistically strong results.

Official scikit learn guidance identifies preprocessing before splitting as a common cause [6]. This procedure is unsafe:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.20, random_state=42
)

The scaler learned from the entire dataset, including the test records. A pipeline keeps learned preprocessing inside training:

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)

model = Pipeline(
    steps=[
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

model.fit(X_train, y_train)
predictions = model.predict(X_test)

Leakage also occurs when related examples cross the split. If each patient has ten scans, a random image split may place scans from one patient in both training and test sets. The model can exploit patient specific properties. The independent splitting unit should be the patient, not the image.

The same principle applies to video frames, authors, satellite scenes, and documents.

Another form uses future information. A discharge code cannot help predict a diagnosis at admission because it does not exist then. Feature availability must be judged at the exact prediction moment.

Suppose a model predicts subscription cancellation within 30 days. The field account_closed_date is strongly related to cancellation but appears after the event. Using it lets the model read part of the answer. A feature audit should record when every feature becomes available.

Weak evaluation is broader. A procedure may be technically valid but answer the wrong question.

Repeated use also weakens a test set. If you inspect test results, modify the model, and test again, that set has entered development. Its filename may still say test, but its role has changed.

6. Metrics can hide failure

Consider a hypothetical dataset in which 990 of 1,000 transactions are legitimate. A model that predicts every transaction as legitimate obtains 99 percent accuracy and detects no fraud.

Precision asks how often positive predictions are correct:

Precision = TP / (TP + FP)

Recall asks how many real positive cases are detected:

Recall = TP / (TP + FN)

No metric is automatically best. A screening system may prioritize recall because missed cases are costly. A manual review system may require higher precision because false alerts consume staff time.

Many classifiers output probabilities. A threshold turns probability into a class. The default value of 0.50 is convenient, not universally correct. Select the threshold on validation data according to the consequences of errors, then evaluate the frozen threshold on the test set.

Probability quality also matters. If predictions near 0.80 are positive about 80 percent of the time, the model is well calibrated at that level. A reliability diagram shows this agreement, while the Brier score summarizes probability error. Calibration does not choose a threshold, but it makes probability based decisions easier to interpret.

Overall results can also hide subgroup failure. A classifier with 90 percent accuracy might score 96 percent for one group and 58 percent for another. Report important classes, sources, devices, and groups separately when sample sizes permit.

7. How to diagnose the problem

7.1 Plot training and validation curves

Record both losses after each epoch. Training loss that falls while validation loss rises is a classic sign that later training is becoming too specific.

7.2 Repeat the experiment

One run can benefit from a favorable split or initialization. Report the mean and standard deviation across several runs:

s̄ = 1/k ⋅ ∑_j=1^k s_j

σ = √1/k ⋅ ∑_j=1^k (s_j - s̄)²

A result reported as 0.84 plus or minus 0.01 is more informative than a best run of 0.85.

7.3 Inspect errors and challenge shortcuts

Read, view, or listen to incorrect predictions. Group them by class, source, condition, and subgroup. Metrics tell you how often the model failed. Error analysis begins to reveal how and why.

Alter a suspicious feature while preserving the intended target. Cover hospital markers, change backgrounds, paraphrase text, or separate acquisition sources. A sharp prediction change suggests that the feature carries more influence than intended.

7.4 Test beyond the original source

An internal test estimates performance under familiar collection conditions. External validation uses independently collected data and tests transfer across sources.

A strong study may include an internal test, a later time period, an external source, subgroup results, and controlled stress tests. Each answers a different question.

8. What usually helps

• More representative data. Additional examples help when they add relevant variation. Near duplicates add less information than new sources and conditions.
• A simpler model. Begin with a clear baseline. Greater capacity is useful only when the data and task justify it.
• Regularization. Regularization adds a preference for simpler parameter values:

  L_total = L_data + λR(θ)

  Too little may have no effect. Too much can cause underfitting. Common methods include weight decay and dropout.
• Early stopping. Save the checkpoint with the best validation result instead of assuming that the final epoch is best.
• Appropriate augmentation. Augmentation should reproduce plausible variation while preserving the correct label. Rotating some objects may be reasonable; rotating text or anatomy may change meaning.

9. Cross validation without false confidence

When a dataset is small, one split may produce a result that depends heavily on which examples happened to enter the validation set. Cross validation reduces this dependence by evaluating several partitions.

In five fold cross validation, the data is divided into five parts. The model trains on four parts and validates on the remaining part. This repeats until every part has served as validation data. The final summary is usually the mean score and its variation across folds.

Cross validation does not repair a poor splitting rule. Patient data still requires patient grouped folds, and forecasting still requires time ordered folds. If preprocessing occurs outside the fold, information can also leak between training and validation.

Using the same cross validation results for both tuning and final reporting can bias the estimate. Nested cross validation separates the roles: inner folds select settings, and outer folds estimate performance. Cawley and Talbot [7] and Varma and Simon [8] explain why this separation matters when tuning is extensive.

For a beginner project with a separate untouched test set, ordinary grouped or stratified cross validation on the development data is often sufficient. Nested cross validation becomes more useful when the dataset is limited, tuning is extensive, and no large independent test set is available.

10. A practical evaluation workflow

1. Define the prediction moment and list the information available then.
2. Define the target population, including users, locations, devices, and time periods.
3. Choose the independent unit, such as patient, user, scene, video, or document.
4. Build a simple baseline before adding complexity.
5. Fit scaling, imputation, feature selection, and sampling using training data only.
6. Track training and validation results together.
7. Repeat runs or folds and report variation.
8. Inspect class, subgroup, and source performance.
9. Freeze preprocessing, model weights, threshold, and postprocessing.
10. Evaluate once on the untouched test set.
11. Add later data, external sources, and stress tests.
12. Monitor input drift and performance after release.

11. Diagnostic guide

12. What the final claim should say

A high training score proves that a model can fit its training sample. It does not prove that the model learned the intended task.

A high internal test score provides stronger evidence, but only within the conditions created by the split. Related records, leaked preprocessing, repeated test use, or different deployment conditions can still make it misleading.

The useful question is not simply:

How accurate is the model?

It is:

How accurate is this frozen model, for which population, under which conditions, at what time, using which metric, and with what uncertainty?

That question is less convenient. It is also much closer to the truth.

References and further reading

1. Geirhos, R., Jacobsen, J. H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence. arXiv:2004.07780
2. Karpathy, A. (2019). A Recipe for Training Neural Networks. karpathy.github.io/2019/04/25/recipe
3. Moreno Torres, J. G., Raeder, T., Alaiz Rodríguez, R., Chawla, N. V., & Herrera, F. (2012). A Unifying View on Dataset Shift in Classification. Pattern Recognition. doi.org/10.1016/j.patcog.2011.06.019
4. Quiñonero Candela, J., Sugiyama, M., Schwaighofer, A., & Lawrence, N. D., editors (2009). Dataset Shift in Machine Learning. MIT Press. mitpress.mit.edu
5. Rabanser, S., Günnemann, S., & Lipton, Z. C. (2019). Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. Advances in Neural Information Processing Systems (NeurIPS). arXiv:1810.11953
6. Scikit learn. Common Pitfalls and Recommended Practices. scikit-learn.org/stable/common_pitfalls.html
7. Cawley, G. C., & Talbot, N. L. C. (2010). On Overfitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research. jmlr.org/papers/v11/cawley10a.html
8. Varma, S., & Simon, R. (2006). Bias in Error Estimation When Using Cross Validation for Model Selection. BMC Bioinformatics. bmcbioinformatics.biomedcentral.com

---

Want to share your own experience? Every member can write here: reach out and we'll help you publish your first post.