What is Overfitting? — AI Glossary

Overfitting is a fundamental problem in machine learning where a model learns the training data too well — including its noise and quirks — and performs poorly on new, unseen data. An overfit model has essentially memorized the answers to specific questions rather than learning the underlying patterns, so it fails when the questions change slightly.

Imagine a student who memorizes every practice exam answer word for word without understanding the material. They ace the practice tests but bomb the real exam when questions are phrased differently. That’s overfitting. The model has learned the training set’s specifics, not the general concept.

Why Overfitting Happens

Overfitting occurs when a model is too complex relative to the amount of training data available. A model with millions of parameters trained on only a few thousand examples has enough capacity to memorize every training example exactly — and it often does, rather than learning generalizable patterns.

Telltale signs of overfitting:

• Very low training loss but high test loss
• Excellent accuracy on training data but poor accuracy on validation or test data
• The model performs well only when inputs are very similar to training examples

The opposite problem — underfitting — occurs when the model is too simple to capture even the training data’s patterns. Finding the right balance of complexity is called the bias-variance tradeoff, a central concept in machine learning theory.

How to Prevent Overfitting

Engineers use several techniques to prevent or reduce overfitting:

• More training data — the most effective remedy. More diverse examples make memorization less useful.
• Regularization — adding penalties for large weights (L1/L2 regularization) discourages the model from becoming too specialized.
• Dropout — randomly disabling neurons during training forces the network to develop redundant pathways and not rely on specific patterns.
• Early stopping — monitoring validation performance and halting training when it starts to degrade.
• Data augmentation — artificially expanding the training set by transforming examples (rotating images, paraphrasing sentences).
• Cross-validation — training and evaluating on multiple data splits to get a stable performance estimate.

Overfitting in Practice

In supervised learning, the standard defense against overfitting is splitting data into three sets: training (used to update parameters), validation (used to tune hyperparameters and apply early stopping), and test (used once at the end for a final unbiased performance estimate).

Overfitting is also a concern in fine-tuning large models. Fine-tuning a large LLM on a small domain-specific dataset can cause it to “forget” its general capabilities while becoming narrowly specialized — a phenomenon called catastrophic forgetting. Parameter-efficient fine-tuning methods (LoRA, adapters) help mitigate this by changing only a small fraction of the model’s weights.

Common Misconceptions

Misconception: Overfitting only happens with small datasets. Even large datasets can be overfitted if the model is complex enough. Large models trained for too many epochs on even substantial datasets will start memorizing rather than generalizing.

Misconception: High training accuracy means the model is good. Training accuracy is not a reliable measure of real-world performance. Always evaluate on held-out data. A model with 99% training accuracy and 70% test accuracy is severely overfit and likely to disappoint in production.

---

Key Takeaways

• Overfitting occurs when a model memorizes training data instead of learning generalizable patterns.
• It shows up as low training loss but high test loss.
• Prevention techniques include regularization, dropout, early stopping, and data augmentation.
• The bias-variance tradeoff describes the balance between underfitting and overfitting.
• Always evaluate model performance on held-out test data, not just training data.

Frequently Asked Questions

What is the difference between overfitting and underfitting?

Overfitting: model is too complex, memorizes training data, fails on new data (high variance). Underfitting: model is too simple, can’t capture patterns even in training data (high bias). Both lead to poor real-world performance; the goal is a model that generalizes well.

What is the validation set and why does it matter?

The validation set is a portion of data held out from training, used to monitor performance during training and tune hyperparameters. It allows early stopping when validation performance starts to decrease — a sign of overfitting beginning. It is crucial for diagnosing overfitting without contaminating the final test evaluation.

Can deep learning models overfit?

Yes, and they are particularly susceptible because they have enormous numbers of parameters. Modern deep learning has developed many regularization techniques (dropout, batch normalization, weight decay) specifically to combat overfitting at scale.

What is double descent?

A counterintuitive phenomenon where models that are massively overparameterized (far more parameters than training examples) can generalize well, even though classical theory predicts they should overfit. Large language models seem to operate in this “interpolation regime” — suggesting the classical overfitting narrative is incomplete for modern deep learning.

How does data augmentation help with overfitting?

Data augmentation creates new training examples by applying transformations — rotating or flipping images, adding noise, paraphrasing text. This artificially expands the training set and forces the model to be invariant to those transformations, making it harder to memorize specific examples.

---

Sources: Grokipedia — Overfitting · Scikit-learn: Overfitting and Underfitting · arXiv: Double Descent in Neural Networks

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI

Sources

This article draws on official documentation, product pages, and industry reporting. Specific sources are linked inline throughout the text.

Last reviewed: April 2026