What is a Loss Function? — AI Glossary

A loss function (also called a cost function or objective function) is a mathematical formula that measures how wrong an AI model’s predictions are. During training, the goal is to minimize the loss — to reduce the gap between what the model predicts and what the correct answer actually is.

Think of the loss function as a scoring rubric. If you predict a house will sell for $300,000 and it sells for $400,000, the loss function assigns a penalty proportional to that $100,000 gap. The training process — via gradient descent and backpropagation — works to shrink that penalty over thousands of iterations by adjusting the model’s parameters.

How Loss Functions Work

A loss function takes two inputs — the model’s prediction and the true label — and outputs a single number: the loss. Lower loss means the prediction was closer to the truth. Training tries to find parameter values that make this number as small as possible across the entire training dataset.

Different tasks require different loss functions:

• Mean Squared Error (MSE) — for regression tasks. Squares the difference between predicted and actual values, penalizing large errors heavily.
• Cross-Entropy Loss — for classification tasks. Measures how different the predicted probability distribution is from the true distribution. Standard for language models (predicting next tokens).
• Binary Cross-Entropy — for binary classification (yes/no, spam/not spam).
• Contrastive Loss — for similarity learning, used in embedding models. Pulls similar examples together and pushes dissimilar ones apart.
• Huber Loss — a blend of MSE and mean absolute error, more robust to outliers.

Why the Choice of Loss Function Matters

The loss function defines what the model optimizes for — it is the statement of “what does good look like?” A poorly chosen loss function trains a model that technically minimizes the formula but doesn’t achieve what you actually want.

A classic example: training a medical AI to minimize false negatives (missing cancer diagnoses) requires a different loss function than minimizing overall prediction error. If false negatives and false positives are penalized equally, the model will optimize for the common case and might miss rare but critical positive diagnoses.

Loss function design is also a key topic in AI alignment. If you optimize an AI agent’s behavior using a loss function that doesn’t fully capture human values, the agent will find ways to maximize the metric that violate the spirit of what you wanted — a form of reward hacking.

Loss Functions in Practice

For language models, the pre-training loss function is typically cross-entropy over next token prediction: given the preceding tokens, how well does the model predict the next token across the training corpus? This simple objective, at massive scale, produces models with broad capabilities.

In reinforcement learning, the “loss” is related to the reward signal — the model is trained to maximize cumulative reward rather than minimize prediction error. RLHF uses human preference scores as the reward signal to fine-tune helpful behaviors in chatbots.

Common Misconceptions

Misconception: Minimizing loss is the ultimate goal. Loss is a proxy metric, not the actual goal. You want the model to perform well in the real world, not just minimize a training loss. A model with very low training loss that overfits will fail in production.

Misconception: There is one correct loss function for each type of problem. Multiple loss functions can be combined. Multi-task learning uses weighted sums of task-specific losses. Custom loss functions that incorporate domain knowledge often outperform standard ones.

---

Key Takeaways

• A loss function measures the gap between predicted and actual values during training.
• Training minimizes the loss by adjusting model parameters via gradient descent.
• Common loss functions include MSE (regression), cross-entropy (classification), and contrastive loss (embeddings).
• The choice of loss function defines what the model optimizes — and must align with real-world goals.
• Loss is a proxy for real-world performance, not a guarantee of it.

Frequently Asked Questions

What is the difference between loss and accuracy?

Loss is a continuous number measuring prediction error across all examples. Accuracy is a discrete metric measuring what percentage of predictions are exactly correct. Loss is used during training because it is differentiable (smooth enough for gradient descent); accuracy often is not. A model can have low loss but below 100% accuracy, or vice versa.

Why does loss decrease during training?

Gradient descent iteratively adjusts parameters in the direction that reduces loss. Each training step moves parameters slightly “downhill” in loss space. Over thousands of steps, this produces parameters that make the model’s predictions progressively closer to the true labels.

What happens when loss stops decreasing?

The model has converged — either to a local minimum (which may or may not be global) or to a plateau. This can mean training is complete, the learning rate needs adjustment, or the model needs a different architecture. If validation loss increases while training loss continues to decrease, overfitting is occurring.

What is regularization in the context of loss functions?

Regularization adds a penalty term to the loss function — typically proportional to the size of the model’s weights. This discourages the model from becoming too complex or specialized, reducing overfitting by building in a preference for simpler solutions.

Can you have multiple loss functions at once?

Yes. Multi-task learning trains a model on multiple objectives simultaneously by summing or weighting multiple loss functions. For example, a model trained on both image classification and object detection might minimize a classification loss plus a regression loss for bounding box coordinates.

---

Sources: Grokipedia — Loss Function · PyTorch: Loss Functions · Google ML Crash Course: Loss

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI

Sources

This article draws on official documentation, product pages, and industry reporting. Specific sources are linked inline throughout the text.

Last reviewed: April 2026