Definition
Loss functions quantify the difference between predicted and actual values, guiding model training.
- **Common Loss Functions:**
- MSE (Mean Squared Error): For regression
- Cross-Entropy: For classification
- Binary Cross-Entropy: For binary classification
- Huber Loss: Robust to outliers
Properties of Good Loss Functions: - Differentiable (for gradient descent) - Minimum at correct prediction - Appropriate for the task
In LLM Training: - Cross-entropy loss on next token prediction - Minimizing perplexity - RLHF reward modeling
Relationship to Metrics: - Loss: Used during training - Metrics (accuracy, F1): Used for evaluation
Examples
Cross-entropy loss measuring how far probability predictions are from true labels.
Related Terms
Want more AI knowledge?
Get bite-sized AI concepts delivered to your inbox.
Free intelligence briefs. No spam, unsubscribe anytime.