AI & Machine Learning · reviewed in July 2026
Gradient descent
Gradient descent is an optimization algorithm that iteratively adjusts a model's parameters in the direction that most reduces its error, computed from the gradient (the slope) of a loss function. It's the engine behind training most machine learning models and neural networks.
Frequently asked questions
What is the learning rate?
It's the size of the step the algorithm takes at each iteration. Too high a rate can make the algorithm overshoot the minimum and fail to converge; too low makes training unnecessarily slow.
What is stochastic gradient descent?
It's a variant that updates the parameters using only a small batch of data (or a single example) at each step, instead of the full dataset, which makes it far faster on large datasets.
Does gradient descent always find the best possible solution?
Not always. It can get stuck in a local minimum on complex loss functions, though in practice, for many models, it finds solutions that are good enough.