Gradient descent · Mathematics

GRADIENT DESCENT SIMULATION

Walking down a mountain in thick fog leaves only one method: feel which way the ground falls, then take a step that way. That is literally the rule x ← x − η∇f running inside every neural-network optimiser. The danger lies in the step length: a small η crawls and never quite arrives, an oversized η leaps clean over the valley floor onto the far slope and diverges, and for a quadratic the safe threshold is exactly η < 2 divided by the curvature. On a landscape of many basins, the starting point decides which local minimum the ball settles into.

▸ Building the loss-function landscape f(x, y)…

▸ Painting the contour map (heatmap)

▸ Computing the gradient ∇f = (∂f/∂x, ∂f/∂y)

▸ Dropping the ball downhill along −∇f

▸ Marking the local & global minima

▸ Ready — Online. ✅

Iteration state

⛰️ Descending

−∇f = steepest descent direction

Iteration—

Position (x, y)—

Loss f—

|∇f| (steepness)—

Learning rate η—

Status—

Tip

The ball always moves against the gradient (−∇f): the gradient points UPHILL, so subtracting it goes DOWN. Each step: x ← x − η∇f. Too large an η → blow-up/oscillation; too small → slow crawl. With many basins → the ball can get stuck in a local minimum.

—

Click the map to drop the ball there · drag Learning rate η to see slow crawl ↔ blow-up · switch the Scenario (single basin · many basins · big step · small step · narrow valley · momentum)

Your browser has canvas disabled.

Loss f & steepness |∇f| by iteration loss f|∇f| steepness