No frames recorded yet — drive a lap in Drive mode first.
Dataset
0frames
| recorded | predicted | error | |
|---|---|---|---|
| steer | — | — | — |
| throttle | — | — | — |
Brightness is influence. Each pixel is lit by how much the prediction would change if that pixel changed — the gradient of the model's output with respect to the input image. Bright means the pixel is load-bearing; dark means the model would say roughly the same thing without it. It is recomputed every epoch, so the heat moving around is the model changing its mind about what matters.
steer and throttle are different questions. Each button shows the gradient of that one output over the same frame. A model can steer off one feature and brake off another, and when the two maps disagree that difference is the interesting part.
What good looks like: heat gathered on the lane edges and the road boundary ahead — roughly where you look when you drive. What to worry about: heat on the sky, the car's hood, or one piece of scenery. That is a shortcut that happens to correlate with the right answer across your recorded laps, and it will not survive a track the model has not seen. A falling loss curve cannot tell these two apart; this can.
In the city, put it on the throttle head as the car comes up to a signal. If the model brakes for a red light but the heat sits on the road surface instead of the light itself, it has learned where you happened to stop rather than what the signal means — and it will sail through the same light somewhere else.
Which frame you are looking at. The frame slider scrubs your whole recording, in the order you drove it, and works before you have trained anything — the picture and the recorded numbers come straight from the tub. It opens on the sharpest steering frame, because a frame where the car is barely turning contains no decision worth explaining, and on a mostly-straight track an arbitrary frame is usually one of those. Whichever frame you leave it on is the one the next run explains itself with, so it is worth lining up a corner before pressing train.
The prediction columns need a model, so they read “—” until one exists. Once a run is under way the reading catches up at the end of each epoch; once it has finished, scrubbing re-predicts and re-explains as you go — the picture immediately, the heat a moment behind it.
Reading limits. Brightness is scaled per frame against its 99th-percentile pixel, so a map shows where the model is looking within this frame — not how strongly, and not comparably between epochs or between the two heads. Raw gradients are also noisy by nature: treat scattered speckle as noise and coherent regions as signal.
Loss is how wrong the model is — the mean squared error between what it predicted and what you actually did. Lower is better, and the only number that means anything is the trend.
The thin pale line is training loss, one point per batch, so it is jagged by nature: each batch is a different random handful of frames and some are harder than others. Watch its downward drift, not its spikes. The orange line and dots are validation loss, one per epoch, measured on frames held out of training that the model never got to learn from.
The y-axis is logarithmic. Loss improves by shrinking toward zero in ever smaller absolute steps, and on a linear axis all the interesting late progress would be squashed flat against the bottom. Equal vertical distance here means equal proportional improvement.
The shape to look for: both lines falling together. If the pale line keeps dropping while the orange one flattens or turns upward, that is overfitting — the model is memorising your specific recorded frames rather than learning to drive, and it will be worse on the track than the training loss suggests. Stop early and record more varied laps.
This is the model driving, next to you driving. The pale line is the steering you recorded across 100 frames; the orange line is what the model predicts for those same 100 frames, re-run after every epoch. The gap between them is the error you are training away, frame by frame, instead of collapsed into a single number.
It is the same 100 frames every epoch, drawn once from the training split and then held fixed, so the two lines stay comparable as the run progresses. They are in shuffled order, not recorded order — neighbouring points are unrelated moments, so read this as 100 separate comparisons rather than as a lap. Watch the orange line start flat near zero and gradually grow into the shape of the pale one.
Where it stays wrong is the useful part. A model that tracks the straights but flattens out the sharp corners has learned the easy majority of your data and skipped the rest — the corners are outnumbered, so averaging them away is a cheap way to lower loss. That usually means recording more of whatever it is missing.
An orange line pinned near zero while the pale one swings is the degenerate case: predicting the average of everything is the safest possible guess, and a model that has not learned anything yet will find it first. It should not still be there by the last epoch.
Press run forward pass to send the frame on the left through the network.
This is one training step, slowed down. The loss graph on the Train page shows that a model learns; it cannot show how. So this takes one frame you recorded, runs it through the network, works out how wrong the answer was, traces that blame back through every layer, nudges the weights once, and runs the same frame again so you can see the error shrink. Real training does exactly this, tens of thousands of times, too fast to watch.
Pick the frame with the slider below. A corner is worth more than a straight here: on a frame where the car is barely steering there is not much of a decision for the network to get wrong, so the error starts small and the whole thing is less dramatic. It opens on the sharpest turn in your recording for that reason, and it is the same frame the Train page explains its saliency map with.
Going forward there is no error yet. The pale bars are activations — how strongly each layer is responding to this picture. Nothing is right or wrong at that point, because the network has not yet been told what the answer should have been. That is why nothing is orange until the very end of the row.
Error is born at the output. Only when the two predictions meet the steering and throttle you actually recorded does a mistake exist: the orange bars in the last column are that gap, and the loss is their squares added together.
Backward, that error is divided up as blame. Each layer's orange bar is how much this frame's mistake is attributable to that layer's weights — the gradient. Notice it does not shrink neatly on the way back. It jumps around, and in the early convolution layers it is often hundreds of times smaller than at the heads. That is vanishing gradient, it is a real problem in deep networks, and it means the first layers learn far more slowly than the last ones. Turn on log bars when the early columns are too short to read.
The learning rate scales the step, not the error. This is the part that is easy to get backwards. The gradient says which direction each weight should move and how urgently; the learning rate decides how far you are willing to go on the strength of one frame. The white sliver inside each orange bar is that step — literally the gradient multiplied by the rate. Drag rate and watch the sliver resize while the orange bar behind it stays exactly where it was.
Then it starts over — and that is worth noticing, because there is no fifth act. Running the frame again is act 1 of the next step, and the shorter error bars turn up at act 2, in the same place the error first appeared. Four acts on a loop is the whole mechanism: not the error shrinking as it travels backward, but the model being slightly less wrong each time it is asked.
Acts 1 and 2 hold their numbers between comparisons. Run the frame forward after an update and the loss and the error bars do not move yet — they still show the previous comparison, and only change when act 2 compares again. That is deliberate: it is what lets you see the bar travel down from where it used to be, rather than watching it vanish and reappear at some other height.
What to watch for: push the rate to the top of its range and keep stepping. The loss stops falling smoothly and starts bouncing — down, then up past where it started — because a step that big sails over the bottom of the valley and partway up the far side. Often it never comes back: a couple more steps and the numbers reach infinity, the model is dead, and only reset revives it. Drag the rate to the bottom instead and you get the opposite failure — every bar is right, every step is real, and nothing visibly happens. Every trained model in this app lives between those two.
Stepping this one frame is memorising, not learning. Hold auto long enough and the loss goes to essentially zero — the model has not learned to drive, it has learned this single picture. Real training averages the gradient over a batch of 64 different frames before taking each step, which is what makes the direction worth trusting. This panel drops that on purpose so there is only one thing moving at a time.
Reading limits. Bars are scaled within each pass against the largest one, so heights compare layers to each other, not to the loss number or to the previous act. The orange backward bar is the gradient with respect to that layer's weights — what the update actually uses — which is related to, but not the same as, the error signal passing through it. And this steps with plain gradient descent so the rate means exactly what it says; the real trainer uses Adam, which adds momentum and per-weight scaling on top and would make the sliver a half-truth. Nothing here touches your trained model: it works on a throwaway copy, and reset puts it back.
Move mouse to steer · scroll or WS for throttle · R reset to line
Plug in a gamepad and just use it — whatever you touch last drives.
The small screen is exactly what your model will see.