I'm behind on a couple of posts I've been planning, but am trying to post something every day if possible. So today I'll post a cached fun piece on overinterpreting a random data phenomenon that's tricked me before.

Recall that a random walk or a "drunkard's walk" (as in the title) is a sequence of vectors <span>_x_1, x_2, ldots_</span> in some <span>_mathbb{R}^n_</span> such that each <span>_x_k_</span> is obtained from <span>_x_{k-1}_</span> by adding noise. Here is a picture of a 1D random walk as a function of time:

Weirdly satisfying

A random walk is the "null hypothesis" for any data with memory. If you are looking at some learning process that updates state to state with some degree of stochasticity, seeing a random walk means that your update steps are random and you're not in fact learning. If you graph some collection of activations from layer to layer of a transformer [...]