Scott Sumner wrote a good post earlier this month on the usefulness of tautologies which got me thinking about tautologies in software development. One which came to mind is

"To reduce intrinsic complexity of a program one must reduce the complexity of the problem it's solving"

I liken this to the no free lunch theorem (hence the title of this post) because when designing ML algorithms, you can only improve performance by limiting the set of problems under consideration (i.e., making assumptions about the problem at hand). Similarly, in software, you can only simplify code by making simplifying assumptions about the problem you're solving.

So what can we learn from this tautology? If you believe that software developers should strive for simplicity (as John Ousterhout argues in "A Philosophy of Software Design"), then you must also believe software developers should focus on making assumptions.

What are assumptions to a software engineer? Assertions, preconditions, and API contracts! We should therefore strive to write methods that looks like

/**
 * Assumes that bar() was called first.
 */
public void foo(Key key)
{
    Preconditions.checkArgument(key != null);
    Preconditions.checkArgument(cache.contains(key));
    Preconditions.checkState(isCacheInitialized());
    ...
}

with the understanding that it will simplify the final program. I've recently found myself writing more and more methods that look like this. When I started out, I would have looked at comments like

"this method assumes that it will only be invoked once"

or

"this only works assuming no more data will be added to this collection"

as "cop-outs" or lazy coding, when in reality they are important design decisions. Every simplification is the flip-side of an assumption, and it's better to document (or code!) assumptions directly than leave them implicit.

A demand-side analysis of AI's impact on the job market

If you collaborate using Git, here are some fun commands to run

git log --format=format: --name-only --since="1 year ago" | sort | uniq -c | sort -nr | head -20
git shortlog -sn --no-merges
git log --format='%ad' --date=format:'%Y-%m' | sort | uniq -c
          

One Dimensional chess

1D-Chess, Rowan Monk

Frontier AI labs are profitable on the margin

Grade inflation hurts students in the long run

Related, one proposed solution to grade inflation: capping the number of A grades awardable, penalizes students for taking difficult courses.

Malus provides "clean room as a service" (i.e., the removal of external dependencies from a software project)

Robin Hanson's "Grabby Aliens" model (2020) suggests that if humans stay put on Earth, we won't discover aliens for 500 million years (median estimate) but once we do, it will only be a few years before they arrive on our doorstep. Fun idea.

Trained to play tennis, humanoid robots can hold a cooperative rally.

This is the most recent jaw-dropping robotics demo I've seen. The footwork is particularly impressive.
LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data, 3/13/26

When it comes to AI's economic impact, always keep your eye on production .

Brain-drain to the U.S. has significantly slowed Canada's economic growth

I’ve spent a while searching for resources to provide new hires on particle filters. There are two main categories: theoretical introductions (e.g., A Tutorial on Particle Filters for Online Nonelinear/Non-Gaussian Bayesian Tracking and conceptual blog posts (e.g., Emma Benjaminson’s Series). I’ve yet to find a resource with the following characteristics

1. Accessible to a Math/CS undergrad
2. Follows a single example, from Bayesian Inference to a Kalman Filter
3. Has visualizations of multiple-dimensions and hidden variables
4. Demonstrates the problems Kalman and Histogram filters encounter

I hope to fill that gap. These notes grew out of a recruiting talk I gave at UIUC and are intended as a conceptual primer for one of the theoretical introductions. A familiarity with calculus, statistics, and linear algebra is useful.

Outline

I’ll follow a single, unifying example: Imagine we’re in a submarine equipped with active sonar. Like a bat, we can emit a sound and listen for its echo. Given the echo’s elapsed time and speed of sound, we can calculate our distance to things. Unfortunately, we don’t have a perfect knowledge of the speed of sound underwater, as it varies depending on temperature, salinity, and depth. We are tasked to find another submarine which is somewhere nearby. We’ll start with the simplest case: Both submarines are stationary and we want to estimate only the distance to the other submarine.

Each act builds on the previous by adding one layer of complexity. A new technique (and limitations of the old one if applicable) will be discussed at each stage

Act 0 provides a recap of Bayes’ rule. Act 6 describes resampling and perturbations: two heuristics for better performance with less computation.

Act 0: Math Recap

The easiest way to remind yourself of Bayes’ theorem is to re-arrange the law of conditional probability (the comma means “and”, and the bar means “given”):

$$P(A, B) = P(A|B)P(B) = P(B|A)P(A)$$

$$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$$

Bayesian inference is a technique which repeatedly uses Bayes’ theorem to understand some outcome/event of interest $(A)$ as we observe events/collect data which tells us something about it $(\textrm{e.g., } B)$. For example, “given that a card is red, what is the probability it is a heart?”

• $P(A)$ is called the prior. This is the initial degree of belief in the hypothesis $A$.
• $P(A|B)$ is called the posterior. It is the degree of belief in $A$ after incorporating the news of $B$.
• $P(B|A)$ is the likelihood. It is the probability of the data given the hypothesis
• $P(B)$ is called the evidence, or marginal likelihood. It is the probability of the data under all hypotheses.

The crux of Bayesian inference is that once we have $P(A|B)$, if we observe another event, $C$ (which is independent of $B$) we can “update” our belief about $A$ by making $P(A|B)$ our prior and starting again to find a new posterior, $P(A|B,C)$. Note that we don’t have to start from scratch each time we get additional evidence, only compute Bayes’ theorem one more time. All of our knowledge thus far about $A$ is contained in the posterior.

What do I mean by “degree of belief”? If you haven’t come across the distinction between Frequentest and Bayesian statistics, a frequentest will take a weighted coin, and will say “flip it 1 million times, and the ratio of $\frac{\text{\#heads}}{\text{\#flips}}$ is the probability it lands heads. A Bayesian will take a weighted coin and say probability is the”degree of belief” I hold that it will land heads. i.e., if I had to place a bet on it, what “probability” would make for a fair betting line?

We can derive Bayes’ theorem for probability density functions similarly:

$$f_{X,Y}(x,y)=f_{X|Y=y}(x)f_Y(y)=f_{Y|X=x}(y)f_X(x)$$

$$f_{X|Y=y}(x)=\frac{f_{Y|X=x}(y)f_X(x)}{f_Y(y)}$$

Remember the evidence, $f_Y(y)$, is the probability density of the data under all hypothesis (possible values of $x$). In the continuous case you would write this as an integral

$$f_Y(y)=\int f_{Y|X=x}(y)f_X(x)dx$$

Act 1: One Measurement of A Stationary Submarine

To our submarine example, let’s make the following simplifications:

• The problem is one-dimensional
• Both submarines are stationary
• The sonar readings are centered on the true distance with some noise. To be precise, they follow the normal distribution with $\mu=\text{true distance}$, and $\sigma=100$. A positive value means the object is in front, negative means it is behind.

   🛥️ ········))       🛥️
<───┼───────────────────┼───> x
    0                  ❓

Given the following measurement

what is the probability distribution of position of the other sub? Let’s approach this as a Bayesian. Call the distance to the other submarine $x$ and the sonar reading $y$. Here’s Bayes’ theorem again:

$$f_{X|Y=y}(x)=\frac{f_{Y|X=x}(y)f_X(x)}{f_Y(y)}=\frac{f_{Y|X=x}(y)f_X(x)}{\int f_{Y|X=x}(y)f_X(x)dx}$$

$$\substack{\text{p-distribution of the distance x} \\ \text{given the sensor reading y}} = \frac{\substack{{\text{p-distribution of the sensor }}\\ {\text{reading y given distance x}}} \times \substack{\text{prior p-distribution}\\{\text{ of the distance x}}}}{\substack{\text{p-distribution of the sensor}\\{\text{reading y across all possible x}}}}$$

Here, $X$ represents the “state” of the other submarine (also called our “hypothesis”) and $Y$ represents the Evidence/Data we have about this state. The core problem is that we have a probability density over possible observations, $f(Y|X)$, but we want a pdf over state, $X$. Bayes’ theorem is what achieves this.

We need two things to calculate our posterior. First, the prior $f_X(x)$. This represents our belief about the other sub’s position before receiving any measurement. Since we don’t have any a-priori knowledge, we can use a Gaussian distribution with extremely high variance (i.e., very flat) which loosely says “it could be anywhere”.

$$f_X(x)=\mathcal{N}(\mu=0, \sigma^2=10^8)=\frac{1}{\sqrt{10^8\times2\pi}}\exp\left(-\frac{1}{2}\left(\frac{x^2}{10^8}\right)\right)$$

Aside: however wide, this prior is not uniform: it has slightly more probability around 0 than at 1000m. We can’t make our prior’s variance infinite as that is ill defined, but we could have used an uninformative, improper prior. “Uninformative” meaning it contains no information (uniform everywhere) and “improper” because it doesn’t integrate to 1. While I won’t go into more detail here, I point it out because sometimes people say you can “initialize the particle filter using the first measurement” (i.e., use the normalized likelihood function of the first measurement as your prior), but really what they’re doing is applying Bayes’ rule with an uninformative prior.

Second, we need our likelihood function, $f_{Y|X=x}(y)$. This is defined as the probability of measuring $y$ given the state, $x$. The problem statement directly gives this to us: “The sonar readings are centered on the true distance with standard deviation of 100”. Here, the “true distance” is $x$

$$f_{Y|X=x}(y)=\mathcal{N}(\mu=x, \sigma^2=10^4)=\frac{1}{\sqrt{10^4\times2\pi}}\exp\left(-\frac{1}{2}\frac{(y-x)^2}{10^4}\right)$$

Now, we can plug these into Bayes’ theorem and solve for the posterior $f_{X|Y=y}(x)$. Fortunately, the product of two Gaussian Distributions is also Gaussian (proof) with mean and variance

$$\sigma=\sqrt{\frac{\sigma_1^2\sigma_2^2}{\sigma_1^2+\sigma_2^2}},\quad\mu=\frac{\mu_1\sigma_2^2+\mu_2\sigma_1^2}{\sigma_1^2+\sigma_2^2}$$

The denominator (evidence), $f_Y(y)$, is a scalar value (remember, $y$ is given). Therefore, the posterior is also a Gaussian. I won’t go through the entire derivation here, but understand the solution is analytical

Aside: an analytical solution is derived by moving around variables with pencil and paper (e.g., anything you did in a high school algebra class). It is exact. Numerical solutions, on the other hand, are calculated via an algorithm and are approximate. They typically incur some discretization or rounding error which approaches zero in the limit of infinite memory or compute time.

I’ll note a few things. First, the prior is hard to see because it is pretty close to a flat line right around 0. Second, the mean of the posterior is very slightly less than the measurement. This is because our prior is centered at zero, so it will still “pull” the posterior towards it, no matter how flat it is. Third, the equation for the standard deviation, $\sigma$, above guarantees that the variance of the posterior is less than or equal to the variance of prior and likelihood Gaussians.

Act 2: Multiple Measurements of A Stationary Submarine

Now, we get a second measurement,

   🛥️ ···))·····))     🛥️
<───┼───────────────────┼───> x
    0                  ❓

And we’d like to update our probability distribution of the other submarine’s position, $f_X(x)$. Recalling the math recap from the beginning, we can use the posterior from Act 1 as our new prior, rinse and repeat.

$$\text{Act 2 posterior} = \frac{\substack{{\text{measurement \#2}}\\ {\text{likelihood}}} \times \text{Act 1 posterior}}{\substack{\text{probability of measurement \#2 }\\{\text{across all possible x}}}}$$

Again, the variance of the posterior is smaller than the two inputs. Since the variance of the prior and likelihood are roughly the same, the mean of the posterior is half way in between the two.

One of the powers of Bayesian inference is that it allows subjective priors. By that, I mean your domain knowledge and lived experience may give you a hunch that the other sub likes to hang out in some area and is therefore probably some distance away. Bayesian inference allows you to turn this “hunch” into a prior. This is possible because (in theory) no matter what prior you choose (so long as it is not zero where it matters) with enough measurement updates eventually your posterior will converge on the true distribution. Therefore we can leverage subjective “hunches” while maintaining some mathematical guarantees.

Act 3: Multiple Measurements of a Moving Submarine

Now, the other submarine is moving, but we don’t know how fast. We take sequential sensor measurements. Our goal is to estimate (at the time of the last measurement) the velocity and range of the other submarine. We’ll make the following assumptions

   🛥️ ···))······))    🛥️💨
<───┼───────────────────┼───> x
    0                  ❓

• Everything is still in one dimension
• The other sub is moving at a constant velocity
• Our prior on the other submarine’s velocity is $\mathcal{N}(\mu=0, \sigma^2=5)$ where a negative value means it is getting closer.

The state of the other submarine is now a random vector $X$:

$$x= \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}= \begin{bmatrix} \text{position} \\ \text{velocity} \end{bmatrix}$$

And its probability distribution is now multivariate Gaussian $X\sim\mathcal{N}(\mu,\Sigma)$. Our prior is

$$\mu =\begin{bmatrix} 0 \\ 0 \end{bmatrix}, \quad \Sigma=\begin{bmatrix} 10^8 & 0 \\ 0 & 5 \end{bmatrix}$$

Again, we will employ Bayesian inference. The problem is now two dimensions, but conceptually the technique is the same. We will treat the first measurement just as we did in Act 2. However, after calculating the posterior $f_X(x)$ of the random vector, instead of immediately turning it into our next prior, we first must update it with time. To do this, we need something called a “motion model” or “state update equation”. Given a state at $t-1$, and our assumption of constant velocity, the state at $t$ is given by

$$\begin{aligned} \textrm{pos}_t&=\textrm{pos}_{t-1}+\text{vel}_{t-1}\Delta t \\ \textrm{vel}_t&=\textrm{vel}_{t-1} \end{aligned}$$

where $\Delta t$ is the time in between each sensor reading. This is a linear transformation we can write in matrix form:

$$\begin{bmatrix} \text{pos} \\ \text{vel} \end{bmatrix}_t = \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} \begin{bmatrix} \text{pos} \\ \text{vel} \end{bmatrix}_{t-1} = A \begin{bmatrix} \text{pos} \\ \text{vel} \end{bmatrix}_{t-1}$$

(I’m sticking with convention here to call this matrix $A$).

Given a state vector we can transform it with time, but how do we transform a continuous distribution? Let’s change our framing slightly; instead of thinking about probability distributions, let’s consider the random variable $X=\{\text{pos},\text{vel}\}$ which this distribution describes. Again, we are fortunate that our variable is Gaussian. Thinking back to an intro stats class, you may remember that multiplying a random variable by a constant $b$ scales its mean by $b$ and its variance by $b^2$. This extends to random vectors. Given any linear transformation $Y=BX$,

$$E[Y]=B E[X]$$

$$Cov(Y)=B Cov(X)B^T$$

Because the Gaussian distribution is defined by this mean and variance, the resulting distribution of our random variable is also Gaussian with

$$\mathcal{N}(\mu,\Sigma)\to \mathcal{N}(A\mu,A\Sigma A^T)$$

So long as our motion model is a linear transformation and our prior and measurements are still Gaussian, the posterior will remain Gaussian, and therefore the entire process can be solved analytically. Say we receive the following three measurements:

To process the first, we apply Bayes’ theorem just as we did in Act 2. I’ve plotted top-down heatmaps of our probability distributions. Getting oriented, an informationless prior would look like a uniform, light orange background. Our prior is a bit better than informationless: we know the velocity is probably between $\pm 4$ m/s.

Our sensor still provides only range information. Therefore it appears as a 1D Gaussian smeared uniformly across the velocity axis.

Aside: I haven’t plotted the actual prior here – it would appear as a single color. I’ve emphasized things for visual effect.

Notice how the posterior mean is slightly lower than 7507. This is because our prior is not truly “informationless” it is just a very flat Gaussian centered at 0, so it will pull the posterior slightly towards 0.

Aside: the first posterior’s covariance matrix has 0s in its off diagonals. This is a fancy way of pointing out that it isn’t slanted. After we apply the motion model, these off-diagonal elements become non-zero and it becomes slanted. It is this covariance matrix that carries the “information” about the relationship between the possible positions and velocities: i.e., “If the sub has a positive velocity it is probably further away now”.

Let’s zoom out for one moment. Our goal is to motivate the particle filter. Until this point we haven’t defined “particle” or “filter”. The approach I’ve just described is is called “recursive Bayesian estimation” or a “Bayesian Filter”. The first phrase describes exactly what we’ve done: recursively apply Bayes’ theorem to estimate the state of the submarine. As for the latter, “filtering” simply means we are estimating the state of the submarine at the time of the last measurement as opposed to during the whole encounter. At this point you know literally 1/2 of “particle filter”, and our conceptual understanding is roughly half way there as well.

In particular, a Bayesian filter where the prior and measurements are Gaussian and the state update is linear is called a “Kalman Filter”.

Notice one more thing about our results. Our final posterior tells us that the other submarine’s velocity is between +2 and +4. This is remarkable because we never received any information about its velocity! We call velocity a “hidden state” because it is never observed. One powerful trait of Kalman Filters is that they can provide estimates of states which are never observed.

Aside: the Kalman filter can also represent Gaussian noise in the state update. For example, imagine the velocity of the submarine depended on the temperature/salinity of the water. We could model the effect of this unknown environment as Gaussian noise (just like we did with the measurements).

Aside: the “extended Kalman filter” (EKF) can use any differentiable function for the state-transition model (rather than linear). This involves constructing a Jacobian (matrix of partial derivatives) to update the covariance with. The EKF is the de facto technique of GPS systems.

Act 4: Histogram Filter

Now imagine that before receiving the first sensor report, we receive an intelligence that the other submarine is between 4,600 and 6,000 meters away.

   🛥️ ···))······))       🛥️
<───┼────────────────┼──────────┼───> x
0                  4,600  ❓  6,000

And then we receive the following three measurements:

We can incorporate the intelligence report into our Bayesian framework as a uniform prior:

$$f_X(x) = \frac{1}{\text{1,400}} \quad \text{for } x \in [\text{4,600}, \text{6,000}]$$

The problem with a uniform prior is that we can no longer solve the problem analytically. What can we do instead? One option divide the x-dimension into 200 “bins” then calculate the prior and likelihood for each bin. Then, using the same linear motion update

$$\begin{bmatrix} \text{pos} \\ \text{vel} \end{bmatrix}_t = \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} \begin{bmatrix} \text{pos} \\ \text{vel} \end{bmatrix}_{t-1} = A \begin{bmatrix} \text{pos} \\ \text{vel} \end{bmatrix}_{t-1}$$

For each bin in step $(t-1)$’s posterior, transform its state $x_1$ into $x_2$ using $A$ and add that probability to step $t$’s prior’s bin at $x_2$.

Aside: alternatively we could have used a Gaussian prior with ($\mu=5,300, \sigma=700)$, called it close enough, and used a Kalman Filter. This sounds hacky, but it’s often often a good option.

Let’s take one more step back. What we’ve just constructed is called a “Histogram Filter”. Our solution is no longer analytical – that is we’ve approximated our continuous distributions with a finite grid. This sacrifices some precision, but allows us to consider non-Gaussian priors.

In this Act we used the same motion model $A$ as in Act 3. However, now that we’ve abandoned the requirement of an analytical solution, our motion model no longer needs to be linear either. So long as we can come up with a way to “update” the probability in the posterior forward in time, we can do so however we wish. For example, say we know that every 10 minutes the other submarine turns around. We could have a literal if-else statement in our motion update that if $t\%(10*60)==0$, negate the velocity. This is impossible with a Kalman Filter.

Finally, note that while our measurements have remained Gaussian they need not have. Once we start solving things numerically, we can drop any requirements about functions being Gaussian.

Act 5: Particle Filter

Now let’s consider a higher-dimensional problem: both submarines move in three dimensions. We would like to track both position $\{x,y,z\}$ and velocity $\{v_x,v_y,v_z\}$. Assume our measurements are 3d Gaussians on $\{x,y,z\}$ and our prior is a uniform distribution over a cube in $\{x,y,z\}$.

If, like before, we split each dimension into 200 bins, we now have $200^6=64\;\text{Trillion}$ bins. Using one float (32 bits) to store each bin’s value would require $32\times 64\times 10^{12} \;\text{bits}\approx250\;\text{Terabytes}$. This is an example of “the curse of dimensionality”.

Let’s think about what’s happening here. In our 2-dimensional example, after 3 measurements we are almost certain the other submarine has a positive velocity. We nonetheless multiply the prior (0) by the likelihood (0) to get the posterior (0) for every bin with negative velocity. This is a tremendous waste of compute. There are some tricks like having a dynamic discretization: have very wide bins where there is no probability and very small grids where the probability is non-zero. Possible, but adds quite a bit of complexity.

Note that the Kalman filter does not suffer from the curse of dimensionality. Since it only needs to keep track of the mean and covariance matrix, which, in 6 dimensions only require 42 floats.

Aside: the idea of assuming a Gaussian distribution to avoid the curse of dimensionality is not unique to the Kalman Filter. In machine learning, Quadratic Discriminant Analysis and Naive Bayes leverage this idea.

The particle filter solves this problem. Instead of moving our probability between bins, leaving many bins with zero probability, why not move the bins themselves? Call these moving bins “particles”. Each particle is comprised of its state $\{\text{position},\text{velocity}\}$ and its probability which we call its “weight”.

To construct our set of particles, sample from the first prior distribution. There is no longer a need to discretize our likelihood functions as we can evaluate them directly at the state of each particle. After applying the likelihood function, we update each particle using our state-transition model $A$. Like the histogram filter, we need not use a linear state-transition model or Gaussian prior/likelihoods. Note that the superscripts denote the particle index, not an exponent. Typically, the subscript is reserved for the time index.

$$\text{particle 1 (state}\ x^1):\quad x^1_t = \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} x^1_{t-1}$$

$$\text{particle 2 (state}\ x^2):\quad x^2_t = \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} x^2_{t-1}$$

$$\vdots$$

At each step the prior is the set of original particles and their weights and the posterior is the set of particles with updated weights. Each particle’s weight is updated according to Bayes’ theorem.

$$\text{particle 1 (state}\ x^1):\quad \text{new weight} \propto \frac{\substack{{\text{likelihood of sensor }}\\ {\text{reading y given }x^1}} \times \text{old weight}} {\text{evidence}} \propto \text{likelihood}\times\text{old weight}$$

$$\text{particle 2 (state}\ x^2):\quad \text{new weight} \propto \frac{\substack{{\text{likelihood of sensor }}\\ {\text{reading y given }x^2}} \times \text{old weight}} {\text{evidence}} \propto \text{likelihood}\times\text{old weight}$$

$$\vdots$$

I’ve written “$\propto$” (proportional to) instead of “$=$” because there is one more step to calculate the new weight. The particles comprise a probability distribution, so the sum of their weights should sum to 1. Therefore, we divide each weight by their sum. The effect of this normalization is the same regardless of any constant factor, so we can ignore the evidence constant.

Aside: people often get lazy and say “the particle’s likelihood”. This is short for “the likelihood of the measurement conditioned on the particle’s state”. Particles don’t have likelihoods.

For consistency, I’ll show the same 2D example as from Act 5. It’s important to understand that this technique can scale to higher dimensions, but things are simpler to visualize in 2D.

Note that in these plots I’ve set a floor on the opacity of each particle to help visualize things. Had I let the opacity go to zero, the very opaque particles would all disappear.

I’ll pause here and say that a particle filter with infinite particles and a histogram filter with infinite bins, will converge on the same analytical posterior provided by a Kalman filter. Of course, we’re doing all this to avoid trillions of cells let alone infinite ones, but it’s good to know things will converge to the correct answer. Everything from here on is a trick to use less compute, not a mathematical requirement.

Aside: what I’ve described here is called a “bootstrap filter”. It is a particular kind of particle filter where certain assumptions are made so that $\text{new weight}\propto \text{likelihood}\times\text{old weight}$. This isn’t always the case. I’ve omitted the mathematical details since I don’t think they are necessary for a conceptual understanding. You can read more on Wikepdia here.

Act 6: Resampling and Perturbations

My critique of the histogram filter was that it wasted effort computing bins which had zero probability. We could make the same critique here: most of the particles have near-zero weight but we keep them around.

The solution is to “resample” the particles. Imagine we have 100 particles. Originally each has weight $1/100$. After performing the measurement update, half have weight zero and half have weight $1/50$. To resample we throw away the particles with 0 weight and duplicate each of the survivors. We’re left with a total weight of 1 and our surviving particles are all in the “region of interest”. I won’t go into more detail about resampling, there is no shortage of explainers on the internet which focus on it.

A common technique which accompanies resampling is to add “perturbations” to the resampled particles. That is, add a bit of Gaussian noise to each perturbed particle’s state. This is because we only have finite particles, so we would like to “smear out” the survivors to more evenly cover the space.

Aside: it’s at this point that things start becoming more of an art than a science. As you progress developing a particle filter, more and more decisions will start to fall in the former category

Here are the results of adding resampling and perturbing to our particle filter:

Fair warning: particle filters fall apart in high dimensions. There just aren’t enough particles to cover the space (curse of dimensionality). Things fare better if things are roughly Gaussian.

Plenty was written on the Pentagon's falling out with Anthropic this week. This was one of the more clear-headed articles.

A good visualization of grid-scale energy being brought online in 2026.

U.S. Energy Information Administration

A two-part series considering the policy implications of recursively improving AI systems.

The vegetables in VeggieTales are not Christian.

Scott Alexander's rebuttal to the "stochastic parrot" argument

Film students can't sit through films.

Bacteria as a treatment for cancer. An idea I hadn't heard before.

Luxury apartment construction is bringing down rent.

Building more housing brings down rent prices (not surprising). More important, rent prices come down for Class-C housing even when most of those additions are in luxury apartments. Austin, Phoenix, and Denver have had an average growth in new units of 6.8%, 4.9%, and 4.3% respectively over the past 5 years.



Luxury Apartments Are Bringing Rent Down in Some Big Cities, Bloomberg

Fewer questions were asked last month on Stack Overflow than during its first month of operation.

Stack Exchange, Anonymous Post

Every contribution to Claude Code in December by its creator was via Claude Code.

India received $140B in personal remittances in 2024. That's more than the profit of their largest 70 publicly traded companies combined ($120B). The U.S. accounts for roughly 30%.

A good blog post on software work estimation

Central Stockholm renters face a 20-year wait due to rent control

Gemini's performance on the "Needle in a Haystack" test blows competition out of the water.

They've got some secret sauce, maybe sub-quadratic attention, maybe RL improvements. Unfortunately, we won't know anytime soon. google seemingly solved efficient attention, Celeste

A 2014 interview with Walter Isaacson at Khan Academy

It would have served Sam Bankman-Fried to separate utility from dollars.

I was given How Not to Be Wrong for Christmas and am enjoying it so far. I've watched plenty of Numberphile and Veritasium and have read Stephen Pinker's similar book, Rationality, yet it introduces many topics I haven't come across before. One of which is St. Petersburg paradox:

1. I put $2 in a pot then flip a coin
2. If tails, the game ends and you win the money. If heads, I double the money
3. I flip the coin again. Repeat step 2

Clearly you should want to play this game, as you win no matter what. The question is, how much should you be willing to stake for the opportunity to play? $10? $20? If you do the math you'll find that the expected value (EV) of this game is infinite — the series diverges. Therefore in theory you should be willing to stake any finite amount to play the game. This feels wrong. I wouldn't stake my entire net worth, let alone one paycheck to play. The mathematics is sound, so the paradox is of human behavior. Perhaps humans underestimate small probabilities (I've never seen a coin land 10 heads in a row) or perhaps the question is nonsense (no house could ever fund the other side of the bet).

The traditional solution is to say the premise of the EV calculation is wrong: the player should calculate the EV of their utility not of the raw dollar amount. Further, their utility should be the logarithm of the dollar amount. I had never heard of doing such a thing: every example in school, every analysis of a casino/board game, every quant interview question calculates EV directly with dollars. However, on consideration, it doesn't seem outlandish. To rappers, money is logarithmic; lines are as often about figures as they are absolute dollars. Further, for most millionaires it would hurt more to go broke than it would feel good making another million. With this conception of utility, the expected value is no longer infinite.

How does this relate to SBF? In Sam's appearance on Conversations with Tyler, he was posed a similar question: if you could push a button with a 51% chance of doubling the number of sentient beings in the universe (e.g., double the number of earths) and a 49% chance of killing all living beings, would you press it? He answered that he would press it repeatedly. Most commentators flawed Sam for lacking empathy or being hyper-rational in his answer. From what I know from Going Infinite the former is fair. But I don't think the latter is. If Sam calculated his EV using the logarithm of the number of living beings (which I'm starting to think is correct), he wouldn't have arrived at the same conclusion and his answer would have been more wonky/"hyper-rational".

Taking this a step further, could solve some of the problems/paradoxes that arise with longtermist utilitarian ideas. If you're not familiar, some people argue that we should not (or should only barely) discount the utility of future sentient being's lives with time. Therefore since there could be many more living people in the future we should prioritize protecting the future above all else. Therefore, even if you choose not to discount with time, you could still avoid some of the problems with expected values and infinity if you consider utility to be the logarithm of the number of beings. Surprisingly, in all I've read about utilitarian philosophy I've never heard this idea, though I'm sure I'm not the first.

For another time, but you could obviously extend this to SBF's Alameda Research. I'm curious to what extent, if at all, hedge funds separate EV/utility from dollars.

School cellphone bans likely have a small, positive impact on test performance

The UK without London is poorer than Mississippi

Tyler Cowen's alternative to Fischer Random Chess

St. Petersburg Paradox

I like how Casey Handmer and Terraform think about resumes

An argument to emphasize culture over process and incentives in software engineering teams

Gregory Clark's The Son Also Rises gets repetitive but is worth the read.

Bryan Caplan's The Case Against Education argues higher-ed is mostly signaling.

An interesting multi-part retrospective on 2010s culture from a little-subscribed Substack

Trader Joe's played a major role in popularizing wine (particularly Californian wine) in America. In 1970, Trader Joe's was the largest wine retailer in California.

Prediction Error Minimization Theory suggests the brain works by refining a world model to minimize the difference between its prediction and what happens next. I had never heard of this; it's supervised learning for cognitive science, which seems too simplistic.

One economist calculates U.S. academic achievement declines will result in 6% lower GDP by the end of the century.

An 8 per cent lifetime 'tax' is coming for students. Washington Post

Last month, I shared an Atlantic article about the De Beers diamond cartel. They just ran a paid post in the NYT. Like their previous campaigns, it doesn't advertise any product; this time they mythologize the story of a single diamond discovery.

The first AI image that made my jaw drop. Courtesy of Nano Banana Pro. Past models made photorealistic people look airbrushed and glossy; we've crossed the uncanny valley completely.

Riley Goodside. X

Taiwan's central bank plays a heavy-handed role in devaluing its currency to subsidise exports. Good news for Joey!

I just started reading Gregory Clark's The Son Also Rises. He tracks social mobility using the relative prevalence of surnames in elite institutions. The rates he calculates are lower than most other estimates.

When I say “Online Master’s” I mean fully remote programs for professionals from accredited institutions. For example, JHU Engineering for Professionals, where I’ve taken courses. These programs provide value via

1. Instruction (materials and guidance provided by the professor)
2. Signaling (proxy for intelligence/conscientiousness – for career advancement)

The introduction of reasoning models represents a drastic decrease in the value of both factors. First, equivalent instruction is possible via the combination of publicly available syllabi and recorded lectures in concert with LLM tutors.

Second, the elephant in the room, reasoning models can get As in most classes. I’ve tested these models on physics, math, and CS problems and suspect this is the case for all STEM subjects. If a student wishes to do so, they can get an A with almost no effort and without learning anything. If they haven’t already, employers and PhD admissions committees will soon realize that online Master’s degrees are only worth the student’s word.

I predict online Master’s programs will lose 90% of their value in the next year as LLM tutors improve and employers wake up. Assuming these programs won’t suddenly fall 90% in price, I expect enrollment to fall.

What next?

I won’t be taking any more classes at JHU. I’m toying with the idea of finding a group of coworkers or friends who want to find a textbook or recorded online course to study together. If I were in charge of my company’s HR department, I would stop compensating employees for online coursework. I would also stop counting online Master’s programs awarded after 2025 as “years of experience” when calculating salary scales. Finally, if I were a director making hiring decisions, I would weigh online Master’s coursework equally to a prospective hire claiming to have self-taught a course via a recorded lecture series and LLM tutor.

Given the following observations:

• What previously costed $2,500 (assuming instruction is 1/2 the value of these courses) can now be offered for the price of an LLM subscription.
• Demand for post-graduate education will increase as students drop out of (or never begin) these programs

There’s an opportunity here for a startup (or open source project). I’m imagining a service that organizes publicly released lectures and syllabi and matches students with others in their city to form study groups. What remains is recovering the lost signaling value. Perhaps peer reviews or in-person final exams at testing centers.

The De Beers diamond cartel ran an advertising campaign to dissuade people from selling their diamonds

Takeaways from Acquired's third part on Google

• In 2000, 15% of Google's data center infrastructure was dedicated to their language model PHIL, used for search and AdSense
• Google has, in total, roughly as many TPUs as NVIDIA ships GPUs per year
• Google could have bought Tesla for 6 billion in 2013
• Zuckerberg tried to buy DeepMind for $800 million in 2014, twice what Google ultimately paid
• The AlphaGo documentary recommended by Ben is worth watching

The manufacture of paper grocery bags is loosely 7.9x worse for the environment than plastic bags

The fastest buildout of nuclear reactors took place in France in the 70s and 80s

Paul Erdos' mathematical output heavily relied on stimulants

In relative terms, data center freshwater consumption is unconcerning

Takeaways from Andrej Karpathy on Dwarkesh Podcast

• Andrej believes that LLM's encyclopedic knowledge is holding them back. He believes removing much of this encyclopedic knowledge, leaving behind a "cognitive core" will improve performance. Tool use (e.g., connecting to Google) would make up for lost factual knowledge.
• Andrej discounts the possibility of a discrete intelligence explosion. He views the impacts of AI as an extension of ongoing computer automation.
• He's starting an AI education company. Incidentally, I wrote a blog post making the case for something exactly like this.

Multi-scale emergence can be quantified using the entropy of a system's transition probability matrices

Coal-fired plants cause 5x more excess deaths per year than total excess deaths from Chernobyl

The US and Canada accidentally kick-started India's nuclear weapons program

Pew Research survey finds increasing distaste for sports betting across America

Gemini 2.5 Pro can one-shot linear programming word problems

Above-market government salaries can compromise economic productivity

A larger fraction of Europeans die from preventable heat death than the fraction of Americans who die from firearms

GPT 5 created this slop website. GPT-5: It Just Does Stuff — Ethan Mollick.

“Conversations with Tyler” is my new favorite interview style podcast

Aircraft carriers may already be a relic

️️10% of 25 to 54-year-old men are not seeking employment

Age-graded classrooms are the worst form of schooling … except for all the others

A great comparison of US renewable energy supply

This advice is general, but probably biased by my background with legacy Java applications.

First, work through MIT’s “The Missing Semester of Your CS Education”

Useful concepts

• Chesterton’s Fence: A hiker comes across a small wooden fence on a beautiful hillside which anyone could step over. He removes it to restore the place’s beauty; it wouldn’t stop trespassers anyway. Once he has left, over the next months the cows from the neighboring farm eat all its grass and trample it into a muddy slope. When building software, if you encounter something that seems poorly designed, figure out why it was designed that way before refactoring or removing it.
• Pareto Principle (power law): Most real-world relationships are non-linear. E.g., 80% of revenue comes from 20% of customers.

One example of the Pareto Principle is methods which speed up typing:

1. Touch typing
2. Vim key bindings
3. Dvoark layout
4. Ergonomic keyboard

Touch typing will get you 80% of the way to a “pro” typist while Vim keybindings will get you another 15%. Changing keyboard layouts could help a little beyond that. Although touch typing provides 5x more value than Vim keybindings, it’s about the same difficulty to learn. My suggestion: if you can’t already touch type, learn now. If you can, learn vim keybindings. Don’t worry about keyboard layouts, taking “The Missing Semester”, reading books about software development, and networking are all better uses of your time.

Minimize configuration. “Every line of configuration is a liability” - ThePrimagen. It’s tempting to spend a lot of time configuring your IDE, terminal emulator, and shell. First, remember Chesterton’s Fence – the creators of dev tools designed their tool’s features and configuration intentionally (often for their own use!). Another drawback is that you will inevitably need to switch computers, help colleagues, or ssh into a server. These are all more difficult if you can’t use default configurations. My suggestion: when starting with a new tool, use its vanilla configuration for the first month (or more). Learn its core features before configuring settings or installing plugins.

Learn the command line versions of tools

1. They are more feature-rich
2. They can be scripted
3. Amortize costs of learning the CLI (terminal emulator, shell) and supporting tools (e.g., tmux, man) across multiple applications.

Try the simple/open source tool first. More often than not it has all the functionality you’ll need and is easier to learn. If you discover a functionality it doesn’t have, you’ll be more prepared to choose a paid offering. For example, VisualVM has suited all my Java profiling needs so far.

I read a few books on software engineering. It was a good use of my time. Ranked (power law applies here):

1. A Philosophy of Software Design – John Ousterhout
2. Code that Fits in Your Head – Mark Seemann
3. Working Effectively with Legacy Code – Michael Feathers
4. Design Patterns – Gamma, Helm, Johnson, Vlissides
5. Refactoring for Software Design Smells – Suryanarayana, Samarthyam, Sharma
6. Clean Code – Robert Martin

And finally, a few tips I try to follow (but beware pithy rules like these)

• Make simplicity the number one priority
• Use pure methods and immutable data whenever possible
• Favor composition over inheritance
• Favor strong typing

Motivation

If held in a plasma of high enough temperature and density, light ions will eventually fuse. Magnetic confinement is an approach to controlled fusion which achieves these conditions using the magnetic force. This force acts perpendicular to ions’ velocity and the magnetic field, causing them to travel in helices around field lines. Tokamaks bend these field lines into a torus, trapping ions like beads on a bracelet. Once this trap is set, ions are injected and heated with electromagnetic radiation to achieve fusion conditions. Unfortunately, the centers of ion orbits drift away from their original field lines causing them to escape confinement. The addition of a poloidal (short way around the torus) component to the magnetic field negates the effect of this drift. This post derives a representative magnetic field using a toroidal/poloidal coordinate system and motivates the additional poloidal component by simulating the trajectory of a single particle.

Toroidal Coordinate System

It’s much simpler to formulate a tokamak’s magnetic field using a toroidal/poloidal coordinate system because its geometry mirrors the problem’s, encapsulating its complexity. This system describes locations relative to a “central circle” of radius $R_0$ using three coordinates: $r$, $\theta$, and $\zeta$. $\zeta$ measures the toroidal angle (long way around), $\theta$ the poloidal, and $r$ the distance from the central circle.

The translation to and from Cartesian coordinates is given by $$\begin{aligned} &x = (R_0+r\cos\theta)\cos\zeta\quad&r = \sqrt{(\sqrt{x^2+y^2}-R_0)^2+z^2}\\ &y = (R_0+r\cos\theta)\sin\zeta\quad&\theta = \arctan2(z, \sqrt{x^2+y^2}-R_0)\\ &z = r\sin\theta\quad&\zeta = \arctan2(y, x)\\ \end{aligned}$$

Where $\arctan2(y, x)$ is the C/Python function which returns the angle between the positive x-axis and the point $(x, y)$ in the plane. Notice that $\sqrt{x^2+y^2}-R_0$ is the signed distance to the central circle in the xy-plane where a negative value indicates the point is within the circle.

The unit vectors in this system (blue vectors in the figures), expressed in Cartesian coordinates, are $$\hat{r}= \left( \begin{array}{c} \cos\theta \cos \zeta \\ \cos \theta \sin \zeta \\ \sin \theta \\ \end{array} \right),\; \hat{\theta}= \left( \begin{array}{c} -\sin\theta\cos\zeta \\ -\sin\theta\sin\zeta \\ \cos\theta \\ \end{array} \right),\; \hat{\zeta}= \left( \begin{array}{c} -\sin\zeta \\ \cos\zeta \\ 0 \\ \end{array} \right)$$

Therefore, to translate a field vector with base ($r$, $\theta$, $\zeta$) and components ($a$, $b$, $c$) to Cartesian coordinates, the base can be found using the translation above, and the vector using the scaled unit vectors: $$\begin{aligned} \mathbf{p}&=a\hat{r}+b\hat{\theta}+c\hat{\zeta}\\ \mathbf{p}&=(a\cos\theta\cos\zeta-b\sin\theta\cos\zeta-c\sin\zeta)\hat{x}\\ &+(a\cos\theta\sin\zeta-b\sin\theta\sin\zeta+c\cos\zeta)\hat{y}\\ &+(a\sin\theta+b\cos\theta)\hat{z} \end{aligned}$$

Basic Tokamak Magnetic Fields

The simplest field to imagine is a torus with constant magnitude in the $\hat{\zeta}$ (toroidal) direction. $$\begin{aligned} \textbf{B}(r, \theta, \zeta)=\begin{cases} 1\hat{\zeta} & \text{if } r < r_0 \\ 0 & \text{otherwise} \end{cases} \end{aligned}$$

Where $r_0$ is the poloidal radius. Remember that the major radius $R_0$ is implicit in the coordinate system. Unfortunately, such a field is impossible to construct. Consider the integral around the central circle. Since the field is constant, the integral will be proportional to the loop’s radius. The integral over a loop with a slightly larger radius will be slightly larger. This contradicts Ampere’s law which tells us the integrals must be equal because the same amount of current (from the toroidal field coils) passes through each loop. In order to obey Ampere’s law, the magnitude of the magnetic field must vary with $1/R$:

$$\begin{aligned} &\oint_{C_t} \textbf{B}_t\cdot d\ell=\mu_0I_{\textrm{enc}}\quad\textrm{(Ampere's law, $C_t$ is a toroidal loop)}\\ &\oint_{C_t} B_t d\ell=\mu_0I_{\textrm{enc}}\quad\textrm{(B is purely toroidal, parallel to }C_t)\\ &B_t\oint_{C_t} d\ell=\mu_0I_{\textrm{enc}}\quad\textrm{(B has toroidal symmetry)}\\ &B_t\int_0^{2\pi} R d\theta=\mu_0I_{\textrm{enc}}\quad\textrm{(R is the radius of }C_t)\\ &B_tR = \frac{\mu_0I_{\textrm{enc}}}{2\pi}=\textrm{Constant}\Rightarrow B_t\propto\frac{1}{R} \end{aligned}$$

Accounting for this fact leads us to the field: $$\begin{aligned} \textbf{B}(r, \theta, \zeta)=\begin{cases} R_0/(R_0 + r\cos\theta)\hat{\zeta} & \text{if } r < r_0 \\ 0 & \text{otherwise} \end{cases} \end{aligned}$$ Since $R=R_0+r\cos\theta$. Choosing a numerator of $R_0$ normalizes this so that the field along the central circle $1$.

Look closely, and you’ll see that the length of the vectors on the inner radius are longer than those on the outer radius.

The addition of a poloidal field helps mitigate particle drift (I’ll explain why in the next section). It’s not possible to create such a magnetic field with external coils, so instead, Tokamaks drive a toroidal current through the plasma itself. The precise mechanism (induction, neutral beam injection, or electromagnetic radiation) is not important here. Assuming this current density $\textbf{J}$ is uniform:

$$\begin{aligned} &\oint_{C_p} \textbf{B}_p\cdot d\ell=\mu_0I_{\textrm{enc}}\propto J\pi r^2\quad\textrm{(Ampere's Law, $C_p$ is a poloidal loop)}\\ &\oint_{C_p} B_p d\ell\propto r^2\quad(B_p\textrm{ is purely poloidal, parallel to }C_p)\\ &B_p\oint_{C_p} d\ell\propto r^2\quad(B_p\textrm{ has poloidal symmetry)}\\ &B_pr \propto r^2\Rightarrow B_p\propto r \end{aligned}$$

Ampere’s law tells us that this poloidal field is proportional to the minor-distance from the central circle, $r$. Because the magnetic field in a vacuum linear, the toroidal $\textbf{B}_t$ and poloidal fields $\textbf{B}_p$ can be combined by addition:

$$\begin{aligned} \textbf{B}(r, \theta, \zeta)=\begin{cases} R_0/(R_0 + r\cos\theta)\hat{\zeta} + (r/r_0) \hat{\theta} & \text{if } r < r_0 \\ 0 & \text{otherwise} \end{cases} \end{aligned}$$

I normalize the poloidal $\hat{\theta}$ term, so it varies from 0 at the central circle to $1$ at the surface.

Particle drift

There are two causes of particle drift due to complexities of the magnetic field. The first is caused by the magnetic field varying in magnitude. This is called grad-B $(\nabla B)$ drift. It causes the orbital centers of particles to move with velocity:

$$\mathbf{v}_{\nabla B}=\pm\frac{1}{2}v_\perp r_L\frac{\mathbf{B}\times\nabla B}{B^2},\quad r_L=\frac{mv_\perp}{|q|B}$$ Where $v_\perp$ is the particle’s velocity perpendicular to the field line, $r_L$ is the Larmour radius (radius of a single orbit around the field line), and $B=|\textbf{B}|$ is the magnitude of the magnetic field. The $\pm$ indicates that the drift is positive for ions and negative for electrons. To calculate this drift for the toroidal field, consider an ion anywhere in the poloidal cross-section at $\zeta=0$. Using the fact that when $\zeta=0$, $\hat{\zeta}=\hat{y}$ and that $x=R_0+r\cos\theta$, we can write the gradient of the field in Cartesian coordinates.

$$\begin{aligned} &\textbf{B}=R_0/(R_0+r\cos\theta)\hat{\zeta}=(R_0/x) \hat{y}\\ &B=R_0/x\\ &\nabla B=\frac{\partial B}{\partial x}\hat{x}=-\frac{R_0}{x^2}\hat{x} \end{aligned}$$

The grad-B drift for an ion with charge $q$ and mass $m$ is then

$$\mathbf{v}_{\nabla B}=\frac{1}{2}v_\perp\frac{mv_\perp}{q(R_0/x)}\frac{(R_0/x)\hat{y}\times(-R_0/x^2)\hat{x}}{(R_0/x)^2}=\frac{mv_\perp^2}{2qR_0}\hat{z}$$

By argument of symmetry (we could have oriented the x-axis however we like) the ion will experience this same drift everywhere in the torus. Notice this is a constant, so this drift is the same for any particle trajectory.

The second type of drift is due to the curvature of the field and arises from the centrifugal force: $$\textbf{v}_R=\frac{mv_\parallel^2}{qB^2}\frac{\textbf{R}_c\times\textbf{B}}{R_c^2}$$ Where $\textbf{R}_c$ is the vector pointing from the center of the torus to the particle, and $v_\parallel$ is its velocity along the magnetic field line. Now consider an ion at $\zeta=\theta=0$, that is, along the $x$-axis so that $R_c=x$. The curvature drift will be $$\textbf{v}_R=\frac{mv_\parallel^2}{q(R_0/x)^2}\frac{R_c\hat{x}\times(R_0/x)\hat{y}}{R_c^2}=\frac{mv_\parallel^2}{qR_0}\hat{z}$$

If the particle has a non-zero $z$-component, this becomes more complex since $R_c\neq x$ and $\textbf{R}_c$ is no longer perpendicular to $\textbf{B}$.

The net effect in the toroidal field is that ions traveling in the direction of the field lines will drift in the $+\hat{z}$ direction until they escape confinement. This is obviously a concern. Fortunately, the addition of a poloidal component of the magnetic field solves this problem. The twisting path causes ions to spend an equal amount of time in the top ($z>0$) and bottom ($z<0$) of the torus. While the ion is in the top, its upward drift causes it to move away from the central circle, but while it is in the bottom, its drift bring it back towards the central circle. Therefore, on average, there is no overall drift!

Simulation Setup

Assume the effect of gravity is negligible and the electric field is zero. The ion experiences only the magnetic force:

$$\begin{aligned} &\textbf{F}=m\textbf{a}=q\textbf{v}\times\textbf{B}\\ &\frac{d\textbf{v}}{dt}=\frac{q\textbf{v}\times\textbf{B}}{m}\\ \end{aligned}$$

The ion’s trajectory will follow this ODE. This cannot be solved analytically, so I use an adaptive Runge-Kutta method to solve it numerically. We can simplify the code by non-dimensionalizing this equation. That is, replace each component of the equation with a characteristic unit (denoted by a subscript $c$) multiplied by a non-dimensional term (denoted by ~). For example, $a=\tilde{a}a_c$

$$\begin{aligned} \left(\frac{v_c}{t_c}\right)\frac{d\tilde{\textbf{v}}}{d\tilde{t}}&=\left(\frac{q_cv_cB_c}{m_c}\right)\frac{\tilde{q}\tilde{\textbf{v}}\times\tilde{\textbf{B}}}{\tilde{m}}\\ \frac{d\tilde{\textbf{v}}}{d\tilde{t}}&=\left(\frac{q_cB_ct_c}{m_c}\right)\frac{\tilde{q}\tilde{\textbf{v}}\times\tilde{\textbf{B}}}{\tilde{m}}\\ \frac{d\tilde{\textbf{v}}}{d\tilde{t}}&=\frac{\tilde{q}\tilde{\textbf{v}}\times\tilde{\textbf{B}}}{\tilde{m}}\\ \end{aligned}$$

Fix $q_c$, $B_c$, and $m_c$ to scales relevant to the problem, and then choose $t_c$ so that this pre-factor becomes 1. Characteristic length is then derived from the other units.

• Magnetic field: $B_c=1$ T (Typical Tokamak field)
• Electric charge: $q_c=1.6\times10^{-19}$ C (proton charge)
• Mass: $m_c=1.67\times10^{-27}$ kg (proton mass)
• Velocity: $v_c=2.2\times10^5$ m/s (typical thermal velocity in a fusion reactor)
• Time: $t_c=\frac{m_c}{q_cB_c}=1.04\times10^{-8}$ s
• Length: $L_c=v_ct_c=0.00230$ m

The simulation will use this simplified, non-dimensional equation. To recover a dimensional-quantity, simply multiply the non-dimensional value by the characteristic unit.

I use the following parameters and initial conditions:

• Major radius: $R_0=2\textrm{m}=870$
• Minor radius: $r_0=0.5\textrm{m}=217$
• Initial position: $\textbf{r}=(R_0+r_0/2, 0, 0)=(979, 0, 0)$
• Initial velocity: $\textbf{v}=(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}, 0)$

Particle Trajectories

At the stating time, $v_\parallel=v_y=v_\perp=v_x$. Since $v_\parallel^2+v_\perp^2=v^2$, we have that $v_\perp^2=v_\parallel^2=v^2/2$, so the drift velocity should be roughly $$\begin{aligned} &\textbf{v}_{drift}=\textbf{v}_{\nabla B} + \textbf{v}_R=\left(\frac{mv_\perp^2}{2qR_0}+\frac{mv_\parallel^2}{qR_0}\right)\hat{z}\\ &\textbf{v}_{drift}=\left(\frac{mv^2}{4qR_0}+\frac{mv^2}{2qR_0}\right)\hat{z}=\left(\frac{3mv^2}{4qR_0}\right)\hat{z}\\ &\textbf{v}_{drift}=189\hat{z}\textrm{ m/s}=0.00086\hat{z} \end{aligned}$$

In the toroidal field, the particle follows a circular path while drifting upward as expected. The rate of this drift is slightly smaller than $0.00086$. I don’t know why. Remember that while the particle loops the torus, it travels in a helical path around the field line, this is why the line in Figure 6 is thick.

The addition of a poloidal field adds a periodic poloidal motion to the trajectory. The amplitude of the particle’s oscillation in the $z$-direction does not grow with time, so the drift has been mitigated.

Resources

I’ve only provided the analytic form of these drifts to keep this short. If you’re looking for a conceptual understanding, I suggest chapter 4 of “The Future of Fusion Energy” by Parisi and Ball. For an introduction to the Runge-Kutta method, see Chapter 6 of Toby Driscoll’s “Fundamentals of Numerical Computation”.

Consider a rectangular dartboard with two circles $A$ and $B$ drawn on it. A monkey is throwing darts at the board, and they fall randomly within it. The probability that the dart falls within the circle labeled $A$ is its area divided by the area of the rectangle, $20/100=0.2$. Now imagine you’re facing the other way and the money throws the dart. The bartender tells you it landed in $B$. What is the probability it also landed in $A$ (i.e., in the purple intersection)? It’s hopefully intuitive this is the area of the purple section divided by the red section, $4/12=0.33$.

Bayes’ rule formalizes this: $$P(A|B)=\frac{P(B|A)P(A)}{P(B)}\tag{1}$$

Where $P(A)$ is the probability of event $A$ occurring and $P(A|B)$ is the probability of $A$ occurring given that $B$ occurred. Plugging in the probabilities from the dartboard example, we get the result we expect: $$P(A|B)=\frac{(4/20)\times (20/100)}{(12/100)}=4/12=0.33\tag{2}$$

One way to derive Bayes rule is by starting with the definition of joint probability: $$P(A\cap B)=P(A|B)P(B)=P(B|A)P(A)\tag{3}$$

If you aren’t convinced of this fact, consider the dartboard again. You can think of $P(A|B)$ as what % of the red circle is covered by the purple section. Or, in other words, the ratio of the area of the purple section to that of the red section. If we then multiply that by the area of the red section, we’re left with the area of the purple section. You can use the same logic to convince yourself of the third term in (3). Finding Bayes’ rule is as simple as dividing (3) by $P(B)$.

Let’s take a step back, and think more about (1). $P(A|B)$ is a probability and therefore bounded between 0 and 1. How does this fraction ensure this is the case? It’s clear that it won’t be negative as the three components are all positive. But if $P(B)\to 0$, why wouldn’t this fraction grow to infinity? The reason is that $P(B|A)P(A)=P(A\cap B)$ which is strictly less than or equal to $P(B)$.

In (1), $P(A)$ is called the prior and $P(A|B)$ the posterior since $P(A)$ was our belief about the probability of event $A$ occurring before observing the event $B$, and $P(A|B)$ is our belief about its probability after (prior to) observing $B$.

Resources

• An Introduction to Bayesian Thinking: Free online textbook from a Coursera course. Good for someone who has taken an undergraduate stats course

Take a Gaussian probability distribution $$f(x; \mu, \sigma)=\frac{1}{\sigma\sqrt{2\pi}}\exp(-\frac{1}{2}(\frac{x-\mu}{\sigma})^2)$$

This is the definition of the likelihood function.

I’ll emphasize that the expression on the right hasn’t changed. A few things fall out of this 1) Previously the integral of $f$ (holding $\mu$, $\sigma$ constant) over $x$ was 1. Now if we integrate $L$ over $\mu$ and $\sigma$ (holding x constant) there is no reason to expect the integral is still 1. 2) $L$ evaluated at $(\mu=a,\sigma=b;x=5)$ is a likelihood. If $f$ happened to be parameterized by $\mu=a,\sigma=b$ and evaluated at $x=5$ the result would be the same (since the expression is the same). Therefore, the value of $f$ evaluated at $x=5$ is the likelihood of its parameters parameterized by $x=5$. That’s a long-winded way to say $f(x)$ is a likelihood. (Remember it is the integral of $f$ which is a probability: $\int_a^bfdx$).

So what does it mean? Look at the line on this surface for $\mu=5.5$. Notice that near $\sigma=0$ it has a low likelihood. As $\sigma$ increases, its likelihood increases for a bit and then declines again. Consider three points along that line ($\mu$, $\sigma$) = (5.5, 0.2), (5.5, 1), (5.5, 2). How did we calculate the value of $L$ at each of these points? We just plugged in these $(\mu, \sigma)$ pairs along with $x=5$ into the expression above. Let’s visualize our doing that, but letting x vary again (notice these curves are PDFs now).

The value of our likelihood surface for each combination $(\mu, \sigma)$ is the intersection of the red line with each of these PDFs. Orange performs the best. Even if we centered the green PDF on $x=5$ it would still not have as high a value as the orange curve. So we can think about the values of the likelihood function like this: “how well does this pair of parameters $(\mu,\sigma)$ fit the data point $x=5$?”

That brings us to Wikipedia’s definition: “A likelihood function (often simply called the likelihood) measures how well a statistical model explains observed data by calculating the probability of seeing that data under different parameter values of the model.” Hopefully this is clear now.

One last thing. Wikipedia’s article continues “It is constructed from the joint probability distribution of the random variable that (presumably) generated the observations”. When I said earlier that the definition of the Likelihood was simply the PDF with its parameters considered as variables that was only partially true. It is actually the joint PDF. In this case we just considered a “joint” PDF with only one part. $$L(\mu, \sigma; x_1,x_2,...x_n)\triangleq \prod_{i=1}^nf(x_i;\mu,\sigma)=\prod_{i=1}^n\frac{1}{\sigma\sqrt{2\pi}}\exp(-\frac{1}{2}(\frac{x_i-\mu}{\sigma})^2)$$ The likelihood function is useful because it helps us guess the parameters of a distribution given some samples from it. Say we have IID samples $x_1,x_2,\cdots,x_n$, from a known distribution with unknown parameters, and we want to find the parameters which maximize the probability of having drawn this sample. The parameters which maximize the likelihood function are those parameters! If this isn’t clear, think back to the one dimensional case and the previous figure.

Here is a simple example of this process (called a maximum likelihood estimate) for three data points assumed to be drawn from a Gaussian distribution

The root cause of a lot of bad code is the overuse of common rules (e.g., DRY, YAGNI, “keep methods under X lines”). These are well-intentioned and distill years of programming expertise. However, even in a world where they’re right 90% of the time, the 10% of times they’re wrong translates to technical debt which compounds quickly. This isn’t to say their creators are at fault; These rules are often introduced with disclaimers about overuse. The problem is that these catchy sayings don’t have room for disclaimers. After learning a new rule, new developers may only remember the acronym and not any of its nuances. Senior developers aren’t innocent either; During code review, it’s easier to cite a rule than explain the underlying issue with the code. Teachers don’t want to overwhelm students, so they may give a blanket statement like “don’t use static methods”. There is a direct relationship between the length of a piece of advice and its value. Longer answers can provide context, edge cases, and describe more sophisticated concepts. It shouldn’t be a surprise that short, general statements can lead us astray.

“Methods should be shorter than X lines”

The professor of my software design course at Georgia Tech suggested methods should be under five lines, and Clean Code claims “The first rule of functions is that they should be small. The second rule of functions is that they should be smaller than that.” In my mind, this rule is just a heuristic to remind us that methods should generally only have a single “concept”. I argue that because the former is easier to remember it gets used (and abused) more frequently. See It’s probably time to stop recommending Clean Code for an example of how this can go too far.

“Don’t repeat yourself”

In my first internship, I inherited a thousand-line R script with code literally copy and pasted half a dozen times with different hard-coded parameters. DRY is a powerful concept when you first learn it, and I’ve met a number of data scientists who could have learned it sooner. However, entry level software engineers understand copy and pasting code is a red flag. However, it can cause more harm than good in the form of premature optimization and over-abstraction.

“Comments are a failure to express yourself through code”

Early on at my first job, I came across this saying in Clean Code. It seemed reasonable, and my team already used comments sparingly, so I accepted it at face value. Over my first year, I wrote nearly no comments. I went so far as to leave PR comments to explain confusing bits because I internalized that putting it in the code was a failure. I’ve since grown a much more favorable outlook on comments. The ultimate irony is that, upon returning to the chapter on comments in Clean Code, I agree almost entirely with it. But because I only remembered this one saying, my outlook on the topic was quite warped.