Deriving forward pass recursive formulae

For an exponential moving average (EMA), we have the following formula to use in our torch.forward() method.

In order to use it in our forward pass, we need to express this in terms of the previous running average, where the upper limit of our summation is ‘T-1’ aka EMA_T-1 as we’ll be recursively calculating this for each step.

1: Write out the Full Summation

Using the formula we’ve been given, we’ll express the EMA at time T as a series of additions:

EMA_T = μ·mean_T + μ(1-μ)·mean_T-1 + μ(1-μ)²·mean_T-2 + … + μ(1-μ)^T-1·mean₁

2: Look at the Previous EMA

Now consider what EMA_T-1 looks like:

EMA_T-1 = μ·mean_T-1 + μ(1-μ)·mean_T-2 + μ(1-μ)²·mean_T-3 … + μ(1-μ)^T-2·mean₁

3: Notice the Pattern

Look carefully and you’ll see that every mean_T-n value is present in both EMA summations but multiplied by (1-μ) for EMA_T. There is also one additional μmean_T.

This gives us:
EMA_T = EMA_T-1 * (1-μ) + μ·mean_t

or in our forward method:

running_avg = running_avg * (1 - momentum) + batch_mean * momentum