Why is it that $p(\sigma) \propto e^{\beta f(\sigma)}$ for the Gibbs’ distribution? Let’s look at two possible reasons.

Pre-requisites: Setup for Statistical Physical Models

Reason 1. Because they maximize entropy for a fixed expected energy. (This argument is a famous one due to Jaynes.)

Suppose $p(\sigma)$ was an unknown distribution. We want to maximize the entropy $$H(p) = - \sum_{\sigma} p(\sigma) \log p(\sigma)$$ We have the constraints $\sum_{\sigma} p(\sigma) = 1$ and $\sum_{\sigma} f(\sigma)p(\sigma) = E$. Lagrange multipliers gives us $$-\log p(\sigma) - 1 + \lambda +\mu f(\sigma)= 0$$ i.e. $p(\sigma) = \exp(-1+\lambda + \mu f(\sigma))$, so $p(\sigma) \propto e^{\mu f(\sigma)}$.

Reason 2. (from here)

Pretend for the moment that energy is allowed be any positive real number $[0,\infty)$.

Suppose we have a large number $N$ of systems (“replicas”) with average energy $1/\beta$. The energy vector thus belongs to the simplex $$\left\{(E_1,E_2,...,E_N): E_i \ge 0, \sum_i E_i = \frac N \beta \right\}.$$ Suppose that any energy vector is equally probable. (For instance, if at each step we pick two indices $i,j\in[N]$ uniformly at random and transferred an infinitesimal packet of energy from replica $i$ to replica $j$).

What is the probability that $E_i \ge E$? It is a slightly smaller simplex with sidelength $\frac N \beta - E$, so $$\mathbb P(E_i\ge E) = \left(1-\frac{E}{N/\beta}\right)^{N-1}$$ so the probability density for $E_i$ is $$p(E_i = E) = \frac{\beta(N-1)}{N}\left(1-\frac{E}{N/\beta}\right)^{N-2} \overset{N\to\infty}\longrightarrow \beta e^{-\beta E}$$ and there we have it!

What if our energy has discrete levels? Then, we have to pretend that we do the same process but we only ever look when all the energies are on valid levels (i.e. take the conditional distribution over valid energy levels), and this gives us the same conclusion that $p(E)\propto e^{-\beta E}$!

Update. I no longer think that the last paragraph is sufficient justification: for one, if we restrict the energy levels, we might not have any support on our fixed level energy surface at all! More to come…