We compute the free energy for the Curie-Weiss model and verify that the mean-field approximation predicts the correct free energy density. This article follows the approach in Information, Physics, and Computation, example 4.11.
Status: still a work a progress.
$\newcommand\E{\mathbb E}$
We set $f(\sigma) = \frac 1 {2n} (\sigma \cdot \mathbf{1})^2$ and we wish to compute $\log Z$ where $$Z = \sum_{\sigma \in \{\pm 1\}^n} e^{\beta f(\sigma)} = \sum_{k=0}^n \binom{n}{k} \exp(\frac \beta {2n} (n -2k)^2).$$
On one hand by Jensen we get $Z\ge 2^n\cdot e^{-\beta/2}$, but $Z\le 2^n \cdot e^{-(\beta / 2)n}$. This tells us that $\frac{1}{n}\log Z$ is bounded.
Can we say more? Let’s try some heuristics.
We use the estimate $$\log \binom{n}{k} \approx nh(k/n),$$ where $h(p) := -p\log p - (1-p)\log(1-p)$ is the binary entropy function. So \begin{align} Z \approx \sum_{k=0}^n \exp(n(\tfrac \beta 2 (1-\tfrac {2k} n)^2 + h(k/n))) \end{align}
Define $m$ to be the average spin $m := 1-\frac{2k}{n}$, then we can approximate $$Z \approx \frac{n}{2} \int_{-1}^1 \exp(n(\tfrac {\beta m^2}{2} + h(\tfrac {1+m} 2))\, dm$$
Laplace method tells us that as $n\to \infty$, $$\int_{-1}^1 e^{nf(m)} dm \sim \sqrt{\frac{2\pi}{n|f''(m_0)|}}e^{n f(m_0)}$$ where $m_0 = \arg\max f(m_0)$, so it remains to analyze the function $$f(m) := \frac{\beta m^2}{2} + h\left(\frac{1+m}{2}\right).$$
Once we do, the answer is simply $$\frac{1}{n}\log Z \to \max_{m\in[-1,1]} f(m)$$
We can do this visually:
[graphs]
The rigorous explanation comes from $$f''(m) = \beta + \frac{1}{x^2 - 1} $$ If $\beta \le 1$, then the function is concave with a unique maximum at $m=0$. Beyond that ($\beta > 1$), the function is concave-convex-concave, so we just find the critical points $$f'(m) = \beta m - \tanh^{-1} m = 0$$
The mean-field approximation gives \begin{align} \log Z &\le \max{ \beta \cdot \E_\xi f(\sigma) + H(\xi) }\ &\le \max_{m_i} \left{ \frac{\beta}{2n}\left(\sum m_i\right)^2 + \frac{\beta}{2n} \sum_i (1-m_i^2) + \sum_i h\left(\frac{1+m_j}{2}\right) \right} \end{align}
For $\sum m_i$ fixed, we get a separable sum of concave functions, so we can replace all the $m_i$’s by their average $m := \frac 1 n \sum_i m_i$ to get \begin{align} & \le n\cdot \max_{m} \left{\frac{\beta}{2} (m^2 + \tfrac 1 n (1-m^2)) + h\left(\frac{1+m}{2}\right) \right} \end{align}
You can really see this is almost the same as our heuristic answer above! In particular, we expect that as $n\to\infty$, this should give the exact same answer for $\frac{1}{n}\log Z$.
Even more specifically, $\log Z$ appears to differ by at most $O(\beta)$. This turns out to be wrong for larger $\beta$ because our other approximations start to become wrong, and the true answer at $\beta=1$ looks more like $O(\log n)$.
Okay, so now we actually want to try and figure out the mean-field gap in various cases.
In the high temperature case ($\beta < 1$), by substituting the optimal $m=0$ we get $$GAP := \log Z - n \log 2 - \frac \beta 2 $$
[show some plots maybe… todo!]
Aside. Why is $\log \binom{n}{k} \approx nh(k/n)$? We can interpret the LHS as the entropy of drawing a vector $X\in \{0,1\}^n$ with exactly $k$ 1’s uniformly. However, each coordinate $X_i$ behaves like a $\mathrm{Bern(k/n)}$, so the RHS is precisely the sum of entropies of each individual coordinate.
It is a known fact that $$H(X) \le H(X_1) + ... + H(X_n)$$
The magic now is that equality roughly holds, which we sort of expect with the philosophy that drawing a uniform vector with $k$ 1’s is really “similar” to just independently drawing $X_i$.