A short note about how energy and entropy appears together.

$\newcommand{\E}{\mathbb E} \newcommand{\Unif}{\mathrm {Unif}} \newcommand{\GAP}{\mathsf{GAP}} \newcommand{\norm}[1]{\left\| #1 \right\|}$

The Gibbs decomposition tells us that log-partition function can be split into “energy” and “entropy”: $$\log Z = \log \sum_\sigma e^{\beta f(\sigma)}\ge \underbrace{\beta\cdot \E_{\mu} f}_{\text{energy}} + \underbrace{H(\mu)}_{\text{entropy}}$$ and furthermore equality holds for the Gibbs distribution $\mu \propto e^{\beta f}$.

What exactly does that mean? The fundamental tension here is as follows:

• in the sum for $Z$, we have $2^n$ terms, which makes us suspect that $Z$ is roughly $2^n$ times of the “average” term.
• for a sum that looks like $$\sum_i e^{a_i n}$$ we expect that for large $n$ this sum is dominated by $e^{n\cdot \max a_i}$.

So which one wins? The largest term, or the combined power of the common folk?

In the two extreme cases:

• $\beta \to\infty$: energy wins out.
• $\beta \to 0$: entropy wins out.

We see this play out in the calculation for The Curie-Weiss Model: take$f(\sigma) = \frac 1 {2n} (\sigma \cdot \mathbf{1})^2$ and $$Z = \sum_{\sigma \in \{\pm 1\}^n} e^{\beta f(\sigma)} = \sum_{k=0}^n \binom{n}{k} \exp(\frac \beta {2n} (n -2k)^2).$$ Here, if we just fix $k$ and look at the log of a single term, we see that it cleanly splits into an energy term ($\tfrac \beta {2n} (n-2k)^2$) and an entropy term ($\log \binom{n}{k} \approx nh(k/n)$). So we see the exact two opposing forces at play: the sum coming from there being many terms is precisely the entropy, and the sum coming from the terms being large is precisely the entropy term.

Which one ends up being the answer? The specific $k$ that maximizes the term.