Channel Capacity — Mathematics Knowledge System

Overview

Channel capacity C is the maximum rate at which information can be transmitted over a noisy communication channel with arbitrarily small error probability. Shannon's noisy-channel coding theorem (1948) is one of the most profound results in information theory: it establishes a sharp threshold. For any rate R < C, there exist encoding/decoding schemes that achieve arbitrarily small error probability. For any R > C, the error probability is bounded away from zero regardless of the code used. For the additive white Gaussian noise (AWGN) channel with bandwidth W and signal-to-noise ratio SNR, the capacity is C = W log₂(1 + SNR) bits per second — the Shannon-Hartley theorem.

Intuition

Imagine a noisy telephone line. You can whisper (low rate, low noise impact) or shout (high rate, more errors). Shannon says there is a sweet spot — the capacity C — below which you can communicate perfectly reliably at that rate using the right code, no matter how noisy the channel. Above C, errors are unavoidable. The key insight is that random coding over many channel uses averages out the noise: the codewords become well-separated in the space of possible channel outputs, so the decoder can tell them apart with high probability even through the noise.

Formal Definition

Definition

A discrete memoryless channel (DMC) is defined by input alphabet X, output alphabet Y, and transition probabilities p(y|x). The capacity is:

C = \max_{p(x)} I(X;Y)

Channel capacity (bits per channel use)

C = \max_{p(x)} \sum_{x,y} p(x)p(y|x) \log \frac{p(y|x)}{p(y)}

Expanded form

C = W \log_2\!\left(1 + \frac{S}{N}\right)

Shannon-Hartley theorem (AWGN channel)

C_{\text{BSC}}(p) = 1 - H_b(p) = 1 + p\log_2 p + (1-p)\log_2(1-p)

Binary symmetric channel capacity

Notation

Notation	Meaning
$C$	Channel capacity (bits per channel use or bits/sec)
$p(y\|x)$	Channel transition probability
$\mathrm{SNR}$	Signal-to-noise ratio
$H_b(p)$	Binary entropy function: -p log p - (1-p)log(1-p)

Theorems

Theorem 1: Shannon's Noisy-Channel Coding Theorem

For a discrete memoryless channel with capacity C, for any rate R < C and \epsilon > 0, there exists a sequence of codes with block length n such that the average probability of error P_e^{(n)} \to 0 as n \to \infty. Conversely, for any R > C, P_e^{(n)} \geq \delta > 0 for all codes.

Theorem 2: Shannon-Hartley Theorem

The capacity of an additive white Gaussian noise channel with bandwidth W Hz and signal power S in noise power N is C = W \log_2(1 + S/N) \text{ bits per second}.

Theorem 3: Binary Symmetric Channel Capacity

For a binary symmetric channel with crossover probability p, C = 1 - H_b(p) \text{ bits per channel use, where } H_b(p) = -p\log_2 p - (1-p)\log_2(1-p).

Worked Examples

1
The BSC capacity formula is C = 1 - Hb(p).
$C = 1 - H_b(0.1)$
2
Compute Hb(0.1) = -0.1 log₂(0.1) - 0.9 log₂(0.9).
$H_b(0.1) = 0.1 \times 3.322 + 0.9 \times 0.152 \approx 0.469 \text{ bits}$
3
Subtract from 1.
$C = 1 - 0.469 = 0.531 \text{ bits per channel use}$

✓ Answer

C ≈ 0.531 bits per channel use. A BSC with 10% error rate can reliably transmit about 53% of 1 bit per use.

Practice Problems

Mediumapplication

A phone line has bandwidth 4000 Hz and SNR = 1000 (30 dB). Compute the theoretical maximum data rate.

Mediumfree response

The BSC has crossover probability p = 0 (noiseless) and p = 0.5 (pure noise). What are the capacities in each case, and interpret them.

Common Mistakes

Common Mistake

Thinking channel capacity is achieved by a specific modulation scheme or code.

Capacity C is a theoretical maximum. Approaching it requires increasingly long random codes; no simple scheme achieves it exactly.

Common Mistake

Confusing bandwidth W (Hz) with capacity C (bits/sec). They are related but not equal.

C = W log₂(1 + SNR). Doubling bandwidth doubles capacity; doubling SNR increases capacity by log₂((1+2SNR)/(1+SNR)), a smaller gain.

Quiz

Historical Background

Prior to Shannon's 1948 paper, engineers believed that reducing transmission error required reducing the communication rate — that noise was a fundamental barrier. Shannon's counterintuitive insight was that error can be made arbitrarily small at any rate below capacity by using clever coding, without reducing the information rate. This separated the concept of channel capacity from bandwidth and established coding theory as a rich mathematical discipline. The Shannon-Hartley formula had appeared earlier in the work of Nyquist and Hartley, but Shannon unified them into the full noisy-channel coding theorem.

1928
Nyquist publishes bandwidth-speed relationship for noiseless channels.
Harry Nyquist
1928
Hartley introduces logarithmic measure of information capacity.
Ralph Hartley
1948
Shannon proves the noisy-channel coding theorem and derives channel capacity.
Claude Shannon
1950s
Hamming, Reed, Solomon begin constructing practical error-correcting codes approaching capacity.
Richard Hamming, Irving Reed, Gustave Solomon
1993
Turbo codes demonstrate near-capacity performance, vindicating Shannon's theorem practically.
Berrou, Glavieux, Thitimajshima

Summary

Channel capacity C = max_{p(x)} I(X;Y) is the maximum mutual information between input and output over all input distributions.
Shannon's noisy-channel coding theorem: reliable communication is possible iff rate R < C; for R > C, error is unavoidable.
For AWGN channels: C = W log₂(1 + SNR) — bandwidth and SNR are the two levers for increasing capacity.
For binary symmetric channels: C = 1 - Hb(p) bits/use, ranging from 1 (noiseless, p=0) to 0 (pure noise, p=0.5).

References

PaperShannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27, 379-423.
BookCover, T. and Thomas, J. (2006). Elements of Information Theory, 2nd ed. Wiley. Ch. 7-8.
WebsiteWikipedia — Channel capacity