Linear Algebra

The same vector can be described by different coordinate lists depending on which basis is used, and a change-of-basis matrix converts coordinates from one basis to another without changing the underlying vector itself. If B is a basis whose vectors are the columns of a matrix P, then a vector's B-coordinates c relate to its standard coordinates x by x = Pc, so converting the other way requires c = P⁻¹x. Change of basis is essential for simplifying linear transformations — expressing a transformation matrix in the 'right' basis (such as an eigenbasis) can turn a complicated matrix into a simple diagonal one.

A set of vectors is linearly independent if no vector in the set can be written as a combination of the others — equivalently, the only way to combine them to get the zero vector is to use all-zero coefficients. If some nontrivial combination does produce zero, the set is linearly dependent, meaning at least one vector is redundant: it lies in the span of the rest. Linear independence is the precise notion of 'no redundancy' that underlies the idea of a basis, and it is tested algebraically by checking whether a homogeneous linear system has only the trivial solution.

A basis of a vector space is a set of vectors that is linearly independent and spans the whole space — every vector in the space can be written in exactly one way as a linear combination of basis vectors. All bases of a given vector space have the same number of elements, and that common number is called the dimension of the space. Dimension is the single most important structural invariant of a vector space: it tells you how many independent 'coordinates' are needed to describe every vector, regardless of which particular basis you choose.

A vector space is a set of objects (vectors) that can be added together and scaled by numbers, obeying a short list of intuitive rules — and nothing else is required. This abstraction is what makes linear algebra so widely applicable: arrows in the plane, polynomials, matrices, and even functions all form vector spaces, so any theorem proven about vector spaces in general applies to all of them at once.

A square matrix A is diagonalizable if it can be written as A = PDP⁻¹, where D is diagonal and P's columns are eigenvectors of A. Diagonalization exists exactly when A has n linearly independent eigenvectors (for an n×n matrix), which is guaranteed, for instance, whenever A has n distinct eigenvalues. Once diagonalized, computing powers of A becomes trivial — Aᵏ = PDᵏP⁻¹, and raising a diagonal matrix to a power just means raising each diagonal entry to that power — which makes diagonalization a central tool for solving recurrences, systems of differential equations, and Markov chain long-run behavior.

The singular value decomposition (SVD) factors ANY m×n matrix A — even non-square, non-invertible ones — as A = UΣVᵀ, where U and V are orthogonal matrices and Σ is diagonal with nonnegative entries called singular values. Unlike eigendecomposition, which only applies (nicely) to square diagonalizable matrices, SVD exists for every real matrix, which is why it is the single most versatile matrix factorization in numerical linear algebra. The singular values measure how much A stretches space along each of a special set of orthogonal input/output direction pairs, and they underlie low-rank approximation, PCA, and pseudo-inverses.

A symmetric matrix is a square matrix equal to its own transpose (A = Aᵀ), meaning the entry in row i, column j always matches the entry in row j, column i. Symmetric matrices are exceptionally well-behaved: the spectral theorem guarantees that every real symmetric matrix has only real eigenvalues and possesses a full set of orthogonal (or orthonormal, after scaling) eigenvectors, so it is always diagonalizable — no defective cases exist. This makes symmetric matrices the workhorse of applications from covariance matrices in statistics to stiffness matrices in engineering, all of which are symmetric by construction.

The eigenspace of a matrix A for a given eigenvalue λ is the set of all eigenvectors with that eigenvalue, together with the zero vector — that is, the null space of A − λI. Every eigenspace is itself a subspace, and its dimension (called the geometric multiplicity of λ) tells you how many independent directions are scaled by that same factor λ. The geometric multiplicity is always at most the algebraic multiplicity (how many times λ is a repeated root of the characteristic polynomial); when they're equal for every eigenvalue, the matrix is diagonalizable.

The rank of a matrix (or linear transformation) is the dimension of its column space — the number of independent directions its output can reach. The nullity is the dimension of its null space (kernel) — the dimension of the set of inputs that get crushed to zero. These two numbers are linked by the rank-nullity theorem, one of the most useful bookkeeping identities in linear algebra: rank plus nullity always equals the number of columns (the input dimension). Rank measures how much information a transformation preserves; nullity measures how much it destroys.

An eigenvector of a linear transformation is a nonzero vector that doesn't change direction when the transformation is applied — it only gets scaled. The scaling factor is the eigenvalue. Eigenvectors reveal the 'natural axes' of a transformation: directions along which its behavior is as simple as multiplication by a single number.

The Gram-Schmidt process converts any linearly independent set of vectors into an orthogonal (or orthonormal) set that spans the same subspace. It works by processing vectors one at a time: each new vector has its projections onto all previously-built orthogonal vectors subtracted off, leaving only the part that is genuinely new and perpendicular to everything so far. This process is the constructive proof that every finite-dimensional inner product space has an orthonormal basis, and it underlies the QR decomposition of a matrix used throughout numerical linear algebra.

An inner product space is a vector space equipped with an inner product — a function that takes two vectors and returns a scalar, generalizing the familiar dot product. The inner product lets us measure lengths (via the induced norm) and angles (via the sign and size of the inner product), even in abstract spaces like polynomials or functions where 'length' and 'angle' have no obvious geometric meaning otherwise. Every inner product must be symmetric (or conjugate-symmetric over ℂ), linear in one argument, and positive-definite, and it automatically satisfies the Cauchy-Schwarz inequality, which makes the notion of 'angle between vectors' well-defined in any such space.

Two vectors are orthogonal if their dot product is zero — the generalization of 'perpendicular' to any number of dimensions. A set of vectors is orthogonal if every pair in the set is orthogonal, and orthonormal if additionally every vector has unit length. Orthogonal sets are enormously convenient in linear algebra because they behave like independent, non-interfering coordinate axes: computing a vector's coordinates in an orthonormal basis reduces to simple dot products instead of solving a linear system.

A norm is a function that assigns a nonnegative 'length' or 'size' to every vector, generalizing the everyday notion of distance. The most familiar is the Euclidean norm (2-norm), the straight-line length given by the Pythagorean theorem, but other useful norms exist, such as the 1-norm (sum of absolute values) and the infinity-norm (largest absolute entry). Every norm must satisfy three axioms: it's zero only for the zero vector, it scales linearly with scalar multiplication, and it obeys the triangle inequality. Norms let us measure error, define convergence, and compare vectors of any dimension.

A vector is a quantity with both magnitude and direction, represented as an arrow in space or, algebraically, as an ordered list of components. Vectors let us describe displacement, velocity, and force in a way that plain numbers (scalars) cannot — knowing a car moves at '60' isn't enough; you need '60 mph north'.

A matrix is a rectangular array of numbers arranged in rows and columns. Matrices are the workhorse data structure of linear algebra: they encode systems of linear equations, represent linear transformations between vector spaces, and store data (images, graphs, networks) in a form that algebraic operations can act on directly. Every matrix has a shape — m rows by n columns, written m × n — and the entry in row i, column j is written aᵢⱼ.

Matrix multiplication is a binary operation that produces a new matrix from two matrices. For the product AB to be defined, the number of columns of A must equal the number of rows of B: an m×n matrix times an n×p matrix yields an m×p matrix. Unlike ordinary number multiplication, matrix multiplication is not commutative in general — AB ≠ BA — because it represents the composition of two linear transformations, and the order in which transformations are applied matters.

The determinant is a scalar-valued function of a square matrix's entries, denoted det(A), det A, or |A|. It characterizes key properties of the matrix and the linear map it represents: the determinant is nonzero exactly when the matrix is invertible (equivalently, when the linear map is an isomorphism). A matrix with determinant zero is called singular and has no inverse. Geometrically, |det(A)| is the factor by which A scales area (2×2) or volume (3×3 and higher).

A linear transformation (or linear map) is a function between two vector spaces that respects vector addition and scalar multiplication. Every linear transformation between finite-dimensional spaces can be represented by a matrix, and applying the transformation to a vector is exactly matrix-vector multiplication — this correspondence is what makes matrices useful for more than just bookkeeping systems of equations.