The History of AI

Support vector machines

Era 3 · From the expert boom to the statistical turn (1980–2011) › Statistical learning and probabilistic models

Lineage
Data and learning
Kind
Publication

Who

AT&T Bell Labs

What happened

Corinna Cortes and Vladimir Vapnik of AT&T's laboratories publish "Support-Vector Networks" in the journal Machine Learning.

Why it mattered

A classifier that looks for the boundary with the widest possible margin between classes and, with the kernel trick, separates data that are not linearly separable. It was the standard method of machine learning until deep networks took off.

The full story

Since the 1960s, in the Soviet Union and with Alexey Chervonenkis, Vapnik had been developing a statistical theory of learning: when a model that gets the training data right will also get new data right. A practical idea came out of that theory. Among all the boundaries that separate two classes, you should choose the one that leaves the widest margin to the nearest points on each side, because that is the one that generalizes best. The points that hold up the boundary are the support vectors.

In 1992, with Bernhard Boser and Isabelle Guyon, Vapnik added the kernel trick: projecting the data into a space with many more dimensions, where they can be separated by a plane, without ever computing the projection, only dot products. The 1995 paper by Cortes and Vapnik completed the method with the soft margin, which tolerates some errors, essential with real data that never separate cleanly.

The result was a method with a solid mathematical foundation, trained by solving an optimization problem with no local minima, that matched or beat the neural networks of the day on problems like handwritten digit recognition.

For a decade, support vector machines and kernel methods were one of the main currents of machine learning, while neural networks stayed in the background.

Two groups of points, circles on the left and squares on the right, separated by a diagonal line. Two parallel dashed lines mark the margin on each side. Two circles and two squares, highlighted, touch those lines: they are the support vectors.

Among all possible boundaries, the machine chooses the one with the widest margin; only the points that touch the margin, the support vectors, determine it.

Under the hood

With examples xᵢ and labels yᵢ = ±1, the boundary is the hyperplane w·x + b = 0. If every point is required to satisfy yᵢ·(w·xᵢ + b) ≥ 1, the margin measures 2/‖w‖, so maximizing it amounts to minimizing ½‖w‖². It is a convex quadratic problem: it has a single solution and no local minima to get stuck in.

The solution depends only on the points that touch the margin, the support vectors, and the data appear only in dot products xᵢ·xⱼ. The kernel trick replaces that product with a function K(xᵢ, xⱼ), such as the Gaussian exp(−‖xᵢ − xⱼ‖² / 2σ²), which is equivalent to a dot product in a space with many more dimensions, even infinitely many, without ever computing it.

The soft margin of 1995 adds slack variables ξᵢ ≥ 0 and minimizes ½‖w‖² + C·Σ ξᵢ. The parameter C decides how heavily each error is penalized against the width of the margin.

Side by side

Support vector machines and neural networks in the 1990s
AspectSupport vector machinesNeural network
TrainingA convex problem, with a single solutionGradient descent, with local minima
What is chosen by handThe kernel and the parameter CThe architecture, the learning rate, and when to stop
FeaturesFixed by the kernelLearned in the hidden layers
With lots of dataCost grows at least with the square of the number of examplesEach pass through the data grows in proportion to the number of examples

Builds on

  • Vapnik-Chervonenkis theory (1968)

    Documented direct influence

    Support vector machines rest on Vapnik and Chervonenkis's statistical learning theory

Sources

Original source

How to cite

“Support vector machines.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/support-vector-machines

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline