Support vector machines
Era 3 · From the expert boom to the statistical turn (1980–2011) › Statistical learning and probabilistic models
- Lineage
- Data and learning
- Kind
- Publication
Who
AT&T Bell Labs
What happened
Corinna Cortes and Vladimir Vapnik of AT&T's laboratories publish "Support-Vector Networks" in the journal Machine Learning.
Why it mattered
A classifier that looks for the boundary with the widest possible margin between classes and, with the kernel trick, separates data that are not linearly separable. It was the standard method of machine learning until deep networks took off.
The full story
Since the 1960s, in the Soviet Union and with Alexey Chervonenkis, Vapnik had been developing a statistical theory of learning: when a model that gets the training data right will also get new data right. A practical idea came out of that theory. Among all the boundaries that separate two classes, you should choose the one that leaves the widest margin to the nearest points on each side, because that is the one that generalizes best. The points that hold up the boundary are the support vectors.
In 1992, with Bernhard Boser and Isabelle Guyon, Vapnik added the kernel trick: projecting the data into a space with many more dimensions, where they can be separated by a plane, without ever computing the projection, only dot products. The 1995 paper by Cortes and Vapnik completed the method with the soft margin, which tolerates some errors, essential with real data that never separate cleanly.
The result was a method with a solid mathematical foundation, trained by solving an optimization problem with no local minima, that matched or beat the neural networks of the day on problems like handwritten digit recognition.
For a decade, support vector machines and kernel methods were one of the main currents of machine learning, while neural networks stayed in the background.
Two groups of points, circles on the left and squares on the right, separated by a diagonal line. Two parallel dashed lines mark the margin on each side. Two circles and two squares, highlighted, touch those lines: they are the support vectors.
Under the hood
With examples xᵢ and labels yᵢ = ±1, the boundary is the hyperplane w·x + b = 0. If every point is required to satisfy yᵢ·(w·xᵢ + b) ≥ 1, the margin measures 2/‖w‖, so maximizing it amounts to minimizing ½‖w‖². It is a convex quadratic problem: it has a single solution and no local minima to get stuck in.
The solution depends only on the points that touch the margin, the support vectors, and the data appear only in dot products xᵢ·xⱼ. The kernel trick replaces that product with a function K(xᵢ, xⱼ), such as the Gaussian exp(−‖xᵢ − xⱼ‖² / 2σ²), which is equivalent to a dot product in a space with many more dimensions, even infinitely many, without ever computing it.
The soft margin of 1995 adds slack variables ξᵢ ≥ 0 and minimizes ½‖w‖² + C·Σ ξᵢ. The parameter C decides how heavily each error is penalized against the width of the margin.
Side by side
| Aspect | Support vector machines | Neural network |
|---|---|---|
| Training | A convex problem, with a single solution | Gradient descent, with local minima |
| What is chosen by hand | The kernel and the parameter C | The architecture, the learning rate, and when to stop |
| Features | Fixed by the kernel | Learned in the hidden layers |
| With lots of data | Cost grows at least with the square of the number of examples | Each pass through the data grows in proportion to the number of examples |
Builds on
Vapnik-Chervonenkis theory (1968)
Documented direct influence
Support vector machines rest on Vapnik and Chervonenkis's statistical learning theory
Sources
Original source
- Support-vector networks. Machine Learning, 20(3), pp. 273-297 (opens in a new tab). Corinna Cortes and Vladimir Vapnik, 1995
Further reading
How to cite
“Support vector machines.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/support-vector-machines
This entry was last reviewed on November 13, 2026.