The roots of the current deep learning boom go surprisingly far back, to the 1950s. While vague ideas of “intelligent machines” can be found further back in fiction and speculation, the 1950s and ’60s saw the introduction of the first “artificial neural net‐ works,” based on a dramatically simplified model of biological neurons. Amongst these models, the Perceptron system articulated by Frank Rosenblatt garnered partic‐ ular interest (and hype). Connected to a simple “camera” circuit, it could learn to dis‐ tinguish different types of objects. Although the first version ran as software on an IBM computer, subsequent versions were done in pure hardware.
image by Uriel SC on unsplash
The increase in computing power together with the development of the backpropagation technique (known in various forms since the ’60s, but not applied in general until the ’80s) prompted a resurgence of interest in neural networks. Not only did computers have the power to train larger networks, but we also had the techni‐ ques to train deeper networks efficiently. The first convolutional neural networks combined these insights with a model of visual recognition from mammalian brains, yielding for the first time networks that could efficiently recognize complex images such as handwritten digits and faces. Convolutional networks do this by applying the same “subnetwork” to different locations of the image and aggregating the results of these into higher-level features. In the ’90s and early 2000s interest in neural networks declined again as more “understandable” models like support vector machines (SVMs) and decision trees became popular. SVMs proved to be excellent classifiers for many data sources of the time, especially when coupled with human-engineered features. In computer vision, “feature engineering” became popular. This involves building feature detectors for small elements in a picture and combining them by hand into something that recog‐ nizes more complex forms. It later turned out that deep learning nets learn to recog‐ nize very similar features and learn to combine them in a very similar way.