Minsky vs. Rosenblatt

Mikel Olazaran’s (1993) “A sociological history of the neural network controversy” is an in-depth presentation from the Minsky-Rosenblatt years through the 1980s. Very good at correcting the “pop” history presented in many other places.

“Rosenblatt’s contributions” from a 2011 workshop at PACE university. It has an intriguing mention of conference debates between Minsky and Rosenblatt, but so far I haven’t been able to find any other written record of them, or talk to witnesses (Jan. 11th 2016)

“During the late 1950s and early 1960s, much to the enjoyment of those in the audience, Rosenblatt and Minsky debated on the floors of scientific conferences the value of biologically inspired computation, Rosenblatt arguing that his neural networks could do almost anything and Minsky countering that they could do little.”

From a colleague of Rosenblatt’s who prefers to remain anonymous (a recent paper on CNNs vs. RNNs for machine translation; see Rosenblatt 1962 and especially 1967 on the C-system):

If you are a good mathematician you may be able to figure out whether his C-system was a precursor to deep learning with convolutional networks.

Web version of The Quest for Artificial Intelligence by Nils Nilsson, which nicely covers Minsky and Rosenblatt (as well as a lot of other relevant AI material).

A very nice presentation of the relationship between the perceptron update rule, the delta rule, and gradient descent, with code.

Rosenblatt (1964) “Analytic Techniques for the Study of Neural Nets” – a concise, fascinating, rarely cited overview of his research.

Rosenblatt (1967) “Recent Work on Theoretical Models of Biological Memory”. A later piece that speculates on links between his modeling work and memory transfer experiments, and seems to be his last published work with perceptrons: an elaboration of the C-system that retains a memory trace through time.

Minsky 1990 / 1991 “Logical vs. Analogical or Symbolic vs. Connectionist or Neat vs. Scruffy” (thanks to Aaron Schein for the reference). Neat vs. scruffy also comes up in this 1994 paper by Wendy Lehnert, that questions the usefulness of Chomskyan linguistics for Natural Language Processing (thanks to Brendan O’Connor).

Cornell’s 1971 memorial for Rosenblatt

NY Times obituary for Minsky.

This Wired piece on Andrew Ng and Deep Learning sets up a Minskyan view of specialized learners vs. a Deep Learning (AKA Rosenblattian) view of a general purpose neural network. It’s otherwise generally ahistorical (i.e. Deep Learning is new, except that they do mention Hinton as its “godfather”).

Gary Marcus (2012) presents a one-sided history of the Rosenblatt-Minsky debate in this New Yorker piece entitled “Is “Deep Learning” a Revolution in Artificial Intelligence”? (It also has a one-sided portrayal of the past-tense debate). There’s a link to a 1958 article in the New Yorker on Rosenblatt (behind a paywall).

This piece from the MIT Technology Review has a discussion of Rosenblatt’s influence on the Deep Learning movement (see also this blog post).

The amazing McCulloch and Pitts story (a kind of Good Will Hunting thing – thanks to Rob Malouf).

Nagy (1991) “Neural networks then and now” – a very useful technical summary of the work by Rosenblatt and colleagues. Nagy’s (1963) description of the Tobermory perceptron, a hardware implementation of a multi-layer perceptron speech recognition system, can be found here. Nagy (1968) on the state of the art in pattern recognition.

This Microsoft overview gives a good recent (up to 2011) history of NNs in speech recognition. Here‘s a 2013 paper on speech recognition with deep learning and RNNs, and here‘s a 2012 state of the art by Hinton et al. Grossberg’s history of RNNs is here (see also the links at the beginning of his article). “Deep Speech 2” on end-to-end speech recognition with NN’s.

On the Origin of Deep Learning (2017): leaves out Rosenblatt (1962) seems to perpetuate the myth that he worked only with single-layer perceptrons.

Perceptron (but not Rosenblatt) makes Rolling Stone (March 10, 2016)

In 1958, when the “perceptron”, the first so-called neural-network system, was introduced, a newspaper suggested it might soon lead to “thinking machines” that could reproduce consciousness. (p. 48)

LeCun was a pioneer in deep learning, a kind of machine learning that revolutionized AI. While he was working on his undergraduate degree in 1980, he read about the 1958 “perceptron” and the promise of neural-network algorithms that allow machines to “perceive” things such as images or words. (p. 50)

A 1958 conference paper by Rosenblatt, with commentary and replies.

Rumelhart and Zipser (1986) have a very useful discussion of Minsky and Rosenblatt

Fusion.net article on Rosenblatt and Deep Learning

Here are the introductory pages of the 1961 Tech Report version of Rosenblatt’s Principles of Neurodynamics, including the TOC. The preface gives a nice glimpse of Rosenblatt’s view of the stormy early years (see e.g. p. 15 of the cited Minsky paper), and clearly expresses his endebtedness to his collaborators in getting the math right. His PhD student George Nagy (who generously gave me his copy of the book) made this related comment (p.c. March 28, 2016): “Rosenblatt was an intuitive mathematician who did difficult and dirty mathematics.  I was glad to have Dave Block on my committee.”

Block’s 1962 perceptron papers (single-layer overview and proof, multilayer analysis) are in this volume of the Review of Modern Physics. His 1970 review of Minsky and Papert is here.

Rosenblatt went on to conduct biological memory transfer experiments. A good overview research on memory transfer, which cites a 1970 paper of Rosenblatt’s, can be found here. See also the site here for further evidence of the mainstream nature of this work in the mid-60’s.

This 1989 paper in Byte by Touretzky and Pomerleau gives an accessible introduction to the analysis of hidden layers, starting with X-OR. Here is a more recent Scientific American paper on the same subject, focusing on Tensor Flow’s visualization too.

A 2014 Frontiers collection: 50 years after the perceptron, 25 years after PDP: Neural computation in language sciences.

A Gallistel blog post: http://nautil.us/blog/stop-saying-the-brain-learns-by-rewiring-itself

Karpathy on RNNs showing how far one can get with character prediction

Deep learning in cognitive science

Recent simulated annealing for training MLPs

From Rosenblatt’s Principles of Neurodynamics (1961/1962). The Mark I is in the Smithsonian, apparently on discplay.

Rosenblatt, with the image sensor of the Mark I Perceptron (Source: Arvin Calspan Advanced Technology Center; Hecht-Nielsen, R. Neurocomputing (Reading, Mass.: Addison-Wesley, 1990).)

The Mark 1 Perceptron (Source: Arvin Calspan Advanced Technology Center; Hecht-Nielsen, R. Neurocomputing (Reading, Mass.: Addison-Wesley, 1990).)