Winter and Quiet Sprouts

The worst-case scenario of “The AI Winter” drawn in a 1984 discussion ── checking the answers afterward

At the conference of the American Association for Artificial Intelligence (AAAI), a panel discussion was held titled “The ‘Dark Ages’ of AI – Can We Avoid Them or Survive Them?” The moderator, Drew McDermott, said that behind the bustle in industry, there was deep unease among people who had researched for many years, from a worry that “perhaps expectations about AI are too high ... will eventually result in disaster”. As the worst-case scenario he drew this picture: in five years America's “strategic computing initiative” fails, Japan's Fifth Generation also hits a dead end, start-up companies go under, funding for AI stops, and everybody changes the names of their research to something else. He introduced the fact that some people call this state “The AI Winter.” But he also said he thought it unlikely. In the discussion, another researcher (Roger Schank), when asked whether he thought a catastrophe would come, said that he told them it would. On the other hand, the view “Will there be an AI Winter? I think there may be an AI dusk” was also put forward (the same person went on to say that the bubble might not burst but shrink a little).

What happened next:Afterward, development of the autonomous land vehicle (ALV), one of the plan's showpieces, was cut off in April 1988. The head of the responsible division, who took up the post in 1987, also cut off parts of the AI program. The plan disappeared from DARPA's budget in 1993. Membership in the American Association for Artificial Intelligence peaked at 16,421 in 1987 and settled at 4,000–5,000 by 1996. Many of the AI companies developing expert systems disappeared, were acquired, or changed their business. Japan's Fifth Generation Computer Project ended its main stage in fiscal 1992. Nilsson writes that these events piled up in the mid-to-late 1980s and came to be called “The AI Winter.”

What we still don’t know:Whether one of the scenarios, “Everybody hurriedly changes the names of their research projects to something else”, really happened in the 1990s has not been confirmed in original sources (a candidate dealt with in u08)

1984: “AI Winter” was spoken of as the name of the worst thing that might come

At a panel discussion at AAAI-84, the moderator, Drew McDermott, said that researchers were worried that expectations for AI were too high and that it would someday end in trouble. As the worst-case scenario he gave this picture: in five years the strategic computing initiative fails miserably, Japan's Fifth Generation also does not work out, the Japanese government quickly pulls out of computers, the start-up companies all go bankrupt, and no money goes to anything related to AI. He described this as “This condition, called the ‘AI Winter’ by some”. He urged them to warn themselves so that it would not come to that, and to tell the world. In other words, “The AI Winter” was a phrase that appeared in the middle of the second boom as a warning about the future. Calling the decline in research funding in the 1970s the “first AI winter” is a label added afterward.

What we still don’t know:Who first said “AI Winter” is not known (even in the discussion it was “by some”). What book the small value for 1983 in the Ngram comes from has not been confirmed.

Companies making AI-specific “Lisp machines” lose to ordinary workstations (late 1980s)

From around 1980, companies such as Symbolics built and sold “Lisp machines,” special computers that run Lisp, a programming language often used in AI research, fast. According to the recollection of one of Symbolics' founders, new “workstations” from Sun, Apollo and others appeared, and even ordinary computers could run Lisp quite well. So the advantage of the dedicated machines suddenly became small. He also writes that most of the sales depended on expectations for expert systems (AI into which experts' knowledge is written), and that the backlash from those expectations swelling too far came around 1988 and became “The AI Winter.” The company repeatedly cut staff. According to a student paper (2001) written for an MIT course, Symbolics' product sales fell to $101.6 million, $82.1 million and $55.6 million in fiscal 1986, 1987 and 1988. In Japan too, the 1991 white paper wrote that the machines used for AI had come to center not on dedicated machines but on ordinary computers, especially personal computers.

What we still don’t know:We have not opened primary company documents such as the annual reports of Symbolics or LMI. We have not obtained MIT's 1999 bachelor's thesis “If It Works, It’s Not AI” because a human-verification screen appeared on MIT's site.

1991, a government-supervised white paper: Japan “did not enter an AI ‘winter era’ like the one seen earlier in the US” (translated)

The “AI White Paper 1991” wrote that Japan did not enter an AI “winter” (translated) like the one seen earlier in the United States. As the reason, it suggests that users may have found for themselves the uses in which AI is effective (it also writes that the United States had already been through its “winter”). In the white paper's survey (December 1989 to January 1990, sent to 7,084 establishments, with 1,844 replying), 37.8% of the responding establishments (26.2% of the establishments that use computers, 379 sites) had introduced AI. The white paper reported this as 17.2 points more than the 20.6% of the 1988 survey. Of the adopters, 46.2% used expert systems (mechanisms in which experts' knowledge is written) and 20.6% used machine translation.

What we still don’t know:This white paper cannot tell how AI use in Japan changed after 1991 (when winter came, or whether it came at all). The survey results are also shares among only the establishments that replied (26% of those sent), not shares for Japan as a whole.

Membership of the Japanese Society for Artificial Intelligence (1986–2003): regular members peaked in 1993 and supporting members in 1991–92. Supporting members fell to under a third by 2003. Student members kept rising until 2002

Lining up the membership figures given in the society's annual general-meeting reports shows the following. Regular members: about 600 at the founding in July 1986 → 2,554 in 1988 → 3,625 in 1990 → 3,893 in 1993 (the most) → 3,269 in 1995 → 3,044 in 1999 → 2,789 in 2003. Supporting members, the companies and other organizations that support the society: about 30 companies → 100 companies in 1988 → 155 companies in 1991 and 1992 (the most) → 96 in 1995 → 65 companies in 1999 → 49 companies in 2003, a fall to less than a third of the peak. Student members, on the other hand, rose and fell but kept an upward trend, from 266 in 1988 to 539 in 2002. The 2003 report wrote as follows. The number of members had been falling little by little for several years, but this year there was a large drop of about 4%. Not only did regular members fall because companies shrank their research and development departments, but student members, who had kept an upward trend until then, also began to fall.

What we still don’t know:As to why members fell, the society's report only writes “shrinking of companies' research and development departments” (translated) in 2003 and gives no reasons for earlier years. How much this has to do with the recollection that “the word AI was disliked” (translated) cannot be told from these numbers (see record → The source of the story that Artificial…).

1990: a washing machine with fuzzy control goes on sale, and “fuzzy” wins the gold prize in the new-word category of the New Words and Buzzwords Awards

On February 1, 1990, Matsushita Electric released the fully automatic washing machine “Aisaigo Day Fuzzy”. It uses an optical sensor to detect how cloudy the wash water is, and decides the washing time with six rules such as “if the saturation time is long and the transmittance is low, make the washing time very long” (translated) (the saturation time is the time until the cloudiness settles, and the transmittance is how easily light passes through). These rules were created from data collected in homes and from engineers' know-how. In the same year's “New Words and Buzzwords Awards”, “fuzzy” was chosen for the gold prize in the new-word category. The organizer explains that this washing machine was the first to apply fuzzy theory to a home appliance and the spark of the boom, creating a large fuzzy market with many companies jostling in.

What we still don’t know:We have not checked how well fuzzy appliances sold afterward (sales numbers and so on)

Laboratory for International Fuzzy Engineering Research (LIFE): 48 companies gather and, with government backing, study technology for handling “vagueness” (translated) for six years (1989–1995)

To apply fuzzy theory, which processes information that contains “vagueness” (translated), to engineering, the technology research association “Laboratory for International Fuzzy Engineering Research (LIFE)” was founded in 1989. Not only computer companies but 48 companies (49 at the end of fiscal 1989), in fields such as automobiles, steel and securities, took part. The research period was about six years, and the research and development cost was expected to be about 5 billion yen (as of the 1991 white paper). The laboratory had three research rooms, studying fuzzy control, fuzzy intelligent information processing and fuzzy computers. According to a 2005 look back, a total of 66 researchers were on staff at the laboratory near Osanbashi pier in Yokohama, and it was active until 1995. Toshiro Terano served as director for all six years.

What we still don’t know:We have not checked the total cost actually spent (the 1991 white paper only writes “the total research and development cost is expected to be about 5 billion yen at present” (translated))

Sendai's subway opens, using “predictive fuzzy control” (translated) (AI with rules that handle vagueness) for automatic train operation

The Sendai city subway (the ninth city subway in Japan) opened on July 15, 1987. The automatic train operation system developed by Hitachi adopted “predictive fuzzy control” (translated). It used AI (knowledge engineering) rules written in “IF–THEN form” (translated) (the form 'if so-and-so, then do so-and-so') to replace the skills of expert drivers. The article says that better ride comfort, better stopping accuracy and energy saving were achieved, and introduces this predictive fuzzy control as “the first in the world” (translated) (the words of the article by the company that made it).

What we still don’t know:“First in the world” (translated) is the words of an article by the company that made it, and has not been confirmed in other sources.

The story of the magazine “Nikkei AI” (1985–1994): the biweekly specialist magazine dropped “AI” from its title in April 1992, became “Nikkei Intelligent Systems”, and ceased publication in October 1994

Nikkei BP began publishing the AI (artificial intelligence) specialist magazine “Nikkei AI” with a “pre-launch special issue no. 1” on September 9, 1985. From issue no. 1 on January 6, 1986, it came out every two weeks (biweekly). From 1990, a quarterly “Nikkei AI Special Edition” also appeared. But after issue 150 (March 23, 1992), the title changed from issue 151 (April 6, 1992) to “Nikkei Intelligent Systems,” and the letters “AI” disappeared from the title. It ceased publication with issue 211 (October 3, 1994) (the special edition also changed in 1992 to “Nikkei Intelligent Systems Special Edition” and ceased publication with the autumn 1992 issue). It is a record of a magazine that was born in the middle of the second boom, changed its title in the early 1990s, and ceased publication.

What we still don’t know:We have not checked why the title was changed to “Intelligent Systems”, why the magazine ceased publication, or how the circulation changed (we have not opened the magazine itself or announcements from the time).

The source of the story that “Artificial Intelligence” was disliked in the 1990s–2000s: a recollection in the 2023 society journal (even renaming the society was discussed), and “The AI Winter” and “the winter era of artificial intelligence” in the society journal of 1996

In 2023, the Journal of the Japanese Society for Artificial Intelligence depicted the history of AI in a series of cover paintings, and the third one took up “the second winter” (translated) (around 1992 to around 2010). In the commentary, Hitoshi Matsubara recalled that there was big news at the time, such as Deep Blue, but the word AI was disliked and it was not seen as part of the AI category. The society too lost members and corporate members one after another and nearly collapsed, and changing its name was even considered. Tadahiro Taniguchi wrote that it was an age when, even if you wanted to research “Artificial Intelligence,” saying so made people take it the wrong way, and that in the 2000s the wind was strong against neural networks. In a roundtable in the November issue the same year, Matsubara said there was also talk that writing “Artificial Intelligence” made it hard to pass the review for national research funding (Kakenhi). On the other hand, in the explanation by the creator of the cover paintings in the same May issue, the reason researchers in this period began to name specific fields rather than “Artificial Intelligence” is explained as “the expansion of the research field itself” (translated). Going back to the May 1996 issue of the same journal (the 10th-anniversary issue), Hiroaki Kitano wrote that the phrases “The AI Winter” and “AI was a failure” (translated) were spreading in some news reports (though he also wrote that the results of expert systems and others had spread widely). On that basis he aimed at “the restoration of AI” (translated). Mikio Shoji, who belonged to a company, also wrote that he sometimes heard the phrase “the winter era of artificial intelligence” (translated). In the roundtable of the same issue, an opinion was also offered that it was fine if new technologies were useful even if they were not called AI.

What we still don’t know:No source has been found that counted whether grant applications that wrote “Artificial Intelligence” were really harder to pass. A record from the time of when and where the society's rename was discussed is also unconfirmed.

The national project after the Fifth Generation: “Real World Computing (RWC)” (1992–2002), moving from symbols and logic to “patterns” and “learning” such as images and sound

In 1992 the Ministry of International Trade and Industry (MITI) launched a 10-year national project, “Real World Computing (RWC)”, as the successor to the Fifth Generation computer project. According to Nobuyuki Otsu, who led the plan, the Fifth Generation pursued the “logical side (symbol processing)” (translated) of human thought, while RWC aimed at the “intuitive aspect (pattern processing)” (translated). In other words, it was technology for handling vague information such as images, sound and everyday language flexibly, through learning (a machine adjusting its own workings from examples) and self-organization. A 1994 explanatory article introduced it as a plan of “about 70 billion yen over the next 10 years” (translated) and gave household robots and electronic librarians as examples. In July 1992 a research partnership (RWCP) was founded, and besides major Japanese companies, research institutions from Germany, the Netherlands, Sweden and Singapore took part. The first five years (1992–1996) explored widely, and the last five years (1997–2001) narrowed the work to two themes, “real-world intelligence” and “parallel and distributed computing”. The 2002 summary gives the budget as about 60 billion yen over 10 years, and lists as results a sign-language recognition system, software that cross-searches sound, video and text without an index, and a music database free for researchers to use. It also records that about 20% (14 items) of the results were adopted in products or practical work even during the project.

What we still don’t know:The sources we opened do not say how RWC's research on pattern processing and machine learning connected, or did not connect, to the later third boom (the deep-learning AI) (the learning methods named in the 2002 summary include multilayer neural networks, Bayesian estimation and genetic algorithms).

Two years after the Fifth Generation: 100 programs released free of charge, with UNIX support (1993–1995)

The main Fifth Generation project ended in March 1993, and in April a successor, the “Research Infrastructure Project”, began. Software that ran only on the parallel inference machines was ported, using a mechanism that translates KL1 into the C language (KLIC) and other means, so that it would also run on the UNIX parallel computers then appearing on the market. The software developed was released free of charge as “ICOT Free Software”, growing from 71 programs in June 1992 to a total of 100 by the end of March 1995. The software, whose copyright belongs to ICOT rather than the government, was released with restrictions that would hinder research removed. About 2.8 billion yen was spent on this successor plan, and all government-funded research and development was completed in March 1995.

What we still don’t know:This record does not confirm how much the 100 released programs were used afterward (for example, download counts); Figure 5 on p.126 of the MITI Research Review shows access counts up to February 1995.

Japan Electronic Dictionary Research Institute (EDR): building by hand, over 9 years, a huge dictionary that lets computers handle the meaning of Japanese and English (1986–1995)

As a foundation for getting computers to understand natural language, a company to develop a large-scale, high-quality “electronic dictionary”, the “Japan Electronic Dictionary Research Institute (EDR)”, was established, and the project began in April 1986. The Japan Key Technology Center invested in it (AI White Paper 1991), and eight computer companies also put in funds (EDR's 1996 paper). The institute made a “word dictionary” describing the grammar and meaning of words, a “concept dictionary” organizing the relations among concepts, a “co-occurrence dictionary” recording how words go together, and a “bilingual dictionary” showing the correspondences between Japanese and English, among others. The dictionaries now introduced on the page of the National Institute of Information and Communications Technology (NICT) are of this size: about 270,000 Japanese words, about 190,000 English words, about 410,000 concepts, about 200,000 Japanese example sentences and so on. The final evaluation of the Fifth Generation project wrote that collecting the vast knowledge that lies outside of words has limits for ICOT alone, and expected much from EDR's dictionaries.

What we still don’t know:The total cost of the project is not stated in the sources we looked at.

Government spending on the Fifth Generation: about 54.1 billion yen over the main 11 years, about 56.9 billion yen with the later 2 years added

According to a table compiled in 1995 by the Ministry of International Trade and Industry's research institute (the MITI Research Institute), government spending on the Fifth Generation Computer project was about 54.1 billion yen over the 11 fiscal years 1982–1992 (initial stage 8.3 billion yen, middle stage 21.6 billion yen, final stage 24.2 billion yen). After that, about 2.8 billion yen was spent over the two fiscal years 1993–1994 on a finishing plan (a plan to make what had been built usable on ordinary computers), bringing the total to about 56.9 billion yen. The commonly seen figures of “about 54 billion yen” and “57 billion yen” differ only in whether these last two years are counted.

What we still don’t know:The table covers only the money paid by the government (MITI). Funds and staff that participating companies bore on their own are not in this table (as far as can be read from the table's notes)

1986: ATR Interpreting Telephony Research Laboratories, research on a telephone that automatically translates speech into the other person's language. In January 1993, an experiment linking Japan, the US and Germany (vocabulary of about 1,500 words)

An “automatic interpreting telephone”, which automatically translates the speech on a phone call into the other person's language, is built by combining speech recognition (listening to a voice and turning it into text), machine translation and speech synthesis (a computer producing a voice). In April 1986, the “ATR Interpreting Telephony Research Laboratories” was set up inside ATR (Advanced Telecommunications Research Institute International). Basic research to check the component technologies began, and in 1988 it had 47 researchers, including visiting researchers from abroad. For translating spoken language, the laboratories also built a large labeled speech database and a database holding a large collection of examples. As a research result it built an experimental speech translation system, ASURA, and on January 28, 1993 it held an international joint experiment on automatic interpreting telephony connecting three countries, Japan, the United States and Germany. ASURA translated sentences spoken with a short pause after each phrase, with a vocabulary of about 1,500 words (the earlier version, SL-TRANS, had under 400 words). The research ended in March 1993 and was taken over by ATR Interpreting Telecommunications Research Laboratories.

What we still don’t know:The source and amount of ATR's research funding are not given in the sources we opened (this site has not checked). We have also not checked how accurately the 1993 international experiment translated, in the first page we opened.

The physicist Hopfield presents a neural network that recalls the whole from a part

The physicist John Hopfield presented a mechanism in which many simple “neurons” (imitations of nerve cells) are all connected to one another. Just by being shown part of something it has memorized, it can restore the whole (associative memory). He explained it with an idea from the physics of magnets, in which the whole system settles into a place of low energy, the paper's “a least (local) E”.

What we still don’t know:This paper does not confirm whether the real brain really recalls memories by this mechanism (the paper's model is based on part of how nerves work and was studied with ideas from physics).

The Boltzmann machine: a network that switches units on and off by chance, and a learning rule for hidden units (1985)

A Boltzmann machine is a network of neurons (units), each in one of two states, on or off, connected pairwise by weights that are equally strong in both directions. Each neuron chooses on or off with a probability set by the states of the neurons around it and the weights. When the temperature (a number that sets how random the choices are) is lowered gradually from a high value, the whole network tends to settle into a low-energy state (the same idea as annealing in physics). The 1985 paper by Ackley, Hinton and Sejnowski gave a rule for training a network that has hidden units, which are not set from outside, as well as visible units, which are set from outside, so that it imitates how the data appears. In the rule, the probability that two connected neurons are on at the same time is compared between two cases: when data is shown and the visible units are held fixed, and when nothing is shown and the network runs freely. The weight is adjusted in proportion to the difference. How to adjust it depends only on the states of those two neurons.

What we still don’t know:We have not checked which of the two 1983 conference papers came out first (the AAAI-83 paper in which the names appear, and Hinton & Sejnowski 1983, which the 1985 paper cites). It is also unknown how far the Boltzmann machine itself was used in practical work in the 1980s.

The restricted Boltzmann machine (RBM) comes from Smolensky's harmonium of 1986: a Boltzmann machine with connections only between two layers

A restricted Boltzmann machine (RBM) divides the neurons of a Boltzmann machine into two layers (a visible layer and a hidden layer) and limits the connections to those between the layers. Because there are no connections within a layer, once the values of one layer are fixed, all the neurons of the other layer can be computed at the same time. It originated as the harmonium proposed by P. Smolensky in Chapter 6 of Volume 1 of the 1986 book Parallel Distributed Processing. It put pieces of knowledge (knowledge atoms) in the upper layer and features of things (representational features) in the lower layer, and updated the whole upper layer and the whole lower layer in turn. The chapter says the name followed pandemonium, the pattern-recognition mechanism of Selfridge and Neisser (1960). In 2002 Hinton devised a fast learning method called contrastive divergence, and in the 2000s it became an important tool for training deep networks one layer at a time (explanation by the Nobel committee).

What we still don’t know:We have not checked who first used the name restricted Boltzmann machine (Hinton 2002 uses the name in discussing a 1992 work by Freund and Haussler). Smolensky's affiliation at the time of the 1986 chapter is not in the chapter's PDF, so we do not state it.

Shun-ichi Amari's “A Theory of Adaptive Pattern Classifiers” (a learning theory that updates weights bit by bit from errors)

Shun-ichi Amari mathematically analyzed a learning method that, each time it sees one data point, updates the weights (how much importance to give each feature) a little according to the error. The abstract of the English paper describes a proof that the weights approach the optimum even for data that cannot be classified cleanly, the trade-off between learning speed and accuracy, and an extension to piecewise-linear (broken-line-like) discrimination. A 1968 Japanese commentary calls this learning method the “stochastic descent method” (translated). Schmidhuber (2022) writes that it is probably the first paper to train a multilayer neural network by stochastic gradient descent (the basic learning method of today's deep learning).

What we still don’t know:How much a 5-layer network could actually learn (the 1968 book and H. Saito's master's thesis) is not known, because the original sources have not been opened.

A paper on training multilayer neural networks with backpropagation appears in Nature

Rumelhart, Hinton and Williams announced a way of training neural networks that have “hidden layers” between input and output. The difference between the output and the correct answer (the error) is passed back from the output side to the input side step by step, and the strengths of the connections are updated bit by bit (passing the errors backward, as in the paper's title “Learning Representations by Back-Propagating Errors”, is backpropagation). They showed by experiments that the hidden layers come to represent features that nobody taught them.

What we still don’t know:People could read what is represented inside the hidden layers in this paper's small example (a family tree). But in large networks, only parts of it are understood even now (this connects to the unit on the fourth boom).

NETtalk: a neural network that “learned by practice” to read English sentences aloud

Sejnowski and Rosenberg developed a neural network (NETtalk) that learns to convert English letters into pronunciation by looking at examples. When it was trained on 1,024 words transcribed from the conversation of a first-grade child, after about 10 passes its pronunciation was understandable when listened to. Tested on the remaining 439 words it had not learned, it scored 78%. Earlier commercial text-to-speech machines had registered the pronunciation of frequent words in a table and read according to fixed rules.

What we still don’t know:According to the paper, even networks with different connection weights had “similar internal representations of letter-to-sound correspondences” in the hidden layer. It is an early example of trying to look inside a neural network.

1989: the Japanese Neural Network Society is founded, growing out of a small study group started in 1980 with 19 people. At that time the mood at US conferences was “the age of AI (artificial intelligence) is over, now is the age of neural networks” (translated)

The “Japanese Neural Network Society (JNNS)”, a society that studies computational models imitating the brain's neural circuits, held its founding conference on July 21, 1989 at the Gakushi Kaikan in Kanda. The first president was Kunihiko Fukushima. It began in 1980, when Fukushima and others from the NHK research laboratories called on people nationwide to form a small study group that was not a society (the Neuro-Information Science Study Group), and 19 people took part in the first meeting. When a “neuro boom” (translated) arose in the United States in the late 1980s, attendance grew rapidly. In 1987 it passed 100, and because the retreat-style (overnight camp) format could no longer be kept up, the study group was developed into a society. Fukushima recalls that the venue of the 1987 international conference on neural networks in the United States was filled with a hopeful mood of “The age of AI (artificial intelligence) is over; from now on is the age of neural networks” (translated).

What we still don’t know:The date and place of the founding conference come from the recollection. This site has not opened announcements or reports from the time (such as the 1989 JNNS News Letter)

Origins of the autoencoder: in 1985–1987, the task of making a network output what it was given through a narrow middle layer

An autoencoder is a neural network trained to output the same data it receives as input. The number of neurons in the middle layer is kept small. Because the input cannot simply pass through, the middle layer learns on its own a code that represents the input compactly with a few numbers. The first half acts as the encoder, turning the input into a code, and the second half acts as the decoder, restoring the original from the code. This structure appears in the encoder problem of the 1985 Boltzmann machine paper and in a chapter of the 1986 book on backpropagation. In the chapter, a network was trained on a task of reproducing 8 inputs as the same 8 outputs through 3 hidden-layer neurons. The hidden neurons then assigned each of the 8 inputs its own code (not only 0 and 1 but also in-between values were used). In 1987 Ballard proposed a method of stacking autoassociative networks, which make the input and output the same. In this scheme, the upper layer receives the representation of the lower hidden layer and learns from it.

What we still don’t know:We have not confirmed the first paper to use the word autoencoder (the oldest we opened is the 1993 NIPS paper by Hinton and Zemel). The 1991 paper on nonlinear principal component analysis by Kramer is paywalled, so we did not open it. We looked but found nothing conclusive about whether, in Japan around the same time, there was research on networks that output the same thing as their input (called hourglass-type networks, learning of the identity mapping, and so on).

Elman's simple recurrent network (1990) and Jordan's network (1986): feed the previous state back as input to remember the flow of time

A recurrent neural network (RNN) is a mechanism that keeps a memory of the past by returning its output to the input of the next time step. In the form Elman proposed in 1990, context units were added to an ordinary three-layer network. The values of the hidden layer at each time step are copied as they are into the context units (the connection weights are fixed at 1 and not learned), and sent to the hidden layer together with the input at the next time step. When the network was trained by backpropagation to predict the next letter or word, in a string of letters the error was large at the start of a word and became smaller as the word went on, and the boundaries between words emerged from a string with no separators written in. For sequences of words too, when the hidden-layer representations were grouped by similarity, they split into clusters such as nouns and verbs, and living and non-living things. Elman writes that this idea began with Jordan's 1986 report. Jordan's network returns the output to state units and produces a sequence of actions from a fixed plan, and it was applied to the linking of sounds in speech (coarticulation).

What we still don’t know:We do not know the order in which Elman's network and other recurrent networks of the same period (1987–1988: Pineda, Williams and Zipser, and others) were published, because we have not opened the original papers.

Bell Labs' LeCun and colleagues present a neural network that uses “convolution” to read handwritten ZIP codes

LeCun and colleagues trained a neural network to identify digits, using 9,298 digits of handwritten ZIP codes on US mail envelopes (that had passed through the post office in Buffalo, New York). By reusing the same small window (a frame for finding small 5×5 features) across the whole image, they greatly reduced the number of parameters to be learned (the technique now called “convolution”). The error rate on 2,007 test data not used in training was 5.0%.

The source of today's image AI (convolution) is the Neocognitron ── how did its inventor and later papers write about it

Kunihiko Fukushima's Neocognitron (1979) is said to be the source of the “convolution” used in today's image-recognition AI. Sorting through later sources: in a 1984 commentary, Fukushima himself wrote that most of the calculation is the repetition of two-dimensional filters, and that it could be tried out on a minicomputer. LeCun and colleagues' 1989 zip code paper cited Fukushima (1980) in just one place. The 1998 LeNet paper wrote plainly that the combination of convolution and subsampling had been built in Fukushima's Neocognitron, but that at the time there was no method (backpropagation) for training the whole network. In 2022 Fukushima himself wrote “the Neocognitron, published in 1979, is said to be the source of such deep CNNs” (translated), and the 2024 Nobel Committee's explanation also named it as a root of CNNs.

The convolutional neural network LeNet-5 is used to read handwritten characters on checks at banks in the United States

LeCun and colleagues presented “LeNet-5”, a 7-layer convolutional neural network that recognizes handwritten digits, and a check-recognition system. This system was built into check-reading machines made by NCR and was actually used at several banks in the United States. The paper also created “MNIST”, a dataset of handwritten digits with 60,000 training images and 10,000 test images; the error rate on the test data was 0.95% (0.8% when distorted training images were added).

IBM presents a translation method that uses probabilities computed from translated text pairs instead of grammar rules

IBM's Brown, Jelinek and colleagues presented a method that treats translation as a calculation of “the probability that a translator will produce T ... when presented with S” (in this paper S is an English sentence and T a French sentence). Using the records of the Canadian Parliament (kept in both English and French, about 100 million words in computer-readable form), they estimated about 81 million numbers from 40,000 pairs of sentences. When they translated 73 new French sentences into English, 5% matched exactly, but 48% were reasonable translations. The paper writes that Weaver proposed translating with numbers in 1949, but computers were then weak and there was not enough computer-readable text.

Self-driving car races run by the U.S. Department of Defense's DARPA: in 2004 no car finished, and in 2005 Stanford University's Stanley won

DARPA held prize races in which autonomous vehicles drove a desert course. In the first race, on March 13, 2004 (about 142 miles, a $1 million prize), 15 vehicles started but none finished, and the longest distance reached was only 7.5 miles. In the second race, on October 8, 2005 (about 132 miles), 195 teams applied and 5 vehicles finished. Stanford University's Stanley won in 6 hours 53 minutes and took the $2 million prize. According to the Stanley paper, the core technologies were machine learning and probabilistic reasoning, and it learned “drivable” ground from human driving data. The third race, in 2007, was held on an urban course; 6 of 11 teams finished, and a Carnegie Mellon University team won.

Tracing the claim “In the 1990s, researchers avoided the word ‘AI’”: a sentence in the 2005 New York Times wrote only “At its low point”

A New York Times article of October 14, 2005 (by reporter John Markoff) reported the win of Stanford University's car Stanley in the self-driving car race run by DARPA, the research agency of the U.S. Department of Defense, as “the re-emergence of artificial intelligence”. In it is a sentence saying that, when AI was at its lowest point, some computer scientists and software engineers shunned the term “Artificial Intelligence” for fear of being seen as people who talk only of pipe dreams. This is the sentence often quoted in the story that “researchers avoided the word AI”. The same article also writes that the success this time came from a shift in research methods from logic and rules to software that uses probability and statistics.

What we still don’t know:This article does not show how many computer scientists and software engineers actually “avoided the term artificial intelligence”, or when (this site has not found any material that counted them). How the number of uses of the word “artificial intelligence” in paper titles and so on rose and fell year by year can be checked together with the word-count graph of u01.

Mixture of experts (MoE): several expert networks and a gate that decides which one to hand the work to (Jacobs and colleagues 1991, Shazeer and colleagues 2017)

In 1991, Robert Jacobs and Michael Jordan of MIT and Steven Nowlan and Geoffrey Hinton of the University of Toronto published the paper “Adaptive Mixtures of Local Experts”. When one big network is trained on a mix of different tasks, the tasks interfere with each other and learning slows down. For this problem, they showed a method that combines several expert networks with a gate network. For each input, the gate gives the probability of which expert to hand the work to (the output is a softmax: a form of probabilities that add up to 1). They set the way of measuring error so that the experts compete rather than cooperate. So as learning proceeds, each expert takes charge of only a part of the inputs. On a vowel-discrimination task, the paper showed that the work splits into several parts, each handled by a simple expert. In January 2017, Noam Shazeer and others (7 people, including Hinton) of Google's research team Google Brain scaled up this idea and built a layer with up to thousands of experts (the Sparsely-Gated Mixture-of-Experts layer). The gate adds noise, then picks only the top k experts with the highest scores, and does not compute the other experts. So even if the number of parameters is greatly increased, the amount of computation does not increase much. For language modeling and translation, they put MoEs with up to 137 billion parameters between LSTM layers, and got better results than the best at the time with less computation.

IBM's TD-Gammon improves through self-play learning to near the top of the world in backgammon

IBM's Tesauro built TD-Gammon, in which a three-layer neural network played hundreds of thousands of games against itself and learned only from the win-or-lose results (“reinforcement learning”, a method of learning from rewards). It improved even when starting from a random initial state, and the version that played 1.5 million games came within one point of the former world champion Robertie over 40 games.

What we still don’t know:The paper also writes that TD-Gammon “demonstrated favorable empirical behavior of TD(λ) ... despite the lack of theoretical guarantees” (TD(λ) is the name of this learning method; an example that worked without theoretical backing).

Othello champion Takeshi Murakami loses 0–6 to the machine-learning program Logistello

Three months after Deep Blue's victory, a six-game match was held at the NEC Research Institute (Princeton) in the United States. Othello world champion Takeshi Murakami (an English teacher in Tokyo) lost 0 to 6 to Logistello, the program developed by Buro. According to its author, Logistello extracted knowledge from millions of board positions and learned opening moves through self-play. It ran on an ordinary personal computer.

Vapnik and colleagues create the support vector machine, which separates data by drawing the widest road between them

When drawing a line to separate two kinds of points, you draw the line down the middle of the widest road, as far as possible from both kinds of points. Then even points seen for the first time are separated well. Vapnik and colleagues made it so that this idea can be computed from only the few points at the edges of the road (the support vectors: the points that hold up the road). On the handwritten digits of MNIST it reached an error rate of 1.1% (Cortes and Vapnik), close to the neural network LeNet. By around 2000, this kind of method had become the mainstream in practical use.

LSTM: a neural network that can remember the earlier parts of a long sentence or sound is announced

Hochreiter and Schmidhuber announced LSTM, a neural network for time-series data (sequences of sounds or letters) with a built-in “memory cell” (a box that remembers) and “gate units” that open and close what goes into and out of the box. With earlier methods, the signal used for correction in backpropagation either faded away (approached zero) or blew up as it went back in time, so they could not remember things from long ago. LSTM could hold on to and use clues from the past more than 1000 steps away.

IBM's chess-only machine Deep Blue beats world champion Kasparov in a six-game match

IBM's chess-only computer “Deep Blue” played a six-game match against world champion Garry Kasparov in New York and won, with 2 wins, 1 loss and 3 draws (3.5 to 2.5). With the same time limits as in ordinary tournaments, it became the first computer to win a match against a reigning world champion. Its strength came from brute-force calculation, examining about 200 million positions per second. The year before (February 1996, in Philadelphia), Kasparov had won 4 to 2.

The checkers program Chinook becomes “World Champion” at the man-machine world championship

Chinook, a checkers program (draughts on an 8×8 board, in the wording of the paper) developed by Schaeffer and his colleagues at the University of Alberta in Canada, challenged world champion Tinsley in 1992 and lost, with 2 wins, 4 losses and 33 draws. They played again in August 1994 at the Computer Museum in Boston. After all 6 games were drawn, Tinsley withdrew partway because of poor health, and Chinook became the “Man-Machine World Champion”. This is regarded as the first case of a computer winning a human world championship title (from the team's paper).

Prediction: “within ten years a digital computer will be the world’s chess champion” (1957 lecture)

Herbert A. Simon and Allen Newell predicted that within ten years a computer would become the world chess champion (unless the rules barred it from competition).

What happened next:As of April 1967, one of the strongest chess programs (MIT's MAC HACK VI) played at the level of a mid-ranked amateur (1450 on the American rating scale). In 1968, International Master David Levy and some researchers made a bet on whether a computer could beat Levy within ten years. In 1978 Levy beat the computer and won the bet. A computer beat a reigning world champion (Kasparov) in a six-game match about 40 years after the prediction, in May 1997 (IBM's Deep Blue, 3.5 to 2.5). Note that a computer has never won the title of human world champion itself (the prediction also says “unless the rules bar it from competition”).

1985–1991: Rodney Brooks, robots that stopped thinking before moving. For simple intelligence, “explicit representations and models of the world simply get in the way”, and in 1990, “small AI companies are folding, and attendance is well down at … conferences”

Rodney Brooks of MIT's Artificial Intelligence Laboratory presented, in a September 1985 report, a method of designing a robot's behavior in “layers”. On top of lower layers, such as one that avoids obstacles, layers of more advanced behavior are stacked, and an upper layer suppresses the output of a lower layer only when needed (subsumption). In the paper “Intelligence without representation”, received in 1987 and published in 1991, he pointed out that when you study very simple intelligence, explicit internal representations and world models “simply get in the way”. He then reported that he had built several robots that move around offices on their own based on this idea. In the 1990 paper “Elephants Don’t Play Chess”, he argued for a path different from the AI of 30-odd years that manipulates symbols in the head (an approach with a body that interacts with the real world). He also mentioned the situation at the time: small AI companies were going bankrupt, and attendance at AI conferences at home and abroad was falling sharply.

What we still don’t know:How far intelligence can be achieved without representations (models of the world) is itself presented as a hypothesis (H) within the papers, and no conclusion has been reached.

2002, the Roomba robot vacuum cleaner at $199.95. The announcement describes a combination of simple movements, repeating spirals, wall-following and straight runs, so that each part of a room is passed over 2 to 5 times (no mention of maps)

On September 18, 2002, iRobot (a company that develops robot technology products; headquarters in Somerville, Massachusetts, according to the announcement) put on sale the home robot vacuum cleaner “Roomba™ Intelligent FloorVac” at $199.95. The announcement called it “the first automatic floor vac available in the U.S.” and explained that it applied “its unique expertise in artificial intelligence technology”, together with low-cost technology, to a home appliance. It first moves in a spiral; when it hits an obstacle it runs along the wall, and then it repeats spirals and straight runs. Cliff sensors detect stairs so it turns away, and over time it passes over each place in a room 2 to 5 times. It ran for 1 to 1.5 hours on its battery, and virtual walls could mark off areas it must not enter.

Sony announces AIBO, a four-legged household robot dog. At 250,000 yen, 3000 units sell out in 20 minutes

In May 1999 Sony announced the sale of AIBO (ERS-110), an “entertainment robot” for the home. It is an animal-shaped robot about 30 cm long with four legs; programs for emotion, instinct, learning and growth let it think and move by itself, and its personality changes depending on how people treat it, such as by petting it. According to its developer, Masahiro Fujita, despite the price of 250,000 yen, the order slots for 3000 units filled up in 20 minutes. The name is short for Artificial Intelligence roBOt, and is also tied to AI, the eye (EYE), and a partner.

Honda announces “ASIMO”, a humanoid robot that walks on two legs (research since 1986)

Honda announced ASIMO, a small, light humanoid robot that can walk in a way close to a human's. It was designed to be 120 cm tall and weigh 43 kg, a height at which it can reach switches and doorknobs indoors (the previous generation, P3, was 160 cm and 130 kg). A walking technology (i-WALK) that predicts the next movement and shifts the center of gravity in advance lets it make smooth turns. Honda began the research in 1986 and had shown the prototype P2 in 1996 and P3 in 1997.

2002, the “working humanoid robots” (translated) HRP-2, built under a national plan: 154 cm tall, 58 kg, 30 degrees of freedom. The announcement says “the potential to become one of the biggest products of the 21st century” (translated)

From 1998 the Ministry of Economy, Trade and Industry ran a five-year plan, “Research and Development of Human-Cooperative and Coexisting Robot Systems” (translated) (HRP: Humanoid Robotics Project). In the first two years it built the research base (the humanoid robot HRP-1, a remote-control cockpit, and a simulator), and in the last three years it studied five application fields. On March 19, 2002, Kawada Industries and the National Institute of Advanced Industrial Science and Technology (AIST), together with Yaskawa Electric and Shimizu Corporation, announced the “HRP-2 Prototype”, a robot meant to work alongside people outdoors. It is 154 cm tall, weighs 58 kg including the battery, and has 30 degrees of freedom (number of joints), including 2 axes at the waist, and was aimed at walking on uneven ground and getting back up after a fall. The announcement wrote that Japan's industrial-robot market has stayed flat at a scale of 500–600 billion yen a year since the 1980s because uses did not expand, and expressed the hope that, just as one of the biggest products of the 20th century was the automobile, “Humanoid robots have the potential to become one of the biggest products of the 21st century” (translated).

What happened next:The deadline is the “21st century” (translated), and the check is still under way (as of September 2026). This site has not confirmed the current market size of humanoid robots. HRP-2 itself was selected in 2017 by the National Museum of Nature and Science as Future Technology Heritage, and Kawada Robotics explains that it developed into NEXTAGE, a robot that works with people on production lines.

2005, the “Robot Project” (translated) at the Aichi Expo: cleaning, security and guide robots work on the site, and 65 kinds aiming at “practical use by 2020” (translated) are shown for 11 days, drawing about 123,000 visitors

At the 2005 Aichi Expo (the 2005 World Exposition, Japan), the Expo association and NEDO (New Energy and Industrial Technology Development Organization) carried out the “Robot Project ~We live in the Robot Age / We Are the Robot Generation~” (translated). According to a 2004 press release, the Ministry of Economy, Trade and Industry and NEDO were researching and developing about 100 robots in the “Next-Generation Robot Practical Application Project” (translated). During the Expo, five kinds of “working robots” (translated) were at work on the site: outdoor cleaning, trash-bin replacement, security, guidance in four languages, playing with children, and a wheelchair that avoids obstacles. From June 9 to 19, for 11 days, the “Prototype Robot Exhibition” (translated) was held, where 65 kinds of robots under development aiming at “practical use by 2020” (translated) were shown under the title “2020: A Town Where People and Robots Live” (translated), and about 123,000 people visited. The press release also said the robot market was expected to reach 1.8 trillion yen in 2010.

What we still don’t know:This site has not confirmed where the press release's “1.8 trillion yen in 2010” (translated) forecast came from (whose prediction it was), what the actual market size in 2010 was, or which of the 65 kinds aiming at “practical use by 2020” (translated) actually came into use (a candidate for checking as a prediction; market figures need care because they change a great deal with the range of what is counted). The 2001 forecast by the Japan Robot Association (3 trillion yen in 2010) and the 2008 check of it are in the record → A prediction: in 2010 Japan's robot mar….

1998, Moravec: “When will computer hardware match the human brain?” The answer: “available in cheap machines in the 2020s”. And AI research from 1960 to 1990 lay in “the frozen winter of 1-MIPS computers”

Hans Moravec of the Robotics Institute at Carnegie Mellon University estimated, in a 1998 paper, the computing speed and memory needed to do what a brain does. Replacing the work of the retina with robot image processing comes to 1,000 MIPS (MIPS = millions of instructions per second). Since the brain is about 100,000 times the size of the retina, he put the whole brain at about 100 million MIPS and its memory at about 100 million megabytes. From the trend of computers getting cheaper, he predicted that “the required hardware will be available in cheap machines in the 2020s”, and that this 100 million MIPS “will then arrive in home computers before 2030”. The same paper points out that the performance of computers available for AI research stayed at about 1 MIPS from 1960 to 1990 without growing, so that a researcher's whole career passed in “the frozen winter of 1-MIPS computers”. It also says that performance has doubled every year since 1990, and that now (1998) it is spring.

What happened next:As of September 2026, the deadline of the prediction (the 2020s, before 2030) has not yet arrived. Also, Moravec's yardsticks (MIPS = millions of instructions per second, a home machine of about $1,000) do not match today's measures of AI computing (floating-point operations per second, FLOP/s, and so on). The result changes depending on how one converts between them, so this site does not judge whether the prediction was right or wrong.

What we still don’t know:The estimate that the brain's work equals 100 million MIPS is one way of thinking, based on replacing the retina's work with robot image processing; how much computation the brain does is still not known (the paper itself says that “our estimates will depend on analogy and extrapolation”; extrapolation means extending the line of past trends into the future to estimate).

1993, Vinge: “Within thirty years, we will have the technological means to create superhuman intelligence”. He would be surprised if it came before 2005 or after 2030. The term “the Singularity” appeared 12 years before Kurzweil's 2005 book

Vernor Vinge, of the Department of Mathematical Sciences at San Diego State University (his affiliation in the paper), contributed a paper, “The Coming Technological Singularity”, to the proceedings of NASA's 1993 event “Vision 21”. In it he wrote that within 30 years the technological means to create intelligence beyond that of humans would be available, and that the human era would end shortly afterward. He named four paths to it: “computers that are "awake" and superhumanly intelligent”, the awakening of large networks, deep connections between humans and computers, and the enhancement of intelligence through biology. He even gave years for the timing: he would be surprised if it happened before 2005 or after 2030. He called this event “the Singularity”. In the paper, however, he introduces earlier examples: von Neumann had used the word “singularity” in the 1950s (as Ulam recounted), and I. J. Good had the idea of the “ultraintelligent machine” in the 1960s. He also adds himself that people have pointed out that AI enthusiasts have been making similar claims for 30 years.

What happened next:The end of the period he gave (2030) has not yet arrived (as of September 2026). The year 2023, which corresponds to “Within thirty years”, has passed, but whether “superhuman intelligence” has already been achieved depends on the evaluation standard, and this site does not judge it (for predictions by company leaders in 2023–2025, see records such as → OpenAI's CEO Altman: It is possible tha… and → Anthropic's CEO Amodei says, of powerfu…).

What we still don’t know:Neither the paper nor anything else we found gives a clear standard for how to measure “superhuman intelligence” (on which test or job a machine would have to beat humans for it to count).

“RoboCup”, started by Japanese researchers: the goal is for a team of humanoid robots to beat the World Cup winners by the middle of the 21st century

“RoboCup”, which makes robot soccer a shared challenge for AI and robotics research, was started by Japanese researchers (Hiroaki Kitano, Minoru Asada, Yasuo Kuniyoshi and others). They conceived it in June 1993 as a Japan-only “Robot J-League”, but within a month people overseas wanted to take part, and it grew into an international project. The first competition was held in Nagoya in 1997, drawing more than 40 teams and more than 5000 spectators. The ultimate goal stated on the official site is that, by the middle of the 21st century, “a team of fully autonomous humanoid robot soccer players shall win a soccer game” against the winner of the most recent World Cup, following the official rules of FIFA.

What happened next:The deadline has not arrived, so there is no result yet. On July 5, 2026, at RoboCup 2026 held in Incheon, South Korea, 11-on-11 matches between humanoid robots were played for the first time (Germany's B-Human beat Germany's HTWK 4–0). But this was a match between robots, not against human champions.

1997: the International Joint Conference on Artificial Intelligence (IJCAI-97) is held in Nagoya, the second time in Japan after Tokyo in 1979. 1,400 people from 48 countries attend, and the first RoboCup is held alongside

The 15th IJCAI (IJCAI-97), the international conference at which AI researchers gather every two years, was held at the Nagoya Congress Center from August 23 to 29, 1997. According to the list of past host cities on the official IJCAI site, this was the second time it was held in Japan, after Tokyo in 1979. According to the report of the Japanese Society for Artificial Intelligence, 1,400 people from 48 countries took part (550 from overseas). Besides 12 invited talks, 216 contributed paper presentations, 30 workshops and 17 tutorials, side events included the robot soccer tournament RoboCup-97. The decision to bring it to Japan was made in fiscal 1991, and the Japanese Society for Artificial Intelligence had been preparing for it (from the society's business report for fiscal 1991). It was a large international conference held at a time when the number of regular members of the Japanese Society for Artificial Intelligence had begun to fall.

A prediction: in 2010 Japan's robot market will be about 3 trillion yen (a 2001 forecast by industry groups). The home and hospital robot market actually did not reach 10 billion yen

In 2001, a report by the Japan Machinery Federation and the Japan Robot Association forecast the robot market in five fields. For 2010 it gave 1.5 trillion yen for daily life, 850 billion yen for manufacturing, 290 billion yen for public use, 260 billion yen for medical care and welfare, and 90 billion yen for biotechnology (about 3 trillion yen in total), and for 2025 about 8 trillion yen in total. According to a 2009 analysis by Hirohiko Arai of the National Institute of Advanced Industrial Science and Technology, this forecast was used as the basis for explaining research and development budgets by the Ministry of Economy, Trade and Industry and others, and it spread through news reports too. A 2005 Aichi Expo announcement also stated that “a market size of 1.8 trillion yen is expected in 2010” (translated).

What happened next:According to Hirohiko Arai's 2009 analysis, the market for next-generation robots outside manufacturing stayed at 1.4 to 6.8 billion yen a year (under 10 billion yen) in surveys of actual conditions by the industry association and others, far from the 2010 forecast (about 2 trillion yen outside manufacturing). In 2006–2008, national agencies and the industry association itself revised the 2010 forecast downward to 50–75 billion yen. In 2015 the government's “New Robot Strategy” gave the size of the robot market as “600 billion yen at present” (translated) and set a new target of 2.4 trillion yen in 2020 (from the Ministry of Internal Affairs and Communications' 2015 white paper). This site has not checked the 2025 forecast.