The Peak of the Third Boom

Batch Normalization: rescale the values entering a layer to mean 0 and variance 1 for each batch of examples, so that training goes faster (2015)

Batch normalization is a method that builds into a network a step that rescales the values entering a middle layer so that, for each group of data used at one time in training (a mini-batch), the mean is 0 and the variance is 1, and then restores the scale and offset with two learned numbers, γ and β (scale and shift). Google's Ioffe and Szegedy thought that learning is slowed because the distribution of values entering the next layer changes every time the weights of the previous layer change (the paper named this internal covariate shift). They wrote that with this method the learning rate can be made much larger, less care is needed in choosing the initial weights, and in some cases dropout is no longer needed. In an image recognition network, it reached the same accuracy with 14 times fewer training steps. For the combined networks, they reported an error of 4.9% on ImageNet's top-5 rule (4.8% on the test set) and wrote that this exceeded the accuracy of the human raters.

What we still don’t know:Why batch normalization works is still being researched after the 2018 paper, and it cannot be said to be settled (papers later than 2018 have not been opened yet).

Knowledge distillation: having a small model imitate the “soft targets” of a large model (2015)

Knowledge distillation is a method for transferring the knowledge learned by a large, heavy network, or by an ensemble of many networks, into a small, light network. The probabilities output by the large network are smoothed by raising the temperature T of the softmax (a function that turns scores into probabilities), and the small network is trained to reproduce these soft targets. The soft targets also contain the differences in probability among the wrong answers (in the paper's example, the probability of mistaking an image of a BMW for a garbage truck is very low, but many times larger than the probability of mistaking it for a carrot), and this gives clues about how the large network generalizes. A March 2015 paper by Hinton, Vinyals and Dean of Google reported that, in a handwritten-digit experiment, even when the small network was distilled without being shown a single image of a 3, it could correctly classify 98.6% of the 3s in the test set once the bias value for 3 was adjusted. In an experiment with an (older version of) Android's voice search model, it states that more than 80% of the accuracy gained by an ensemble of 10 networks could be transferred to a single distilled network. The paper begins with an analogy to insects, whose body structure differs between larva and adult, to argue that the form of a network may differ between training and use.

What we still don’t know:We have not yet checked in the original sources how widely knowledge distillation is used today for small AI that runs on devices such as smartphones.

word2vec: turning words into vectors so that closeness of meaning and relationships can be calculated (2013)

word2vec is a method, and a program, that represents each word as a set of a few hundred numbers (a vector). It is trained on large amounts of text so that words used in similar ways come close together. Tomas Mikolov and colleagues at Google published it in 2013. There are two designs. CBOW predicts the middle word from the surrounding words, and Skip-gram is trained to predict the words before and after from the middle word. Compared with earlier neural-network language models (such as the 2003 paper by Bengio and colleagues), it lightened the computation by removing the nonlinear processing in the hidden layer. They state that vectors with 300 dimensions for 1 million words could be built from a 1.6 billion-word data set in less than a day on a single CPU. With these vectors, relationships between words can be expressed by addition and subtraction. The paper measured performance by how often the word nearest to the vector biggest − big + small was smallest. The follow-up paper in October added tricks to train Skip-gram quickly (such as negative sampling) and the handling of phrases like Air Canada. It also showed an example in which adding the vectors for Russia and river gives something close to Volga River. The program was released under the name word2vec.

What we still don’t know:I have not yet checked in the original sources how the answers change if the three words of the question are not excluded from the candidates (whether the nearest word is often the word in the question itself). I have also not checked at which 2013 conference word2vec was presented.

An AI that learns to play video games just by looking at the screen (DeepMind's DQN, 2013 and 2015)

In December 2013, DeepMind Technologies (whose address in the 2015 paper is London, UK) announced an AI that plays “classic Atari 2600 games”, golden-oldie video games. Using only the pixels on the screen and the score as clues, it learns by combining reinforcement learning, which improves through trial and error, with deep learning. It beat earlier methods in 6 of 7 games, and exceeded a skilled human player in 3. In a paper in the journal Nature in February 2015 (affiliation: Google DeepMind), the same single system was trained on 49 games and became about as strong as a professional game tester. In 29 games it scored at least 75% of the human score.

Google buys DeepMind, an AI company in London (2014)

In January 2014 it was reported that Google would acquire DeepMind, an AI company in London. Google confirmed the acquisition to Re/code and DeepMind confirmed it to TechCrunch, but they did not disclose the price. DeepMind is a company founded by Demis Hassabis, Shane Legg and Mustafa Suleyman (TechCrunch's wording). In later papers the affiliation became “Google DeepMind” (the 2015 DQN paper).

DeCAF: freezing the middle layers of a CNN trained on ImageNet and reusing them as features for other tasks (Donahue et al., 2013–2014)

In October 2013, Jeff Donahue, Yangqing Jia and 5 others at the University of California, Berkeley and ICSI froze the weights of a CNN trained on ImageNet's 1,000-class classification (the network structure of Krizhevsky and colleagues, 2012). They then tested whether the responses of its middle layers could be used as features (DeCAF) for other tasks. Just by putting a linear classifier on top of the layer near the output (DeCAF6, 4,096 numbers), it got 86.9% right on Caltech-101 with 30 training images per class, beating the then best result, which combined five handcrafted features, by 2.6%. Even with just 1 image per class it got 33.0%. It also beat the then best on scene recognition (SUN-397, 40.94%) and on identifying bird species. The CNN layers were not fine-tuned, and they released it, including the network's weights, as software anyone can use.

Measuring, one layer at a time, how far up a deep network its features can be moved to another task (Yosinski et al., 2014)

In November 2014, Jason Yosinski of Cornell University, Jeff Clune of the University of Wyoming, Yoshua Bengio of the Université de Montréal and Hod Lipson of Cornell University measured, one layer at a time, how many layers of a deep network hold general features that are useful for any task. They randomly split the 1,000 ImageNet categories into two sets, A and B, of 500 categories each (about 645,000 images each). They copied the bottom n layers of an 8-layer network trained on A into a network for B, and trained the rest on B. Layers 1–2 could be moved almost as they were. For layers 3–5, performance dropped because features that had been fragilely co-adapted with their neighboring layers were split apart. For layers 6–7, the main cause of the drop was that the features were specialized to task A. When the copied layers were also fine-tuned on B, accuracy was higher by 1.6% on average (2.1% when 5 or more layers were copied) than for a network trained from scratch on B alone.

What we still don’t know:As for why the effect of the earlier training remains even after fine-tuning, the paper stops at a plausible explanation: the influence of having seen the original data remains.

Trying many kinds of image recognition with only the features of a trained CNN (Razavian et al., 2014)

In March 2014, Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan and Stefan Carlsson of KTH (the Royal Institute of Technology) in Sweden took features of 4,096 numbers from OverFeat, a public model trained on ILSVRC13 classification. Just by putting a linear SVM on top of them, they tried tasks that move step by step away from the original task: object classification, scene recognition, fine-grained recognition, attribute recognition and image retrieval. They wrote that on the classification tasks it beat all the carefully tuned state-of-the-art methods of the time, and concluded that the features of a deep CNN should be the first candidate for most image recognition tasks.

Can Object Detection Models Be Trained from Scratch Without ImageNet Pretraining? A Test (He et al., 2018)

In November 2018, Kaiming He, Ross Girshick and Piotr Dollár of Facebook AI Research (FAIR) re-examined whether the combination of ImageNet pretraining and fine-tuning is really needed for object detection. On COCO object detection and region segmentation, models whose weights were initialized randomly (from scratch) reached results no worse than pretrained and fine-tuned models if trained for more iterations. The same held even when the COCO training images were reduced to 10%. The benefit of pretraining was that it speeds up early training. However, with only 1,000 training images, training from scratch scored 3.5 against 9.9 with pretraining (scores are AP), far lower (with 3,500 images, 16.0 and 9.3 respectively).

What we still don’t know:How much target data is needed before pretraining is no longer required depends on the task. For COCO, the paper's estimate puts it between 3,500 and 10,000 images.

In an explanatory article in the Journal of the Japanese Society for Artificial Intelligence, Toshihiro Kamishima summarizes transfer learning in Japanese (2010)

In July 2010, Toshihiro Kamishima of the National Institute of Advanced Industrial Science and Technology summarized the ideas and methods of transfer learning in Japanese in an explanatory article in the Journal of the Japanese Society for Artificial Intelligence. Transfer learning is a method that takes from data that is related but partly different only the knowledge that is also useful for the target problem, and so obtains more accurate prediction rules. He laid out that similar ideas have been called by many names, such as inductive transfer, domain adaptation, multitask learning and learning to learn. He also introduced research on neural networks that share their middle layers across several tasks (Caruana, 1997), and research on negative transfer, where using the original data actually lowers prediction accuracy.

R-CNN: making about 2,000 candidate regions, turning each into features with a CNN, and detecting objects (Girshick et al., 2013–2014)

In November 2013, Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik of the University of California, Berkeley (shown as UC Berkeley in the paper) and ICSI brought the CNN (convolutional neural network), which had shown its power in image classification, to object detection. Object detection is the technology of drawing a rectangle (a bounding box) around each object in an image and identifying its kind. First, selective search, which involves no learning, makes about 2,000 candidate rectangles per image where an object might be; each is warped to 227×227 pixels and passed through the CNN to become a 4,096-dimensional feature. A linear SVM for each class turns that into a score, and rectangles that overlap a higher-scoring rectangle of the same class by a large amount (IoU: overlapping area ÷ combined area) are removed (non-maximum suppression). The CNN was first pre-trained on ImageNet classification (ILSVRC 2012) and then fine-tuned on detection data. In the conference version (CVPR 2014), it reached an mAP (mean average precision) of 53.3% on PASCAL VOC 2012, more than 30% better in relative terms than the previous best. On the other hand, computing the candidates and features took 13 seconds per image on a GPU and 53 seconds on a CPU.

YOLO: split an image into a 7×7 grid and detect position and type together in a single network pass (Redmon et al., 2015–2016)

In June 2015, Joseph Redmon and colleagues at the University of Washington presented YOLO (You Only Look Once), which solves object detection with one computation of one neural network. The image is resized to 448×448 pixels and split into a 7×7 grid, and for each cell the network outputs 2 boxes with their confidence and the probabilities of 20 classes at the same time (the output is 7×7×30 numbers, and there are 98 boxes in total; R-CNN has about 2,000 region proposals). Because it does not process the detection of candidate regions and their classification separately, it is fast. In the conference (CVPR 2016) version, using a Titan X GPU, it reached an mAP of 63.4% on PASCAL VOC 2007 at 45 images per second. The small version, Fast YOLO, got 52.7% at 155 images per second. By comparison, in the same paper Fast R-CNN got 70.0% at 0.5 images per second. YOLO made more localization errors (19.0% of top predictions had localization errors, against 8.6% for Fast R-CNN), but it less often mistook background for an object (4.75% against 13.6%).

Faster R-CNN: the candidate boxes are also made by a network, speeding up object detection (Ren et al., 2015)

In June 2015, Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun of Microsoft Research presented Faster R-CNN. It is a method that has a network also do the creation of candidate boxes, which had been the cause of slowness in the R-CNN line of work. A small network (the RPN, region proposal network) that shares the CNN features of the whole image with object recognition outputs, for each of 9 reference boxes (anchors: 3 sizes × 3 aspect ratios) on the feature map, an objectness score and a position correction. With VGG-16 and 300 candidates per image, the whole pipeline ran on a GPU at 5 images per second and recorded an mAP of 73.2% on PASCAL VOC 2007. According to the paper, selective search, which had been widely used until then, took 2 seconds per image on a CPU.

ChainerCV: People at the University of Tokyo and Preferred Networks Release a Library of Parts That Runs Object Detection on Chainer as in the Papers (2017)

In August 2017, Yusuke Niitani and Toru Ogawa of the University of Tokyo and Shunta Saito and Masaki Saito of Preferred Networks wrote a paper on ChainerCV, a collection of parts for using object detection and region segmentation networks together on Chainer, a deep learning tool. It even includes programs that train Faster R-CNN and SSD (a different, single-shot detection method) to reach the same performance as in the original papers. On PASCAL VOC 2007 mAP, they reported 70.5% for Faster R-CNN against 69.9% in the original paper, and 77.5% for SSD300 against 77.5%. They also implemented non-maximum suppression, which removes overlapping detection boxes, to run on a GPU.

Variational autoencoder (VAE): squeezing data into a few numbers and making new data from them (Kingma and Welling, 2013)

In December 2013, Diederik Kingma and Max Welling of the University of Amsterdam released their paper “Auto-Encoding Variational Bayes” on arXiv (a place where researchers post papers early). It brings the ideas of probability into the autoencoder, a neural network that combines an encoder, which squeezes the input into a small set of numbers called latent variables, with a decoder, which rebuilds the input from them. The encoder outputs not a single point but a mean and a spread (standard deviation). When a point is chosen from them in the form mean + spread × random number, the random number is separated out of the calculation. This makes it possible to train the whole network at once with backpropagation (the method of passing the output error from back to front to correct the weights) (the paper calls this the reparameterization trick). During training, the aims are to make the error between the rebuilt input and the original input small and, at the same time, to bring the distribution of the latent variables close to a normal distribution with mean 0 and spread 1. As a result, after training, new data can be generated by passing points drawn from that normal distribution through the decoder. The paper tested it on handwritten digits (MNIST) and face photos (Frey Face).

What we still don’t know:Which of the two research groups had the idea first is not known; there are no sources apart from the dates the papers appeared on arXiv (2013-12-20 and 2014-01-16).

Origins of the autoencoder: in 1985–1987, the task of making a network output what it was given through a narrow middle layer

An autoencoder is a neural network trained to output the same data it receives as input. The number of neurons in the middle layer is kept small. Because the input cannot simply pass through, the middle layer learns on its own a code that represents the input compactly with a few numbers. The first half acts as the encoder, turning the input into a code, and the second half acts as the decoder, restoring the original from the code. This structure appears in the encoder problem of the 1985 Boltzmann machine paper and in a chapter of the 1986 book on backpropagation. In the chapter, a network was trained on a task of reproducing 8 inputs as the same 8 outputs through 3 hidden-layer neurons. The hidden neurons then assigned each of the 8 inputs its own code (not only 0 and 1 but also in-between values were used). In 1987 Ballard proposed a method of stacking autoassociative networks, which make the input and output the same. In this scheme, the upper layer receives the representation of the lower hidden layer and learns from it.

What we still don’t know:We have not confirmed the first paper to use the word autoencoder (the oldest we opened is the 1993 NIPS paper by Hinton and Zemel). The 1991 paper on nonlinear principal component analysis by Kramer is paywalled, so we did not open it. We looked but found nothing conclusive about whether, in Japan around the same time, there was research on networks that output the same thing as their input (called hourglass-type networks, learning of the identity mapping, and so on).

GAN (2014): two AIs compete, like “a team of counterfeiters” and “the police”, to make data that looks just like the real thing

In June 2014, Ian Goodfellow and colleagues at the Université de Montréal published a paper on “Generative Adversarial Nets” (GAN), a method that trains two AIs at the same time: a generator G that makes data, and a discriminator D that tells whether data is real or made up. The paper compared the generator to “a team of counterfeiters, trying to produce fake currency” and the discriminator to “the police,” and explained that as they compete, both get better until the fakes can no longer be told from the real thing. The experiments used images such as handwritten digits and face photos.

RNN Encoder–Decoder and GRU: two RNNs that convert sequences, and a simplified LSTM-like structure (Cho et al., 2014)

In June 2014, Kyunghyun Cho and colleagues at the Université de Montréal presented the RNN Encoder–Decoder, which combines two RNNs (recurrent neural networks: networks that process sequence data step by step while keeping a record of past states). The first RNN (the encoder) squeezes a sequence of words of any length into a fixed-length vector, and the second RNN (the decoder) generates a different sequence of words from that vector. The same paper also proposed a new structure (later called the GRU) that makes the LSTM simpler to compute. This structure has two gates (mechanisms that control, with a value from 0 to 1, how much information passes through). The reset gate decides how much of the past state to ignore and rebuild from the current input, and the update gate decides how much of the past state to carry over unchanged to the next step. The paper tried it on English-to-French translation. However, it was used inside a conventional statistical machine translation system (a method that counts the probabilities of how chunks of words are translated, from a large amount of translated text pairs), as an auxiliary clue for scoring pairs of phrases (chunks of words).

seq2seq (sequence to sequence): one LSTM reads a sentence into a single vector, and another LSTM writes out the translation (Sutskever et al., 2014)

In September 2014, Ilya Sutskever, Oriol Vinyals and Quoc Le of Google published the paper “Sequence to Sequence Learning with Neural Networks” on arXiv. A 4-layer LSTM (an RNN designed to remember earlier information even over long sequences) reads the input sentence one word at a time and packs it into a fixed-length vector. Another 4-layer LSTM writes out the translation from that vector, one word at a time. They also discovered that feeding in the words of the input sentence in reverse order makes learning easier (BLEU rose from 25.9 to 30.6). BLEU is a measure that scores how many of the same word sequences a machine's translation shares with a human's translation. In the WMT'14 English-to-French translation task, a translation from an ensemble of 5 LSTMs scored 34.8, beating the statistical machine translation system it was compared with (33.3). When an LSTM re-ranked the 1000 candidates produced by the statistical system, the score reached 36.5. One LSTM had 380 million parameters, and 12 million sentence pairs were used for training.

What we still don’t know:The paper says it cannot fully explain why reversing the input improves accuracy, and offers a guess: the distance between the first words of the input and the first words of the translation becomes shorter (Section 3.3).

The origin of attention: translation that decides, each time it outputs one word, how much to focus on each word of the original sentence (Bahdanau et al., 2014)

In September 2014, Dzmitry Bahdanau, KyungHyun Cho and Yoshua Bengio published the paper “Neural Machine Translation by Jointly Learning to Align and Translate” on arXiv. The older encoder-decoder squeezed the whole input sentence into a single fixed-length vector, and the paper saw this as a bottleneck (a place that narrows like the neck of a bottle) limiting performance. So the encoder keeps, for each input word, a summary (an annotation) made by reading the sentence in both directions, forward and backward. Each time the decoder outputs one translated word, a small neural network gives each word's summary a score for how important it is, turns the scores into weights that add up to 1, and the decoder uses the weighted sum of the summaries. The paper explained that, intuitively, this gives the decoder a mechanism of attention. In English-to-French translation, scores did not drop much even as sentences got longer. When limited to sentences with no unknown words, the scores were about the same as Moses, a statistical machine translation system (BLEU 36.15 and 35.63). When the weights were drawn as a figure, the English European Economic Area could be seen matching the French zone économique européenne, which has the words in the reverse order.

Google Translate switches to “Neural Machine Translation”, which reads a whole sentence at once (2016). People noticed that translations suddenly became more natural, even in Japanese

In September 2016, Google announced neural machine translation (GNMT), a method that reads the whole sentence as one unit. Until then, Google had used statistical machine translation, which translates a sentence part by part, since it started Google Translate about 10 years earlier (the research blog says “Ten years ago”, and the Japanese blog says it “appeared as a pioneer of statistical machine translation” (translated)). GNMT was first used for all Chinese-to-English translation (about 18 million translations a day). In human evaluations, Google said it reduced errors by 60% on average. In November it was extended to 8 languages, including Japanese and Korean. According to ITmedia, “the improvement in accuracy had been a topic of talk in Japan since around the 11th” (translated). There is a technique that splits words into small pieces (wordpieces) so that rare words can be handled. According to Google's technical report, Google originally created it to solve the problem of segmenting Japanese and Korean in speech recognition, and it is also used in BERT in 2018.

Japanese government goal: put speech translation in 10 languages to practical use by 2020 (2014). Next, simultaneous interpretation by 2025

In April 2014 the Ministry of Internal Affairs and Communications announced the “Global Communication Plan”. With the mission of “eliminating the language barriers of the world” (translated), it decided to achieve highly accurate speech translation in 10 languages for conversations on travel, disaster prevention, medicine and daily life by 2020, and to “welcome them without making them feel any language barrier” (translated), meaning the athletes and spectators gathering from around the world for the 2020 Tokyo Olympics and Paralympics. The core was NICT's technologies such as VoiceTra. In the next plan, in March 2020, it said that in the first phase (2014–2019) it had “achieved a practical level in 12 languages” (translated), and, aiming at the 2025 Osaka-Kansai Expo, set a new goal of going “from consecutive translation to simultaneous interpretation” (translated) (consecutive translation means translating one sentence at a time). According to the 2025 White Paper, a translation app in 30 languages using NICT's technology and simultaneous interpretation at seminars and other events were used at the Expo.

What happened next:The Ministry's own plan of March 2020 says “practical level achieved in 12 languages” (translated). The next goal is simultaneous interpretation by 2025, and the 2025 White Paper writes that the Expo put into use a translation app in 30 languages and simultaneous interpretation at seminars and other events. How the “practical level” is measured is not written in the sources I opened. How it was used at the Olympics and Paralympics has not been checked on this site.

Adam: a learning method that automatically adjusts the update size for each individual weight (2014)

Adam is an optimization method that improves on gradient descent (a method that updates the weights little by little in the direction that shrinks the error against the correct answers). For each weight it keeps a moving average of the gradient (which way, and how far, to move) as the first moment, and a moving average of the squared gradient as the second moment, and it updates the weight by the average divided by the square root of the average of the squares. As a result, weights whose gradients are always large are updated cautiously and weights whose gradients are small are updated more boldly, and the update size stays within a roughly fixed size (the step size). Because the moving averages start at 0, the values lean toward 0 early on, so Adam also includes a calculation to correct this bias (bias correction). The name comes from adaptive moment estimation. The paper explains that it combines the strengths of AdaGrad, which works well with sparse gradients, and RMSProp, which works well with problems that change over time. As the source for RMSProp, the paper cites not a paper but Lecture 6.5 by Hinton and colleagues in an online course (Coursera). A footnote in the paper says the two authors contributed equally and that the order of the author names was decided by a coin flip over a video call (Google Hangout).

What we still don’t know:We have not checked in which version between v1 and v9 the way of choosing β changed (only two versions, v1 and v9, were opened). The original RMSProp lecture itself (Coursera Lecture 6.5) has not been opened.

ResNet (residual network): skip connections that jump over layers make networks as deep as 152 layers possible (2015)

ResNet is a neural network that repeatedly builds in skip connections (shortcut connections), which add the value entering a group of layers directly to the output of those layers. A block of layers does not have to learn the target transformation H(x) itself; it only has to learn the difference from the input, F(x) = H(x) − x (the residual), and the output is F(x) + x. Because a skip connection simply passes the value along, it adds neither weights nor computation. Kaiming He and colleagues at Microsoft Research showed that with plainly stacked layers, a 56-layer network had larger errors than a 20-layer one even on the training data (a decline caused not by overfitting but by difficulty of training; the paper calls it degradation). Using the residual form solved this, and on ImageNet they could go as deep as 152 layers (8 times deeper than the earlier VGG). An ensemble of networks reached an error rate of 3.57% and took first place in the 2015 ILSVRC image classification. On CIFAR-10, a small image dataset, they also examined networks of 100 and 1,000 layers.

What we still don’t know:As to why the residual form is easier to train, the paper presents it as a hypothesis. Later explanations have not yet been checked in the original sources.

U-Net: a U-shaped network born for segmenting microscope images (2015)

U-Net is a neural network for segmentation, which classifies what is shown at each pixel of an image. In the first half, the contracting path repeats convolution and downsizing to capture wide context; in the second half, the expanding path repeats upsizing to return to the original size. Because detail-preserving features from the first half are copied and joined to the layers of the same size in the second half, it can pinpoint where objects are. The two halves are almost symmetric, giving the shape of the letter U. Ronneberger, Fischer and Brox of the University of Freiburg in Germany presented this network together with a method of training on few images by distorting them (elastic deformation) to multiply them, because in medical and biological imaging it is impossible to prepare thousands of images with correct labels. In the 2015 ISBI cell tracking challenge, it won by a wide margin, with an average overlap with the correct answers (IOU) of 92% on a dataset of 35 partially annotated images (second place: 83%) and 77.5% on a dataset of 20 images (second place: 46%). Segmenting a 512×512 image took less than a second on a GPU of that time.

What we still don’t know:We have not yet checked figures for how widely U-Net is used in medical-imaging research.

In the image-recognition competition, the top team's “Error (5 guesses)” fell from 28% to 2% in 8 years. The “human: 5.1%” figure is one person's measurement

In a competition to identify 1,000 kinds of objects from images (with a rule allowing up to 5 guesses), the winning error rate fell: 28.2% in 2010, 25.8% in 2011, 15.3% in 2012 (16.4% with only the supplied data), 11.7% in 2013 (11.2% when outside data was also used), 6.7% in 2014, 3.6% in 2015, 3.0% in 2016 and 2.3% in 2017. The organizers also measured the human error rate. It was a test with two people (A1 and A2): A1, who had practiced on 500 images, scored 5.1% on 1,500 images, and A2, who had practiced on only 100 images, scored 12.0% on 258 images.

“In 2015, AI that recognizes images surpassed humans”: who was that “human”?

In February 2015, Kaiming He and colleagues at Microsoft Research reported an error rate of 4.94% on data from an image competition (ImageNet). This is the share of cases in which the correct answer was not among 5 guesses, and the paper announced that it “surpasses for the first time the reported human-level performance” (5.1%). In December of the same year, a combination of 152-layer deep neural networks (ResNet) recorded an error rate of 3.57% and took first place in the 2015 competition.

What we still don’t know:The “human error” rate changes greatly depending on who is trained, how, and on how many images it is measured (even for the two people in the same paper it was 5.1% and 12.0%). Many numbers that compare people with AI have the same problem.

In the image-recognition competition ILSVRC 2012, the University of Toronto's deep convolutional neural network (SuperVision) wins by a wide margin. The runner-up is a University of Tokyo team

In a competition to identify 1,000 kinds of objects from photos, the team “SuperVision” of Krizhevsky, Sutskever and Hinton held its error rate, under a rule allowing 5 guesses, to 15.3% (16.4% if only the data supplied by the competition is used). The runner-up, a University of Tokyo team (ISI), scored 26.2%, a gap of about 11 points. They trained an 8-layer neural network with 60 million parameters on 2 gaming GPUs (graphics chips, made by NVIDIA). In a later paper, the competition's organizers wrote that “This was a turning point for large-scale object recognition” for research on recognizing a wide variety of objects.

“In 2016, AI reached human level at transcribing conversation”: who measured that “human” error figure, and how?

In October 2016, Microsoft researchers measured the errors (word error rate) of professional transcribers on tests of transcribing telephone conversations (Switchboard and CallHome). The results were 5.9% and 11.3%, and their own AI got 5.9% and 11.1%, and they announced “the first time that human parity has been reported for conversational speech”. The paper also wrote that “human performance on this task has been widely cited as being 4%”, but that the source of that figure is a “personal communication” and is unclear. In March 2017, IBM researchers (including researchers from the Tokyo lab as co-authors) measured the human error again. Three transcribers wrote separately and a fourth checked. The best person scored 5.1% (6.8% on CallHome), and they wrote that even their own AI (5.5%) was “albeit not at what we measure to be human performance”.

What we still don’t know:There is no settled way to measure the human error rate that defines “human parity” (whom to use, how many times, and how to have them check). The IBM paper points out that human error may be considerably smaller than had been thought.

“Turing Test was passed for the very first time by computer programme” (2014): where did the passing standard come from?

In June 2014, the University of Reading announced at an event held at the Royal Society in London that the chat program “Eugene Goostman” had “passed for the very first time” the Turing test (a test in which a person talks only through text on a screen and tries to tell whether the other side is a human or a machine). The program imitated a 13-year-old boy. The university's standard was “mistaken for a human more than 30% of the time … five minute keyboard conversations”, and it said Eugene made 33% of the 30 judges think it was human.

What we still don’t know:How to decide whether we can say “machines thinking” still has no settled answer (Turing himself wrote in the same paragraph that his original question, “Can machines think?”, is too meaningless to deserve discussion)

Style transfer: keep a trained CNN as it is and adjust the image instead, combining the content of a photo with the style of a famous painting (Gatys et al., 2015)

In August 2015, Leon Gatys, Alexander Ecker and Matthias Bethge of the University of Tübingen (Germany) and other institutions published the paper “A Neural Algorithm of Artistic Style”. It is a way to make an image that combines the content of a photo (what is where) with the style of a painting (its colors and brush strokes). It uses as it is a CNN (convolutional neural network: a network that applies small filters to an image many times to extract features; here, the 19-layer VGG) already trained for object recognition. The responses of deep layers serve as the representation of the photo's content. In several layers, it computes the correlations (Gram matrices) of how strongly features appear together, and uses them as the representation of style. Starting from an image of white noise, it adjusts not the network's weights but the pixels of the image, by gradient descent (a method that moves things little by little in the direction that reduces the error), so that both the content error and the style error become small. The paper's figure shows a photo taken along the Neckar river in Tübingen rendered in the style of Van Gogh's The Starry Night, Munch's Der Schrei and other works. In Japan, 15 days later, on September 10, 2015, Mr. Matsumoto, a researcher at Preferred Networks (PFN), released a program implementing this method with the deep learning tool Chainer. In his blog post “An Algorithm That Converts Painting Style”, he explains the mechanism in Japanese.

Diffusion models: learn to reverse a process that gradually adds noise and destroys an image, then make images from noise (Sohl-Dickstein et al. 2015, Ho et al. 2020)

In March 2015, Jascha Sohl-Dickstein and colleagues at Stanford University, inspired by non-equilibrium statistical physics (physics that deals with the motion of many particles in states that are not in balance, that is, not in equilibrium), presented a method that learns a diffusion process that gradually destroys the structure of data and the reverse process that restores it (diffusion is a random spreading-out motion, like ink dropped in water). In the forward process, small Gaussian noise (random numbers following a normal distribution) is added to the data again and again until it becomes pure noise. A neural network learns each step of the reverse (a process that removes a little noise). After training, new data can be generated by repeating the reverse process starting from plain noise. The 2015 paper ran experiments on a two-dimensional swiss roll, handwritten digits, and small photos (CIFAR-10). In June 2020, Jonathan Ho and colleagues at the University of California, Berkeley published the DDPM (Denoising Diffusion Probabilistic Models) paper. Using a 1,000-step process and a training method that predicts the noise added at each step, it reached an FID of 3.17, the best at that time, for unconditional CIFAR-10 image generation (FID measures the difference between a set of generated images and a set of real images; smaller is better).

Chainer from Japan's Preferred Networks: Define-by-Run, which builds the network while the program runs (2015)

Chainer is a tool (framework) developed by the Tokyo company Preferred Networks (PFN) and released in June 2015 as open source (software anyone can use). It lets you write deep learning networks in Python and train them. Many earlier tools (such as Caffe) worked in two stages: first the whole structure of the network (the computation graph) was fixed, and only then was data run through it for training (the Chainer paper named this Define-and-Run). Chainer, on the other hand, records the path of the computation as it runs the forward calculation on data, and traces it backward to compute gradients; it called this Define-by-Run. As a result, even networks whose shape changes with each input, such as recurrent neural networks handling sentences of different lengths each time, can be written with ordinary Python loops and if statements, and ordinary debuggers (tools for finding errors in programs) can be used. It was presented as a paper at the NIPS workshop on machine learning systems in December 2015. The 2019 PyTorch paper later wrote that Chainer and DyNet pioneered this approach. On December 5, 2019, PFN announced that it was moving its research and development platform from Chainer to PyTorch, and that Chainer would enter maintenance with v7, released that day.

What we still don’t know:We have not checked which came first, the idea of DyNet or of Chainer, or whether each knew of the other. We also have not yet checked concrete numbers on how much Chainer was used by researchers and companies in Japan.

Japanese AI company Preferred Networks (PFN) is founded (2014). In 2015 it releases the deep-learning tool “Chainer” free of charge

On March 26, 2014, Toru Nishikawa and Daisuke Okanohara co-founded Preferred Networks. In June 2015 they released “Chainer”, a tool for building deep-learning AI, as software anyone can use (open source). In 2015 the company formed capital alliances with Fanuc and Toyota Motor. At the 2017 World Computer Shogi Championship, the runner-up shogi program “Ponanza Chainer” used this Chainer.

The two papers behind “Monte-Carlo Tree Search”, which made Go programs strong

In Go, there are many possible moves in a single position, and evaluating a position is hard. So search methods like those used for chess did not make programs strong. In 2006, a paper by Coulom of France presented a new search method (Monte-Carlo Tree Search, MCTS). It plays out a game at random many times from a position to the end (Monte Carlo), compares moves by their win rates, and examines the more promising branches more deeply. With his 9×9 Go program Crazy Stone, Coulom won a computer Go tournament (KGS). In the same year, Kocsis and Szepesvári of Hungary proposed UCT, a method for deciding which branch to try and how much. Ten years later, the AlphaGo paper (2016) cited these two as the MCTS literature, and combined MCTS with two neural networks (one predicting the next move and one evaluating positions).

Go AI “AlphaGo” wins its first game against a professional with no handicap (5 wins, 0 losses against Fan Hui)

AlphaGo, a Go AI developed by DeepMind of London, England, played a five-game match with no handicap (no stones placed in advance) against Fan Hui, the European champion and a 2-dan professional, and won 5 to 0. The paper said it was “first time … defeated a human professional player, without handicap”, in a feat “previously believed to be at least a decade away”. It was made public about three and a half months later, in January 2016, as a paper in the science journal Nature.

AlphaGo beats Lee Sedol, one of the world's top players, 4 games to 1 (Seoul)

In March 2016 in Seoul, South Korea, AlphaGo played a five-game match with no handicap against Lee Sedol, a 9-dan player who had won 18 world titles. AlphaGo won 4 games to 1. In Game 4, Lee Sedol won with a surprising move at move 78. DeepMind announced that it would donate the 1 million dollar prize to education and Go organizations and to UNICEF.

Is it true that “AlphaGo learned and got strong entirely on its own”?

It is often said that “AlphaGo got strong by playing against itself.” But according to the original papers, the AlphaGo that played humans from 2015 to May 2017 (the matches against Fan Hui, against Lee Sedol, and Master) first learned from human games as models, imitating where a person would play. Strengthening through self-play came after that. The first to learn without using a single human game was AlphaGo Zero, announced in October 2017.

“AlphaGo Zero”, Which Learns Only from Playing Itself, Without Human Game Records

In October 2017, DeepMind announced “AlphaGo Zero.” It uses no human game data. It is given only the rules of Go and learns by playing against itself over and over. After 72 hours of training, it won 100 games to 0 against the version that beat Lee Sedol. After about 40 days of training, it also won 89 of 100 games against Master, which had recorded 60 straight wins.

What we still don’t know:Even though Zero is described as “without Human Knowledge”, humans supplied the rules (where moves can be played and how to judge a win or loss) and the structure that treats the board as a 19×19 image (paper PDF page 22, the Domain Knowledge section).

“AlphaZero”, Which Learns Go, Chess and Shogi with One System, Wins 90 of 100 Games Against the Shogi World Championship Winner elmo

In December 2017, DeepMind announced “AlphaZero,” a development of the AlphaGo Zero method. Given only the rules, it learns Go, chess and shogi by self-play alone. In shogi, it played 100 games against “elmo” (made by Makoto Takizawa), the software that won the World Computer Shogi Championship that year (held in Kawasaki by the Computer Shogi Association), and recorded 90 wins, 8 losses and 2 draws (with one minute per move). The paper explains shogi as “shogi (Japanese chess)”.

Secret AlphaGo “Master” Wins 60 Straight Games Against Top Professionals in Online Play

In January 2017, a player calling itself “Master” appeared on internet Go servers and went 60 wins and 0 losses against top players from around the world. DeepMind announced in a paper and a blog post that this player was a version of AlphaGo (AlphaGo Master). These were unofficial internet games, not official tournament games (DeepMind's explanation).

AlphaGo Beats the World's Number 1 Ranked Player Ke Jie 3–0; AlphaGo Retires from Matches

In May 2017, at the “Future of Go Summit” held in Wuzhen, China, AlphaGo played a three-game match against Ke Jie, 9-dan, then ranked number 1 in the world, and won 3–0. Games in which a human was paired with AlphaGo, and a game against a team of five Chinese professional players, were also played. DeepMind announced that this was AlphaGo's final match and that it would move its research from Go to other challenges.

DeepMind's AlphaStar Beats Professionals at the Strategy Game “StarCraft II” (2019) — But at First It Saw the Whole Map Without Moving the Screen

In January 2019, DeepMind announced “AlphaStar”, an AI that plays the real-time strategy game “StarCraft II.” In a test match on December 19, 2018, it beat the professional player MaNa 5–0, and before that it had also beaten TLO. According to the official blog, in these matches AlphaStar did not move a screen (camera) the way a human does but played by directly taking in the visible units across the whole map. A version that moves the screen like a human was also built, but a prototype trained for only 7 days lost to MaNa in a public match.

OpenAI Five Beats the World Champion Team at the Team Game “Dota 2” (2019)

On April 13, 2019, OpenAI's AI “OpenAI Five” beat the world champion team OG at the team game “Dota 2.” Networks with the same structure each control one of the team's five characters (heroes). The paper reports that it was trained with reinforcement learning through self-play, continued for 10 months with large-scale computing.

Results of games between professional players and shogi software (2007–2017): ten years in which wins and losses switched

In public games between professional players and computers, the professional (Ryuo Akira Watanabe) won in 2007. In 2010 the software “Akara 2010” beat Ichiyo Shimizu, the Women's Ōshō title holder. In the Den-O-sen, Kunio Yonenaga, Eternal Kisei (a retired player), lost in 2012. The 5-versus-5 of 2013 was 1 win, 3 losses and 1 draw for the professionals, and 2014 was 1 win and 4 losses, but in 2015 the professionals came out ahead with 3 wins and 2 losses. In the two-game matches of 2016 and 2017, one player against one, the software (Ponanza) won both games in each of two years in a row.

The shogi program Bonanza, which learns from professional game records, wins a world championship and the next year plays Ryuo Akira Watanabe in a public game

The shogi program Bonanza was developed by Kunihito Hoki, a chemistry researcher, and released free of charge in June 2005. Unlike many earlier shogi programs, it used “full-width search” (translated), which reads out every move allowed by the rules. It also adopted “machine learning” (translated), training the way positions are evaluated so as to play the same moves as professionals' game records. The author has said that he applied the idea from his research on controlling chemical reactions to learning position evaluation. In May 2006, at its first entry in the 16th World Computer Shogi Championship, it won the round robin of the 8 finalists with 6 wins and 1 loss. On March 21, 2007, it played a public game against Ryuo (Dragon King title holder) Akira Watanabe, with 2 hours of thinking time each. Bonanza was also said to have the upper hand in the middlegame, but the Ryuo won in 112 moves. Later its method (and in 2009 the inside of the program) was made public, and shogi programs based on Bonanza, such as Bonkras, were born.

Prediction: computer shogi winning more games than it loses against top professionals “achievable by 2015” (translated) (2007, Hitoshi Matsubara)

In March 2007, right after Ryuo (Dragon King) Akira Watanabe beat Bonanza, the researcher Hitoshi Matsubara wrote in a commentary that a computer would “win just once” (translated) against a top professional within a few years, and that it would take somewhat longer for it to “win more than it loses” (translated), but that this could be achieved by 2015. In the same text he wrote that computer Go was at below amateur 1-dan level and that it would “still take quite a while” (translated) before it beat a top professional.

What happened next:In the 2nd Shogi Den-O-sen in 2013 the active professionals' side finished with 1 win, 3 losses and 1 draw, and in the 3rd in 2014 with 1 win and 4 losses (in the 2015 FINAL the professionals' side won 3 and lost 2). In October 2015 the Information Processing Society of Japan declared the project finished, based on “the analysis result that as of 2015 [computer shogi] has caught up with top professional players” (translated) (according to the declaration, “caught up” means “if many games are played, it is statistically likely to win more than it loses” (translated)). But the same declaration also states that “a match between a top professional player and computer shogi has not taken place” (translated). A game against the Meijin took place in 2017, when the Meijin, Amahiko Sato, lost two games in a row to PONANZA.

1st Shogi Den-O-sen: Kunio Yonenaga, Eternal Kisei, loses in 113 moves to the shogi software “Bonkras”

On January 14, 2012, at the Shogi Kaikan (Shogi Hall) in Tokyo, Kunio Yonenaga, president of the Japan Shogi Association and a retired professional holding the lifetime Kisei title (Eternal Kisei), played the shogi software “Bonkras”, developed by Hidenori Ito of Fujitsu Laboratories, and Bonkras won in 113 moves. Bonkras is a version of the publicly available shogi software Bonanza, improved to search in parallel on several computers; that day it read about 18 million moves per second on 6 servers. The name is a coined word combining Bonanza and “cluster”, which means a parallel computer (a system that calculates on several computers at the same time) (as explained in Fujitsu Laboratories' announcement).

At the 2nd Shogi Den-O-sen (2013) the professionals' side finishes with 1 win, 3 losses and 1 draw. In the final game, “GPS Shogi”, which used about 680 computers at the University of Tokyo, beats Class A 8-dan Hiroyuki Miura

The 2nd Shogi Den-O-sen in 2013 was a team match between 5 active professional players and 5 shogi programs. Before the games the forecast was “favoring the professionals” (translated), but it ended with 1 win, 3 losses and 1 draw for the professionals (Japan Shogi Association column). On April 20, in the final 5th game at the Shogi Kaikan (Shogi Hall) in Tokyo, “GPS Shogi”, developed mainly in a game programming seminar of faculty members and students of the University of Tokyo Graduate School, played 8-dan Hiroyuki Miura and won in 102 moves. GPS Shogi did distributed processing on about 680 computers at the University of Tokyo's Komaba campus and read about 270 million moves per second (University of Tokyo article).

The Information Processing Society of Japan declares an end to its project, saying computer shogi has “caught up with top professional players”

From 2010 the Information Processing Society of Japan (IPSJ) had been running a project aiming to achieve “computer shogi that beats top professional players” (note 1). In October 2015 it declared the project finished, saying “analysis results show that it has caught up with top professional shogi players as of 2015”. “Caught up” did not mean winning every time; it was explained to mean that, over many games, the computer is likely to come out ahead. At that point, however, a direct match against a top professional had not yet taken place.

What happened next:About a year and a half later, on April 1 and May 20, 2017, the shogi software PONANZA beat Amahiko Sato, the Meijin (the top title holder; he played as Eiō), in both games. The organizers ended the Denōsen matches with this contest.

Shogi Software PONANZA Beats Meijin Amahiko Sato Twice in a Row; the Den-O-sen Ends

In 2017, in the “Den-O-sen”, a series of games between professional shogi players and software, the shogi software PONANZA (made by Issei Yamamoto and Akira Shimoyama) beat Meijin (the top shogi title holder) Amahiko Sato in both games (game 1 lasted 71 moves, game 2 lasted 94 moves). Sato played as the winner of the Eio-sen tournament while still holding the Meijin title. It was the first time a title holder had played in the “Den-O-sen.” The organizers ended the “Den-O-sen,” which had run for six years from 2012, with this match.

Handicap stones between Go professionals and software fall from 5 to 0 in 5 years (Japan's Denseisen, 2012–2017)

For five years from June 2012, the University of Electro-Communications and the Nihon Ki-in (Japan Go Association) held the “Denseisen” (translated), official games between professional Go players and Go software, every year to measure the software's strength. At first the software was given a handicap, stones placed on the board in advance. It was 5 and 4 stones in 2012, 4 stones in 2013–2015 (3 stones also in 2015), and 3 stones in 2016. In December 2016 the organizers said that “the format of regular handicap-game matches against professionals has served its purpose” (translated) (a handicap game is one played with handicap stones placed first), and made the final round in March 2017 an even game with no handicap. In that round Ryo Ichiriki, Ryusei (a title), lost to two programs, DeepZenGo and “Zetsugei” (translated).

Japan's Go AI project “DeepZenGo” (2016–2018). It beat Oza title holder Yuta Iyama in 2017

The “DeepZenGo” project began in March 2016, with support from Dwango, to strengthen “Zen”, the Go software of the team of Hideki Kato and Yoji Ojima. In November 2016 it played Honorary Meijin Chikun Cho with no handicap and lost, winning 1 and losing 2. At the “World Go Championship” in March 2017 (hosted by the Nihon Ki-in), it played top players from Japan, China and South Korea, finished third with 1 win and 2 losses, and beat Japan's representative, Oza title holder Yuta Iyama. The project, which the Nihon Ki-in called “Japan’s Go AI development project” (translated), ended with a game in April 2018.

The First AlphaFold (CASP13, 2018): A Convolutional Network Predicts Distances Between Amino Acids and Builds the Shape — Its Inside Is Completely Different from the 2020 AlphaFold2

A protein's three-dimensional structure is decided when a chain of linked amino acids folds up. At the 13th CASP (2018, with about 100 research groups taking part), a protein structure prediction contest held every two years, DeepMind of London entered AlphaFold under the team name A7D. This first AlphaFold (now called AlphaFold1) used a CNN (convolutional neural network), the same kind used in image recognition. From a table lining up the sequences of similar proteins (a multiple sequence alignment), it predicted a map of probabilities for the distance between each pair of amino acids. From that map it built a scoring formula (a potential) and assembled the shape by adjusting it little by little with gradient descent. In the category that predicts without using similar structures as templates (FM), its total score from the judges was 68.3, far above the second place's 48.2. However, it reached high accuracy (GDT_TS of 70 or more) on only 11 of 43 targets. According to the Nobel Committee's explanation, AlphaFold2 at CASP14 two years later (2020) was rebuilt. It dropped convolution and used the Transformer's attention mechanism, and it changed to output the positions of atoms directly. The score (GDT) on hard problems rose from about 60% to about 90%.

Oxford University study: “47 percent of total US employment is in the high risk category” (risk of being replaced by computers; 2013)

In September 2013, Carl Benedikt Frey and Osborne of the University of Oxford published a paper estimating the probability that computers would replace workers in 702 occupations in the United States. It says that 47% of U.S. employment falls “in the high risk category”. The figure was later taken as “a prediction that jobs will disappear” and became a target of checking (for example, the title of a 2022 ITIF article, “Oops: The Predicted 47 Percent of Job Loss From AI Didn’t Happen”, from a separate record this site keeps, checking predictions about work against what happened). The details and the later checking are recorded in a separate study, apart from this site, on checking predictions about work in the age of AI.

Nomura Research Institute: “around 49% of Japan’s working population could be replaced technologically by artificial intelligence and the like” (2015)

On December 2, 2015, Nomura Research Institute (NRI) announced joint research with Osborne and Carl Benedikt Frey of the University of Oxford. For 601 occupations in Japan, it estimated the probability of being replaced by AI or robots. The estimate was that in 10 to 20 years, occupations employing about 49% of Japan's working population could technically be replaced. However, the announcement notes that this is technical replaceability, and that whether replacement actually happens depends heavily on social factors such as labor supply and demand.

“Torobo-kun” (the Todai Robot): What Happened in 2016 to the Goal of Earning a Passing Score on the University of Tokyo Entrance Exam by 2021?

The National Institute of Informatics (NII) project “Can a Robot Get into the University of Tokyo?” (Torobo-kun, the Todai Robot) began in 2011. A 2012 article stated its goals: “scoring high on the university entrance Center Test by 2016” (translated) and “achieving a passing-level score on the University of Tokyo second-stage exam by 2021” (translated). In its November 2016 announcement, it recorded 525 points (deviation score 57.1) on a mock Center Test and a deviation score of 76.2 on the University of Tokyo mock exam in mathematics (science track). At a briefing held the same day, Noriko Arai said that the effort to solve the Center Test the way a person does would end this fiscal year. The reason was that “understanding meaning” (translated) is hard for AI.

What happened next:In the 2016 mock exams it recorded 525 points and a deviation score of 57.1 on the Center Test (by Benesse's assessment, 535 universities had an 80% or higher chance of accepting it) and a deviation score of 76.2 on the University of Tokyo mock exam in mathematics (science track). At the briefing in November 2016, it was announced that the effort to solve the Center Test the way a person does would be frozen this fiscal year. According to that day's news coverage, NII said it would next aim to develop the areas where AI is strong and apply them in industry. The challenge in this form ended without waiting for the target year (2021) for passing-level scores in all subjects of the University of Tokyo second-stage exam.

What we still don’t know:The 2016 view was that reading (understanding meaning) is hard for AI. How generative AI (the fourth boom) later changed that view is a candidate topic for u11 and u12.

“A novel written by AI passed the first round of a literary prize”: how much did the AI really write?

On March 21, 2016, a research team that has AI write fiction (Future University Hakodate, the University of Tokyo and others) announced four works it had entered in the 3rd Nikkei “Shinichi Hoshi Award”. Some passed the first round of screening and were rejected in the second round. The award also accepted entries from non-humans (AI and the like). According to an article later written by Hitoshi Matsubara, who led the effort, the storyline (a “scenario” including the twist ending) was created by a member of Satoshi Sato's laboratory at Nagoya University. The program used sentence parts prepared by humans to assemble Japanese text following that storyline.

What we still don’t know:There is still no clear line for what counts as written by AI (how much human editing makes it a human work). Matsubara's article also touches on the argument that if AI wrote 100%, it would not be protected under current copyright law.

Hinton: “people should stop training radiologists now”, “within 5 years deep learning is going to do better than radiologists” (2016)

In 2016, Geoffrey Hinton spoke at a conference in Toronto about radiologists (doctors who read images such as X-rays and CT scans to find disease). His main points were that training radiologists should stop right now, and that it is obvious that within 5 years (it might be 10) deep learning will do better. An 84-second video recording the occasion was published, and it has been quoted repeatedly in later articles (AuntMinnie 2025, Fortune 2026 and others).

What happened next:In the US residency recruitment (the official NRMP tables), positions in radiology (diagnostic) did not fall. Positions starting in the second year went from 982 in 2016 to 1,057 in 2025, and positions starting in the first year went from 151 in 2016 to 151 in 2025. Almost all the positions in 2025 were filled (1,042 of the 1,057 second-year positions).

What we still don’t know:How AI's ability to read images relates to the whole work of a radiologist (talking with patients and other doctors, deciding on tests, and so on) was not well understood in 2016. Even now, how this job will change with AI is still being studied.

Ford announces a plan to run large numbers of fully self-driving cars with no steering wheel or pedals in ride-hailing services in 2021 (2016)

In August 2016, the automaker Ford announced (in Palo Alto) a plan to run large numbers of “Level 4” vehicles (fully self-driving in limited areas) in ride-hailing and ride-sharing services in 2021. The plan was for the cars to have no steering wheel, no accelerator and no brake. Its president said “The next decade will be defined by automation of the automobile”. Around the same time, Japan's national plan and carmakers also announced self-driving plans targeting around 2020 (→ A national plan: self-driving on expres…).

What happened next:A service of this kind did not begin in 2021. On October 26, 2022, Ford announced that it had judged that putting Level 4 to profitable use at scale in the industry would come later than first expected. It booked a loss of 2.7 billion dollars on its investment in the self-driving company Argo AI, and shifted its development funds to Level 2+ and Level 3 (driver-assistance technologies in which a person watches over the driving).

A national plan: self-driving on expressways and driverless mobility services in limited areas, both “realized by 2020”

In its national strategy for self-driving, the “Public-Private ITS Initiatives and Roadmap” (revised every year since 2014), the Japanese government set the 2020 Olympic and Paralympic Games as its target date. Based on the Prime Minister's remarks of November 2015, the 2016 edition set two goals. (1) Start studying the rules so that cars driven by the system on expressways (Level 3) can be sold on the market around 2020. (2) Realize, by 2020, mobility services with no driver on board in limited areas such as depopulated regions.

What happened next:(1) On November 11, 2020, the Ministry of Land, Infrastructure, Transport and Tourism granted type designation to the Honda Legend, the world's first Level 3 car driven by the system (under set conditions such as expressway traffic jams). (2) For Level 4 mobility services with no driver, the vehicle was approved on March 30, 2023, and operation began on May 21, 2023 in Eiheiji, Fukui Prefecture, the first in Japan (about 3 years after the 2020 target).

Even a government white paper carries the talk of the singularity: “the technological singularity will be reached around 2045” (2016)

The Ministry of Internal Affairs and Communications' 2016 White Paper on Information and Communications, in the first section of the special feature in Part 1, “The Evolution of ICT”, wrote that beyond deep learning, a “technological singularity”, a boundary at which ICT surpasses human intelligence, is predicted to arrive. It also noted that some researchers point out that the technological singularity will be reached around 2045 and that rebuilding social systems will become unavoidable. It is a record that, at the peak of the third boom, such a grand outlook appeared even in a government document.

What we still don’t know:The white paper does not say what standard is used to measure “exceeds human intelligence”, or whether the year 2045 has any basis (it also names no researchers). That year has not yet arrived, and the outcome is not known (a candidate prediction awaiting an answer, handled in u12 and u13).

How the “2045 problem” spread in Japan — a 2007 translation → a 2013 paperback, “2045 problem” → in 2015 the first mention of the “singularity” in the Diet. Along the way, “one billion times more powerful than all human intelligence today” shrank to “surpasses the human brain” (translated)

Kurzweil's 2005 book came out in Japanese in January 2007 from Japan Broadcast Publishing as “The Birth of the Post-Human: When Computers Surpass Human Intelligence” (translated title). In January 2013 Takuya Matsuda's paperback “The 2045 Problem: The Day Computers Surpass Humanity” (translated title) was published (in the National Diet Library, it is the oldest of the books and articles with “2045 problem” in the title). In the May 2014 issue of the Journal of the Japanese Society for Artificial Intelligence, Matsuda explained that, according to Kurzweil, a computer passes the Turing test in 2029 and the technological singularity, when computer intelligence surpasses all of humanity, comes in 2045. At the House of Councillors Committee on Internal Affairs and Communications on March 24, 2015, a member referred to the NHK documentary “Next World” (translated title) and asked: “It is said that this turning point, when computers surpass the human brain, will arrive in 2045” (translated), “It is only thirty years away” (translated); the Minister for Internal Affairs and Communications answered that she had also seen the program. This is the first time the “singularity” appears in the Diet record.

What we still don’t know:Who first used the phrase “2045 problem” (translated) is not known (all we can say is that the oldest in a National Diet Library title search is Matsuda's 2013 paperback; blogs and newspapers were not checked).

US government report “Preparing for the Future of Artificial Intelligence”: for machines to show broad intelligence is “very unlikely ... comparable to or exceeding that of humans in the next 20 years”

In October 2016, the National Science and Technology Council (NSTC) of the US Executive Office of the President released the report “Preparing for the Future of Artificial Intelligence.” In 2016 it held five public workshops, and 161 comments came in from a public request for information. The report sets out 23 recommendations, and a national AI research and development strategic plan was released at the same time. The cover letter says that machines will increasingly exceed humans at particular tasks, but that for them to show broad intelligence is “very unlikely ... comparable to or exceeding that of humans in the next 20 years”. The main text also says that private-sector experts broadly agree that General AI (AI that can do any intellectual task like a human) “General AI will not be achieved for at least decades”, and that the committee agrees. It also says that concern about far-future superintelligence should have little impact on current policy.

What happened next:The target year (2036) has not yet arrived (as of September 2026). Since then, in 2023–2025, forecasts of around 2026–2030 have appeared from company representatives and company papers (records → OpenAI's CEO Altman: It is possible tha…, → Anthropic's CEO Amodei says, of powerfu… and → Google DeepMind paper: highly uncertain…). The results of expert surveys vary greatly depending on how the question is asked (record → A survey of 2,778 AI researchers in aut…).

What we still don’t know:The report itself notes that society would benefit from “accurate and timely methods for monitoring and forecasting AI developments”, and a note admits that this may be difficult. It then recommends continuing to survey experts' views regularly (pdf page 33).

Criticism of the cover design of the Japanese Society for Artificial Intelligence's journal prompts the society to publish its view (2014)

In January 2014, the Japanese Society for Artificial Intelligence renamed its journal “Artificial Intelligence” and redesigned the cover. The cover, chosen by a vote from about 100 proposals gathered so that the journal would be read more widely, showed a female-shaped robot cleaning with a broom. Criticism followed one after another on the internet, asking “isn’t this sexist?” (translated). The society's editorial committee said “there was no intention to discriminate against women” (translated), but apologized for giving “the impression that a woman is cleaning” (translated), saying “we were not sufficiently considerate as a highly public academic body, and we deeply reflect on this” (translated). In the March issue it ran a small special feature that organized this very debate as a subject of study.

The Japanese Society for Artificial Intelligence Publishes Its “Ethical Guidelines”; the Last, Article 9, Is “Requirement that AI comply with ethics” (a Clause That Asks AI Itself to Comply)

On February 28, 2017, the Japanese Society for Artificial Intelligence (JSAI) published “Ethical Guidelines” for researchers. Eight items (contribution to humanity, compliance with laws, privacy, fairness, safety, acting with integrity, responsibility to society, and dialogue with society) are what JSAI members are expected to keep in mind. The last, Article 9, is addressed to AI itself and says that to become a member of society, AI must follow these guidelines just as JSAI members do. Yutaka Matsuo, chair of the ethics committee, has written that one of the reasons the ethics committee was founded was the problem of the cover of the society's journal in 2014.

What we still don’t know:Article 9 supposes a future in which AI becomes “a member of society or something equivalent” (translated). Even the authors do not know when or in what form such AI will appear (it is written as a condition).

Microsoft's chat AI “Tay” is exploited and shut down just after its release on Twitter (USA, 2016)

In March 2016, Microsoft released “Tay”, an AI that talks on Twitter, aimed at 18- to 24-year-olds in the United States. It was designed to learn from conversations with people, but within 24 hours of release some people coordinated to exploit its weakness. Tay began posting highly inappropriate remarks and images, and the company took it offline. Microsoft published an apology and lessons learned on its official blog, and also mentions the experience with XiaoIce in China and in Japan.

What we still don’t know:The company itself says that it is impossible to predict in advance every way an AI that learns from conversations might be abused (a problem of society, not only of technology).

Japan Microsoft's chat AI “Rinna” appears on LINE (2015)

In summer 2015, Japan Microsoft released “Rinna,” an AI that chats like a friend on LINE. It combined deep learning and cloud machine learning developed with the search engine Bing. It aimed not at helping with work but at an emotional connection through small talk (in the company's words, “prioritizing EQ (emotional quotient, the heart’s intelligence) over IQ (the brain’s intelligence quotient)” (translated)). At first it drew attention as an account that did not reveal its true identity, posing as a “high-school girl” (translated), and in December the number of LINE friends exceeded 1.85 million.

AI and robotics researchers issue an open letter calling for “a ban on offensive autonomous weapons beyond meaningful human control”

On July 28, 2015, timed with the opening of IJCAI 2015, an international conference on artificial intelligence, AI and robotics researchers published an open letter. The letter defined autonomous weapons as “Autonomous weapons select and engage targets without human intervention”. It wrote that, setting aside the legal questions, they are technically feasible “within years, not decades”, and are described as “the third revolution in warfare”, after gunpowder and nuclear arms. It said a military AI arms race would not benefit humanity, and called for a ban on offensive autonomous weapons beyond “meaningful human control”. The signatures numbered 34,378 on the page of the Future of Life Institute (FLI), which hosts the letter (the number seen on September 23, 2026). Talks among countries had begun the year before, in May 2014, as an informal meeting of experts under the Convention on Certain Conventional Weapons (CCW), and Japan took part. In December 2023, the United Nations General Assembly adopted a resolution on lethal autonomous weapons systems, stressing the need for the international community to take urgent action.

What we still don’t know:How far the letter's outlook that such weapons would be usable “within years, not decades,” legal issues aside, turned out to be right (whether fully autonomous weapons have actually been used) has not been confirmed in the sources opened for this site. The UN General Assembly resolution is also not a ban; it is a stage of deciding that the international community needs to act urgently and of gathering the views of countries.

2016: at the G7 Kagawa-Takamatsu ICT Ministers' Meeting, Japan proposes discussing “Eight Principles for AI Research and Development”

On April 29–30, 2016, the G7 ICT Ministers' Meeting was held in Takamatsu, Kagawa Prefecture, with Japan's Minister of Internal Affairs and Communications as chair. Japan proposed creating principles for AI research and development that could serve as an international reference, and presented eight as a starting draft: “transparency”, “user assistance”, “controllability”, “security”, “safety”, “privacy”, “ethics” and “accountability”. The countries agreed to move forward with international discussion.

The Asilomar AI Principles (2017): Researchers Discuss and Draft Principles for “Beneficial AI”

In 2017, at the conference “Beneficial AI 2017” held at Asilomar, the Future of Life Institute (FLI) acted as organizer, and AI researchers and others drafted the “Asilomar AI Principles”. The 23 principles cover research issues (for example, that the goal of AI research should be to create “not undirected intelligence, but beneficial intelligence”), ethics and values such as safety and transparency, and longer-term issues (for example, not assuming upper limits on the capabilities of future AI). Only principles that more than 90% of the conference participants agreed with were kept.

What we still don’t know:Principle 19, “Capability Caution”, says to avoid strong assumptions about the upper limits of future AI capabilities, because there is no consensus on them (it turns the present state of not knowing into a principle).

The Japanese Government's “Social Principles of Human-Centric AI” (2019)

On March 29, 2019, the Japanese government adopted the “Social Principles of Human-Centric AI.” It set out three ideals for the society it aims for: “a society in which human dignity is respected” (translated), “a society in which people with diverse backgrounds can pursue diverse forms of happiness” (translated) and “a sustainable society” (translated). It also presented seven principles that society should follow (human-centric, education and literacy, privacy protection, ensuring security, fair competition, fairness, accountability and transparency, and innovation).

Yutaka Matsuo's “Will Artificial Intelligence Surpass Humans?”: a book that divided the history of AI into chapters on the “first, second and third AI booms”

In March 2015, the book “Will Artificial Intelligence Surpass Humans? What Lies Beyond Deep Learning” (KADOKAWA) by Yutaka Matsuo of the University of Tokyo was published. In its table of contents, Chapter 2 is the era of “inference” (translated) and “search” (translated) (the first AI boom), Chapter 3 is that adding “knowledge” (translated) makes AI smarter (the second AI boom), and Chapters 4 and 5 cover “machine learning” (translated) and deep learning (the third AI boom). In the following year, 2016, the Japanese government's White Paper on Information and Communications noted in a footnote that its section on the history of AI was written with reference to this book and others.

What we still don’t know:The book is sometimes called “the book that popularized counting the booms as first to third,” but this site has not looked into earlier uses or citation counts (the way of counting itself is handled in another record that looks into how booms are counted). What was confirmed is that the government's 2016 white paper states that it used this book as a reference for its history section.

The government's White Paper on Information and Communications (2016) explains the history of AI with a chart of the “first, second and third booms” and the winters

The Ministry of Internal Affairs and Communications' 2016 (Heisei 28) White Paper on Information and Communications took up AI in a special feature. In Chapter 4, Section 2, “History of Artificial Intelligence (AI) Research”, it used Figure 4-2-1-5, “History of Artificial Intelligence (AI)”, and explained that AI research since the 1950s has seen “alternating booms and winters” (translated), and that we are now in the third boom. It puts the first boom in the late 1950s to the 1960s (search and inference), the second in the 1980s (knowledge representation; Japan's Fifth Generation Computer also belongs here) with a winter again from around 1995, and the third from the 2000s (machine learning).

SoftBank unveils the humanoid robot “Pepper” (2014) and puts it on general sale (2015)

On June 5, 2014, SoftBank Mobile and the French company Aldebaran announced “Pepper”, a humanoid robot that reads emotions from people's facial expressions and tone of voice. From the next day it was placed in SoftBank stores. When general sales began on June 20, 2015, the first 1,000 units reached the planned number within one minute of the start of orders. At that time the company announced that Pepper had come to “have emotions of its own” (translated).

What we still don’t know:What the company calls “having emotions” (translated) is a mechanism in which a program runs variables modeled on how human emotions work (in the company's words, “simulated brain secretions” (translated)). Whether the robot feels emotions in the same sense that people do is not known (it is also the question of what an emotion is).

The Turing Award Goes to the Three Who Grew Deep Learning (Bengio, Hinton and LeCun) (for 2018, announced in March 2019)

On March 27, 2019, the Association for Computing Machinery (ACM) announced that the 2018 Turing Award would go to Yoshua Bengio, Geoffrey Hinton and Yann LeCun. The reason was major advances in concepts and techniques that made deep learning a critical component of computing. The announcement said that in the early 2000s the three were among the few researchers who kept working on this method, that their efforts to regain interest were at first met with skepticism, but that it has now become mainstream in the field.

In 2019, a Japanese Research Firm Places Artificial Intelligence in the “Trough of Disillusionment”

In “Hype Cycle for Technologies in Japan: 2019”, published by Gartner Japan, artificial intelligence was placed in the “Trough of Disillusionment” (the period when disappointment spreads after expectations have grown too high). It explains that this was because trial uses had led people to face reality instead of expectations.

What we still don’t know:The shape of the Hype Cycle curve is the research firm's own view and has not been confirmed as a law.

[Japan] In the Diet and in book titles, the second peak is in 1990 (about 3 years later than in English) and the third in 2019; rising again from 2023

From the Diet’s records of speeches and from Japanese book titles, we counted how many times “artificial intelligence” and “AI” (recounted with parts of other words removed, so that it means AI) appear. In book titles, “artificial intelligence” is highest in 1990 (37 books). In the Diet, “artificial intelligence” peaks in 1985 (7 mentions) and 1990 (9), and was almost 0 from 1993 to 2014. In the Diet, “AI” meaning AI went 11 in 2015 → 593 in 2019 → down to 336 in 2021 → 761 in 2023 → 2,424 in 2025 (the year the AI law was made). “Generative AI” was 0 until 2022 and reached 172 in 2023.

What we still don’t know:The reason the counts fell in 2020–2021 cannot be pinned down to one cause (it overlaps with the period of the infectious disease epidemic).