Munich 1991: the Roots of the Current AI Boom

David Ha, Sakana AI Jürgen Schmidhuber, KAUST & IDSIA 18 June 2026 @hardmaru @SchmidhuberAI AI Blog

Munich 1991: the Roots of the Current AI Boom Preface by David Ha

When we look at the massive scale of today’s Artificial Intelligence boom, it is easy to forget that the foundations of this trillion-dollar industry were laid down over 30 years ago in Munich.

Today, the world's top tech companies are investing hundreds of billions into scaling up Large Language Models (LLMs) such as ChatGPT. Yet, outside of a few history buffs or old-school folks in the Machine Learning community, people might not realize that virtually every core building block of these modern systems was published in a span of just a few months back in 1991. Incredibly, they all emerged from a single lab at the Technical University Munich led by Jürgen Schmidhuber.

Before that year ended, his team had essentially mapped out the modern era of deep learning. They published the very first Transformer variant (see ChatGPT's "T"), introduced the concept of unsupervised pre-training (ChatGPT's "P"), and pioneered neural network distillation. They also introduced deep residual learning, the centerpiece of both LSTMs and ResNets, the most cited AI papers of the 20th and the 21st century, respectively. These four techniques power today's most advanced LLMs. Furthermore, they laid the early groundwork for generative adversarial networks, foundational for "Generative AI."

Jürgen’s contributions have deeply shaped my own thinking over the years, from my time at Google Brain to our recursive self-improvement (RSI) research we're currently pushing at Sakana AI. I am especially proud to have helped popularize World Models back in 2018, building directly on concepts his lab introduced in the 1990s.

It is amazing to see how well some of these ideas have stood the test of time, scaling up to be fully embraced by the global AI community!

For those interested in the real history of deep learning, Jürgen has put together a detailed timeline below of exactly how these seeds were planted in Munich in 1991.

David Ha, June 2026

Jürgen Schmidhuber's 1991 Timeline, with Annotated References

I am proud of the work my team did in 1991 in my home city when compute was millions of times more expensive than today [RAW], and of all the great people I worked with there and afterwards. Check out TU Munich's following key AI publications dated 3/26/1991—8/31/1991.

★ 26 March 1991: the first kind of Transformer (see the T in ChatGPT)—now called the unnormalized linear Transformer [ULTRA][FWP0-6][WHO10][DLH]: the predecessor of the normalized quadratic Transformer [TR1]. ULTRA is still important, also because of its efficiency: its computational costs scale linearly in input size, rather than quadratically.

★ 30 April 1991: Pre-Training for deep neural networks (NNs)—the P in ChatGPT [UN0][UN1][UN2][UN][DLH]. This enabled very deep learning [WHO5].

★ 30 April 1991: Neural network distillation—central to the famous 2025 DeepSeek "Sputnik" and other Large Language Models (LLMs) [UN0][UN1][UN2][WHO9][DLH].

★ 15 June 1991: deep residual learning with residual connections for very deep NNs [WHO11] (see Sepp Hochreiter's diploma thesis [VAN1]): the core ingredient of Long Short-Term Memory [LSTM1], the most cited AI of the 20th century, basis of the first LLMs in the 2010s (ELMO, ULMFiT).

The most-cited scientific article of the 21st century [MOST25-26] is also about deep residual learning, focusing on a variant of our LSTM-inspired deep residual Highway Net [HW1-25b] that was 10 times deeper than previous feedforward NNs [WHO11][DLH]. Deep residual learning is now being used in virtually all LLMs.

★ 31 August 1991: first peer-reviewed publication [GAN91] on generative & adversarial networks [GAN90-25] for neural world models [WM26,WM26b] trained through artificial curiosity & creativity—now controversially used for deepfakes and other applications of Generative AI [WHO8][DLH].

As of January 2026, the two most frequently cited papers of all time (with the most citations within 3 years—manuals excluded) are directly based on the work of 1991 [MOST26][MOST][MIR]. In 1991, however, it was already totally obvious that LLM-like NNs alone are not enough to achieve Artificial General Intelligence (AGI). No AGI without mastery of the real world [DLH]! That's why we started working on additional techniques required to achieve AGI, e.g., planning with adaptive world models [PLAN1-6][WM26,WM26b] created by artificial scientists [AC] (since 1990 at TU Munich), meta learning & recursive self-improvement (since 1987) [META1][META], and others [DLH][AIB].

Around the same time, Munich also was the origin of the first self-driving cars in traffic [AUT] (by Ernst Dickmanns's team), going up to 175 km/h. The city was truly the epicenter of AI. In the past 3 decades, however, most of commercial AI has shifted to the Pacific Rim, far away from Munich. How could that happen? Can anything be done about it? See [95-25] for answers!

See [WHO3-11] for the broader historical context [DLH] of the work published in 1991 [MIR]. I am still hoping that I may live to see our great field of Machine Learning realize my 1970s teenager vision of building something much smarter than myself, such that I can retire.

Jürgen Schmidhuber, June 2026

Acknowledgments

Thanks to several expert reviewers for useful comments. (Let us know if you can spot any remaining error.) The contents of this article may be used for educational and non-commercial purposes, including articles for Wikipedia and similar sites. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Annotated References

[95-25] J. Schmidhuber (AI Blog, 2025). 1995-2025: The Decline of Germany & Japan vs US & China. Can All-Purpose Robots Fuel a Comeback? In 1995, in terms of nominal gross domestic product (GDP), a combined Germany and Japan were almost 1:1 economically with a combined USA and China, according to IMF. Only 3 decades later, this ratio is now down to 1:5! Self-replicating AI-driven all-purpose robots may be the answer. Based on a 2024 F.A.Z. guest article.

[AC] J.  Schmidhuber (AI Blog, 2021, updated 2025). 3 decades of artificial curiosity & creativity. Schmidhuber's artificial scientists not only answer given questions but also invent new questions. They achieve curiosity through: (1990) the principle of generative adversarial networks, (1991) neural nets that maximise learning progress, (1995) neural nets that maximise information gain (optimally since 2011), (1997) adversarial design of surprising computational experiments, (2006) maximizing compression progress like scientists/artists/comedians do, (2011) PowerPlay... Since 2012: applications to real robots.

[AIB] J. Schmidhuber's AI Blog. With lessons on the history of AI & computing, e.g.: Who invented deep learning? Who invented backpropagation? Who invented convolutional neural networks? Who invented artificial neural networks? Who invented generative adversarial networks? Who invented Transformer neural networks? Who invented deep residual learning? Who invented neural knowledge distillation? Who invented the computer? Who invented the transistor? Who invented the integrated circuit? ...

[ATT] J. Schmidhuber (AI Blog, 2020, updated 2025). 30-year anniversary of end-to-end differentiable sequential neural attention. Plus goal-conditional reinforcement learning. Schmidhuber had both hard attention for foveas (1990) and soft attention in form of Transformers with linearized self-attention (1991-93).[FWP] Today, both types are very popular.

[AUT] J.  Schmidhuber (AI Blog, 2005). Highlights of robot car history. Around 1986, Ernst Dickmanns and his group at Univ. Bundeswehr Munich built the world's first real autonomous robot cars, using saccadic vision, probabilistic approaches such as Kalman filters, and parallel computers. By 1994, they were in highway traffic, at up to 180 km/h, automatically passing other cars.

[DLH] J. Schmidhuber. Annotated History of Modern AI and Deep Learning. Technical Report IDSIA-22-22, IDSIA, Switzerland, 2022, updated 2025. Preprint arXiv:2212.11279. Tweet.

[DLP] J. Schmidhuber. How 3 Turing awardees republished key methods and ideas whose creators they failed to credit. Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 14 Dec 2023, updated 2025. Tweet of 2023.

[DS1] DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

Preprint arXiv:2501.12948. See the popular DeepSeek tweet of Jan 2025.

[FWP] J.  Schmidhuber (AI Blog, 26 March 2021, updated 2025). 26 March 1991: Neural nets learn to program neural nets with fast weights—like Transformer variants. 2021: New stuff! See tweet of 2022.

[FWP0] J.  Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, 26 March 1991. PDF. First paper on neural fast weight programmers that separate storage and control: a slow net learns by gradient descent to compute weight changes of a fast net. The outer product-based version (Eq. 5) is now known as the unnormalized linear Transformer or the "Transformer with linearized self-attention."[ULTRA][FWP]

[FWP1] J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992. Based on [FWP0]. PDF. HTML. Pictures (German). See tweet of 2022 for 30-year anniversary.

[FWP2] J. Schmidhuber. Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam, pages 460-463. Springer, 1993. PDF. A recurrent extension of the unnormalized linear Transformer,[ULTRA] introducing the terminology of learning "internal spotlights of attention." First recurrent NN-based fast weight programmer using outer products to program weight matrices.

[FWP3a] I. Schlag, J. Schmidhuber. Learning to Reason with Third Order Tensor Products. Advances in Neural Information Processing Systems (N(eur)IPS), Montreal, 2018. Preprint: arXiv:1811.12143.