Theories of Deep Learning

This field has been blessed with exponential empirical success in the form of architectures and algorithms that simply worked through scaling, while the theory lagged behind1. Although in the past few years, the “gap“ seems to be diminishing, and we are getting multiple theories for different aspects of deep learning. This essay would be a simple high-level overview of all the theories I’ve come across.NOTE: These are mathematically dense frameworks which either provide a language for formalizing the field itself or aim to explain the observed phenomena. This essay is my personal understanding of the theories, after a lot of back and forth with GPT. Hence, kindly take everything with a grain of salt, and kindly mention any mistakes and misunderstandings in the comments !NOTE: I tend to use the words “theory“ and “framework“ inter-changeably in this essay. If that seems annoying or unnecessary, or is mis-leading in some manner, please DM or comment as well !Before we dive into the actual theories and frameworks, I’d like to sketch out the different “sub-domains” in deep learning here, since we shall see that each sub-domain has a different framework which are largely independent of each other2. We have, broadly :Architecture theory : Or the theoretical foundations which includes all the model architectures, and provides for a common language for us to express popular architectures like transformers, RNNs and CNNs as specific instances of a more general framework.Optimization theory : as the name suggests, theories here try to explain and predict how different optimizers behave based on model architectures and data distributions, in order to find more efficient and better optimizers like Adam, AdamW or Muon. Functional Theory : theories in this domain focus on the neural network as a whole simulating a function, and try to understand it’s behaviours from that perspective. Frameworks here aim to explain how and why models generalize and understand phenomena like grokking/double descent. These aren’t formal categories of the field of deep learning per se, as one cannot be isolated from the other, but the papers and their authors have dealt with only one single aspect from above at a time3, hence I’ve mentioned them here.Let us being with,Categorical Deep Learning for architecture theory.

Paper : Position, Categorical Deep Learning is an Algebraic Theory of All ArchitecturesAuthor(s) : Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, Petar VeličkovićI originally came across this paper while exploring the generalizations of deep learning through Geometric Deep Learning, to then Topological Deep Learning4 before learning about the Position paper. While the first two fields focus on generalizing the data distributions with non-trivial internal structures by introducing certain constraints and modifications to model architectures, in the CDL paper, the authors provide a general framework for describing all model architectures and building a bridge between this theory and implementation through categorical concepts, which can also be implemented in functional programming concepts. In their own words (emphasis added by me),… lack of a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building such a bridge, we propose to apply category theory—precisely, the universal algebra of monads valued in a 2-category of parametric maps—as a single theory elegantly subsuming both of these flavours of neural network design.As I mentioned earlier, the authors focus on generalizing all neural network architectures, by utilizing tools from category theory5 in order to cover all the operations seen in contemporary deep learning research, while focusing on implementations through grounding it in functional programming. (I will be exploring the implementations in upcoming posts!). In brief: the authors utilize the compositionality of monads to represent many things, like sequential computation or recurrence. A 1-category has morphisms or transformations between two objects (mathematical objects, like vector spaces) and a 2-category has morphisms over morphisms, which can help represent high-order operations like parameter-sharing, different variations of the same operation or operations like automatic differentiation. Hence, the authors propose an algebra of monads in a 2-category space as the mathematical framework for deep learning. Modular Duality for Optimization theory.Paper : Modular Duality in Deep LearningAuthor(s) : Jeremy Bernstein, Laker NewhouseThe optimization of deep learning models was generally done by using a standard algorithm like stochastic gradient descent or the Adam optimizer. While these algorithms worked well in practice, a lack of foundational framework meant researchers were compromising on efficiency and speed, as the main method of research was empirical experimentation.

The authors sought to propose a framework meant to shed light on the optimization procedure of training a neural network, while also deriving faster algorithms for the same. At the heart of this framework is the mathematical concept of a norm, which is a measure of any notion of size for a matrix (or, more generally speaking, a vector space). The advantage of taking norms into consideration while training a neural network comes from the primary fact that we want the optimization procedure to be well-behaved and efficient, which means the updates to parameters must not cause unexpectedly large changes to the output6. This goal naturally leads us to ask questions about the norm of the spaces we are concerned with (namely the Input, Output and Parameter spaces). The authors also point out that norms differ from operation to operations and each induces a unique geometry, as a self-attention layer would reside in a different normed space then a feed-forward network. Hence, the primary goal would also be to understand which norms to use in order to optimize each layer and perform weight updates accordingly7. Here the Modular norm is the composition of all the norms of a given neural network architecture (with all it’s layers), constructed recursively, and which helps us derive a better geometry-aware optimization algorithm. The name comes from the fact that each network layer is considered a separate module which contribute their own norms to the overall structure of the network, from which a modular norm could be constructed. And finally, we get to the Modular Dualization paper. This paper attempts to point out an error which deep learning researchers had been ignoring, and solving it, while simultaneously building this framework (as a deep learning library!). In short, the updates and the gradient itself, reside in different vector spaces, hence subtracting one from the other does not make much sense, and leads to an inefficient training run. Hence, the authors propose creating a Duality Map which transforms the update matrices into the correct parameter space, before subtracting it from the weight matrix. Here, our choice of a norm determines what our duality map would be, since our weight updates would then reside in the norm’s dual space 8. Using this information, the authors propose that we can find faster and more efficient optimizing algorithms for essentially all neural network architectures, including ones which we will create in the future.

A Theory of Generalization in deep learning.Paper : A Theory of Generalization in Deep LearningAuthor(s) : Elon Litman, Gabe GuoThe main objective of training any statistical model was to learn the inner distribution of the dataset, utilizing various parametric and non-parametric functions, while understanding the famous bias-variance tradeoff. While the trade-off worked well in classical machine learning problems, various phenomena during the training of deeper models violated the assumptions made using the bias-variance tradeoff, which called for the need of a better theory for their generalization capabilities. The observed phenomena in question are: Benign overfitting, Double descent, Implicit bias and Grokking, which seem to puzzle practitioners. The authors start by simply shifting the space which we observe. They insist that we should, instead studying the Parameter space (observing how the weight matrices change), we should focus more on Output space (or Prediction space), We propose a radical Vereinfachung[simplification]: abandoning the parameter space entirely. Instead, we analyze the network as a dynamical system strictly in output space, focusing on how predictions evolve and where error flowsThe authors ask us to study not just parameter dynamics, but rather study the trajectory of neural network as a function, and understand how it’s predictions change throughout training. This allows us to move away from trying to understand how billions of parameters change in the latent space to understanding how the model’s predictions change, and how they influence each other. The mathematical tool utilized by the authors is the eNTK : the empirical neural tangent kernel. Using this, the authors speculate training run dynamics based on the eigenvalues of the eNTK, and offer an unified explanation for the above mentioned phenomena. The main intuition comes from recognizing a signal channel and a reservoir9. The authors suggest that the Output/Predictions space contain two regions: the higher mobility signal channel and the lower mobility reservoir10. These abstractions help us provide viable explanations for training dynamics we generally observe:Benign overfitting. the important features are transferred into the signal channel space while the noise stays in the reservoir. This lets a model overfit to a dataset without contaminating the test accuracy, as the contributions of anything from the reservoir are vanishingly small.Double Descent. as the model size increases, so does it’s reservoir, which can now handle more noise.