Matrix Calculus (for Machine Learning and Beyond)(Free PDF)

 

Machine learning is often described through algorithms, datasets, and programming frameworks. However, behind many of the most important techniques in modern machine learning lies a deeper mathematical foundation: calculus and linear algebra.

As machine-learning models become more sophisticated, ordinary single-variable calculus is often not enough to understand how derivatives behave when inputs and outputs are vectors, matrices, functions, or even solutions to differential equations.

Matrix Calculus (for Machine Learning and Beyond) by Paige Bright, Alan Edelman, and Steven G. Johnson is a set of lecture notes based on MIT's matrix-calculus course. The work was published on arXiv in January 2025 and is intended for undergraduate learners who already have a foundation in elementary calculus and linear algebra.

The material focuses on extending ordinary differential calculus to more general mathematical spaces and connecting those ideas to machine learning, optimization, backpropagation, adjoint methods, and automatic differentiation.

Download the PDF for free: https://arxiv.org/pdf/2501.14787

Understanding Matrix Calculus

Matrix calculus extends familiar ideas from calculus to situations where the variables themselves may be vectors or matrices.

In ordinary calculus, we might study how a scalar function changes when its scalar input changes.

In data science and machine learning, however, a function may take a vector or matrix as input and produce another vector, matrix, or scalar as output.

This creates a more complicated differentiation problem.

Matrix calculus provides a systematic framework for understanding these relationships.

Why Matrix Calculus Matters in Machine Learning

Modern machine-learning models contain enormous numbers of parameters.

These parameters can be represented naturally using vectors, matrices, and tensors.

During training, the model needs to determine how changes in these parameters affect its output and its loss.

This is fundamentally a differentiation problem.

The connection can be summarized as:

Model Parameters → Function → Loss → Derivatives → Parameter Updates

Understanding matrix calculus therefore helps explain what happens underneath optimization algorithms and neural-network training.

From Ordinary Calculus to Higher Dimensions

Traditional calculus often begins with functions of one variable.

As the number of variables increases, derivatives become more general mathematical objects.

For a vector-valued function, the derivative can be represented using a Jacobian matrix.

For scalar-valued functions, the derivative can be represented through gradients.

For second-order analysis, we encounter Hessian matrices.

The course takes this progression further by treating derivatives as linear operators rather than simply memorizing collections of partial derivatives.

Derivatives as Linear Operators

One of the central ideas of the material is to rethink what a derivative actually represents.

Instead of viewing a derivative merely as a formula containing partial derivatives, the derivative can be understood as a linear operator that describes the first-order change in a function.

This perspective becomes particularly powerful when dealing with matrix-valued functions.

It allows the same conceptual framework to be applied across scalars, vectors, matrices, and more general vector spaces.

Linear Approximation

At its core, differentiation is about approximation.

When an input changes by a small amount, the derivative describes how the output changes approximately.

This idea of linearization is fundamental to the course.

The MIT material emphasizes that understanding derivatives as linearization provides a way to approach complicated objects such as matrix determinants, matrix inverses, and solutions of differential equations.

Jacobians

The Jacobian is one of the most important concepts in multivariable calculus and machine learning.

For a function mapping multiple inputs to multiple outputs, the Jacobian captures how each output changes with respect to each input.

It can therefore be viewed as a generalized derivative for vector-valued functions.

Jacobians are particularly important when studying:

• Neural networks
• Computational graphs
• Optimization
• Sensitivity analysis
• Automatic differentiation

The course develops Jacobians beyond their traditional introductory treatment and connects them with matrix-valued functions and vectorization.

Matrix-Valued Functions

One of the most interesting aspects of matrix calculus is that the input itself can be a matrix.

Functions involving matrix operations can therefore have derivatives with respect to matrices.

Examples of important matrix functions include:

• Matrix multiplication
• Matrix inverse
• Matrix determinant
• Matrix factorizations
• Eigenvalue problems

The material develops techniques for handling these derivatives systematically rather than treating each case as an unrelated formula.

Matrix Inverses and Determinants

Matrix inverses and determinants are common mathematical operations in linear algebra.

Their derivatives become considerably more interesting when the matrix itself is treated as a variable.

Understanding these derivatives provides useful mathematical insight into optimization, numerical algorithms, sensitivity analysis, and machine-learning systems.

The MIT lecture sequence includes dedicated material on derivatives of matrix determinants and inverses.

Vectorization

Matrix expressions can sometimes be transformed into vector-based representations through vectorization.

Vectorization converts a matrix into a vector according to a defined ordering.

This can make certain matrix derivatives easier to express using conventional Jacobian notation.

The course discusses vectorization together with matrix functions and Kronecker products.

Kronecker Products

The Kronecker product is an important operation involving matrices.

It provides a structured way to construct larger matrices from smaller matrices.

In matrix calculus, Kronecker products become particularly useful when expressing derivatives of matrix-valued functions in vectorized form.

The course includes Kronecker products as part of its treatment of higher-dimensional derivatives.

Gradients

A gradient describes how a scalar-valued function changes with respect to multiple variables.

In machine learning, gradients are fundamental because optimization algorithms use them to determine how model parameters should change.

For a loss function, the gradient indicates a direction in parameter space associated with increasing or decreasing loss.

This makes gradients one of the central mathematical concepts behind machine learning.

Inner Products and General Vector Spaces

The material goes beyond ordinary Euclidean vectors and considers derivatives in more general vector spaces.

Inner products provide a way to define geometric relationships between elements in these spaces.

This broader perspective helps explain why gradient concepts can be generalized beyond simple coordinate-based representations.

Chain Rule in Higher Dimensions

The chain rule is one of the most important ideas in calculus.

It explains how derivatives behave when one function is composed with another.

In machine learning, models are often constructed from many layers of functions:

Input → Layer → Layer → Layer → Output

The overall derivative depends on how the derivatives of these individual components interact.

This makes the multidimensional chain rule fundamental to neural-network training.

Computational Graphs

A computational graph represents a complicated calculation as a sequence of smaller operations.

For example, a neural network can be represented as a graph in which each node performs a mathematical operation.

Differentiation can then be performed systematically through this graph.

The MIT course specifically connects chain rules with computational graphs and neural networks.

Backpropagation

Backpropagation is one of the most important applications of the chain rule in machine learning.

It efficiently propagates derivative information backward through a computational graph.

Instead of independently calculating every possible derivative, backpropagation reuses intermediate results.

This makes gradient calculation practical for large neural networks.

The course explicitly connects reverse-mode differentiation, adjoint differentiation, and backpropagation.

Forward-Mode Differentiation

Forward-mode differentiation propagates derivative information in the forward direction through a computational graph.

It can be particularly useful when there are relatively few input directions of interest.

The material discusses both manual and automatic forward-mode differentiation, including approaches based on dual numbers.

Reverse-Mode Differentiation

Reverse-mode differentiation works in the opposite direction.

It first evaluates the computational graph and then propagates derivative information backward.

This is especially powerful for machine learning because many models have a large number of parameters but relatively few scalar outputs, such as a single loss value.

Backpropagation is essentially a highly optimized application of reverse-mode differentiation.

Automatic Differentiation

Automatic differentiation, or AD, is another major theme.

Automatic differentiation is different from symbolic differentiation and finite-difference approximation.

Instead, it systematically applies differentiation rules to the operations within a computation.

Modern machine-learning frameworks rely heavily on automatic differentiation to calculate gradients.

The course introduces modern AD techniques and explains the mathematical ideas behind forward- and reverse-mode approaches.

Finite-Difference Approximations

Finite differences provide a numerical way to approximate derivatives.

Instead of analytically deriving a derivative, the method estimates the rate of change using small perturbations.

This makes finite differences useful for:

• Checking derivative implementations
• Numerical experimentation
• Understanding approximation error
• Validating automatic differentiation

The MIT lecture sequence includes finite-difference approximations and discusses accuracy and numerical error.

Optimization

Optimization is one of the main practical applications of matrix calculus.

Machine-learning training is frequently formulated as an optimization problem in which a loss function is minimized.

Matrix calculus provides the derivatives needed to understand how optimization algorithms navigate the parameter space.

The material connects matrix calculus with nonlinear root-finding, optimization, and large-scale computational problems.

Newton's Method

Newton's method is an important optimization and root-finding technique.

It uses both first- and second-order information to approximate solutions.

In multiple dimensions, this requires Jacobians and Hessian-related concepts.

Studying Newton's method provides a useful example of how multivariable calculus directly translates into numerical algorithms.

Gradient Descent

Gradient descent uses first-order derivative information to iteratively improve a solution.

The gradient indicates how the objective changes, while the optimization process uses that information to update parameters.

Gradient-based optimization is fundamental to machine learning and deep learning.

Matrix calculus provides the mathematical language needed to understand these updates beyond simple scalar examples.

Hessian Matrices

The Hessian contains second-order derivative information.

While gradients describe the local slope of a function, Hessians describe aspects of its local curvature.

This can provide more information about the structure of an optimization problem.

The course includes second derivatives, Hessian matrices, bilinear forms, quadratic approximations, and quasi-Newton methods.

Quadratic Approximations

A complicated function can sometimes be approximated locally using a quadratic expression.

Second-order information from the Hessian is central to this type of approximation.

Quadratic approximations are important in numerical optimization because they provide more information about local curvature than first-order methods alone.

Eigenvalue and Eigenvector Derivatives

Eigenvalues and eigenvectors are fundamental concepts in linear algebra and appear in many areas of data science.

The course goes beyond simply using eigenvalue decompositions and studies derivatives associated with eigenproblems.

This is an advanced topic that connects matrix calculus with spectral methods and numerical linear algebra.

Singular Value Decomposition

Singular Value Decomposition, or SVD, is another important matrix factorization.

It has applications in:

• Dimensionality reduction
• Principal component analysis
• Numerical linear algebra
• Data compression
• Recommendation systems

Understanding how matrix factorizations behave under differentiation is useful for advanced machine learning and optimization.

The course includes derivatives of matrix factorizations such as eigenvalue and SVD-related problems.

Differentiating Ordinary Differential Equations

One particularly advanced topic is differentiation involving solutions to ordinary differential equations.

Instead of differentiating a simple algebraic expression, the material considers how the solution of a differential equation changes when its parameters change.

This extends the idea of sensitivity analysis into more complex mathematical systems.

The lecture resources include dedicated material on differentiating ODE solutions and calculus of variations.

Adjoint Methods

Adjoint methods provide an efficient way to calculate derivatives in certain large-scale problems.

They are closely connected to reverse-mode differentiation and backpropagation.

This is particularly important when a system has many parameters but relatively few outputs.

The course connects adjoint methods with optimization, differential equations, and machine learning.

Derivatives of Random Functions

The material also extends differentiation concepts to functions involving randomness.

This introduces another layer of complexity because the function itself may depend on random variables or stochastic processes.

Such ideas can become relevant in advanced machine learning, stochastic optimization, probabilistic modeling, and scientific computing.

The lecture sequence includes derivatives of random functions as a dedicated topic.

Matrix Calculus and Deep Learning

Deep learning provides one of the clearest practical motivations for matrix calculus.

A neural network can contain millions or billions of parameters.

Those parameters interact through layers of matrix and vector operations.

Training requires calculating gradients of a loss function with respect to these parameters.

Therefore:

Linear Algebra + Calculus + Chain Rule + Optimization → Neural Network Training

Understanding matrix calculus can make the mathematical structure of deep learning considerably clearer.

Matrix Calculus and Automatic Differentiation Frameworks

Modern frameworks such as PyTorch and other machine-learning systems can calculate gradients automatically.

This convenience can sometimes hide the mathematical complexity underneath.

Matrix calculus helps learners understand what these frameworks are actually computing.

Instead of seeing automatic differentiation as a mysterious software feature, it can be understood as an implementation of mathematical differentiation rules applied efficiently to computational graphs.

Course Structure

The arXiv material is based on an MIT course taught by Alan Edelman and Steven G. Johnson, with lecture notes prepared by Paige Bright. The original MIT course was offered through the mathematics department and is structured around lectures, notes, problem sets, and supporting resources.

The lecture sequence progresses through:

Derivatives and Linear Operators

↓

Jacobians and Matrix Functions

↓

Vectorization and Kronecker Products

↓

Finite Differences

↓

General Vector Spaces

↓

Optimization and Adjoint Differentiation

↓

Matrix Determinants and Inverses

↓

Automatic Differentiation

↓

ODE Sensitivity

↓

Random Functions

↓

Hessians

↓

Eigenproblems

This progression demonstrates that the material goes substantially beyond elementary matrix differentiation.

Prerequisites

This is not an absolute beginner mathematics resource.

The MIT syllabus recommends prior study of:

• Linear algebra
• Multivariable calculus

The arXiv description similarly states that the course is intended for undergraduates familiar with elementary calculus and linear algebra.

A learner should therefore already be comfortable with vectors, matrices, derivatives, partial derivatives, and basic multivariable calculus.

Who Should Study This Material?

Machine Learning Students

Students who want to understand the mathematical foundations of machine learning can benefit significantly.

Deep Learning Students

Anyone studying backpropagation and optimization can use matrix calculus to strengthen their mathematical understanding.

Mathematics Students

The material provides a direct connection between advanced calculus, linear algebra, and computational applications.

Data Science Students

It can help learners move beyond library-based machine learning toward mathematical understanding.

AI Researchers

Researchers working with optimization, differentiable systems, or advanced machine learning can benefit from the general treatment of derivatives.

Engineers and Scientists

The applications to optimization, differential equations, and sensitivity analysis make the material relevant beyond machine learning.

Strengths of the Material

Strong Mathematical Foundation

The course develops derivatives from fundamental principles rather than relying only on memorized formulas.

Machine-Learning Relevance

The material directly connects matrix calculus with optimization, backpropagation, and machine learning.

Advanced Topics

It goes beyond basic Jacobians and gradients into matrix factorizations, ODEs, stochastic derivatives, adjoint methods, and automatic differentiation.

MIT Academic Foundation

The material is based on an MIT mathematics course taught by Alan Edelman and Steven G. Johnson.

Free Learning Resources

MIT OpenCourseWare provides lecture notes, videos, problem sets, and solutions associated with the course.

Limitations

The mathematical depth is also the primary challenge.

Readers without a solid foundation in linear algebra and multivariable calculus may find the material difficult.

It is also not primarily a practical machine-learning programming course. The emphasis is on mathematical understanding of differentiation, rather than on building complete machine-learning applications.

Learners looking for Python-based machine-learning projects, data preprocessing, model deployment, or deep-learning engineering will need additional resources.

How to Use This Resource

The best way to study matrix calculus is gradually.

First, strengthen:

Linear Algebra

↓

Multivariable Calculus

↓

Vector Calculus

Then study:

Derivatives as Linear Operators

↓

Jacobians

↓

Matrix Functions

↓

Gradients

↓

Chain Rule

↓

Backpropagation

↓

Automatic Differentiation

↓

Optimization

↓

Hessians and Second-Order Methods

This progression makes the advanced material much easier to understand.

Why This Resource Is Important for AI Learners

Modern AI increasingly relies on differentiable computation.

Neural networks, optimization systems, scientific machine learning, differentiable simulations, and many modern AI architectures require efficient calculation of derivatives.

Matrix calculus provides the mathematical foundation for understanding these operations.

The important shift is from thinking of derivatives merely as formulas to thinking of them as operators that propagate information through complex computations.

That perspective connects classical mathematics directly with modern machine-learning systems.

Download the PDF for free: https://arxiv.org/pdf/2501.14787

Final Verdict

Matrix Calculus (for Machine Learning and Beyond) is an excellent mathematical resource for learners who want to understand what happens underneath modern machine-learning optimization and automatic differentiation.

Its central contribution is the systematic extension of calculus from ordinary scalar functions to vectors, matrices, general vector spaces, matrix factorizations, differential equations, and stochastic functions.

The material is especially valuable for understanding the mathematics behind Jacobians, gradients, Hessians, backpropagation, reverse-mode differentiation, adjoint methods, and automatic differentiation.

It is best suited to learners who already have a foundation in linear algebra and multivariable calculus and want to move toward mathematically rigorous machine learning.