Machine learning is often learned through Python, notebooks, datasets, and ready-made libraries. While practical implementation is extremely important, there is another side of machine learning that is sometimes overlooked: mathematical reasoning.
Pen and Paper Exercises in Machine Learning, written by Michael U. Gutmann, takes a different approach. Instead of concentrating primarily on programming, it provides a collection of mostly pen-and-paper exercises designed to strengthen the mathematical understanding behind machine-learning methods.
The work was submitted to arXiv in June 2022 and covers topics including linear algebra, optimization, graphical models, message passing, hidden Markov models, model-based learning, sampling, Monte Carlo integration, and variational inference.
The main idea is simple: sometimes the best way to understand a machine-learning algorithm is to work through its reasoning by hand.
Download the PDF for free:
Pen and Paper Exercises in Machine Learning
Why Pen-and-Paper Learning Matters
Modern machine-learning libraries can perform complicated calculations almost instantly.
A few lines of Python can train a model, calculate gradients, perform optimization, or make predictions. This is extremely useful, but it can also hide the reasoning behind the algorithm.
When students solve a problem manually, they are forced to understand:
What the algorithm is actually doing
Why each step is necessary
How different mathematical concepts connect
Where assumptions are being made
How the final result is obtained
Why an algorithm behaves differently under different conditions
The exercises in this work are designed around this type of deeper understanding. The author specifically explains that the exercises are intended to strengthen mathematical skills and complement, rather than replace, machine-learning courses or textbooks.
The Mathematical Side of Machine Learning
Machine learning is not only a programming discipline. It combines several areas of mathematics and statistics.
A machine-learning student may encounter:
Linear algebra
Calculus
Probability
Statistics
Optimization
Graph theory
Numerical methods
Information theory
These subjects are not isolated from machine learning. They provide the tools used to design models, understand data, perform inference, and optimize algorithms.
This is why mathematical exercises can be extremely valuable for someone studying machine learning at an advanced level.
Linear Algebra
The first major topic covered is linear algebra.
Linear algebra forms the foundation of many machine-learning algorithms because data and model parameters are frequently represented using vectors and matrices.
The exercises explore concepts such as:
Gram–Schmidt orthogonalization
Linear transformations
Eigenvalue decomposition
Symmetric matrices
Trace
Determinants
The power method
These concepts are important in many areas of machine learning, including dimensionality reduction, optimization, numerical computation, and representation learning.
Why Linear Algebra Matters
A strong understanding of linear algebra allows learners to understand what machine-learning software is actually calculating.
Instead of treating matrix operations as mysterious commands inside a programming library, students can understand their geometric and computational meaning.
This becomes particularly important when studying algorithms such as PCA, graphical models, neural networks, and optimization methods.
Optimization
Optimization is another fundamental part of machine learning.
A machine-learning model generally has some objective that it wants to improve. Training involves searching for parameter values that provide better results according to that objective.
The optimization section helps readers develop the mathematical reasoning required to understand this process.
Important concepts include:
Gradients
Optimization objectives
Gradient-based methods
Local behavior of functions
Parameter updates
Convergence
Optimization challenges
Optimization is particularly important because many machine-learning algorithms are essentially optimization procedures wrapped around statistical or mathematical models.
Graphical Models
One of the strongest themes of the collection is probabilistic graphical models.
Graphical models provide a visual and mathematical framework for representing relationships between variables.
They can help describe:
Dependencies
Conditional independence
Probabilistic relationships
Hidden variables
Inference problems
Structured data
The collection covers both directed graphical models and undirected graphical models.
Directed Graphical Models
Directed graphical models use directed connections to represent relationships between variables.
They are useful for representing probabilistic dependencies and reasoning about how variables influence one another within a structured model.
Studying these models helps learners understand concepts such as:
Conditional independence
Dependency structures
Probabilistic reasoning
Inference
Graph-based representations
These ideas are useful in areas ranging from probabilistic AI to Bayesian reasoning.
Undirected Graphical Models
Undirected graphical models represent relationships without assigning directional relationships between variables.
They are particularly useful when the relationships between variables are symmetric or when the goal is to represent a network of dependencies.
Learning both directed and undirected approaches allows students to understand that probabilistic modeling is not based on a single representation.
Different structures are useful for different types of problems.
Understanding Independence
One of the most important concepts in probabilistic machine learning is independence.
Machine-learning models often need to determine whether knowing one variable provides information about another variable.
Graphical models provide a structured way to reason about these relationships.
Understanding independence is important because it can simplify complex probabilistic problems and make inference computationally more manageable.
The exercises therefore encourage students to reason about relationships between variables rather than simply applying formulas mechanically.
Expressive Power of Graphical Models
Another interesting topic is the expressive power of graphical models.
Different model structures can represent different kinds of relationships.
A simple model may not be able to express complicated dependencies, while a more sophisticated structure may represent them efficiently.
Understanding expressive power helps answer an important machine-learning question:
What kinds of relationships can a particular model represent?
This idea connects directly to modern machine learning, where model architecture and representation capacity strongly influence what a system can learn.
Factor Graphs and Message Passing
The collection also explores factor graphs and message passing.
Factor graphs provide a structured representation of complex probabilistic relationships.
Message passing algorithms then allow information to move through the graph so that different variables can influence one another during inference.
This is an important concept because many probabilistic inference problems would be extremely difficult to solve directly.
Message passing provides a systematic way to break complicated problems into smaller computational components.
Hidden Markov Models
Another important topic is Hidden Markov Model inference.
Hidden Markov Models are used when the system being studied contains hidden states that cannot be directly observed.
Instead, we observe outputs generated by those hidden states and attempt to infer what is happening internally.
This idea has applications in:
Speech recognition
Sequence analysis
Natural language processing
Time-series modeling
Biological sequence analysis
Pattern recognition
Studying HMM inference gives learners an important introduction to reasoning about sequential and hidden information.
Model-Based Learning
The collection also examines model-based learning.
Model-based approaches attempt to construct a mathematical representation of how data is generated or structured.
Instead of treating the model purely as a prediction machine, the learner attempts to understand the underlying data-generating process.
This perspective is particularly valuable in probabilistic machine learning because it emphasizes understanding the structure behind observations.
Independent Component Analysis
One of the topics included under model-based learning is Independent Component Analysis, commonly known as ICA.
ICA attempts to discover underlying independent components within observed data.
A classic intuition is the problem of separating several mixed signals into their underlying sources.
This idea has connections with:
Signal processing
Representation learning
Blind source separation
Feature extraction
Unsupervised learning
ICA demonstrates how mathematical assumptions about data can be used to discover hidden structure.
Unnormalised Models
The collection also discusses unnormalised models, an important concept in probabilistic modeling.
In some probabilistic models, calculating the normalization factor directly can be computationally difficult.
Rather than avoiding such models completely, researchers can develop learning and inference techniques that work with the unnormalised representation.
This topic is particularly interesting for advanced machine-learning students because it introduces challenges that arise when probability distributions become mathematically or computationally difficult to handle.
Sampling
Sampling is another major area covered by the collection.
In many machine-learning problems, calculating an exact probability or expectation can be difficult.
Sampling provides an alternative approach.
Instead of calculating everything exactly, an algorithm can generate representative samples and use those samples to estimate the quantity of interest.
This idea forms the foundation of many statistical and probabilistic methods.
Monte Carlo Integration
Monte Carlo methods use randomness and repeated sampling to estimate quantities that may be difficult to calculate analytically.
The basic intuition is powerful:
Instead of solving a complicated problem exactly, we can sometimes approximate its solution by generating enough representative random samples.
Monte Carlo methods are widely used in:
Bayesian inference
Statistical estimation
Simulation
Numerical integration
Probabilistic modeling
Scientific computing
The collection includes sampling and Monte Carlo integration as part of its broader focus on probabilistic machine learning.
Variational Inference
Variational inference is another advanced topic included in the work.
It is used when direct probabilistic inference is computationally difficult.
The central idea is to transform a difficult inference problem into an optimization problem.
Instead of trying to calculate a complicated probability distribution directly, we construct a simpler approximation and optimize it so that it becomes as useful as possible.
This idea has become extremely important in modern machine learning.
Unsupervised Learning
A particularly important feature of the collection is its strong emphasis on unsupervised learning.
In supervised learning, models receive examples with known target outputs.
Unsupervised learning is different. The model attempts to discover useful structure from data without being explicitly given the desired answers.
This can involve:
Discovering hidden patterns
Finding groups
Learning representations
Identifying latent variables
Modeling probability distributions
Understanding relationships within data
The author notes that the collection focuses strongly on unsupervised methods, inference, and learning rather than attempting to comprehensively cover every area of machine learning.
Inference in Machine Learning
Inference is one of the central ideas running through the collection.
In probabilistic machine learning, inference generally means determining what can be concluded from available information.
For example, a model may contain hidden variables, incomplete observations, or uncertain relationships.
Inference attempts to answer questions such as:
What is likely to have happened?
What hidden state is most probable?
How are variables related?
What information can be inferred from observations?
How uncertain is the conclusion?
Learning and inference are closely connected but represent different computational tasks.
Learning Through Detailed Solutions
An important feature of the collection is that the exercises come with detailed solutions.
This makes the resource more than simply a question bank.
Students can:
Attempt an exercise independently.
Work through the problem manually.
Compare their reasoning with the provided solution.
Identify where their understanding differs.
Revisit the underlying theory.
Try the exercise again.
This process encourages active learning rather than passive reading.
Why Solving Problems Is Different From Reading Theory
Reading a machine-learning textbook can provide conceptual understanding, but solving problems requires a different level of engagement.
When reading, it is easy to think:
“I understand this.”
When solving a problem, the learner has to demonstrate that understanding.
This exposes gaps in knowledge.
For example, a student may understand the general idea of eigenvalues but struggle to perform an eigenvalue decomposition. Similarly, someone may understand gradient descent conceptually but struggle to reason about how the gradient changes during optimization.
Pen-and-paper exercises expose these gaps.
Mathematics Before Coding
The resource does not argue that coding is unimportant.
Instead, it offers a complementary approach.
The author explains that while coding and computer simulations are important in machine learning, pen-and-paper exercises can strengthen mathematical skills, and the two approaches are ideally combined.
A strong learning strategy can therefore be:
Understand the theory → Solve manually → Implement in Python → Experiment with data
This approach provides both conceptual and practical understanding.
Combining Pen-and-Paper With Python
After solving an exercise manually, students can implement the same concept in Python.
For example, after studying:
Matrix operations
Optimization
Sampling
Graphical models
Hidden Markov Models
a learner can implement simplified versions of those ideas in a Jupyter Notebook.
This creates a powerful connection between mathematics and programming.
The learning cycle becomes:
Theory — Understand the concept
Pen and paper — Work through the reasoning
Python — Implement the concept
Experimentation — Observe its behavior
Analysis — Connect results back to theory
Who Should Use This Resource?
This collection is particularly useful for learners who already have some foundation in mathematics.
It is suitable for:
Machine-learning students
Data science students
AI students
Mathematics students
Computer science students
Researchers
Graduate students
Teachers
Advanced self-learners
The work assumes that readers have already encountered relevant theory and concepts and want to deepen their understanding through exercises.
What Makes It Different From a Typical ML Tutorial?
Most modern machine-learning tutorials focus heavily on implementation.
You may see:
Python code
Dataset loading
Model training
Visualization
Performance metrics
Library APIs
This collection focuses on something different.
It asks the learner to think through the machine-learning problem.
That makes it particularly useful for developing the kind of mathematical intuition that is difficult to obtain by simply running machine-learning libraries.
Key Topics Covered
The work brings together a broad set of mathematical and probabilistic machine-learning topics.
Major areas include:
Linear algebra
Optimization
Directed graphical models
Undirected graphical models
Graphical-model expressive power
Factor graphs
Message passing
Hidden Markov Models
Model-based learning
Independent Component Analysis
Unnormalised models
Sampling
Monte Carlo integration
Variational inference
These topics are explicitly listed in the paper's abstract and contents.
Benefits for Machine Learning Students
Working through these exercises can develop several important skills.
Mathematical Thinking
Students become more comfortable reasoning about mathematical structures rather than memorizing algorithms.
Problem-Solving
Exercises force learners to break complicated problems into smaller steps.
Algorithmic Understanding
Manually working through algorithms helps reveal what happens internally.
Statistical Intuition
Probabilistic exercises develop a better understanding of uncertainty and inference.
Model Understanding
Students learn to think about what a model can represent and what assumptions it makes.
Research Preparation
A stronger mathematical foundation can be valuable for reading machine-learning research papers.
From Beginner ML to Advanced ML
A learner's journey through machine learning often begins with basic concepts such as:
Data
Features
Labels
Regression
Classification
Model evaluation
As the learner progresses, mathematical concepts become increasingly important.
Advanced topics such as graphical models, probabilistic inference, variational methods, and unsupervised learning require significantly deeper mathematical reasoning.
This resource is therefore particularly useful as a bridge between introductory machine learning and more theoretical machine learning.
Download the PDF for free:
Pen and Paper Exercises in Machine Learning
Final Thoughts
Pen and Paper Exercises in Machine Learning offers a refreshing approach to learning machine learning in an age dominated by programming frameworks and automated tools.
Its central philosophy is valuable: do not only run the algorithm—understand the algorithm.
By working through problems manually, learners can develop stronger intuition for linear algebra, optimization, probability, graphical models, inference, and unsupervised learning.
The collection does not attempt to replace a machine-learning textbook or course. Instead, it works best alongside them, providing the active problem-solving practice needed to turn theoretical knowledge into deeper understanding.
For anyone who wants to move beyond simply using Python libraries and begin understanding the mathematical and probabilistic foundations of machine learning, this is a highly useful resource.
.png)
.png)
.png)
Posts
0 Comments:
Post a Comment