Math 217, Spring 2024
Topics in Applied Mathematics: Mathematics of Artificial Neural Networks

Lectures: 3:00 pm - 4:20 pm Wednesdays and Fridays, AP&M 5829.
Instructor: Bo Li (Office: AP&M 5723. Office phone: (858) 534-6932. Email: bli@math.ucsd.edu)
Office hours: 4:30 - 5:00 Wednesdays and Fridays, or by appointment.

Course Description

This course focuses on mathematics of artificial neural networks. Here is a tentative list of topics to be covered (and this list will be revised constantly):
  • Basics of (feedforword) neural networks (MLPs)
  • Approximation theory of neural networks (both universal apprxomation and error bounds)
  • Training neural networks (methods and analysis)
  • Structures and landscapes of neural networks
  • Neural networks for dynamical systems and partial differential equations
  • Application in biological physics (if time permits)
Background needed for taking this course includes basic measure theory and numerical analysis, both at the mathematics graduate level. Functional analysis and probability theory are useful but not required.

Expected Work for Students

Students are expected to participate in class discussions, read some papers with a possible presentation, and do a course project.

A list of possible topics for the course project will be provided later. Students can also design their own course projects related to this course. In case so they are encouraged to discuss their topics with the instructor. No completed research project can be used to replace the course project.

The course project is due 11:59 pm, Friday, June 14, 2024.

Lecture Notes

  • Lecture 1: Course description, definition of NNs, example of activation functions.
  • Lecture 2: Nonuniqueness of representations, contruction/operations of NNs.
  • Lecture 3: Stone-Weierstrass, Kolmogorov, and Wiener's. Universal Approx. Theorems (UAT).
  • Lecture 4: UAT w.r.t. uniform approximations, proofs and remarks.
  • Lecture 5: More on UAT w.r.t. uniform approximations, proofs and remarks.
  • Lecture 6: Universal approximations with Lp norms.
  • Lecture 7: Reviews, examples, and remarks. Approximation for classification.
  • Lecture 8: Universal approximations of Ck and Wk,p functions.
  • Lecture 9: Best approximations. Existence/nonexistence. Other unusual properties. Weight explosion.
  • Lecture 10: Bounds of approximations. Some examples.
  • Lecture 11: Bounds for approx. with a single hidden layer. Proof.
  • Lecture 12: Bounds for approx. with a single hidden layer. Proof continued.
  • Lecture 13: Univ. approx. by DNNs with fixed width and a general activiation.
  • Lecture 14: Univ. approx. by DNNs with fixed width - proof continued and some calculations for ReLU nets.
  • Lecture 15: Learning with NNs. Loss functions. Gradient descent.
  • Lecture 16: Differentiation. Backpropagation.
  • Lecture 17: Learning with the stochastic gradient descent (SGD). Convergence.
  • Lecture 18: Moment-based adaptive SGD. ADAM and its variants.

Selected Reference Books

  1. C. C. Aggarwal, Neural Networks and Deep Learning, A textboo, Springer, 2018.
  2. M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical Foundations, Cambridge Univ. Press, 1999.
  3. O. Calin, Deep Learning Architectures: A Mathematical Approach, Springer, 2020
  4. S. Haykin, Neural Networks, A Comprehensive Foundation, 2nd ed., Pearson Education, 1999.

Last updated by Bo Li on April 8, 2024.