Math 217, Spring 2024
Topics in Applied Mathematics: Mathematics of Artificial Neural Networks

Lectures: 3:00 pm - 4:20 pm Wednesdays and Fridays, AP&M 5829.
Instructor: Bo Li (Office: AP&M 5723. Office phone: (858) 534-6932. Email: bli@math.ucsd.edu)
Office hours: 4:30 - 5:00 Wednesdays and Fridays, or by appointment.

Course Description

This course focuses on mathematics of artificial neural networks. Here is a tentative list of topics to be covered (and this list will be revised constantly):
  • Basics of (feedforword) neural networks (MLPs)
  • Approximation theory of neural networks (both universal apprxomation and error bounds)
  • Training neural networks (methods and analysis)
  • Structures and landscapes of neural networks
  • Neural networks for dynamical systems and partial differential equations
  • Application in biological physics (if time permits)
Background needed for taking this course includes basic measure theory and numerical analysis, both at the mathematics graduate level. Functional analysis and probability theory are useful but not required.

Expected Work for Students

Students are expected to participate in class discussions, read some papers with a possible presentation, and do a course project.

A list of possible topics for the course project will be provided later. Students can also design their own course projects related to this course. In case so they are encouraged to discuss their topics with the instructor. No completed research project can be used to replace the course project.

The course project is due 3:00 pm, Wedensday, June 12, 2024.

Lecture Notes

  • Lecture 1: Course description, definition of NNs, example of activation functions.
  • Lecture 2: Nonuniqueness of representations, contruction/operations of NNs.
  • Lecture 3: The Stone-Weierstrass and Kolmogorov Superposition Theorem. Universal Approx. Theorems.
  • Lecture 4: Universal Approx Theorems: proofs and remarks.
  • Lecture 5: More on Universal Approx Theorems: proofs and remarks.
  • Lecture 6: Universal approximations with Lp norms.

Selected Reference Books

  1. M. Anthony and P. L. Bartlett, Neural Network Learning: Theoretical Foundations, Cambridge University Press, 1999.
  2. O. Calin, Deep Learning Architectures: A Mathematical Approach, Springer, 2020
  3. A. L. Caterini and D. E. Chang, Deep Neural Networks in a Mathematical Framework, Springer, 2018.
  4. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016. (Online version)
  5. J. M. Phillips, Mathematical Foundation for Data Analysis, Springer, 2021.
  6. B. Scholkopf and A. J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, The MIT Press, 2002.
  7. S. Shalev-Shwartz and S. Ben-David, Understanding Machine Learning: From Theory to Algorithms Cambridge Univ. Press, 2014.

Last updated by Bo Li on April 8, 2024.