A field that does inspire a lot of deep learning folks and never gets mentiond in this sort of thing is the theory of physical dynamical systems. Attractor is a term that came from here, for example, and much of the mathematics behind the numerical fuckery behind deep nets is dynamical in nature. RNN's are entirely dynamical systems. Classic there is Strogatz book (https://www.amazon.com/Nonlinear-Dynamics-Chaos-Applications...).
There is also information theory, of course, which is part of the MacKay source.
Many of the earlier papers in deep learning-land are really nontrivial to read, because the terminology and worldview of everybody has changed so much. So reading original Werbos or Rumelhart is really difficult. This is really not the case for Sutton and Barto, "RL: An Introduction" (http://webdocs.cs.ualberta.ca/~sutton/book/the-book.html). Two editions, apparently the second edition is basically getting with the program on shoving DL into everything.
Schmidhuber often mentions that Gauss was the original shallow learner. This is a technically correct statement (best kind of statement), but you definitely should probably know linear and logistic regression like the back of your hand before starting on DL too much.
http://www.amazon.com/Nonlinear-Dynamics-And-Chaos-Applicati...
In fact everything by that author is great, buy and read it all:)
http://www.amazon.com/Steven-H.-Strogatz/e/B001KHB290/ref=dp...
Just make sure you've read up on your differential equations before you start:) This isn't a topic for someone without a strong background in the mathematics!
Unfortunately I have yet to find a good introductory course, video, book, pdf....anything for this topic. I had to jump in with a specific model in mind and keep banging away until I iterated my way towards the solution.
Suggestions for introductory material are welcome!!!
See also: non-linear dynamics, dynamical systems
The study of Complexity in the US Sciences is fairly recent, with the notable kickoff of the Santa Fe Institute [1]. They have all the resources there, from classes to scientific conferences.
A few book recommendations I've picked up over the years:
- Lewin, Complexity: Life at the Edge of Chaos [2] (recommended for a gentle introduction with history)
- Gleick, Chaos: Making a New Science [3]
- Holland, Emergence: From Chaos To Order [4]
- Holland, Signals and Boundaries: Building Blocks for Complex Adaptive Systems [5]
- Holland, Hidden Order: How Adaptation Builds Complexity [6] (unread but part of the series)
- Miller, Complex Adaptive Systems: An Introduction to Computational Models of Social Life [7]
- Strogatz, Nonlinear Dynamics And Chaos: With Applications To Physics, Biology, Chemistry, And Engineering [11] (edit: forgot i had this on my desk)
Also a shoutout to Stephen Wolfram, reading one of his articles [8] about underlying structures while stuck at an airport proved to be a pivotal moment in my life.
- Wolfram's New Kind of Science, as an exploration of simple rules computed to the point of complex behavior. [9]
- Wolfram's Physics Project, a later iteration on NKS ideas with some significant and thorough publication of work. [10]
Links:
[1] https://www.santafe.edu/
[2] https://www.amazon.com/gp/product/0226476553
[3] https://www.amazon.com/gp/product/0143113453
[4] https://www.amazon.com/gp/product/0738201421
[5] https://www.amazon.com/gp/product/0262525933
[6] https://www.amazon.com/Hidden-Order-Adaptation-Builds-Comple...
[7] https://www.amazon.com/gp/product/0691127026
[8] https://writings.stephenwolfram.com/2015/12/what-is-spacetim...
[9] https://www.wolframscience.com/nks/
[10] https://wolframphysics.org/
[11] https://www.amazon.com/Nonlinear-Dynamics-Chaos-Applications...