Introduction to Machine Learning

Jump to: navigation, search



Several people have asked me what do they need to do to start research in Machine Learning. While there is no easy answer to that question, I have tried to put down what I feel are basic areas of Mathematics and Computer Science that a person needs to know to be able to do effective research in ML. These are by no mean sufficient conditions, but what I feel are necessary.

Probability and Statistics

While a rudimentary knowledge of probability theory is needed to even understand much of the work in the area, one has to have a good understanding of probability theory, and stochastic approximations to be able to add do effective work in ML. The current state of affairs is such that ML has effectively become applies statistics and probability and hence spending time with these topics is essential

Suggested Books/Courses:

  • Introduction to Probability and Statistics for Engineers - Sheldon Ross [1]
  • All of Statistics - Larry Wasserman [2]
  • Probability, Random Variables and Stochastic Processes - Athanasios Papoulis [3]
  • Introduction to Probability Models - Sheldon Ross [4]
  • Probability and Random Processes - Geoffrey Grimmett and David Stirzaker [5]
  • Statistical Inference - Casella, G. and Berger, R.L. [6]
  • Probability: Theory and Examples - Durrett, R. [7]
  • Condensed Notes on Probability and Statistics for quick reference [8]
  • Probability : The Analysis of Data, Volume 1 [9]
  • ThinkStats - Probability and Statistics for Programmers [10]

Linear Algebra

Again an essential ingredient to even understanding the language of ML.

Suggested Books/Courses:

  • Linear Algebra - MIT OCW - Gilbert Strang [11]
  • Introduction to Linear Algebra, 4th Edition, Gilbert Strang [12]
  • Matrix Computations - Golub and Van Loan [13]
  • Linear Algebra - Hoffman and Kunze [14]

Convex Optimization

Everything in the world is an optimization problem. You just need the right objective function. A lot of work in ML is about finding the right optimization techniques

Suggested Books/Courses:

  • Convex Optimization I [15] & Convex Optimization II [16] - Lecture's by Stephen Boyd
  • Convex Optimization @ CMU [17]
  • Convex optimization - Stephen Boyd [18]
  • Linear and non-linear optimization - David G. Luenberger, Yinyu Ye [19]
  • Introduction to Linear Optimization - Bertsimas, D. and Tsitsiklis, J. [20]
  • Convex Analysis and Optimization by D. Bertsekas, A. Ozdaglar, and A. Nedic. [21]
  • Convex Optimization @ IITM [22]

Real Analysis

Can come in handy when studying convergence and stability properties of algorithms

Suggested Books/Courses:

  • Principles of Mathematical Analysis - Walter Rudin [23]
  • Analysis: With an Introduction to Proof - Steven. R. Lay [24]

Functional Analysis.

Suggested Books/Courses:

  • Introductory Functional Analysis with Applications - Kreyszig, E [25]
  • Introduction to Functional Analysis - Vladmir Kisil's Lecture Notes [26]

References for Machine Learning

Suggested Books/Courses:

  • Elements of Statistical Learning - Hastie, Tibshirani and Friedman [27]: A fantastic mathematical introduction to statistical learning.
  • Pattern Recognition and Machine Learning - Chris M Bishop [28]: Somewhat of a bible for many ML and pattern recognition concepts.
  • Information Theory, Inference, and Learning Algorithms - David MacKay [29]
  • Machine Learning: a Probabilistic Perspective by Kevin Patrick Murphy [30]

Other useful links

Suggested Books/Courses:

  • Andrew Ng's ML lectures.[31] After you go through all the above basic material, if you listen to this course, you will get a lot more out of it. :).
  • Sargur Srihari's Machine Learning Lectures [32]
  • Alex Smola's blog [33]
  • John Langford's blog [34]
  • Max Welling's Classnotes in Machine Learning [35]
  • Mark Reid's blog on Machine Learning and Statistical Inference [36]
  • Machine Learning course at CMU[37]
  • Machine Learning course at UPenn [38]
  • Machine Learning Summer School at Purdue 2011[39]
Personal tools