> Maybe, maybe not. But even if you use a machine learning toolkit like TensorFlow or PyTorch, what you will take away from this book is an appreciation for how the fundamentals work.
There are two books that do exactly this:
1. Deep Learning from Scratch
2. Data Science from Scratch
In these books, you implement each part of the ML/DL pipeline, from scratch, in Python.
There is also a GitHub project call Minitorch that teaches you the inner workings of a framework like PyTorch.
And then there are several other good resources for exactly this.
What he claims to have as a content is neithet new nor unique.
Even if you think so, Python is really an easy language, and you can easily port the code to something else.
If you already have the basic ideas about the parts of a Neural Network pipeline, you can just search google "implement part-X in Y language", and you will get well written articles/tutorials.
Many learners/practitioners of Deep Learning, when they have the big enough picture, write an NN training loop in their favorite language(s) and post it online. I remember seeing a good enough "Neural Network in APL" playlist in YT. It implements every piece in APL and gains like 90%+ accuracy in MNIST.
I also remember seeing articles in Lisp (of course!), C, Elixir, and Clojure.
I am writing one in J-lang in my free time.
So if you use a library for matrix multiplication, inverse, transpose, ... with a nice syntax, you're good to go.
Do you reimplement matmul or other basics?
Do you reimplement auto-diff?
Maybe PyTorch or TensorFlow using auto-diff is a good "from scratch" basepoint, without using predefined optimizers, or modules/layers, or anything. Just using the low-level math functions, and then auto-diff.
Yes, in those books, you do implement matmul, auto-diff, etc.
I just wanted to point out that "from scratch" is not really well defined. There is always some arbitrary line. I just found it interesting to discuss and think about where to draw this line exactly. Obviously it's never really from scratch, i.e. you don't reinvent the hardware level, for example. Or you don't start with teaching quantum physics. So you start from somewhere.
And I was wondering whether auto-diff is maybe something which could also already be the starting point, or also matmul. Reimplementing an efficient matmul on CUDA is not easy, and might distract from the main deep learning content. But it depends also where you want to have the focus on.
