Honestly the issue is that most ML programs are taught as being some kind of additive skill set: the more courses you take the better or selection of the right kind of courses gets you some where.
In reality:
1. most real world problems are also about subtraction knowing what not to try and why it might not work. Like when I ask people about Recommendtaion engines for recommending colocated things, people pile on embeddings, in reality its about finding good false negatives to train datasets, calibration of classifier output and those are really hard problem. Embeddings may be necessary but are the least of your worries.
2. Most companies will not teach you about the fundamentals of stats; you will be lucky if you can get a mentor in a company that has both the theoretical rigour and the practical implementation skill to solve problems.
3. Most ML problems require engineering to work as well, for example you can't use Bayesian MCMC to do most things at scale. Its why Topic models that used statistical models like simulating posterior were crazy expensive on large datasets.
The reason is not that using shapley values is bad, they are great, but you can get a lot of insight by having some base models that are simpler to debug.
The reason a lot of the theoretical basis is taught is because you need to get the skills to learn why things work, when to use them, when they fail, when not to use them, and __most importantly__ their limitations. The problem is, most of this isn't explained explicitly. Maybe just this process happening for a few decades and momentum. Or that teaching isn't a priority and so no one tries to fix it. (there are exceptions to this. You've all probably met professors that are outstanding and make boring things seem fascinating)
But what you're talking about is part of this "when to use, what to use" part. It is also why those classes are so boring, because they aren't properly motivated. But it is also why we're running into so many problems: because evaluation is fucking hard. You see models perform really well on research papers but not in the real world but you'll also see researchers evaluating papers purely on singular benchmarks. "In reality" you're forced to come to terms with the limitations of the limitations of datasets, as datasets are just proxies and what you are about is the actual generalization. But if we're not discussing and evaluating on actual generalization in research then we get this dichotomy.
There's definitely more efficient (tractable) posterior estimators that work at large scale but just a lot of stuff isn't really known unless you're in that niche yourself. Statistics is often taught from the reference of "here's a bunch of tools and when to use them" rather than "here's the problems, our assumptions, and the main tool we use to solve them. It looks different in different settings, but they are actually the same thing." So it is kinda problematic, but then again, to get there requires a lot more work and most people aren't going to bother with things like metric theory. So a middle ground approach is taken and it gets jumbled.
This is true in most fields. I view school as giving you a broad overview of everything that you might need in your field, but for any given problem it will be on you to narrow it down to the solutions you actually need and then to learn that specific set of solutions well enough to apply it.
People fresh out of college will usually try to apply everything all at once until they learn—either from a mentor or their own hard experience—to filter it down. It might be that ML has it worse than other fields right now not because it's taught wrong but because it's new enough that there aren't enough mentors with decades of war stories.
Any recommendations for a _good_ linear regression class / course?
There are other people in that department like Ben Goodrich who are also good at teaching bayesian statistics
What you think you are targeting : 'business owners researching startups and investing'.
And somehow your ads get pushed to 'people with interest in dogs' when you go with the recommendations.
There are a lot of courses out there on AI from esteemed institutions at that. What do people recommend as a curriculum for someone with a formal univ education in CS albeit from a while ago and who has programmed extensively though not in Python.
The goal at the end is to have a deep understanding of the LLM space and its adjacencies.
I myself loved it and learned a lot, but YMMV
This is kinda a hard thing to quantify. How are we defining deep? Like you want to understand how they work? The Karpathy videos are good for that. But I wouldn't call this "deep".
If you want to get down into the weeds and into the mud, you need a hell of a lot more than 13hrs of education. You're also going to have a hard time doing this because most people are going from an engineering perspective of "enough to work with it" rather than "I fundamentally want to understand all inner workings". If you are the former, then the fastai course and others are great for you. If you want to really get deep though, you're going to need a lot more than programming. You're going to need some pretty advanced maths too: high dimensional statistics, metric theory, and optimization theory are some. (Most researchers aren't doing this btw) But if you do go down this path you'll also be able to understand the full spectrum of generative models and have a clearer picture. But I should also say that there is still a black box element to these models as they are so large that they are near impossible to analyze. But it is definitely achievable to learn a 2 layer transformer autoregressive network and fully understand its inner workings. But programming skills alone won't get you there.
After doing one of the fastai courses you will have some applied Python project experience and you can hone in deeper on a particular part of the project you are more interested in intellectually.
I ask because it’s pretty difficult to get through the math of backprop without a firm grasp of these. The Python part is trivial by comparison, the main difficulty being the matching of dimensions.
Honestly this is all moving so fast you can do well by reading the news, following a few reddits/substacks, and skimming the prompt engineering papers as they come out every week (!).
https://www.latent.space/p/ai-engineer provides an early manifesto for this nascent layer of the stack.
Zvi writes a good roundup (though he is concerned mostly with alignment so skip if you don’t like that angle): https://thezvi.substack.com/p/ai-18-the-great-debate-debates
Simon W has some good writeups too: https://simonwillison.net/
I strongly recommend playing with the OpenAI APIs and working with langchain in a Colab notebook to get a feel for how these all fit together. Also, the tools here are incredibly simple and easy to understand (very new) so looking at, say, https://github.com/minimaxir/simpleaichat/tree/main/simpleai... or https://github.com/smol-ai/developer and digging in to the prompts, what goes in system vs assistant roles, how you guide the LLM, etc.
The cool thing, is it helped me put a more structured thought process on how I should pursuing AI leanings...
I couldnt find anything concise out there - and this helped me to better think through everything:
If anything - its a good primer for getting your own thought process on the subject going...
https://chat.openai.com/c/7a36b5dc-0016-4b4c-bf1a-c3a66dac7c...
It walks a very thin line between too little info and *just* enough to get you to the point where you know what you don't know (the productive point) and it explains the Math in code samples. It really is a very good way of teaching. When I was reading, I thought the theory covered was too far from the Mathematical base, but I found my self being surprised at how I could hold my own in discussions that moved in to theory.
That said, this likely won't be enough for you to be a researcher - but I imagine for a lot of people tempted by courses like the OP - that isn't the actual end goal anyway.
1. Was one of the first members of the #ai slack channel inviting some people I had in person conversations about AI with.
2. I posted _a lot_ in there. Stuff about regulatory updates, people using co-pilot, cool github repos, little demo projects I was working on.
3. Now this was pure luck and probably the best thing to push me over the boundary, there was a hackathon. I thought "Hmm if I make a kick ass demo showcasing generative AI here, a lot of high up people will see it" - that 100% happened, CTO reached out to me saying demo was great and that people will be in touch.
4. I started really digging in to how I could provide value to our existing data team - be that code, deploying things, bringing some of my engineering know how to that team. This point the #ai channel really started to grow and the head of data and engineering started talking to me and directing people my way based on what they saw at the hackathon.
5. Did a demo of my hack in the company all hands which the CEO was MC'ing.
6. Started having fortnightly 1 to 1s with head of data at this point
7. Floated idea of team taking a little subset of good and motivated people from other teams for a short time to investigate and implement LLMs in some small way into our apps. That team has now grown to effectively investigate any and all use cases (internal and external) for generative AI.
8. I started reading more theory and also following a bit of a road map for things I should learn to have a better picture of how to actually bring LLMs in some form to production (fine-tuning, vector dbs, functions, guard rails).
9. Now I am just building some quick feature in the mobile app to show case the value of the team to exec as quick as I can, which should give us few months cover to work on the thing I am really interested in - multi-arm bandit LLM that uses our existing models.
This was pretty much it. Seems trivial, but in between each points was lots of reading, tinkering, working on weekends, but its totally possible. The ML + AI focused PhD's in your company likely need help from engineering but don't know it - bringing those two groups together quickly shows how you can be useful.
This post was helpful; https://blog.gregbrockman.com/how-i-became-a-machine-learnin...
https://stanford.edu/class/ee364a/
https://www.youtube.com/playlist?list=PL3940DD956CDF0622
It's one of the best courses to take if you want to obtain some fundamental understanding of the mathematical concepts behind AI.
Yes there's much more to it than NNs/transformers/'Attention is all you need' paper/whatever else is trendy right now. No, don't expect to do important, as in employable, work if you won't be spending some time truly understanding the mathematical foundations.
People only seem to want the former.
There's also Coursera courses that are much of the same content (taught by Andrew Ng as well in many cases). They have specializations for Machine Learning [1], Deep Learning, etc. These are paid via Coursera subscription, but financial assistance is available
[1] https://www.coursera.org/specializations/machine-learning-in...
If anyone is interested, I’ve published detailed notes and my submissions for the lab assignments:
https://github.com/pmulard/machine-learning-specialization-a...
Most of these look like they're just the slides and syllabus, correct me if I'm wrong.
Now, a few years later, transformers are the tech, and GPT-2's successors are the most hyped technologies of the century so far.
All of which is to say that I wouldn't assume that coursework without immediate application is useless. I'm in a much better position to jump in on the latest AI stuff than I would be if I hadn't taken that course.
Which course was this?
When I try to use transformers or any AI thing on a toy problem I come up with, it never works. Even Fizz-Buzz which I thought was easy doesn't work (because division or modulo is apparently hard to represent for NNs). And there's this blackbox of training that's hard to debug into. Yes, for the available resources, if you pick the exact same problem, the exact same NN architecture and exact same hyperparameters, it all works out. But surely they didn't get that on the first try. So what's the tweaking process?
Somehow this point isn't often talked about in courses and consequently the ones who've passed this hurdle don't get their experience transferred. I'd follow an entire course on this if it were available. An HN commenter linked me to this
https://karpathy.github.io/2019/04/25/recipe/
which is exactly on point. But it'd be great if it were one or more tutorials with a specific example, wrapped in code and peppered with many failures.
https://mlu-explain.github.io/neural-networks/
See also here:
If you're saying FizzBuzz doesn't work, presumably you mean that encoding the n directly doesn't work. Neither does encoding n from 0 to 1 or between -1 and 1 (and don't forget: obviously don't use relu with -1 to 1). It doesn't.
Neural networks can do a LOT of things, but they cannot deal with numbers. And they certainly cannot deal with natural or real numbers. BUT they can deal with certain encodings.
Instead of using the number directly, give one input to the neural network per bit of the number. That will work. Just pass in the last 10 bits of the number.
Or cheat and use transformers. Pass in the last 5 generations and have it construct the next FizzBuzz line. That will work. Because it's possible.
To make the number-based neural network for FizzBuzz "perfect" think about it. The neural network needs to be able to divide by 3 and 5. They can't. You can't fix that. You must make it possible for the neural network to learn the algorithm for dividing by 3 and 5 ... 2, 3 and 5 are relative primes (and actual primes). So "cheat" and pass in numbers in base 15 (by one-hot encoding the number mod 15 for example).
PM me if you'd like to debug whatever network you have together over zoom or Google meets or whatever.
https://en.wikipedia.org/wiki/One-hot
This may be catastrophically wrong. I only have a master's in machine learning (a European master's degree, meaning I've written several theses on it (didn't pass first time, had to work full time to be able to study), and I was writing captcha crackers using ConvNets in 2002. But I've never been able to convince anyone to hire me to do anything machine learning related.
You mention bag of tricks and that's indeed one issue but its worse than that because it includes knowing what "silent problems" needs a trick applied to it in the first place!
Indeed, despite using vectors everywhere, NN are bad with numerical input encoded as themselves! Its almost like the only kind of variables you can have are fixed size enums. That you then encode into vectors that are as far apart as possible, and unit vectors ("one hot vectors") do this. But that's not quite it and sometimes you can still some meaningful metric on the input that's preserved in the encoding (example: word embeddings). And so its again unclear what you can give it and what you can't.
In this toy example, I have an idea of what the shape of the solution is. But generally I do not and would not know to use a base 15 encoding or to send it the last 5 (or 15) outputs as inputs. I know you already sort of addressed this point in your last few paragraphs.
I'm still trying out toy problems at the time so it might be a "waste" of your time to troubleshoot these but I'm happy to take you up on the offer. HN doesn't have PMs though.
Do you remember when you first learned about the things you are using in your reply here? Was it in a course or just asking someone else who worked on NN for longer? I learned through by googling and finding comment threads like these! But they are not easy to collect or find together.
Oh wow, those are great credentials. I'm surprised that you haven't run across a position yet. Maybe it is a matter of your location? It seems like a lot of these jobs want onsite workers, which can be a real problem.
TBH, I get the feeling that a lot of us without such credentials are in a similar position right now. Slowly trying to work our way towards what seems to be a big new green field, but having a really unclear path to getting there...
However, can you point me to the lectures where training happen (and architecture choices, hyperparam selection, and debugging happens.). I'm less familiar with SD but at a quick glance it seems like we're using a pretrained model and implementing bits that will eventually be useful for training but not training a new model, at least in the beginning of the deep dive notebook and first few lessons (starting at part 2, lesson 9).
(as the CS237A Principles of Robotic Autonomy I link seems to be broken)
entropy of a single signal, say a sequence of letters, "ababababab" is the scaled "average" surprise per letter. So if they are uniformly distributed, each letter is equally likely/unlikely to come next in the sequence, where if instead one letter only 1/1000th of the time (aaa....aaa...aa..a.z.aaaa), then when the rare beast shows up, it is a big surprise, so the total amount of surprise available in the sequence is high.
That's entropy.
The same thing would be true for a sequence of numbers.
But what if there is some relationship? if aaabaa occurs frequently with 111211, if you line up the sequences by timestamp?
In this simple case, if you know the letters and you can spot the relationship, then there is zero surprise in the number sequence. The cross entropy "letters plus numbers" has the same entropy as "letters" or "numbers" in isolation.
And as you move away from the 1:1 correspondence, you'll see the cross entropy increase until it reaches its max at "entropy(letters) + entropy(numbers)" -- no information shared between the two systems.
To bring it home, I think of cross entropy as the amount of information shared between two signals.
Others might think of it slightly differently.
> if instead one letter only 1/1000th of the time (aaa....aaa...aa..a.z.aaaa), then when the rare beast shows up, it is a big surprise, so the total amount of surprise available in the sequence is high
…when a Bernoulli distribution is skewed, the maximum surprise is high, yes, but the average surprise (= entropy) is low. The entropy of a Bernoulli distribution is maximized when p = 0.5 and falls off to either end:
https://en.wikipedia.org/wiki/Binary_entropy_function
For your examples, if the sequence is uniformly distributed (Bernoulli(1/2)), the entropy is log(2) ≈ 0.693 bits per symbol; if instead one letter occurs 1/1000th of the time, the entropy is about 0.0079 bits per symbol.
