Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.
That is, for example, why it is possible to find people presumably seriously suggesting to:
3. Flashcard the Deep Learning Book (4-6m)
4. Flashcard ~100 papers in a niche (2m)
I mean, it's clear to me that the language deployed in the article is ostensibly about teaching yourself to do machine learning research when what it's really discussing is how to get hired by one of the companies that are curently paying six-figure salaries for machine learning engineers etc.
Or I'm just old and cynical. Wait, let me find my false teeth so I can chew that over.
MIT has takes 3,000 students, Canadian universities take 30,000 students. (Remember Canada has 30 million people and US has 300 million.)
- https://web.mit.edu/facts/enrollment.html
- https://www.univcan.ca/universities/facts-and-stats/enrolmen...
But I'm not sure what that has to do with buying expensive formal education credentials.
If it's easy to see that a piece of output (a paper, code library, machine learning model, whatever) or a job candidate is great, then the credentials behind it don't matter much. However if it's challenging to evaluate quality, then people will shift to looking at secondary signals such as credentials, price, etc.
As to why a lot of economists go with the signaling model of education, well, it might just say something about their field and how much they got out of their own educations.
https://en.wikipedia.org/wiki/The_Case_Against_Education#Rev...
Bryan Caplan back and forth with Noah Smith on the book: https://www.econlib.org/archives/2015/04/educational_sig_1.h...
Bryan Caplan back and forth with Bill Dickens on the book: https://www.econlib.org/archives/2010/08/education_and_s.htm...
I wonder:
— Is math a problem for non-academic researchers?
Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but math nonetheless.
Not that you can't learn these on your own, but how big is the gap in practice, on the job, compared with actual PhDs in ML/math? (how much of a hinderance, a problem it is for the self-taught researcher)
— "Contracting" in the field of AI sounds great but, how exactly? Especially solo: what type of clients and how/where to find them, what type of 'business proposition' as a freelancer do you offer, what's the pricing structure of such gigs?
I mean, I can sell you websites and visuals and stuff, but AI? I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny and demand scripting skills to sort out (extract business value), not AI, so the value proposition is low for both parties; it's still early adoption so 90% don't even consider spending 1 cent on "AI" unless as a SaaS (they actually don't need to know if it's AI or programming).
I can imagine tons of fantastic research to do with SMBs, as partners or 'interested sponsors' (should they reap benefits on a low investment), but really not much yet in the way of "freelancer products" to market and sell for a living. I'm eagerly anticipating those days, but it's more like 2025-2030 as I see it.
I would love to hear first hand takes on this.
It takes a while to figure out how to read academic papers, but it's largely about learning the notation. In the end, it maps back to the code you write anyway in most cases, so it's just another way of writing stuff you already know.
It's not so much linear algebra you need, since much of that is not relevant to AI. It's really matrix calculus. Which is largely about multiplying things together and adding them up. Terence Parr and I tried to create a "all you need to know" tutorial here: https://explained.ai/matrix-calculus/ .
You certainly don't need topology (unless you happen to be interested in that particular sub-field).
It might be wrong but I tend to see vectors and matrices as two notations for the same mathematical object[1]. So I indeed meant matrices! However I didn't see calculus itself as such a big requirement, as it all felt pretty "linear" to me (regressions etc). Are we talking things that e.g. "Calc 2"[2] should cover?
I feel reassured by your first paragraph. This can be done.
I'll definitely work on your tutorial; I assume it's a good benchmark for math pre-requisites in the field. Thanks a lot for the work, and advice.
[1]: That was particularly reinforced with Geometric Algebra, which I'm currently diving in. https://en.wikipedia.org/wiki/Geometric_algebra
> Most papers strike me as requiring a non-trivial knowledge of linear algebra
I think this is correct, if you consider college level linear algebra and an intuition for applying it to novel problems to be non-trivial knowledge
Things like topology (e.g. TDA, persistent homology, etc.) aren't really mainstream yet, but even then most of it isn't really "hardcore" math in the sense that you can get away with a basic understanding, e.g. what a Vietoris-Rips complex is and why we use it instead of a Cech complex in TDA. Plus most DL research nowadays is pretty (advanced) math-light. That being said, taking the time to understand the math is absolutely worthwhile in my experience.
It should also be noted that a lot of real world ML/AI projects in industry aren't really about brand new algorithms using advanced math, but rather more about applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
Basically, i want a book like Statistical Rethinking or Blitzstein's Introduction to Probaiblity, but for linear algebra. And i havent been able to find it.
Thanks for the pointer! (link[1] for those interested)
> you can get away with a basic understanding
Great news to me!
> taking the time to understand the math is absolutely worthwhile in my experience.
Strongly agree — for any topic, any field. My concerns are practical indeed, and less about the 10-year horizon (well enough to become skilled at anything) than the early stages of that, the best way to propel oneself far/fast enough on year 1, then 2, etc.
> applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
I hear that. I actually do like the sound of that, hence concerns that I was biased.
[1]: https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra...
Thanks to transfer learning, tiny datasets are not a major issue to developing AI solutions. Fast.ai makes it super easy to overcome that hurdle.
Will investigate. Thanks a lot!
1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research"
2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1. What might be termed "basic" or "pure" research.
I'm not placing one of these above the other in terms of importance. They are both necessary, and they form a virtuous feedback loop between the two that, one without the other, would see the other wither on the vine.
In the example of this particular person, Emil Wallner, he appears to be doing #1, and perhaps doing so in a way that might help inform more of #2.
But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of overlap. When Leo Breiman [0] did his work that led to gradient boosting and tools like Xgboost, that was firmly #2.
It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications.
To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
I think this is what we try to capture as “expanding human knowledge”.
IMO the more isolated the result (“technique x gave good results for problem y, the end”), the less like “research” it is. Though plenty such papers get into good conferences every year. A nice story and a little reviewer luck go a long way.
#1 might ask about the performance of a deep neural network in approximating a given model in a specific application. Alphafold, on the front page currently, is an example of #1.
Personally I don't know enough about AlphaFold or the problems of protein folding to be remotely confident in my judgment on it
The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups).
The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.
Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reali...
http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T...
And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText
Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)
while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal),
fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough.
Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
There is still plenty you can do with a reasonable personal budget, however.
You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.
So don't feel bad about your life just because someone on the internet pretends to have a more interesting one. Those people are usually just attention seekers and for some reason need the outside validation to feel good about their achievements. And remember that not needing that validation can be a strength too!
I realize that these articles suffer from the connecting the dots thing, where people make connection in the present which they would never have in the past. But that is besides my point, even if he failed at all those things, I am still jealous he had the chance to try all these things.
We're about the age you describe, late-20s, early-30s. But our friends of around the same age with kids seem to have a lot less of this 'freedom'. Responsibilities take over.
Not saying you can't do those things with kids - but it does seem harder.
Here in Northern Europe you're expected to do most of your traveling in your 20's, but we also have pretty decent vacations - so the solo / friend / backpacking type of traveling gets replaced with more family friendly stuff when you start getting kids.
I have lots of friends in their late 20's / early 30's that still travel the world, many times a year. But they don't have kids, and their travels (outside summer vacation) tend to be shorter, as in long-weekends etc.
As I write this, my (admittedly limited) understanding of how Western society works makes me think these would not be problems but my biggest assets.
Don't believe it. Like him being king of the village??? LOL
Either way, this whole focus on "portfolios are everything and credentials are meaningless" spits in the face of all the work I did to get my university education. And it didn't involve "copying assignments". And you come out with one hell of a portfolio if you take your education seriously.
I mean I don't think self-educated people are without merit. I happen to think they're really important. But I only ever see them rag on higher education, despite them having "never been there".
Just another example of wunderkin super genius knows all because he was able to follow a non-standard path and make it. Glad he was smart enough to become a Google employee. But I question whether he should be giving advice on paths to get there when there's always many paths to a position. And especially after reading his brief comments on how credentials imply you're a liar.
Then actually pay attention to the arguments they're making instead of talking about how offended you are because it goes against your self-interest as a degree holder. It's not as if the people bashing modern education are some kind of elusive minority.
I've got a master's degree and I've always though our education system is stupid, and at least in the U.S. not unlike a giant pyramid scheme given the cost of tuition these days. Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
This is categorically false. Face-to-face time with an expert is incredibly valuable and incredibly expensive outside of an academic setting. In fairness, you have to show some initiative in college to get quality face-to-face time with a professor, but it still takes a lot less motivation than self-studying a complex subject for a nontrivial amount of time.
Which brings me to my second point. There's an enormous amount of free stuff you could learn from. But actually doing it is a completely different matter and the overwhelming majority will fail. For instance, the bulk of a university-level education in pure mathematics is over a century old, and free resources are easy to find. With stackexchange, you can even get expert feedback on your work! Yet most people who try (who are already a very self-selected sample) do not in fact succeed in teaching themselves undergraduate level mathematics. Even Ph.D. students taking a few years off for whatever reason find it highly (but not impossibly) difficult to do any significant amount of self-study for a prolonged period of time. And these are precisely the people who are training to become independent researchers!
I don't know who "you" is (perhaps you in particular are very gifted) or what you personally learned in college, but on my end in college I specifically took particular classes to learn topics that I had previously tried and failed to learn on my own, so, if it's meant to be generic, I'm pretty confident your claim is false.
Do you have any idea where the majority of massive breakthroughs in technology come from? They come from university.
AI started in university before it was even a thing. Autonomous vehicles were a university funded DARPA project. Nearly everything, even this guys research, is an important derivative of this.
Why do you think this guy chose to publish his paper in a academic journal? It's because it's peer reviewed. The Internet is not peer reviewed, it's public reviewed, as in anyone with an opinion can say whatever they want and get a million other opinions accepting or rejecting that opinion with little evidence. That's essentially what the entirety of Hacker News is. Very rarely do I see a post, including my own, that's properly sourced.
Let me ask you, do you think it's stupid that I paid money, like most people, to build a solar power management system? Do you think it's stupid that I built a memory management system, and text message system from bare bones hardware? Do you think it's stupid that I built my own shell? I did all that in school with equipment that I could only dream of owning with people that spent more time helping me than writing blog posts trying to get famous.
It's funny, I was actually glad this guy got where he wanted. It must be nice to be a genius. And to be honest, I'm probably not as smart as this guy. It's great that he had a lot of drive and achieved greatness. But he doesn't need to imply that I'm some liar because I chose to go to university. I worked incredibly hard to get my degree, spending hours and hours in the lab doing assignments. Hours thinking I was dumb and that I'd never make it. Months trying to find a job.
And all I see is people that did it a non-standard way and then have these warped views of the traditional way despite the decades of people lifted out of poverty because of it. And despite having never even attending a university! I see all these smart people and how they're the only ones that matter. College did a lot for me. And I try and do my best every single day because of it.
Sure, I may not be able to write an ML paper in a year like this guy, but that doesn't mean I'm not going to defend myself when he essentially implies I cheated my way in because I got a degree.
I'm not sure I have heard anyone claim that "higher education is useless" in general. Education is never useless in general.
> I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to self-fund your research.
Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020? How many are going to place in a Kaggle competition with just half a year of experience? More importantly, just how many people are privileged/secure enough to put their all into learning, with no sense of security or peer support?
> I started working with Google because I reproduced an ML paper, wrote a blog post about it, and promoted it. Google’s brand department was looking for case studies of their products, TensorFlow in this case. They made a video about my project. Someone at Google saw the video, though my skill set could be useful, and pinged me on Twitter.
So what really mattered was self-promotion, good timing, and luck.
> Tl;dr, I spent a few years planning and embarking on personal development adventures. They were loosely modeled after the Jungian hero’s journey with the influences of Buddhism and Stoicism.
Why does the author have to present his life like one would in a fucking college essay?
[0] https://medium.com/@andreas_madsen/becoming-an-independent-r...
Yes. He seems like someone who is good at self-promotion and networking. Well, good for him, but I think he underplays the role these have in his success.
> Why does the author have to present his life like one would in a fucking college essay?
I guess that's the self-promotion. And humble-bragging. Like this bit:
"I started working as a teacher in the countryside, but after invoking the spirit of their dead chief, they later annotated me the king of their village."
Exactly. Good for Emil, but it's always frustrating to hear survivorship bias preaching. Even the interviewer starts off by saying:
"By the way, I really love your CV - the quirks section was especially fun to read."
It's even more frustrating when I hear non-POC's talk about their journey to some non-western country (and subsequent conquering of fantastical goals like gaining the approval of locals) or pursuit of some sense of foreign culture. It's almost a given that they have internalized and appropriated the ideas (i.e. Buddhism or even worse post-retreat Buddhism). Good for the author to receive such positive feedback for such signaling, but it makes me sad to know that I might not receive the same.
I don't think the idea is to look for an internship after the course but an additional 4 months of personal projects. After applying state of the art deep learning for 4 months full time you'll have some very cool projects, and you could probably convince some company to take you on as an intern for a certain amount of time.
Also, Emil's approach to learning will create a flawed sense of expertise. Look at how the article presents him as if he has a deep-domain expertise which might not be true.
One important thing to consider is to look at the article more like content marketing tactic, that FloydHub is using promote its brand which might not serve well for engineers as it lacks some aspect of truth.
Is that really the case? Apart from the obvious "maybe that's what they're intrinsically interested in" - if you start with a problem, try to solve it, and "pure mathematics" (whatever exactly that is supposed to be, anyway) is required to arrive at a solution, it becomes part of the intrinsic motivation. And if you keep solving problems without it the question that eventually comes up is "is it really useful at that point?"
I do however agree that if you're looking at someone who's qualification is primarily his "portfolio", you do actually need to check whether it includes interesting problems, or at least projects that are related to what you need them to do at your company. But if that is the case, I really don't see a problem.
Few ex:
Like, reading literature is a purely fun and mentally draining activity that might/might not have any goal attached to it.
Like travelling, is a purely fun activity and might/might not have any intrinsic goal attached to it.
I started learning Algebra out of the random, without any intrinsic goal. Because, it was purely out of fun. Playing chess is an activity without any intrinsic goal.
Also finance... really? I'm not sure I would agree that is a subject with a less capitalistic motivation.
I don't get a lot of the bitterness here. I mean not everyone here creates their own programming language, or would know how to write a database, yet we are fine using them as tools. Why must we understand all the code and concepts in a neural network to apply it?
> Why must we understand all the code and concepts to apply it?
That is one of the reason why our engineers are so sub-par because we were told to just shut up and write shit. We could have become a force to be reckoned with because of our expertise, because of our ability to solve complex problems. Yet we are living in an industry were there is huge disparity in salary, structure, and principles.
I don't necessarily agree with why one shouldn't understand the concepts, I'm more a guy who says why not?
Because, if you are standing on the shoulders of pioneers and claiming to be improving their work at least do it with compassion and honesty.
Sorry about the rant.. would love to hear alternative opinions!
I think it's good that companies are willing to look into non-trad candidates, that may not have found their "calling", so to speak, until their late 20's / 30's or whatever. But it does start to sound contrived when a bunch of 'em have the same type of alternate-route stories, which involves traveling to Africa / India / SE Asia to help out kids, create some startup aimed at climate / poverty / equality / etc. I guess it makes you sound passionate and legit - no-one can say that you wasted your time on chasing those things.
Nothing on the quirks list is actually a quirk. They're interesting things he's done that other people wrote books about, received praise for, and then he followed their newer, well-traveled path.
It's not a non-traditional background. He's not a refugee who managed to learn coding. He's not volunteering at a needle exchange clinic. I think that's what's bothering me; he's pretending to be interesting, and taking the room which could be going to someone else.
Thank you for helping me get to why something felt off. Appreciated, internet stranger.
I would rather build a small company by solving a real problem than work for a big company spinning my wheels.
I think if you look at history this is also evident: the inventions of the late 18th century were a function of necessity, the invention of semis (not just in the US but how Taiwan developed)...this isn't to say academia is pointless but there is just far more going on (I think if you look at some of the East Asian nations that get great academic results, their progress on actual R&D innovation is far less impressive).
For a thoughtful counterpoint to the necessity argument, see: https://jnd.org/technology_first_needs_last/ (previously discussed on HN)
Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)? I can't help but wonder a bit.
1. Solves previously unsolved problems
2. Publishes papers sharing those solutions
without regard to the kind/spirit/scope of problems solved.
Since conference publications don’t have the same number constraints as journal papers, and are accepting of application-specific results, this explosion of what is considered “research” is somewhat inevitable. Also, there are a lot of people chasing this given the prestige associated with the title.
From his GH profile looks like he's a competitive applicant for ML engineering positions or perhaps a fellowship/residency/PhD program.
So, a junior researcher at the level of a decent second or third year PhD student. A researcher, maybe someone you'd trust to build a prototype or product, lots of potential, but probably not someone you'd trust to run a research program.
3 months learning FastAI, 3-12 months personal projects and consulting, 2 months flashcards of ~100 papers, 6 months to publish a paper
What does he mean by ‘paper’? A Medium post? NeurIPS?
But yeah, going from 6 months of programming experience with C, to a Deep Learning internship - that sounds a bit far stretched.
I have some friends in Oxford who are DPhil/Postdocs in highly reputable research departments specializing in ML and if they sometimes struggle to get more than a poster session at the leading conferences, with the addition of well known professors names attached, then there's just no way I can believe Joe Bloggs who just learnt python 12 months ago is able to do the same.
I almost want to follow his guide just to check.
> Early evidence of practical knowledge often comes from usage metrics on GitHub, or reader metrics from your work blog. Progress in theoretical work starts by having researchers you consider interesting engage with your work.
> Taste has more to do about character development than knowledge. You need taste to form an independent opinion of a field, having the courage to pursue unconventional areas and to not get caught up in self-admiration.
When I study abstract interpretation or lattices, I'm doing so because I find those subjects interesting and beautiful, and studying math relaxes me. I can lie to myself and say that it's improving my problem solving ability and that it's like doing mental yoga and will make me better at my job or some baloney, but that's not why I do it.
I can spend time with a plant in my garden, take a cutting, root it and replant it, and watch it grow, learn the ebbs and flows of its watering needs through the seasons, learn what its seed pods look like, and eventually watch it die through some misstep of my own or otherwise.
And in doing so, I am learning, and building a mental model for this plant and an intuition for it, but I'm not "creating value" in some weird capitalist sense, which I feel always underlies these sorts of opinions about learning and education, and people who self-identify as "makers" in general. It rubs me the wrong way because it encourages a very narrow view of the human experience and what it means to learn and why we should learn.
FAKE.
They're free to do so of course, but they should not give advice based on it.
Regardless of that, I suppose the bar for being a "researcher" has been stooped so low. According to this guy publishing an ML paper is equivalent to writing a blog post or making a video about "AI".
Seems to me the difficult part is how to support yourself financially while spending your time doing interesting learning and research, or how to get paid to do it
Maybe the most important detail in the story is "He co-founded a seed investment firm that focuses on education technology" but it is not discussed further
nice to be able to work for free and not starve
1) There is the general phenomena or collective project, where hardware, algorithms and human insights are improved to approach the situation of man-made intelligent machines.
2) There are the people who are designing algorithms, using mathematical intuition and knowledge, analogies with physics, etc... Most people would agree these people are doing optimization / machine learning "proper".
3) There are the people working on improving hardware for machine learning / optimization purpouses, by looking at the most performant algorithms, breaking them down into primitive operations and requirements for hardware, there are also people working on the algorithms themselves and finding computational shortcuts (which can end up in software or hardware, can end up as proprietary knowledge or common knowledge, ...). The distinction between hard and software is somewhat blurry, since hardware designers can optimize or implement a section of software into hardware. A lot of this can still be considered ML "proper".
4) Then there are the people who apply the ML frameworks and their exposed choices and settings to a specific problem domain. Many of them don't need to understand the internals if they don't need state of the art results. Many would nevertheless benefit from understanding the internals, and the requisite math. What I propose is to stop calling their activity as Machine Learning, and instead call it Machine Teaching. They are teachers, and just like elite schools they can choose which specific type of available student they will teach, and they can tweak (or filter from a large family of students) which student they select to teach the task at hand. There are bound to be many advantages of having actual human teachers get involved in machine teaching. These people will not be proficient in designing novel families of students unless they also know the requisite math, and identify those ML papers that are ML "proper" instead of ML "teacher". When trying to find important foundational insights in ML "proper" one is typically overwhelmed by a large surplus of ML "teacher" type papers. These are important datapoints, and necessary to advance human insight into ML "proper", but they are data, not knowledge. There are actual ML "proper" knowledge papers out there that explain why a certain phenomena is such and so, and they get very little attention because they necessarily lag the breakthrough ML datapoint paper, and most ML "teachers" don't have the math background to understand them. So the probability that a given ML "proper" researcher fundamentally improves the state of the art is much higher than the probability that a given ML "teacher" will fundamentally improve the state of the art. At the same time the probability that a given fundamental breakthrough was achieved by an ML "teacher" is higher than the probability that a given fundamental breakthrough was achieved by an ML "proper" researcher:
P( Breakthrough | Proper ) > P ( Breakthrough | teacher)
while
P ( Teacher | Breakthrough ) > P ( Proper | Breakthrough )
Since most people don't have the broad math / physics / ... knowledge to draw on, the number of ML "teachers" is much higher than ML "proper" researchers.
[1] well, really, some actors have vested interests in conflating those together...
EDIT: just to be clear, I am not complaining about ML Teachers, we need the ML Teachers, and their breakthrough datapoints. What I am complaining about, is conflating both activities of ML Proper and ML Teaching. This makes it harder for the few ML Proper researchers to find each other's insights.