Finding Signal in the Noise: Machine Learning and the Markets (Jane Street) (opens in new tab)

I don't think they'd ever call someone like me back, not that I'd ever be able to pass a coding round with them.

People like me are "NPCs" in their parlance.

Any time a notebook needed to be tested, refactoring it into a module and testing that was the better choice.

Notebooks in my opinion are by far the best tool for interactive and exploratory data work. I've been using Jupyter/IPython for 10 years and it takes very little discipline to keep them clean and clear. I've never bothered trying to deploy them in any meaningful way.

The quirks with version control are annoying, that's one reason I've switchex to marimo in the last few months.

When you are exploring something, experimenting, showing.. it’s great; train-of-thought structure, APIs like Pandas optimised for writing and terseness etc.

But when you have a piece of code that will lose a million dollars a minute if someone ships a bug, and which will be maintained by many engineers over many years, then you really want a format that’s optimised for long-term maintenance, incremental change, testability, and APIs optimised for readers.

Writing clean data code is one aspect. Filling in knowledge gaps is another. Covertly teaching software engineering best practices to folks who "aren't programmers" yet sit down and write code in Jupyter all day is another.

I should probably write a blog post or record a short video. (I just taught a week long course on this for a client last week.)

This error seems to be a particularly common (and often lauded!) trait among those who work in high-conjecture low-evidence fields (eg, economics). The prominent thinkers become skillful at deploying this fallacy: “see, it’s there, therefore <insert-personal-belief> is certainly the cause!”, using their credentials and esteem to mask the error. Listeners think, “well he’s a smart, respected guy,” and nod along despite the missing logical link.

I greatly appreciate Cowen’s podcast, and I definitely respect him as a thinker and inquisitor – so I don’t mean to discard his work or opinions (in fact, I appreciate his occasional brashness because it exposes the underlying thought/principle). However, many of his aggressive-yet-speculative statements (like the one you roughly quoted) are best received with an understanding of the error.

Complete garbage. The same way that Jane Street hires smart people that don't know anything about trading and those people contribute, the same would be true if there was money in curing cancer.

I don't think there's a gene for playing esoteric minigames on the options market while you literally suck at everything else

Derivatives have great value to industry. Derivatives that require the fuel of 15 math Phds to lock in fractions of a percentage pricing inefficiencies so their firm can pocket the difference do not have much value at all.

It's like employing Harvard Med surgeons to remove gold dust from gold market sidewalks, and calling it "gold market efficiency".

We have far, far too much of our economy sunk into a sector that fundamentally produces nothing of value, one that only shuffles value around and chooses to whom it will be allocated (coincidentally, often the people doing the allocating!).

Compare our economy and its woes to China. Do they spend nearly as much of their human capital figuring out how to get an extra bp on some statistical arb? Thethe origins of DeepSeek are illustrative in this regard.

But, ok, Spread Networks spent $300m around 2010 to build a somewhat straighter glass fibre data connection between NY and Chicago. Other companies spent more to build a micro wave connection, since microwaves move faster through air than light through fibre.

Is that money really well spent? And I'm afraid a lot of what happens in finance is similarly private gain, without much social welfare increase.

This has been the case for well over 200 years and is not done by the same kind of people that modern finance/fintech employs.

Imagine the world 150 years ago and imagine were you'd allocate the smart people that's the highest paid these days. They'd be doing science and developing technology instead of doing finance. Why is finance that much more important now? (is it just because it makes rich people richer?)

In practice, if the availability of the apples you're eating is dependent on finance people, they're commodity grade, not specialty varieties.

I don't eat apples, but I'd rather have a good heirloom tomato from the farmer's market than the commodity grade stuff that's at the grocery store.

See, if you're a fishmonger and someone comes to you and say "why don't you trade flowers instead", you ignore them because they are not your customer. They won't trade fish with you, and in fact they wouldn't trade flowers either. They're just useless relative to your trade and only yapping so just turn your back on them and leave them be. Tell them to fuck off and find a flower monger if they really want it or mong those those flowers themselves because you are sticking to monging fish.

The inverse is true too: lots of things provide value that no one pays for.

I find it shocking that anyone that programs would think this way considering how widespread and common open source tooling is. I don't know how you can get through your day without using OSS or even free websites like stack overflow.

I'd argue that pulling these people out of moving society forward is just another form of externalized cost. The stanford guy who figured out how to halve the cost of solar in 2012 never got to realize it because $750k/yr to do stochastic modeling of cattle feed to (secretly) overcharge farmers for futures contracts was just too enticing.

No, this would be the alternative to securities and derivatives markets being electronic.

>If it didn’t provide value, nobody would pay for it.

The parent was remarking on the value provided to society, not the value provided to the firms.

See for example “effective altruism”, which turns out to have been neither, but more of a self-deluded justification for insane greed, coupled with a god complex.

I'd definitionally describe all voluntary transactions free of coercion to imply the buyer values the utility of what they're buying (i.e. true wealth - piles of currency are not true wealth, they're what you exchange for true wealth) more than the currency they're trading for it, no?

Involuntary transactions featuring coercion on the other hand, like the government demanding you pay taxes under threat of imprisonment (enforced at gunpoint, if necessary) are clearly extractive, by my definition.