undefined | Better HN

0 pointseterm4y ago0 comments

We never get a good sense of how much time was actually saved with that change not least because the original function calls "initialise weights" inside every loop, the new function does not. It would have been interesting to see what difference that alone made.

The takeaway of the article, that computers are blindingly fast and we make them do unecessary work (and often sit around waiting on I/O) with most their time is true of course.

I'm currently writing a utility to do a basic benchmark of data structures and I/O and it's been a real learning experience for me in just how fast computers can be, but also just how slow a little bit of overhead or contention can cause things, but that's better left for a full write up another day.

0 comments

1 comments · 1 top-level

geph20214y ago

   We never get a good sense of how much time was actually saved with that change not least because the original function calls "initialise weights" inside every loop, the new function does not.

Good point. Furthermore to your point, I would assume a library like pandas has fairly well optimized group and sort operations. It would not occur to me that pandas is the bottleneck, but the author does clarify in his footnote that pandas operations, by virtue of creating more complex pandas objects, can indeed be a bottleneck.

   [1] Please don't get me wrong. Pandas is pretty fast for a typical dataset but it's not the processing that slows down pandas in my case. It's the creation of Pandas objects itself which can be slow. If your service needs to respond in less than 500ms, then you will feel the effect of each line of Pandas code.

j / k navigate · click thread line to collapse