undefined | Better HN

0 pointsmicro_cam9y ago0 comments

Have you had performance issues getting things to conform to functional paradigms?

For example i've found that as a pipeline gets optimized for production use it needs to preallocate all of its output space and then modify things in at each step (like a one hot encoder flipping a few bits in specific rows of a zeroed array instead of allocating new ones and copying them in).

I find it difficult to reconcile this sort of code with a "pure functions without side effects" philosophy and still have it perform an an acceptable level.

0 comments

SatvikBeri9y ago

We're mostly doing ETL on large datasets, so the code needs to parallelize well, but beyond that performance isn't really a big concern. We use ML in research, but no models in production, because the costs of increased maintenance/lost transparency generally outweigh the benefits in our use case.

In jobs that were heavy on ML, I would use high-performance tools for the models (imperative code, numeric computing packages etc.) and functional code for the ETL, which worked pretty well–no need to be dogmatic about it, a 70% pure codebase is still generally easier to reason about than a 20% pure codebase.

j / k navigate · click thread line to collapse

0 pointsmicro_cam9y ago0 comments

Have you had performance issues getting things to conform to functional paradigms?

I find it difficult to reconcile this sort of code with a "pure functions without side effects" philosophy and still have it perform an an acceptable level.

0 comments

SatvikBeri9y ago

j / k navigate · click thread line to collapse