undefined | Better HN

0 pointstetha2y ago0 comments

> Not in the context of game development, however! There you typically don't care about wall time. Instead, you work towards a set frame rate meaning you have a set slice of time (usually 16.7 or 33.3 ms) to go through the entire game and render loop each time.

I was about to bring a similar example: Some of our really old daily data processing at work might need some attention and optimization in the future, because we're running out of hours in the day. And we haven't found a place we can buy more hours in the day from yet.

0 comments

karamanolev2y ago

If you can run the daily batch processing in parallel, can't you have one batch start at T+0, the next one starts at T+24h, then the first one finishes at T+28h and so on?

tethaOP2y ago

Sadly, the resource constraints/setup prevent us from parallelizing this. And the customers of that system expect the data to be processed and available after 24 hours.

In part, it's a somewhat rewarding topic. Thinking about queries, joining a bit differently, adding another index based on new data patterns can cut hours of runtime without incurring further resource cost.

But on the other hand, it's yet another project someone dumped on the floor and we were forced to adopt it "because of the customer". And the second or third project of PD trying to "do it right" is teetering on failure once again. Cron running shell scripts held together with chicken wire and duct tape is too strong of a stack I guess.

antoinealb2y ago

That leads to an infinite backlog no ? If you need more than 24h to process 24h of data ?

Sayrus2y ago

That may depend on the context and data but you may end the first job at T+28 (runtime of 28 hours) and the second at T+52 (28 hours as well, started at T+24).

If jobs must be executed one after another, then you absolutely create an infinite backlog.

j / k navigate · click thread line to collapse