undefined | Better HN

0 pointsbombcar3y ago0 comments

The total size of all item data for a store like Target can't be much more than what, a few gigabytes? Or at least the "UPC -> Price" dataset. So download the whole dataset each night, if you can't get delta changes to work.

And if that had failed somehow, it would have been noticed immediately upon the new code roll-out.

The internet was designed to be extremely resilient to host/route losses, we've made it so reliable we assume all machines are reachable at all times.

(To be fair, apparently they "do" have this but the dataset is printed on the items and the cashiers had to enter it by hand)

0 comments

jaywalk3y ago

Their system was basically designed the way you're saying, with a fallback to grab the data from the central location if it's missing locally. What you're asking for is the same system without a fallback, which doesn't make any sense.

kevan3y ago

It's counterintuitive but when you're dealing with distributed systems lots of things are: https://aws.amazon.com/builders-library/avoiding-fallback-in...

bombcarOP3y ago

Exactly - they had a fallback system that worked well enough for the testing to pass, but not well enough for the main system to operate on it.

Interestingly enough the Amazon example there is basically exactly what happened to Target.

bombcarOP3y ago

The fallback was the problem - design it without it, or with a manual window that pops up saying "ITEM NOT FOUND, QUERY TARGET ORACLE" or something, and the fault wouldn't have taken down the whole company.

If suddenly every cashier is being forced to hit OK on every item, people would hear about it immediately from the test rollout instead of when it hit everything (of course, assuming you have good methods for detecting things like this and don't just completely ignore associates' complaints).

j / k navigate · click thread line to collapse

0 comments

jaywalk3y ago

kevan3y ago

It's counterintuitive but when you're dealing with distributed systems lots of things are: https://aws.amazon.com/builders-library/avoiding-fallback-in...

bombcarOP3y ago

Exactly - they had a fallback system that worked well enough for the testing to pass, but not well enough for the main system to operate on it.

Interestingly enough the Amazon example there is basically exactly what happened to Target.

bombcarOP3y ago

j / k navigate · click thread line to collapse