undefined | Better HN

0 pointssrean1y ago0 comments

> It's how every major object storage platform works.

Very interesting. Could you name a few, am curious. I would be happy if erasure codes are actually being used commercially.

What I find interesting is the interaction of compression and durability -- if you lose a few compressed bytes to reconstruction error, you lose a little more than a few. Seems right up rate-distortion alley.

0 comments

Twirrim1y ago

That I know of (and is public so I'm not breaching any NDA) AWS S3[1], Azure[2], GCP[3], Backblaze[4], Facebook's storage layer uses it[5][6], and Oracle Cloud's Object Storage Platform[7].

The economies of scale mean that you really have to have something like erasure encoding in place to operate at large scale. The biggest single cost for cloud providers is the per-rack operational costs, so keeping the number of racks down is critical.

[1]https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...

[2]https://www.usenix.org/system/files/conference/atc12/atc12-f...

[3]https://cloud.google.com/storage/docs/availability-durabilit...

[4]https://www.backblaze.com/blog/reed-solomon/

[5]https://www.usenix.org/conference/hotstorage13/workshop-prog...

[6]https://research.facebook.com/publications/a-hitchhikers-gui... they even do some interesting things with erasure encoding and HDFS

[7] https://blogs.oracle.com/cloud-infrastructure/post/first-pri...

pas1y ago

Ceph has a very stable EC feature. And lot of companies use Ceph as a storage backend. Unfortunately I cannot find any straightforward statement about a commercial offering, but I would bet that DreamHost's DreamObjects does use it.

While it's not "commercial", but CERN uses it and many other institutions.

https://indico.cern.ch/event/941278/contributions/4104604/at... --- 50PB

...

ah, okay, finally an AWS S3 presentation that mentions EC :)

https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...

j / k navigate · click thread line to collapse