undefined | Better HN

0 pointsAlotOfReading11mo ago0 comments

Yes, it's a pretty important error case with storage devices. It's so common that modern storage devices and filesystems include their own protections against it. Your system may or may not have these and bit flips may happen after that point, so WAL redundancy wouldn't be out of place.

Sure, the benefits to the incomplete write use case are limited, but there's basically no reason to ever use a fletcher these days.

It's also worth mentioning that the VFS checksums are explicitly documented as guarding against storage device bitrot and use the same fletcher algorithm.

0 comments

4 comments · 1 top-level

lxgr11mo ago· 3 in thread

It would be absolutely out of place if your lower layers already provided it, and not enough if they didn't (since you'd then also need checksums on the actual database file, which SQLite does not provide – all writes happen again there!)

AlotOfReadingOP11mo ago

sqlite does actually provide database checksums, via the vfs extension I mentioned previously.

There's no harm to having redundant checksums and it's not truly redundant for small messages. It's pretty common for systems not to have lower level checksumming either. Lots of people are still running NTFS/EXT4 on hardware that doesn't do granular checksums or protect data in transit.

Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

lxgr11mo ago

> There's no harm to having redundant checksums

There sure is: Redundant checksums need extra storage and extra processing. SQLite often runs on embedded systems, where both can come at a premium.

> Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption. At a very minimum, you'd also need checksums of the actual database file.

AlotOfReadingOP11mo ago

There's no extra storage because it's already storing checksums. There's technically extra processing, but a good CRC is faster than reading memory on virtually all modern systems (even in embedded) so there's no reason to prefer a fletcher unless the few extra cycles of latency to get the CRC running are deeply concerning to you.

    That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption.

A CRC wouldn't provide full resistance anyway, so this clearly isn't what I'm arguing. What I'm saying is that a fletcher is strictly worse than a CRC here. Not an advanced algorithm, a bog-standard CRC. Fletcher actually takes more space in this case, has worse error detection, and doesn't allow recovery. A CRC would allow you to recover some of the time, from some kinds of errors, in some scenarios. It's simply better than nothing.

1 more reply

j / k navigate · click thread line to collapse

0 comments

4 comments · 1 top-level

lxgr11mo ago· 3 in thread

AlotOfReadingOP11mo ago

sqlite does actually provide database checksums, via the vfs extension I mentioned previously.

Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

lxgr11mo ago

> There's no harm to having redundant checksums

There sure is: Redundant checksums need extra storage and extra processing. SQLite often runs on embedded systems, where both can come at a premium.

> Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption. At a very minimum, you'd also need checksums of the actual database file.

AlotOfReadingOP11mo ago

    That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption.

1 more reply

j / k navigate · click thread line to collapse