FC8 – Faster 68K Decompression (2016) (opens in new tab)

(bigmessowires.com)

68 pointselectricant1y ago10 comments

10 comments

I'm working on a game for Amiga (another 68k-based platform) and settled on ZX0 to decompress assets on the fly: https://github.com/einar-saukas/ZX0

I was originally using LZ4, but I switched to ZX0 after learning that it can do in-place decompression, which means I don't have to allocate separate memory for the compressed data. I'm very happy with the compression ratio, and decompression of large assets (~48kb) only takes a few frames on a 7MHz 68000.

Also of note is LZ4W, included in Sega Genesis Dev Kit (and discussed in the comments section of OP's article), a variant of LZ4 that only uses word-aligned operations. That makes it much faster on the 68000, which can struggle to efficiently handle 8-bit data. More info here: https://github.com/Stephane-D/SGDK/blob/master/bin/lz4w.txt

geon1y ago

I wrote a naive lz77 packer/unpacker in C for a c64 game. https://github.com/geon/woorm/blob/master/tools/lz77.c

Not fast, but the compression ratio was decent, and made it easy to fit a bunch of levels into the game.

bananaboy1y ago

Nice! When you say “on the fly” are you literally decompressing assets from disk during gameplay? Can you do asynchronous IO on the Amiga?

dansalvato1y ago

As of now, I'm loading all the level assets in advance, but many of the assets in RAM stay compressed until they're needed, such as backgrounds and fullscreen graphics. I decompress those asynchronously into video memory when I need them. The second video in this blog post shows the decompression happening onscreen in realtime: https://dansalva.to/coding-the-anime-woosh-screen-on-amiga/

1 more reply

the-rc1y ago

It's actually harder to do synchronous floppy I/O on the Amiga. Data transfer is done over DMA, then you perform MFM decoding. The latter can be done by the CPU, or, asynchronously again, by the blitter, in which case you can't use it at the same time for graphic operations.

1 more reply

miohtama1y ago

In “faster than memcpy” series we have also Blosch for modern CPUs

https://www.blosc.org/pages/blosc-in-depth/

I have never been able to use Blosch myself. But it sounds really interesting, outperforming RAM. Not sure what are the applications - columnar data processing, Parquet files, etc?

jeramey1y ago

It gets used a fair amount in the weather data space. Forecasting and climate reanalysis grids are typically large (gigabytes) N-dimensional arrays of float32 values and Blosc provides enough tunable knobs that it's fairly easy to find a combination that performs acceptably without writing a bunch of custom handling code to keep track of which underlying compression schemes and settings were used. Additionally, it supports byte- and bit-shuffling filters which can really help boost the compressibility of certain data sets.

thechao1y ago

Why doesn't BLOSC have a little chart comparing itself to LZ4, Zstd, etc? Kinda like this:

https://stackoverflow.com/questions/37614410/comparison-betw...

Because it seems like such a trivial chart to make.

j / k navigate · click thread line to collapse

10 comments

dansalvato1y ago

I'm working on a game for Amiga (another 68k-based platform) and settled on ZX0 to decompress assets on the fly: https://github.com/einar-saukas/ZX0

geon1y ago

I wrote a naive lz77 packer/unpacker in C for a c64 game. https://github.com/geon/woorm/blob/master/tools/lz77.c

Not fast, but the compression ratio was decent, and made it easy to fit a bunch of levels into the game.

bananaboy1y ago

Nice! When you say “on the fly” are you literally decompressing assets from disk during gameplay? Can you do asynchronous IO on the Amiga?

dansalvato1y ago

1 more reply

the-rc1y ago

1 more reply

miohtama1y ago

In “faster than memcpy” series we have also Blosch for modern CPUs

https://www.blosc.org/pages/blosc-in-depth/

I have never been able to use Blosch myself. But it sounds really interesting, outperforming RAM. Not sure what are the applications - columnar data processing, Parquet files, etc?

jeramey1y ago

thechao1y ago

Why doesn't BLOSC have a little chart comparing itself to LZ4, Zstd, etc? Kinda like this:

https://stackoverflow.com/questions/37614410/comparison-betw...

Because it seems like such a trivial chart to make.

j / k navigate · click thread line to collapse