undefined | Better HN

0 pointsmort965mo ago0 comments

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158

I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch of random stuff really.

I compressed them all with 'zstd --ultra -22', 'brotli -9', 'xz -9' and 'gzip -9'. Here are the results:

    +------+------+-----+------+--------+
    | none | zstd | xz  | gzip | brotli |
    +------|------|-----|------|--------|
    | 47M  | 45M  | 39M | 38M  | 37M    |
    +------+------+-----+------+--------+

Here's a table with all the files:

    +------+------+------+------+--------+
    | raw  | zstd | xz   | gzip | brotli |
    +------+------+------+------+--------+
    | 12K  | 12K  | 12K  | 12K  | 12K    |
    | 20K  | 20K  | 20K  | 20K  | 20K    | x5
    | 24K  | 20K  | 20K  | 20K  | 20K    | x5
    | 28K  | 24K  | 24K  | 24K  | 24K    |
    | 28K  | 24K  | 24K  | 24K  | 24K    |
    | 32K  | 20K  | 20K  | 20K  | 20K    | x3
    | 32K  | 24K  | 24K  | 24K  | 24K    |
    | 40K  | 32K  | 32K  | 32K  | 32K    |
    | 44K  | 40K  | 40K  | 40K  | 40K    |
    | 44K  | 40K  | 40K  | 40K  | 40K    |
    | 48K  | 36K  | 36K  | 36K  | 36K    |
    | 48K  | 48K  | 48K  | 48K  | 48K    |
    | 76K  | 128K | 72K  | 72K  | 72K    |
    | 84K  | 140K | 84K  | 80K  | 80K    | x7
    | 88K  | 136K | 76K  | 76K  | 76K    |
    | 124K | 152K | 88K  | 92K  | 92K    |
    | 124K | 152K | 92K  | 96K  | 92K    |
    | 140K | 160K | 100K | 100K | 100K   |
    | 152K | 188K | 128K | 128K | 132K   |
    | 188K | 192K | 184K | 184K | 184K   |
    | 264K | 256K | 240K | 244K | 240K   |
    | 320K | 256K | 228K | 232K | 228K   |
    | 440K | 448K | 408K | 408K | 408K   |
    | 448K | 448K | 432K | 432K | 432K   |
    | 516K | 384K | 376K | 384K | 376K   |
    | 992K | 320K | 260K | 296K | 280K   |
    | 1.0M | 2.0M | 1.0M | 1.0M | 1.0M   |
    | 1.1M | 192K | 192K | 228K | 200K   |
    | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M   |
    | 1.2M | 1.1M | 1.0M | 1.0M | 1.0M   |
    | 1.3M | 2.0M | 1.1M | 1.1M | 1.1M   |
    | 1.7M | 2.0M | 1.7M | 1.7M | 1.7M   |
    | 1.9M | 960K | 896K | 952K | 916K   |
    | 2.9M | 2.0M | 1.3M | 1.4M | 1.4M   |
    | 3.2M | 4.0M | 3.1M | 3.1M | 3.0M   |
    | 3.7M | 4.0M | 3.5M | 3.5M | 3.5M   |
    | 6.4M | 4.0M | 4.1M | 3.7M | 3.5M   |
    | 6.4M | 6.0M | 6.1M | 5.8M | 5.7M   |
    | 9.7M | 10M  | 10M  | 9.5M | 9.4M   |
    +------+------+------+------+--------+

Zstd is surprisingly bad on this data set. I'm guessing it struggles with the already-compressed image data in some of these PDFs.

Going by only compression ratio, brotli is clearly better than the rest here and zstd is the worst. You'd have to find some other reason (maybe decompression speed, maybe spec complexity, or maybe you just trust Facebook more than Google) to choose zstd over brotli, going by my results.

I wish I could share the data set for reproducibility, but I obviously can't just share every PDF I happened to have laying around in my downloads folder :p

+------+------+-----+------+--------+ | none | zstd | xz | gzip | brotli | +------|------|-----|------|--------| | 47M | 45M | 39M | 38M | 37M | +------+------+-----+------+--------+

+------+------+------+------+--------+ | raw | zstd | xz | gzip | brotli | +------+------+------+------+--------+ | 12K | 12K | 12K | 12K | 12K | | 20K | 20K | 20K | 20K | 20K | x5 | 24K | 20K | 20K | 20K | 20K | x5 | 28K | 24K | 24K | 24K | 24K | | 28K | 24K | 24K | 24K | 24K | | 32K | 20K | 20K | 20K | 20K | x3 | 32K | 24K | 24K | 24K | 24K | | 40K | 32K | 32K | 32K | 32K | | 44K | 40K | 40K | 40K | 40K | | 44K | 40K | 40K | 40K | 40K | | 48K | 36K | 36K | 36K | 36K | | 48K | 48K | 48K | 48K | 48K | | 76K | 128K | 72K | 72K | 72K | | 84K | 140K | 84K | 80K | 80K | x7 | 88K | 136K | 76K | 76K | 76K | | 124K | 152K | 88K | 92K | 92K | | 124K | 152K | 92K | 96K | 92K | | 140K | 160K | 100K | 100K | 100K | | 152K | 188K | 128K | 128K | 132K | | 188K | 192K | 184K | 184K | 184K | | 264K | 256K | 240K | 244K | 240K | | 320K | 256K | 228K | 232K | 228K | | 440K | 448K | 408K | 408K | 408K | | 448K | 448K | 432K | 432K | 432K | | 516K | 384K | 376K | 384K | 376K | | 992K | 320K | 260K | 296K | 280K | | 1.0M | 2.0M | 1.0M | 1.0M | 1.0M | | 1.1M | 192K | 192K | 228K | 200K | | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.2M | 1.1M | 1.0M | 1.0M | 1.0M | | 1.3M | 2.0M | 1.1M | 1.1M | 1.1M | | 1.7M | 2.0M | 1.7M | 1.7M | 1.7M | | 1.9M | 960K | 896K | 952K | 916K | | 2.9M | 2.0M | 1.3M | 1.4M | 1.4M | | 3.2M | 4.0M | 3.1M | 3.1M | 3.0M | | 3.7M | 4.0M | 3.5M | 3.5M | 3.5M | | 6.4M | 4.0M | 4.1M | 3.7M | 3.5M | | 6.4M | 6.0M | 6.1M | 5.8M | 5.7M | | 9.7M | 10M | 10M | 9.5M | 9.4M | +------+------+------+------+--------+

0 comments

13 comments · 5 top-level

terrelln5mo ago· 3 in thread

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M |

Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file size by more than a very small percent, like any compressor. Am I interpreting it correctly that you're doing something like `zstd -22 --ultra $FILE && wc -c $FILE.zst`?

If you can reproduce this behavior, can you please file an issue with the zstd version you are using, the commands used, and if possible the file producing this result.

mort96OP5mo ago

Okay now this is weird.

I can reproduce it just fine ... but only when compressing all PDFs simultaneously.

To utilize all cores, I ran:

    $ for x in *.pdf; do zstd <"$x" >"$x.zst" --ultra -22 & done; wait

(and similar for the other formats).

I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization:

    $ for x in *.pdf; do zstd <"$x" >"$x.zst" --ultra -22; done

That one file becomes 1.1M, and the total size of *.zst is 37M (competitive with Brotli, which is impressive given how much faster it is to decompress).

What's going on here? Surely '-22' disables any adaptive compression stuff based on system resource availability and just uses compression level 22?

terrelln5mo ago

Yeah, `--adaptive` will enable adaptive compression, but it isn't enabled by default, so shouldn't apply here. But even with `--adaptive`, after compressing each block of 128KB of data, zstd checks that the output size is < 128KB. If it isn't, it emits an uncompressed block that is 128KB + 3B.

So it is very central to zstd that it will never emit a block that is larger than 128KB+3B.

I will try to reproduce, but I suspect that there is something unrelated to zstd going on.

What version of zstd are you using?

1 more reply

Zekio5mo ago

doesn't zstd cap out at compression level 19?

1 more reply

noname1205mo ago· 3 in thread

Could you add compression and decompression speeds to your table?

mort96OP5mo ago

I just did some interactive shell loops and globs to compress everything and output CSV which I processed into an ASCII table, so I don't exactly have a pipeline I can modify and re-run the tests with compression speeds added ... but I can run some more interactive shell-glob-and-loop-based analysis to give you decompression speeds:

    ~/tmp/pdfbench $ hyperfine --warmup 2 \
    'for x in zst/*; do zstd -d >/dev/null <"$x"; done' \
    'for x in gz/*; do gzip -d >/dev/null <"$x"; done' \
    'for x in xz/*; do xz -d >/dev/null <"$x"; done' \
    'for x in br/*; do brotli -d >/dev/null <"$x"; done'
    Benchmark 1: for x in zst/*; do zstd -d >/dev/null <"$x"; done
      Time (mean ± σ):     164.6 ms ±   1.3 ms    [User: 83.6 ms, System: 72.4 ms]
      Range (min … max):   162.0 ms … 166.9 ms    17 runs
    
    Benchmark 2: for x in gz/*; do gzip -d >/dev/null <"$x"; done
      Time (mean ± σ):     143.0 ms ±   1.0 ms    [User: 87.6 ms, System: 43.6 ms]
      Range (min … max):   141.4 ms … 145.6 ms    20 runs
    
    Benchmark 3: for x in xz/*; do xz -d >/dev/null <"$x"; done
      Time (mean ± σ):     981.7 ms ±   1.6 ms    [User: 891.5 ms, System: 93.0 ms]
      Range (min … max):   978.7 ms … 984.3 ms    10 runs
    
    Benchmark 4: for x in br/*; do brotli -d >/dev/null <"$x"; done
      Time (mean ± σ):     254.5 ms ±   2.5 ms    [User: 172.9 ms, System: 67.4 ms]
      Range (min … max):   252.3 ms … 260.5 ms    11 runs
    
    Summary
      for x in gz/*; do gzip -d >/dev/null <"$x"; done ran
        1.15 ± 0.01 times faster than for x in zst/*; do zstd -d >/dev/null <"$x"; done
        1.78 ± 0.02 times faster than for x in br/*; do brotli -d >/dev/null <"$x"; done
        6.87 ± 0.05 times faster than for x in xz/*; do xz -d >/dev/null <"$x"; done

As expected, xz is super slow. Gzip is fastest, zstd being somewhat slower, brotli slower again but still much faster than xz.

    +-------+-------+--------+-------+
    | gzip  | zstd  | brotli | xz    |
    +-------+-------+--------+-------+
    | 143ms | 165ms | 255ms  | 982ms |
    +-------+-------+--------+-------+

I honestly expected zstd to win here.

noname1205mo ago

Thanks a lot. Interestingly Brotli’s author mentioned here that zstd is 2× faster at decompressing, which roughly matches your numbers:

https://news.ycombinator.com/item?id=46035817

I’m also really surprised that gzip performs better here. Is there some kind of hardware acceleration or the like?

terrelln5mo ago

Zstd should not be slower than gzip to decompress here. Given that it has inflated the files to be bigger than the uncompressed data, it has to do more work to decompress. This seems like a bug, or somehow measuring the wrong thing, and not the expected behavior.

1 more reply

noname1205mo ago· 2 in thread

Why not use a more widespread compression algorithm (e.g. gzip) considering that Brotli barely performs better at all? Sounds like a pain for portability

mort96OP5mo ago

I'm not sold on the idea of adding compression to PDF at all, I'm not convinced that the space savings are worth breaking compatibility with older readers. Especially when you consider that you can just compress it in transit with e.g HTTP's 'Content-Encoding' without any special PDF reader support. (You can even use 'Content-Encoding: br' for brotli!)

If you do wanna change PDF backwards-incompatibly, I don't think there's a significant advantage to choosing gzip to be honest, both brotli and zstd are pretty widely available these days and should be fairly easy to vendor. But yeah, it's a slight advantage I guess. Though I would expect that there are other PDF data sets where brotli has a larger advantage compared to gzip.

But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

ksec5mo ago

>But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

I may dislike Google. But my support of JPEG XL and Zstd has nothing to do with competition tech being Google at all. I simply think JPEG XL and Zstd are better technology.

mort96OP5mo ago

Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size.

Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size):

    +---------+---------+--------+--------+--------+
    |  none   |  zstd   |   xz   |  gzip  | brotli |
    +---------|---------|--------|--------|--------|
    | 47.81M  | 37.92M  | 37.96M | 38.80M | 37.06M |
    +---------+---------+--------+--------+--------+

These numbers are much more impressive. Still, Brotli has a slight edge.

gcr5mo ago

If you're worried about double-compression of image data, you can uncompress all images by using qpdf:

    qpdf --stream-data=uncompress in.pdf out.pdf

The resulting file should compress better with zstd.

j / k navigate · click thread line to collapse

0 comments

13 comments · 5 top-level

terrelln5mo ago· 3 in thread

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M |

If you can reproduce this behavior, can you please file an issue with the zstd version you are using, the commands used, and if possible the file producing this result.

mort96OP5mo ago

Okay now this is weird.

I can reproduce it just fine ... but only when compressing all PDFs simultaneously.

To utilize all cores, I ran:

    $ for x in *.pdf; do zstd <"$x" >"$x.zst" --ultra -22 & done; wait

(and similar for the other formats).

I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization:

    $ for x in *.pdf; do zstd <"$x" >"$x.zst" --ultra -22; done

That one file becomes 1.1M, and the total size of *.zst is 37M (competitive with Brotli, which is impressive given how much faster it is to decompress).

What's going on here? Surely '-22' disables any adaptive compression stuff based on system resource availability and just uses compression level 22?

terrelln5mo ago

So it is very central to zstd that it will never emit a block that is larger than 128KB+3B.

I will try to reproduce, but I suspect that there is something unrelated to zstd going on.

What version of zstd are you using?

1 more reply

Zekio5mo ago

doesn't zstd cap out at compression level 19?

1 more reply

noname1205mo ago· 3 in thread

Could you add compression and decompression speeds to your table?

mort96OP5mo ago

    ~/tmp/pdfbench $ hyperfine --warmup 2 \
    'for x in zst/*; do zstd -d >/dev/null <"$x"; done' \
    'for x in gz/*; do gzip -d >/dev/null <"$x"; done' \
    'for x in xz/*; do xz -d >/dev/null <"$x"; done' \
    'for x in br/*; do brotli -d >/dev/null <"$x"; done'
    Benchmark 1: for x in zst/*; do zstd -d >/dev/null <"$x"; done
      Time (mean ± σ):     164.6 ms ±   1.3 ms    [User: 83.6 ms, System: 72.4 ms]
      Range (min … max):   162.0 ms … 166.9 ms    17 runs
    
    Benchmark 2: for x in gz/*; do gzip -d >/dev/null <"$x"; done
      Time (mean ± σ):     143.0 ms ±   1.0 ms    [User: 87.6 ms, System: 43.6 ms]
      Range (min … max):   141.4 ms … 145.6 ms    20 runs
    
    Benchmark 3: for x in xz/*; do xz -d >/dev/null <"$x"; done
      Time (mean ± σ):     981.7 ms ±   1.6 ms    [User: 891.5 ms, System: 93.0 ms]
      Range (min … max):   978.7 ms … 984.3 ms    10 runs
    
    Benchmark 4: for x in br/*; do brotli -d >/dev/null <"$x"; done
      Time (mean ± σ):     254.5 ms ±   2.5 ms    [User: 172.9 ms, System: 67.4 ms]
      Range (min … max):   252.3 ms … 260.5 ms    11 runs
    
    Summary
      for x in gz/*; do gzip -d >/dev/null <"$x"; done ran
        1.15 ± 0.01 times faster than for x in zst/*; do zstd -d >/dev/null <"$x"; done
        1.78 ± 0.02 times faster than for x in br/*; do brotli -d >/dev/null <"$x"; done
        6.87 ± 0.05 times faster than for x in xz/*; do xz -d >/dev/null <"$x"; done

As expected, xz is super slow. Gzip is fastest, zstd being somewhat slower, brotli slower again but still much faster than xz.

    +-------+-------+--------+-------+
    | gzip  | zstd  | brotli | xz    |
    +-------+-------+--------+-------+
    | 143ms | 165ms | 255ms  | 982ms |
    +-------+-------+--------+-------+

I honestly expected zstd to win here.

noname1205mo ago

Thanks a lot. Interestingly Brotli’s author mentioned here that zstd is 2× faster at decompressing, which roughly matches your numbers:

https://news.ycombinator.com/item?id=46035817

I’m also really surprised that gzip performs better here. Is there some kind of hardware acceleration or the like?

terrelln5mo ago

1 more reply

noname1205mo ago· 2 in thread

Why not use a more widespread compression algorithm (e.g. gzip) considering that Brotli barely performs better at all? Sounds like a pain for portability

mort96OP5mo ago

But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

ksec5mo ago

>But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

I may dislike Google. But my support of JPEG XL and Zstd has nothing to do with competition tech being Google at all. I simply think JPEG XL and Zstd are better technology.

mort96OP5mo ago

Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size):

    +---------+---------+--------+--------+--------+
    |  none   |  zstd   |   xz   |  gzip  | brotli |
    +---------|---------|--------|--------|--------|
    | 47.81M  | 37.92M  | 37.96M | 38.80M | 37.06M |
    +---------+---------+--------+--------+--------+

These numbers are much more impressive. Still, Brotli has a slight edge.

gcr5mo ago

If you're worried about double-compression of image data, you can uncompress all images by using qpdf:

    qpdf --stream-data=uncompress in.pdf out.pdf

The resulting file should compress better with zstd.

j / k navigate · click thread line to collapse