undefined | Better HN

0 pointstextmode8y ago0 comments

"... there are some metadata extensions that allow this)."

Where to find these extensions? Are they portable between Linux and BSD?

The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files.

Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.

0 comments

9 comments · 3 top-level

barrkel8y ago· 6 in thread

A zip file is a concatenation of gzipped files. A .tar.gz is a gzip stream of concatenated files. Anything that could do random access into the contents of a zip file entry could do similar things with a tarball.

jahewson8y ago

Not so simple. A bundle of streams is not the same as a stream of bundles.

sakuronto8y ago

What about .gz.tar, in which all the files are gzipped, then tarballed? It seems like it would be a slightly fatter .zip file.

barrkel8y ago

With a transparent random access overlay, the difference mostly disappears, reducing to whether the stream needs to be scanned or whether it's indexed, which is itself orthogonal - zip file directory at the end is redundant.

1 more reply

gkfasdfasdf8y ago

Zip files have a directory which lists the files in the archive and their offsets, etc. No such feature in a tar archive.

barrkel8y ago

If we're talking about something that indexes gzip streams, it's not a leap for it to also index the inner tar.

Mikhail_Edoshin8y ago

AFAIK a compressor like zip builds a dynamic running table of frequent byte sequences; the resulting archive is written in such a way that when you decompress it, you re-build the table in the process.

So if you concatenate files A, B, and C and then compress the result, then by the time the compressor starts compressing the data of C, it will have that table built from A and B. To extract C, you'll need to re-build the same table and thus you'll need first to decompress A and B.

In a zip file each entry is compressed individually; this gives random access, but worse compression rate, because the table is not re-used between files.

pwg8y ago

> Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Yes. If the tar file is on a seek-able medium, just read the headers only and build an index to the offsets of the file contents from the header data.

Then use the offsets index to seek to just the item of interest and read out only it and nothing more.

Now, does such a utility already exist? The answer seems to be yes: https://github.com/devsnd/tarindexer

Hello718y ago

afaik, only pixz does this; they store the index in xz.

j / k navigate · click thread line to collapse

0 comments

9 comments · 3 top-level

barrkel8y ago· 6 in thread

jahewson8y ago

Not so simple. A bundle of streams is not the same as a stream of bundles.

sakuronto8y ago

What about .gz.tar, in which all the files are gzipped, then tarballed? It seems like it would be a slightly fatter .zip file.

barrkel8y ago

1 more reply

gkfasdfasdf8y ago

Zip files have a directory which lists the files in the archive and their offsets, etc. No such feature in a tar archive.

barrkel8y ago

If we're talking about something that indexes gzip streams, it's not a leap for it to also index the inner tar.

Mikhail_Edoshin8y ago

In a zip file each entry is compressed individually; this gives random access, but worse compression rate, because the table is not re-used between files.

pwg8y ago

> Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Yes. If the tar file is on a seek-able medium, just read the headers only and build an index to the offsets of the file contents from the header data.

Then use the offsets index to seek to just the item of interest and read out only it and nothing more.

Now, does such a utility already exist? The answer seems to be yes: https://github.com/devsnd/tarindexer

Hello718y ago

afaik, only pixz does this; they store the index in xz.

j / k navigate · click thread line to collapse