Hi,
just discovered that the (sometimes excessive large) golang mod disk cache is - in opposite to the compiler cache - not (yet) properly deduplicated.
Try yourself: Made a little hack/tool to deduplicate data via hardlinks (unix/i-node-filesystems tested only, not windows yet).
https://github.com/paepckehh/fsdd
(less than 1kLOC, 100% golang stdlib, no external deps)
go install paepcke.de/fsdd/cmd/fsdd@latest
cd $GOMODCACHE && fsdd --hard-link .
FSDD [start] [/usr/store/gomod][hash:MAPHASH]
FSDD [_done] [time: 39.000221ms]
FSDD [stats] [files:13329] [inode(s): 8680] [sym.valid: 0] [sym.invalid: 0] [data blocks: 277.1 Mbytes]
FSDD [_info] [new deduplication savings][inode(s): 4649] [data blocks: 66.9 Mbytes]
If you can reproduce and has it shows no downsides (remove/add should auto-dereference) it should be part of src/cmd/go/internal/mod* pack?
If you can confirm/field test, please report back!
I would love to write a quick stdlib/proposal.
At golang scale, it could make a real resource / enviromental dent!
It least for me, it cut my disk cache in half and reduced vfs fs mem-cache required as well, so should scale even on deduplicated fs like zfs/btrfs. Made my batch builds fast and snappy again!
Michael
(edit:first-post-ever-style-fixes)
