Show HN: ZFS Implementation in Python (opens in new tab)

(github.com)

411 pointsalcari6y ago86 comments

86 comments

Note: this was implemented without referencing any ZFS source code and should not be subject to the CDDL.

So we port this python to something not slow, and all the kernel-people can shut up about ZFS being terrible ;)

I can Port from Python -> Common Lisp at a rate of ~100 LoC per hour, and that's a pretty friendly port (the yield expression is the only thing that can't be just done line-by-line; truthyness is the only real "gotcha" as there are a lot of "false" values in Python, but only 1.5 in Lisp[1]).

1: I say 1.5 because there is only one false value, but it has 2 idiomatic meanings: nil (equivalent to python's None) and the empty list.

j88439h846y ago

Dont forget, pypy is fast.

6 more replies

newnewpdro6y ago

Would the CDDL matter for a python implementation that will never become part of the kernel?

CaliforniaKarl6y ago

You'd want a reverse-engineering lawyer, so be certain. But my (IANAL) guess is: If this is a proper reverse-engineered implementation, you could then convert _this_ implementation to C, and contribute _that_ into the kernel.

Except, it seems this is BSD-licensed, so I'm not sure how that would work in the kernel (which is GPLv2).

2 more replies

dr0verride6y ago

It's clear to me that the solution is to integrate python into the kernel.

1 more reply

lelf6y ago

Bad title. It’s not a ZFS implementation, so hold your horses.

AHTERIX50006y ago

What do you mean? Can't you mount & read ZFS filesystems with this one?

ivanbakel6y ago

Pierre Menard, author of The Filesystem.

But I'm surprised this is possible without a specification - how can you test a filesystem through hexdumps? The effects of some operations are going to pretty far-reaching, surely?

randrus6y ago

Might be interesting/useful to aim for zfs send/receive compatibility :)

And thanks for the Borges callout.

cryptonector6y ago

There are lots of blog posts, lots of docs. There's ZFS code in GRUB that is GPL, etc.

aerovistae6y ago

One of my favorite short stories, and nobody else has ever read it. So glad to see someone else reference it.

hueving6y ago

Link?

1 more reply

mfsch6y ago

Does someone know whether it would be legal for someone to go through the ZFS code and write a specification of the features this author hasn’t figured out yet? I.e. could someone write a detailed description of the missing functionality that doesn’t include any details about the implementation so other people can implement it in non-CDDL code?

ummonk6y ago

Edit: yeah that is how you avoid copyright infringement https://en.wikipedia.org/wiki/Clean_room_design

Original comment: I could swear this was actually the standard practice for writing an implementation of an unknown file format or interface without infringing on copyright. But I don't remember the term for it.

atomicwrites6y ago

That's called a clean room implementation and was the standard way to make x-compatible products (like for example, the bios on an IBM PC clone). Not sure what the current legal standing of that method is.

EDIT: Ninjad because I left the reply in a tab without posting.

zymhan6y ago

Reverse engineering is legal in the US, but you had better have detailed records proving no one who knew the insides of the original product ever influenced the clone. And be prepared to explain that in court.

Someone6y ago

The on-disk format is available (http://www.giis.co.in/Zfs_ondiskformat.pdf)

That PDF says ”Unless otherwise licensed, use of this software is authorized pursuant to the terms of the license found at: http://developers.sun.com/berkeley_license.html”*. That link is broken, but it seems that’s Berkeley license (whatever that means for a specification, and for which variant?)

According to http://open-zfs.org/wiki/Developer_resources, its outdated, but still useful.

I think I would use that, rather than spend months diffing disk images.

AlexanderDhoore6y ago

What?! How?! Why?!
This is the greatest thing ever. I wish I could just write code for the fun of it. Every time I wonder whether people will use it and give up before I even get started.

js26y ago
I wrote this to scratch my own itch but mostly just for the fun of it and some people ended up using it.
https://github.com/jaysoffian/eap_proxy
This was a really simple project but tickled all my fancies: Python, low-level, networking, reverse engineering, system administration.
Just do it! Who cares if people use it?
Alternatively, contribute something to some open source project you use. I’ve done that too. Just small stuff here and there but that’ll guarantee someone uses your code if that’s what’s important to you. It only takes 39 commits to get on this page:
https://github.com/git/git/graphs/contributors
:-)

max05636y ago
I had this problem too. I’ve been able to get over it by from coming up with a scenario, even if it’s completely fabricated, where what I am doing can be useful. I also make sure that I incorporate something new that I want to learn in the project. Whether it’s a language, library, whatever. Then I give myself a date I can quit. Normally it’s about two months. This makes me really consider whether I want to take something on because if I do I force myself to dedicate two months of time to it. If I enjoy it still at the end of two months then I continue otherwise I move on to another idea. At least in that time I because a little better at whatever I was trying to do. That’s the real goal anyway.

mirceal6y ago
why not? writing code can fall into one of a few bucket. one of them is play.
anon42426y ago
Exactly! Write code that you find interesting and/or need for something and then share it. If someone uses your stuff, then great, if not, at least you've become a slightly better programmer! It's a win-win!
ehsankia6y ago
Absolutely, some of the most fun I've had coding was reverse engineering/implementing known protocols. Although something this big may be a little overboard :)

craftyguy6y ago
> What?! How?! Why?!
The readme literally answers this..

lelf6y ago

No ARC/L2ARC?
Edit: of course not. This is actually just it’s just a ZFS user-facing ”front-end”, not a ZFS implementation.

lunixbochs6y ago
It's capable of doing IO against a real ZFS array without any other code. ARC is an implementation detail and not necessary for correctness. If you removed ARC from ZoL it would still work, just slower. ARC is far from the most interesting milestone for a reimplementation effort because an ARC implementation doesn't need to be anything like the Sun version internally, as long as it offers similar performance.
This project is cool not because you're going to run the Python in your kernel today, but because someone can use it as a documented reference implementation of all of the data structures and transactions that is not covered by the CDDL, so another implementation based on this can live in the Linux kernel without problem.
hnlmorg6y ago
If the GPLv2 GRUB ZFS code[1] wasn't enough to get someone started then I doubt this will make any different in porting ZFS to GPL given there would be more work involved in turning this into a usable kernel driver.
Not taking anything away from the work that the author has done though. It's a nice project. I just think a little pragmatism is needed before we get carried away with the ZFS GPL comments.
[1] https://blogs.oracle.com/solaris/zfs-under-gplv2-already-exi...
chasil6y ago
It would certainly be wonderful if this led to a Linux kernel module that was free of Oracle.
lelf6y ago
You don’t need CDDL to use ZFS. Which is all the library does. It does not implement ZFS.

4oo46y ago

This is awesome! Do you plan on blogging anything about how you went about reverse-engineering?

PaulHoule6y ago

i love userspace implementations of filesystems.
note that the issues are entirely different from those with a kernel implementation since you aren't having to think about page cache et all.

burmecia6y ago
There is an userspace implementations of filesystems ZboxFS: https://github.com/zboxfs/zbox.

gaze6y ago

Hell yeah, dude. This is awesome.

foxhop6y ago

Hey alcari!
I know you from ICV - we used to hang out online on the forums and IRC.
Nice work on this project, I'm looking forward to diving into the codebase!

rashkov6y ago
Just curious, what is / was ICV?
foxhop6y ago
It was / is an community of people who formed around a book called 1337 h4x0r h4ndb00k by tapeworm.
ICV was the name of the forum for the book, now defunct (icodeviruses.com)
https://www.amazon.com/1337-h4x0r-h4ndb00k-tapeworm/dp/06723...

RantyDave6y ago

This is all kinds of funny. I'm awash with awe and admiration.

adamnemecek6y ago

There's also TFS a ZFS inspired FS in Rust https://github.com/redox-os/tfs

garmaine6y ago

Simultaneously fucking awesome (that you pulled it off at all), and fucking useless (performance...).
Thanks for sharing though. Maybe could be useful in making a suite of zfs inspection tools?
Is the OP here? How difficult would it be creat a zfs reshaping tool, allowing for the offline expansion of a vdev?

j / k navigate · click thread line to collapse

86 comments

lunixbochs6y ago

Note: this was implemented without referencing any ZFS source code and should not be subject to the CDDL.

josteink6y ago
So we port this python to something not slow, and all the kernel-people can shut up about ZFS being terrible ;)
aidenn06y ago
I can Port from Python -> Common Lisp at a rate of ~100 LoC per hour, and that's a pretty friendly port (the yield expression is the only thing that can't be just done line-by-line; truthyness is the only real "gotcha" as there are a lot of "false" values in Python, but only 1.5 in Lisp[1]).
1: I say 1.5 because there is only one false value, but it has 2 idiomatic meanings: nil (equivalent to python's None) and the empty list.
j88439h846y ago
Dont forget, pypy is fast.
6 more replies

newnewpdro6y ago
Would the CDDL matter for a python implementation that will never become part of the kernel?
CaliforniaKarl6y ago
You'd want a reverse-engineering lawyer, so be certain. But my (IANAL) guess is: If this is a proper reverse-engineered implementation, you could then convert _this_ implementation to C, and contribute _that_ into the kernel.
Except, it seems this is BSD-licensed, so I'm not sure how that would work in the kernel (which is GPLv2).
2 more replies
dr0verride6y ago
It's clear to me that the solution is to integrate python into the kernel.
1 more reply

lelf6y ago
Bad title. It’s not a ZFS implementation, so hold your horses.
AHTERIX50006y ago
What do you mean? Can't you mount & read ZFS filesystems with this one?

ivanbakel6y ago

Pierre Menard, author of The Filesystem.
But I'm surprised this is possible without a specification - how can you test a filesystem through hexdumps? The effects of some operations are going to pretty far-reaching, surely?

randrus6y ago
Might be interesting/useful to aim for zfs send/receive compatibility :)
And thanks for the Borges callout.

cryptonector6y ago
There are lots of blog posts, lots of docs. There's ZFS code in GRUB that is GPL, etc.

aerovistae6y ago
One of my favorite short stories, and nobody else has ever read it. So glad to see someone else reference it.
hueving6y ago
Link?
1 more reply

mfsch6y ago

ummonk6y ago
Edit: yeah that is how you avoid copyright infringement https://en.wikipedia.org/wiki/Clean_room_design
Original comment: I could swear this was actually the standard practice for writing an implementation of an unknown file format or interface without infringing on copyright. But I don't remember the term for it.

atomicwrites6y ago
That's called a clean room implementation and was the standard way to make x-compatible products (like for example, the bios on an IBM PC clone). Not sure what the current legal standing of that method is.
EDIT: Ninjad because I left the reply in a tab without posting.
zymhan6y ago
Reverse engineering is legal in the US, but you had better have detailed records proving no one who knew the insides of the original product ever influenced the clone. And be prepared to explain that in court.

Someone6y ago
The on-disk format is available (http://www.giis.co.in/Zfs_ondiskformat.pdf)
That PDF says ”Unless otherwise licensed, use of this software is authorized pursuant to the terms of the license found at: http://developers.sun.com/berkeley_license.html”*. That link is broken, but it seems that’s Berkeley license (whatever that means for a specification, and for which variant?)
According to http://open-zfs.org/wiki/Developer_resources, its outdated, but still useful.
I think I would use that, rather than spend months diffing disk images.