One of the bosses working for one of our industrial partners was completely clueless about the details of the mission and the context. One day, he entered the office of one of his employees:
- Hi Bob [fake name], do you know anything about this Planck stuff?
- Sure, I'm working full time on Planck.
- Good! And do you know what WMAP is?
- Of course I do!
- Great! Now listen, if WMAP run on Windows, you should definitely install it on my computer, so I can play with it.
Do you mean that it would be impossible for a team of experts in C and rust to rewrite pg in 5 years? That is essentially what he is proposing. (Which probably isn't a great idea, but I think the work scope estimate is in the right ball park.)
I'm just asking possibly stupid questions and having possibly
stupid theories, trying to understand why such a project would
be possible or not.
Because of the interop ease, it's not hard to imagine a multi-year peicemeal rewrite. Postgres' code has some pretty good idioms that could be nicely encoded in Rust, and a good separation of concerns
Postgres has a great focus on solving real problems for real people. Losing that focus would be a disaster.
That's putting the cart before the horse. The right question is:
a) What's the argument in favor of porting Postgres to Rust?
and
b) What's the opportunity cost?
company in reference https://en.wikipedia.org/wiki/Trustly
If the author finds C overcomplicated I look forward to finding out what they think of a complete database system.
> Well that's going to be a show-stopper right there. For a proper
> port, a deep understanding of the current source code is necessary.
> You'd need a team expert in both C and Rust to pull it off.The weird thing is that they clearly have the sense to recognize that a rewrite would be a non-trivial project, but they seem to completely misunderstand why.
Well, yes and no.
On the one hand unsafe Rust is roughly similar to C[1], and unsafe Rust is a subset of Rust. But unsafe Rust is not the most widely used subset of Rust, stable Rust is ! And stable Rust is arguably way easier to understand and work with than C. And you can write fairly complicated software without writing a single line of unsafe code.
[1]: even though, the more expressive syntax and type system of Rust makes a lot of things less error-prone in Rust, like pattern matching or Result<> for error handling instead of switch and goto.
There's a reason many, many person-years have been devoted to tooling: offloading as much of the "simplicity" as we can onto computers is the best way we know how to write correct C at scale.
(That said, PostgreSQL clearly works well enough and has a system for writing new code that works well enough, so proposing a wholesale rewrite into a different language seems... unproductive, even moreso than usual.)
For open-source projects, being written in a scary[1] language is a liability in the long run.
For many tasks, we used to have no good alternative than using C (or C++ which is even scarier), but this is no longer the case now thanks to Rust.
I'm obviously not saying we should rewrite PostgreSQL in Rust, but the ability to grow/maintain the contributor base is an argument that should not be underestimated.
[1]: no matter how difficult or not it is in practice, the C is scary for many people and most of them will never feel confident enough to suggest changes in a C code base.
I think people do have this feeling, but likely it's because they don't actually know much about writing databases, so they see one issue but not the other.
OTOH it might be possible to contribute small things (e.g. improving the database client program, adding a test) if one can write some code in $LANG_MORE_ACCESSIBLE_THAN_C, even without knowing what a btree is or what "cache locality" means.
Scary is having to run from a lion to save our life.
By my reading for most of the submitted patches language issues aren't in the top three of problems preventing the patches from being merged; although such issues certainly exist. That said, that obviously doesn't say anything about patches never getting submitted.
Personally the biggest issue with growing the contributor base in postgres isn't language related, it's that a) the easy thing have been done long ago, making the learning curve steep b) lack of time from the experienced people/committers. Most of us have other stuff and our own projects to deal with :(
Waiting for a flamewar on the LKML on this.
"Hey, Linux devs. I heard there is this nice language called Rust. Do you mind rewriting your OS in this? Ok, might not be possible over the weekend, but maybe over a year? Maybe use some Deep Learning to automagically transform the code?"
I smell an upcoming Linus Rust rant.
Note that the OP has, afaik, no involvement in the rust project. Nor has he done much hacking on postgres. Don't think the rust people can be blamed for that one.
This way you could also say: hey! I have successfully ported a DB written in C in the past, and it has been an extremely positive experience, how about trying the same with Postgres?
it seems many people re-implement redis (I recall at least also Go and Scala).
It looks like it's the new "backend language todo list project".
The logical decoder: http://pgxn.org/dist/jsoncdc
The bindings: https://github.com/posix4e/rpgffi
The extensibility of Postgres is really profound -- there are so many things you can do, and so much internal functionality you can access, without having to recompile the database. The way extensions are loaded also mitigates the risk of what can be admittedly tricky issues relating to safety. Extensions run in backend processes, not the postmaster.
If rewriting Postgres happens, I hope it would be gradual. You can replace a lot of the core with extensions, and the scope for extension is ever increasing. So say you wrote a storage engine in Rust. Writing a foreign data wrapper for it would give you a way to use it from Postgres for all your tables; but you wouldn't have to write a SQL parser, client manager, permissions system, or any of the rest of that stuff.
Although the C stuff is a little gritty, a lot of what goes on in a database is gritty "because it is" and that low-level control is needed to deliver on tail latency, fault tolerance or simple storage efficiency. For example, Postgres has code to recover from malloc failing.
Extensions could be neat, there doesn't seem to be a high-level Rust API for it (in the style of ruru or rust-cpython) yet though, only incomplete-looking bindgen: https://github.com/posix4e/rpgffi
[0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql
Nowadays most developers comes from memory-managed languages => for us, C is a scary land where it's really hard to get things right when you don't have all the prerequisite knowledge (what's the stack ?) => the language is really old, you don't have all the fancy and useful tools and syntactic sugar (package manager, proper error handling, generics) => then we don't want to invest a lot of time to learn a difficult language that is so unproductive. => and even if we do out of curiosity, we'll most likely never feel confident enough to dig into an open-source project and contribute.
Rust on the other hand, is like C but with a personal IA assistant (the compiler) and all the syntactic sugar you're used to (and even more if you're not familiar with the ML family). That's what's attracting so many web developers to Rust !
In my opinion it means that, in a somewhat near future, the amount of potential contributors for projects written in Rust could be higher than for projects written in C. And those contributor are also likely to be more productive, thanks to the language features and tooling.
A guy randomly asks a question on a mailing list, and then his mail goes in the front page, and then people start insulting him: «blah blah, stupid politic CTO … understand nothing, blah blah blah».
(TLDR: CTO comes in and says "Angular and TypeScript is old. Python is new, and Google is writing everything with it. Python is the best practices for development." Tells IT department to rewrite everything in Python. “If our team still needs to use Angular, just use the Python version.” After failing to convince CTO this is absurd, IT realizes they can do nothing and tell the CTO they totally rewrote everything in Python, since he won't know the difference anyway.)
This case isn't quite that bad, obviously. The request is more reasonable, but still seems to betray a pretty fundamental misunderstanding of the situation.
A CTO of a payment company, so the inquiry to see if it would be possible to replace a large C project in their stack with a drop-in replacement written in a more safe language makes sense from where he's standing.
I just realised Python is roughly my age.
The recent hype of the language is great, but I think it's really important to start seeing substantial large projects receiving productivity gains otherwise it'll attract derision and then loose its momentum.
But they would need to have a contributor and maintainer base sufficiently motivated to learn Rust and stick with it. Which will happen more and more if Firefox can be visibly shown to benefit from including Rust in their toolkit.
I don't understand this statement. Is he saying he want to contribute to postgres, but he doesn't know C well enough, so he kindly asks the team to consider the idea of a rewrite in Rust to make it easy for him?
That's... underselling it somewhat a little.
github.com/posaix4e/rpgffi (logical decoders and foreign data wrappers in wrust) github.com/posix4e/jsoncdc ( an example logical decoder in rust)
We are going to be adding bigtable and s3 fdw soon.
I did find a basic one: