Ask HN: Why isn't every programming language interoperable?

3 pointsmanlymuppet1mo ago5 comments

I've been programming for a while, but I've avoided this question despite my curiosity because it sounds quite stupid. In the spirit of asking stupid questions anyways though, here goes: why isn't every programming language interoperable? Why hasn't somebody built a system which allows for at least better interoperability?

I was reading the Swift 6.3 release, and better C interoperability was one of the main new features. As a Swift developer, I know that a lot of people love Swift because of how great it is to work with alongside C. So this brings forward the question: why doesn't Swift also work with Rust, or Python, or many other languages?

(Specifically with Swift, I don't know how useful interoperability with Rust or Python would be (probably not very useful), and it certainly wouldn't be a trivial engineering challenge, but aside from that: why not?)

More generally, why isn't interoperability between languages a bigger thing? There are of course challenges, like interpreted vs compiled, or JIT vs AOT compilation, but I think there are serious benefits to be had. Perhaps in a perfectly setup project you wouldn't need interoperability, but in the real world plumbing between languages is a constant problem. To demonstrate this, if one language could easily communicate with many others, here are some possible use cases.

- All libraries of all languages become compatible; this is the big one. It would be pretty freeing (although perhaps not always architecturally wise) if when searching for libraries to do what you want, you could pick from literally anything. - Certain languages are suited to certain applications, with interoperability, you could be incredibly versatile. If one language is really good at one thing, and another language is really good at another thing, and you're project needs to do both thing, you don't have to compromise if both languages are interoperable. - Multi-language projects could not only be possible, but even easy. If you could do backend in Go and frontend in TypeScript, it would be phenomenal if you could freely import abstracts in-between languages. Imagine end-to-end type safety even, with no additional work. - Performance. Simple languages are faster to write, though sometimes run into problems later when performance becomes a bottleneck. This would be solved if say, Rust and Python could work together. You could write everything in Python, then when you need the power, switch to Rust. This already exists with Maturin and PyO3, and as a frequent user of this structure, it's pretty great. Imagine if it was available for all languages.

There are just some of the overlapping benefits, but I am sure there are more. In any case, these are things that would be great to have! Why hasn't somebody, perhaps in a performant low-level language, built a plumbing layer which connects many different programming languages? We live in an age with really powerful IDEs, that, if given the right tools, could definitely support much deeper interoperability.

What if programming with two languages in one project didn't feel different from just using one? Sure, some people just want to get from point A to point B, but I think the freedom to use anything could really create a lot of productivity.

While I am well aware that this would be a behemoth engineering project, I think a small group of ambitious developers could make significant headway. It wouldn't need to be all the way, and you'd be wise to take it slow, but I feel the benefits of such a project would be well worth the considerable effort.

5 comments

recursivecaveat1mo ago

They are, in the sense that basically all of them expose some C FFI. Of course that lowest common denominator interface makes it very challenging to do anything more than call some free functions with opaque pointers and numbers. In general if you want some data to go from language A->B->C->D it has to be losslessly represented in all of them, which prevents you from utilizing any of the unique features (including the very basic and everyday ones) of these languages.

The accelerator language concept works oookay because you're generally offloading pretty formulaic processing of large amounts of flat data to the lower level language. In most cases of similar performance I don't think the differences between languages justifies the pain of splitting your codebase. It is attractive at least in scenarios where it merges 2 languages like eliminating a front/back-end split though.

gogurt20001mo ago

The paradigms from different programming languages aren't always compatible. The differences in those paradigms are the strengths and weaknesses that make languages better or worse for various applications. It's hard for me to imagine low-level plumbing that would give you the languages' strengths when mixing languages -- some mixes just don't make sense.

As an example... why would I want to call python from C? If I'm writing in C, its because I want high performance with low-level control of resources. To call a python function, I suddenly need to spin up the python virtual machine including its baggage like its garbage collector -- there goes my performance and low-level control, I'm suddenly running a massive stack that I didn't write!

Alternately, you could compile the python code into a bytecode (different than PVM bytecode) that C can call performantly... but to do that you lose the benefits of python. Suddenly your python code needs to be compiled after every change (slowing down development, one of Python's greatest benefits) and you're required to declare and enforce strict types (no more easy duck typing!) so that C can count on getting back data in the expected format.

In contrast, calling C from Python does make sense and can already be done.

And writing plumbing to handle generic interoperability between 2 languages would be a lot of work. Writing parts of a project or system in different languages and then having them communicate through a standard interface like an API is much less work.

PaulHoule1mo ago

Like interoperable in the sense that I could write a function in C and call it in Rust?

manlymuppetOP1mo ago

Yes, essentially. It's been a while since I've written Rust, but I am pretty sure that's (calling C from Rust) already possible.

So imagine whatever Rust has with C but with many other languages.

PaulHoule1mo ago

I guess I'll say that minicomputer and mainframe implementations of languages like Pascal and FORTRAN and COBOL and even BASIC back in the 1970s were frequently built with interoperability in mind, like you bought all the compilers in one big bundle. (Maybe like gcc or the the llvm-based compiler suite today?) It might have been easier because most of these languages had minimal runtime systems and the science of activation records and language implementation was mostly figured out by 1975 when Scheme came out with closures.

More modern languages have data structures that are similar but different that are a challenge. For instance the list types in Java and Python as they are used in common software are similar in a lot of ways but different. C has arrays in the stdlib but not that kind of expandable list, C developers might write their own or get one out of a library, I guess C++ has std::list but that doesn't have a fast way to get the n-th element of a list. Common Lisp has its own idea of a "list" and Clojure lives in the JVM and can access a Java List just fine but has its own immutable list which has a totally different API than thoses other language (e.g. no nconc!)

In general it is not so hard to access those objects through the 'foreign function API' or whatever you have but you either write stuff in the host language which is built especially to work with the guest language, or you make a wrapper, or you copy the data structures wholesale. It is never going to be trivial.

Similarly a language like Java has garbage collection and Python and Rust have their own forms of reference counting, either way you will either do your own memory management in the guest language or use some APIs to participate in the memory management of the host but it is a hassle.

For a long time the standard architecture for complex systems has been a scripting language + a systems language. Like Lua and C or Python and C or maybe Clojure and Java.

j / k navigate · click thread line to collapse

5 comments

recursivecaveat1mo ago

gogurt20001mo ago

In contrast, calling C from Python does make sense and can already be done.

PaulHoule1mo ago

Like interoperable in the sense that I could write a function in C and call it in Rust?

manlymuppetOP1mo ago

Yes, essentially. It's been a while since I've written Rust, but I am pretty sure that's (calling C from Rust) already possible.

So imagine whatever Rust has with C but with many other languages.

PaulHoule1mo ago

For a long time the standard architecture for complex systems has been a scripting language + a systems language. Like Lua and C or Python and C or maybe Clojure and Java.

j / k navigate · click thread line to collapse