What prevents GCC and Clang (and co from Intel/MS) saving a sidecar file next to a program/.so with all the functions and their argument types, struct structures, etc. in some agreed upon format for easy parsing / use? Could even be emitted based on some annotation/pragma like #export before the function declaration.
Is it just that nobody really cares to coordinate such a solution?
Better FFI's generate their definitions from such header files, called header parsers. (based on some CPP code). This problem was solved in the 80ies, but I guess forgotten by now.
Some languages already use that for the purpose, but it's an ill fit. It's meant for the internal parser of the language, so you need to parse C code (even if it's just the header file syntax). And it wasn't created with FFI in mind, in the first place.
What I propose is a file intended for others wanting to FFI, and thus trivial to parse (could be a schema for a format with ubiquitus parsers, like XML, JSON, YAML, TOML, and co).
That would have the benefit of:
(1) being cross-language (every language that exposes functions with C calling conventions could produce such a file to allow others to interface with it.
(2) being trivial to parse and integrate with ffi facilities of different languages (e.g. ctypes)
(3) being extensible beyond the C-header files give (e.g. metadata about usage, or valid int ranges, even function documentation, etc).
Yup, there's also the more common approach of implementing extensions in C (or equivalent), possibly with an intermediate language.
For example, Cython is a much more scalable approach, in my experience, than attempting to describe the interface in Python and using a libffi-style binding.
See: Vulkan's vk.xml
"Making Libraries Consumable for Non-C++ Developers"
Our argument for this weird symbiosis was that we wanted the popularity of the NodeJS ecosystem at the time with the performance of C++ - and we achieved this to an at least satisfactory degree.
When it came to building binaries based on the mixed codebases, especially for windows, things were a lot more challenging.
To tie it back to the piece, it would be pretty easy to generate code that outputs function definitions from an apache arrow scheme in both C and a target interpreted language such as python (e.g. ctypes bindings).
