They're all practically useless or delegated to specific tasks. At this point you'd need to present incredible evidence that an alternative compiler can be useful. Personally I find it comical how many developers are still eluded by a promise of performant python. I hope you achieve your goals, good luck.
His project seems to be generating generic C code which is much easier to port to any weird platforms. In fact, it might be perfect for my use-case where dynamic linking is just extra attack surface.
I understand that the project is still in the early stages, but I will be paying close attention to it. If at some point it will be possible to write "regular" Python in it (minus most of the standard library and imports), then it could be a candidate for an edge computing platform.
C code is performant and that is a fact. Python code is not.
When building mission critical systems why don't programmers just use C itself instead of coding in another programming language and having it transpiled for them? Why introduce such tools all the time?
I am against this because the tools programmers use are becoming too bloated compared to 10-20 years ago.
Want to build an Android App? Use Java/Kotlin.
Want to build an iOS App? Use Swift.
Want to build a Web App? Use a Single JS Framework (Why millions of frameworks?)
Want to build a Windows Desktop App? Use C#.NET Either with WinForms or WPF.
I really see tools and technologies coming up all the time to solve a problem that most of the time doesn't exist.
soft real time systems, for sure. if it runs any hard real time systems, get out of the lab.
We could of course debate the boundary between hard and soft, but I'd rather not.
Writing C code directly is as a good option, as long as your code is not too complex to develop, maintain and extend.
And, again, Python here is intended to be the host language for an embedded domain specific language that gets compiled into C. It does not need to be efficient it needs to be expressive and easy to analyse and transpile.
Everything has a cost. This may not be ideal but learning to do C properly as an experienced Python dev will have a time cost as well. This may just be the best way to get from A to B.
I remember when I did a one-off project with a PIC microcontroller. I only had an assembler and I spent 2 days getting nowhere.
Then i found a C compiler and I had the whole thing running in 2 hours. The compiler turned out to much more efficient in speed as well as code size than my hand-written assembler.
Why C and not assembler?
> Why introduce such tools all the time?
C compilers are one of those tools.
I took a cursory look at:
https://github.com/zanellia/prometeo/blob/master/prometeo/cg...
It seems quite similar in spirit to
https://github.com/adsharma/py2many/blob/main/pyrs/transpile...
I'm not spending much time on py2many last few months (started a new job). Let me know if any of it sounds useful - especially the ability to transpile to 7-8 languages including Julia, C++ and Rust.
https://github.com/adsharma/py2many/blob/main/py2many/cli.py...
Maybe I'm not understanding your use case well enough and maybe your approach is actually a locally optimal solution, and I honestly greatly respect your effort and want to see your project succeed, cuz I love Python too. I even wanted to make a faster python myself at some point. Just, the more I learn about Nim, the more I appreciate its design decisions and think to myself: this is the faster version of python I'm looking for. Now, we just need the large package ecosystem that python has, but I'm willing to both wait and participate in making it come about.
This presentation about writing keyboard firmware in Nim may be helpful, if you're willing to give Nim some more consideration:
There is also another talk about embedded programming in Nim from the same conference, here:
Do you envisage this being a conduit for tight loop optimisation in Python? Or is it rather "you'd like a C program but can't write C good"?
And if the former, how do you compare to Nuitka and Cython? I read your README but couldn't quite make sense of it :)
yep
> Do you envisage this being a conduit for tight loop optimisation in Python? Or is it rather "you'd like a C program but can't write C good"?
There are already plenty of options for calling high-performance libraries from Python. Now 1) interpreting Python programs that use, e.g., NumPy, can be slow. 2) Compiling these programs using, e.g., Cython or Nuitka, can speed up the code across calls to high-performance libraries, but the resulting code will still rely on the Python runtime library, which can be slow/unreliable in an embedded context.
Coming to the second part of the question, writing C code directly is definitely an option, but, after doing a bit of that, I realized how tedious/error prone it is to develop/maintain/extend relatively complex code bases for embedded scientific computing (e.g. this one https://github.com/acados/acados). Or, to put it as Bjarne Stroustroup once said "fiddling with machine addresses and memory is rather unpleasant and not very productive". The good news seemed to be that many of the code structures necessary to write that type of code are rather repetitive and can hopefully be generated automatically to some extent.
> And if the former, how do you compare to Nuitka and Cython? I read your README but couldn't quite make sense of it :)
This table (from the README) shows some computation times for Nuitka, prometeo, Python and PyPy.
CPU times in [s]:
Python 3.7 (CPython) : 11.787 Nuitka : 10.039 PyPy: 1.78 prometeo : 0.657
Other than performance, the main difference is, again, the runtime library dependency.
https://github.com/zanellia/prometeo/blob/master/benchmarks/...
It's kinda of sad how much effort is put on the creation of new Python compilers to make it slight faster while the problem of latency to compile that people hate at Julia is not tracked because of the lack of manpower to improve Julia's interpreter.
https://youtu.be/IlFVwabDh6Q?t=2530 (tldr: The Julia interpreter is currently about 500x slower than JIT code and there are a lot of low-hanging fruit work there that could easily give it a 10x speedup - this could make more viable to switch between compiler and interpreter depending on the work)
It's kind of why C is still king in this space.
The single threaded thing is not an issue because you can still call the same function on each CPU and use the CPU ID to target parts of the computation, like a compute kernel function.
There's ongoing work to reduce runtime dependencies of Julia (for example in 1.8, you can strip out the compiler and metadata), but then it's only approaching Go/Swift and other static languages with runtimes.
Generating standalone runtime free LLVM is another path, that is actually already pretty mature as it's what is being done for the GPU stack.
Someone just has to retarget that to cpu LLVM, and there's a start here: https://github.com/tshort/StaticCompiler.jl/issues/43
You could also do it at the LLVM level: https://github.com/JuliaComputingOSS/llvm-cbe
One cool use case is in https://github.com/JuliaLinearAlgebra/Octavian.jl which relies on loopvectorization.jl to do transforms on Julia AST beyond what LLVM does. Because of that, Octavian.jl. a pure julia linalg library, beats openblas on many benchmarks
small comment - related to the benchmarks:
- in Julia: it has a newer ricatti solver (in package)
https://github.com/andreasvarga/MatrixEquations.jl/blob/mast...
After reading: this is really cool. If I understand this, I think you should be able to beat Cython without breaking a sweat. I'm quite excited to use this.
> Cython is a programming language whose goal is to facilitate writing C extensions for the Python language. In particular, it can translate (optionally) statically typed Python-like code into C code that relies on CPython. Similarly to the considerations made for Nuitka, this makes it a powerful tool whenever it is possible to rely on libpython (and when its overhead is negligible, i.e., when dealing with sufficiently large scale computations), but not in the context of interest here.
I.e., it’s a python-like DSL that does not depend on the Python runtime.
Thanks for sharing OP, this is pretty cool.
Many of us think of clusters as high-performance scientific computing, which are about as far from embedded as it gets.
Please note that I am not being snarky, just curious!
I noticed your disclaimer at the bottom of the linked page [0], and wanted to get an idea of how far you were looking to take this. Will it go beyond maths into normal functions (string handling, etc) ? Do you eventually plan on supporting most of python - for instance, do you think I could write a web server using your tool in the future?
[0] - "Disclaimer: prometeo is still at a very preliminary stage and only few linear algebra operations and Python constructs are supported for the time being."
I can't speak much about the code itself or the aims of the projects. Personally I would recommend more informative commit messages.
I do this myself, especially working on personal stuff, but writing commit messages that succinctly explain what each commit does is a good practice and gives a serious impression.
I often find myself hacking away and periodically going back to flesh out messages using rebase.
Finally, although it does not use Python as source language, we should mention that Julia too is just-in-time (and partially ahead-of-time) compiled into LLVM code. The emitted LLVM code relies however on the Julia runtime library such that considerations similar to the one made for Cython and Nuitka apply.
prometeo is an experimental modeling tool for embedded high-performance computing. prometeo provides a domain specific language (DSL) based on a subset of the Python language that allows one to conveniently write scientific computing programs in a high-level language (Python itself) that can be transpiled to high-performance self-contained C code easily deployable on embedded devices.
The package is still rather experimental, but I hope this concept could help making the development of software for high-performance computing (especially for embedded applications) a little easier.
What do you think of it? Looking forward to receiving comments/suggestions/criticism :)
