You can see this attitude of "no" on this very HN thread. Read the comments! Instead of talking about ways we can make Python startup faster, we're seeing arguments that Python shouldn't be fast, we shouldn't try to make it faster, and that programs (and, by implication, programmers) who want Python startup to be fast are somehow illegitimate. It's a dismal perspective. We should be exercising our creativity as a way to solve problems, not finding creative ways to convince ourselves to accept mediocrity.
Here are some other things we can say "yes" to:
- Rewrite as much of Mercurial in Rust as possible, which will provide performance improvements well beyond what Python can possibly offer. https://www.mercurial-scm.org/wiki/OxidationPlan
- Spend resources on developing PyPy, which (being a JIT) has relatively slow startup but much faster performance in general, for people who want fast performance.
- Write compilers from well-typed Python to native code.
- Keep CPython easy to hack on, so that more people with a "can do" spirit can successfully contribute to CPython instead of it being a mess of special cases in Guido's head.
Will you join me in saying "yes" to these things and not convincing ourselves to accept mediocrity?
There's also no reason to believe that startup improvements would make the interpreter incomprehensible; the unstated assumption that improvements in this area must hurt hackability is interesting. IME, optimizations frequently boost both simplicity and performance, usually by unifying disparate code paths and making logic orthogonal.
MicroPython put together a Python in 250kb. Why the hell can't we make an LLVM frontend for Python that can use type hints for optimization? Sure, you lose some dynamic features as you optimize for speed, but that's the dream. Quickly write a prototype, not caring about types, optimize later with adding types and removing dynamicism.
I'm currently learning Racket and LLVM and I have about 70 more years to live. I'm gonna try make Python fast on slow weekends 'til I die.
> time pypy -c 'print "Hello World"'
Hello World
pypy -c 'print "Hello World"' 0.08s user 0.04s system 96%
cpu 0.120 total
> time luajit -e 'io.write("Hello World!\n")'
Hello World!
luajit -e 'io.write("Hello World!\n")' 0.00s user 0.00s
system 0% cpu 0.002 totalI would like to note, the cpython ties to the PyObject C abi seem to stymie rather than encourage “hacking”. Cpython seems to have traditionally valued stability over all else.... see the issues pypy has had chasing compatibility with c and retaining speed.
So: normally i’m with you and a language should lean into its strengths, but i’ve always listed startup time as a primary strength of python!
Optimization is very, very rarely completely „free“ - and usually a concious trade of some property for another trait that‘s deemed more important in a specific case.
Simplicity for performance. Code size for compilation speed. Startup time for architectural complexity. UX for security.
For a great product, you need to say „no“ much more often than not. Do one thing and do it well. Be Redis, not JBoss.
I love how this article gets down to the essence of it: https://blog.intercom.com/product-strategy-means-saying-no/
I read that article and I'm still wondering: why Rust?
I would think developers have the experience to realize this isn't true but I see it all the time on these forums.
You are literally bringing an attitude of "no" to the question of whether you are being an attitude of "no" to the discussion....
Python is a big ship, it may not be as nimble as a young FOSS project but it is always improving and investments in things like start up time pays dividends to a large ecosystem.
[1] https://mail.python.org/pipermail/python-dev/2018-May/153300...
The Python 2/3 transition is a great example of this. Python 2 continued an earlier tradition of saying "yes" to almost everything from one particular group of programmers: people working on Unix who wanted a high-level language they could use to write Unix utilities, administrative tools, daemons, etc. In doing that, Python said "no" to people in a lot of other domains.
Python 3 switched to saying "yes" to those other domains much more often. Which came with the inherent cost of saying "no" (or, more often, "not anymore") to the Unix-y crowd Python 2 had catered to. Life got harder for those programmers with Python 3. There's been work since then to mitigate some of the worst of it, but some of the changes that made Python nice to use for other domains are just always going to be messy for people doing the traditional Unix-type stuff.
Personally, I think it was the right choice, and not just because my own problem domain got some big improvements from Python 3. In order to keep growing, and really even to maintain what it already had, Python had to become more than just a language that was good for traditional Unix-y things. Not changing in that respect would have been a guaranteed dead end.
This doesn't mean it has to feel good to be someone from the traditional Unix programming domain who now feels like the language only ever says "no". But it does mean that it's worth having the perspective that this was how a lot of us felt in that golden age when you think Python said "yes" to everything, because really it was Python saying "yes" to you and "no" to me. And it's worth understanding that what feels like "no" doesn't mean the language is against you; it means the language is trying to balance the competing needs of a very large community.
Could you elaborate on this?
I thought Python was pretty good about supporting non-Unix OSes from early on. It was originally developed on SGI IRIX and MacOS. From the README for version 0.9:
> There are built-in modules that interface to the operating system and to various window systems: X11, the Mac window system (you need STDWIN for these two), and Silicon Graphics' GL library. It runs on most modern versions of UNIX, on the Mac, and I wouldn't be surprised if it ran on MS-DOS unchanged. I developed it mostly on an SGI IRIS workstation (using IRIX 3.1 and 3.2) and on the Mac, but have tested it also on SunOS (4.1) and BSD 4.3 (tahoe).
though it looks like there wasn't "painless" DOS support until 1994, with the comment "Many portability fixes should make it painless to build Python on several new platforms, e.g. NeXT, SEQUENT, WATCOM, DOS, and Windows."
I also thought that PythonWin had very good Windows support quite early on. The 1.5a3 release notes say:
> - Mark Hammond will release Python 1.5 versions of PythonWin and his other Windows specific code: the win32api extensions, COM/ActiveX support, and the MFC interface.
> - As always, the Macintosh port will be done by Jack Jansen. He will make a separate announcement for the Mac specific source code and the binary distribution(s) when these are ready.
...I can't really think of many, if any at all. Sometimes you just say "no" to "inertia".
I can't imagine anyone would object if Python could magically be 10x faster. I can't say the same thing for the Haskell thing.
Surely you're not implying that improving Python's performance would preclude finding interesting algorithms, nor that this is a suitable rationale for keeping Python slow? Anyway, algos can only get you so far when they're built on slow primitives (all data scattered haphazardly across the heap, every property access is a hash table lookup, every function call is a dozen C function calls, etc).
I'm sorry but that makes zero sense. Haskell is defined by immutability. People want to use haskell because of that characteristic. I don't want to use python because it is slow.
Look at this post in the same list thread: https://mail.python.org/pipermail/python-dev/2018-May/153300...
Python 3.6 is trying an enormous number of potential paths that code for imports might be found at. Why is that fixed in stone? Couldn't Python 3.(n+1) change that, if it's slow and historical, cutting out a bunch of slow system calls?
As someone who makes use of Python to deploy software, it's entirely possible that could cause me a few issues... which I'd fix quite easily. It should be totally reasonable to expect the community using the software to cope with those sorts of changes after a major release; the alternative is ossification.
Django suffered from maintaining too much compatibility, and releasing too slowly, and they fixed it. Three or four years ago everyone was talking about moving away from it; now they release often, deprecate stuff when they need to, and the project is as vibrant as it ever was. Time for cPython to learn the same lesson.
The mail included both
> At some point, we'll likely replace Python code with Rust so the build system is more "pure" and easier to maintain and reason about.
and
> Since I am disproportionately impacted by this issue, if there's anything I can do to help, let me know.
Remember that Python2 was faster.
Part of the reason for the language's success is because it made intelligent tradeoffs that often went against the grain of the opinions of the commentariat and focused on its strengths rather than pandering to the kinds of people who write language performance comparison blog posts.
If speed were of primary importance then PyPy would be a lot more popular.
It's surprising but frequently true that startup latency has a greater effect on the perception of performance than actual throughput. Nobody likes to type a command and then be kept waiting, even if the started program could in principle demonstrate amazing feats of computation once warmed up.
Simply try something like unpickling a 10 GB data structure while keeping your GUI in the main thread responsive. You cannot do that because the GIL locks up everything while modifying data structures. Move the data to another process instead of another thread. Great, your GUI is responsive but you can't access the data from the main thread.
You can say that such a humongous data structure is wrong or that a GUI isn't meant to be responsive or programmed in Python or that I'm holding it wrong. Probably right.
The programming language R does not, as I understand it, optimize for speed, because they are optimizing for ease of exploratory data analysis. R is growing quite rapidly. So is python, actually. It doesn't mean that either one is good at everything, and it's probably the case that both are growing because they don't try to be good at everything. A good toolbox is better than a multi-tool.
While the "maybe you shouldn't use Python" comment could be construed as trolling to some, there is definite truth to your line of reasoning and I agree with comment.
I absolutely love Python as a programming language for the space it is in. But as someone who needs to think long term about maintaining large projects with lifetimes measured in potentially decades, Python has a few key weaknesses that make it really difficult for me to continue justify using it for such projects. Startup time is one. The GIL is the other large one (not being able to achieve linear speedups on CPU-bound code in 2018 with Moore's Law dead is unacceptable). General performance disadvantages can be adequately addressed with PyPy, JITs, Cython, etc. Problems scaling large code bases using a dynamic language can be mitigated with typing and better tools.
Python can be very competitive against typed systems languages. But if it fails to address its shortcomings, I think more and more people will choose Rust, Go, Java, C/C++, etc for large scale, long time horizon projects. This will [further] relegate Python to be viewed as a "toy" language by more serious developers, which is obviously not good for the Python ecosystem. So I think "maybe you shouldn't use Python for this, then" is a very accurate statement/critique.
Startup time is a problem for Python. But concurrency is much more complex than you state: threading is not the only or best concurrency model for many applications. And certainly removing the GIL will not just enable Python "to achieve linear speedups on CPU-bound code". Distributed computing is real. One of Python's problems for a long time was not the GIL, it was the sorry state of multi-process concurrency.
The speed issues that JITs solve for other languages may not be solvable in Python due to language design.
Python is so slow that it forces you to be fast.
Consider data analysis: on modern machines, you're almost always better off with a columnar approach: if you have a struct foo { int a, b, c; }, you want to store int foo_a[], foo_b[], foo_c[], not struct foo data[]. It's better for the cache, better for IO, and better for SIMD.
numpy makes it much easier to use the latter than the former, whereas in C, you might be tempted with the former and not even realize how much performance you were leaving on the table. Likewise for GPU compute offloading, reliance on various tuned libraries for computationally intensive tasks, and the use of structured storage.
I am primarily a Python programmer, but if Rust, Go, etc. take over as the language of choice in certain cases, I don't think that's a bad thing. Which doesn't mean one shouldn't write an article to highlight this cost of not having short startup time, just in case this cost wasn't understood by Guido, et al. But my guess (and it's only a guess), is that it was.
I wouldn't say I construed it as trolling. More like, "You might be right, but where does that get us?" Not trolling, but also not that constructive, because it's extremely easy to write something like "maybe you shouldn't use Python" but likely hard and time-consuming to make it so.
There are a lot of questions when considering such a move. For example:
- What's the opportunity cost of migrating $lots_of Python to Rust, or some other language?
- Is that really where you can add (or want to add) the most value?
- And what does having to do that do to your roadmap? Maybe it enables it, but surely it's also stealing time from other valuable work you could be doing?
- Longer term, are we sacrificing maintainability for performance? (In your case it sounds like the opposite?)
- How easily can we hire and onboard people using $new_tech? (Again, it sounds like you might reduce complexity.)
Basically I suppose what I'm saying is I find it a little trite when people say, "well, maybe you should do X," without having weighed the costs and benefits of doing so. And in a professional environment, if that's allowed to become a pattern of behaviour, it can contribute to the demotivation of teams. Hence, I found myself a bit irritated by the grandparent post.
That’s a problem if that’s an area python wants to compete in.
I mean, I was always into little scripts, but I used Tcl and then Perl.
I have been using Python for small scripts for 20+ years and haven't had this issue. The JVM on the other hand was historically slow to start.
I've been to quite a few PyCons and never heard anyone espousing this view, but I'm open to the possibility that I have missed it. Can you link me to a piece of media that you think persuasively makes the case that this is what Python is supposed to be for?
Python seems to be mainly optimized for web servers, scientific computing and machine learning tasks. None of those care about startup time.
Meanwhile, Ruby is making great strides in performance and even has JIT coming in 2.6.
Edit: I'm aware that there are solutions that put everything a program touches into a kind of executable archive. A single file several hundred Megabytes in size. I've tested it. It doesn't really pre-compile the modules. The startup time was exactly the same.
- it compiles your program and make it stand alone so you can distribute just the exe
- it makes it start faster
- it makes it run faster
- it's fully independant of the system python. Actually your system doesn't even need a python at all
I don't get why it's not used, it's very robust, compatible with 3.6 and on some of my script I get about x4 speed up just on start up alone.
Is Nuikta compatible with numpy, pickle, etc? I remember that numpy was very problematic with compilers like pypy for a long time.
Any time I see something like this, I feel like I'm hearing about some homeopathic cancer cure. If Nuitka actually does what it says it does, it's solving a big recurrent problem for the Python community, so why is nobody talking about it?
Cython is a complicated beast but I feel like it just needs a more friendly wrapper for this to be more widespread.
https://stackoverflow.com/questions/22507592/making-an-execu...
https://github.com/cython/cython/wiki/EmbeddingCython
Why Cython isn’t in the stdlib (I think it could easily replace ctypes) is beyond me sometimes
We were able to solve most of the problem by zipping up the Python standard library and the our application.
That is, if you look at sys.path you'll see something like:
>>> sys.path
['', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', ...]
This should also include getting access to the pre-compiled byte code.
You can also have Python byte-compile all of the .py files in a directory as part of your build/zip process.
python -m compileall --help find . -type f -name "*.py" -delete
Also note calls to imp.load_source need to change to imp.load_compiled, and any .py files references directly in code need to be changed to .pyc (this is with 2.7, not sure about 3.x)
https://github.com/dart-lang/sdk/wiki/Snapshots
It's what Flutter uses on iOS since you can't run JITed code; AOT compile it and load it as just another shared library.
There are many great reasons to use Python, but execution speed is not one of them.
For instance, for many many CLI tasks a python script will be many times faster than a Java tool, just due to the JVM startup. It doesn't really matter if the Java would even run INSTANTLY... the JVM startup time just kills speed for small CLI invocations.
luckily these problems actually do get addressed slowly via AoT/Graal and Substrate VM.
Here comparing a simple hello world program one is written in java and uses Substrate VM to create a binary and compare it with python:
$ time ./hello.py hello world!
real 0m0.041s user 0m0.017s sys 0m0.023s
$ time ./hello.main hello world!
real 0m0.019s user 0m0.008s sys 0m0.010s
of course the comparsion is unfair
In fact, large parts of git were written in shell until they realized that shell is only fast on UNIX because of co-evolution (you can fork without exec, and fork is quite fast), and on other platforms like Windows, existing shell implementations are much slower and there isn't a well-tuned production-ready shell that does things completely differently. Then they started rewriting everything in C.
I use a C curses application (dokia) to store some oft-used commands, but anything that won't be run 500x/day or runs longer than ~1/10th of a second I'll write in python for easier/more powerful development
That interpreted programs frequently start slower than their compiled equivalents reflects badly on interpreter implementations, not the concept of interpretation itself.
Um, I think there are lots of examples where using pythons internal data structures as they were designed results in code that is fast enough.
Even though the language implementation is interpreted, lots of common things can be optimized under the hood using data structure and data type specific execution paths and so on.
Imagine you import tons of modules which often are only available in Python. This gets you going really quickly with your project and it runs very smoothly. Transferring this to C++ would probably take so long you won't even finish to find out before you run out of funding.
I have hopes that Rust or some descendant of Rust will get us there in maybe 10 years but in the meantime it would be better to get Python up to speed as good as possible.
Indeed. As hg is moving to use more and more Rust:
No love for Haskell[0]? It does look like the best system scripting language out there right now... I have just never tried it to be sure :(
0: http://www.haskellforall.com/2015/01/use-haskell-for-shell-s...
> Changing gears, my day job is maintaining Firefox's build system. We use Python heavily in the build system. And again, Python startup overhead is problematic. I don't have numbers offhand, but we invoke likely a few hundred Python processes as part of building Firefox. It should be several thousand. But, we've had to "hack" parts of the build system to "batch" certain build actions in single process invocations in order to avoid Python startup overhead. This undermines the ability of some build tools to formulate a reasonable understanding of the DAG and it causes a bit of pain for build system developers and makes it difficult to achieve "no-op" and fast incremental builds because we're always invoking certain Python processes because we've had to move DAG awareness out of the build backend and into Python. At some point, we'll likely replace Python code with Rust so the build system is more "pure" and easier to maintain and reason about.
I'm really curious why there hasn't been much of a desire to change this and it even got worse as time progressed which is odd.
When I used to care about the problem, I looked into it (https://cournape.wordpress.com/2015/09/28/distributing-pytho...) and got somewhere. It improves somewhat startup times, and allow distribution of a single binary.
Lots of libraries are terribly slow to import (e.g. requests), but right now there is little incentive to fix it as there is no toolchain to build good, self-contained python CLI apps.
Theses experience are a major turn-down from Python for me.
LWN gave some excellent coverage late last year, in this piece:
Quite the contrary, as I stated in my other comment, we now have nuikta.
Not what some fringe tool can or cannot do.
- Imports can't be parsed statically.
- Startup time has two major components: crawling the file system for imports, and running all the init() functions of every module, which happens before you get to main(). The first is only fixable through breaking changes, and the second is hard to fix without drastically changing the language.
The import code in CPython was a mess, which was apparently cleaned up by importlib in Python 3, through tremendous effort. But unfortunately I think importlib made things slower?
I recall a PyCon talk where as of 3.6, essentially everything about Python 3 is now faster than Python 2, EXCEPT startup time!
This is a shame, because I would have switched to Python 3 for startup time ALONE. (As of now, most of my code and that of my former employer is Python 2.) That would have been the perfect time to address startup time, because getting a 2x-10x improvement (which is what's needed) requires breaking changes.
I don't think there's a lack of interest in the broader Python community, but there might be a lack of interest/manpower in the core team, which leads to the situation wonderfully summarized in the recent xkcd:
FWIW I was the one who sent a patch to let Python run a .zip file back in 2007 or so, for Python 2.6 I think. This was roughly based on what we did at Google for self-contained applications. A core team member did a cleaner version of my patch, although this meant it was undocumented until Python 3.5 or so:
https://docs.python.org/3/library/zipapp.html
The .zip support at runtime was a start, but it's really the tooling that's a problem. And it's really the language that inhibits tooling.
Also, even if you distributed self-contained applications, the startup time is not great. It's improved a bit because you're "statting" a zip file rather than making syscalls, but it's still not great.
In other words, I have wondered about this "failure" for over a decade myself, and even tried to do something about it. I think the problem is that there are multiple parts to the solution, the responsibility for these parts is distributed. I hate to throw everything on the core team, but module systems and packaging are definitely a case where "distributed innovation" doesn't work. There has to be a central team setting standards that everyone else follows.
Also, it's not a trivial problem. Go is a static language and is doing better in this regard, but still people complain about packaging. (vgo is coming out after nearly a decade, etc.)
I should also add that while I think Python packaging is in the category of "barely works", I would say the same is true of Debian. And Debian is arguably the most popular Linux package manager. They're cases of "failure by success".
AFAIK importlib is entirely written in Python and kinda portable across Python implementations, while previously most was C code. It's not surprising something gets slower when written in Python.
> Also, even if you distributed self-contained applications, the startup time is not great. It's improved a bit because you're "statting" a zip file rather than making syscalls, but it's still not great.
PyQt applications on Windows typically take two or more seconds before they can do anything, including Enterprise's favourite start-up pastime, splashscreens. Except maybe if you rolled your own .exe wrapper that displayed the splash before invoking any of the Python loading.
That's really, really poor in the age of 4 GHz CPUs from the factory, RAM big enough to fit multiple copies of all binaries on a PC and SSDs with at the very least tens of thousands of IOPS.
m is the number of libraries you're importing, and n is the length of the PYTHONPATH.
So it gets really bad, and it's not just one person's "fault". It's a collusion between the Python interpreter's import logic and how package managers use it.
Also, even if you distributed self-contained applications, the startup time is not great. It's improved a bit because you're "statting" a zip file rather than making syscalls, but it's still not great.
Exactly. There is no silver bullet. The problem is how much code gets run on startup, and how Python's dynamic nature makes traditional startup speedup strategies impossible. Is this even fixable?
Just calling `hg root` takes 200ms with hot cache. The equivalent code in fish-script takes about 3. Which enables us to turn on hg integration in the prompt by default.
The equivalent `git rev-parse` call takes about 8ms.
But 8ms is still too slow for me. :) I implemented the Git recognition code myself in my own prompt using the minimal amount of FS operations [1], and it renders in 5 ms from start to finish, including a "git:branch-name/47d72fe825" display.
[1] https://github.com/majewsky/gofu/blob/master/pkg/prompt/git....
One of the reasons git-rev-parse takes slightly longer than your implementation is that you just unconditionally truncate the SHA-1 to 10 bytes. E.g. run this on linux.git:
git log --oneline --abbrev=10 --pretty=format:%h |
grep -E -v '^.{10}$' |
perl -pe 's/^(.{10}).*/$1/'
Which is not to say that there isn't a lot of room for improvement. The scope creep of initialization time is one of the things that tends to get worse over time without being noticed, but Git unlike (apparently) Python makes huge use of re-invoking itself as part of its own test suite (tens of thousands of times), so it's naturally kept in check somewhat.
If you have this use-case I'd encourage you to start a thread on the Git mailing list about it.
We also do the latter, but dirstate format isn't easily readable just with shell builtins (lots of fixed-length fields with NUL-byte padding, also we don't even have a `stat` builtin and the external program isn't a thing on macOS AFAIK), so we still fire up `hg status` for that - but only after we decide that there is a hg repo.
My git status line uses timelimit (https://devel.ringlet.net/sysutils/timelimit/) to automatically stop if any of the git status parts (dirty/staged/new files) take > 0.1 seconds to finish:
https://github.com/justuswilhelm/pufferfish/blob/master/fish...
For me, personally, I use python to automate tasks - or to quickly parse through loads and loads of data. To me, startup speed is somewhat irrelevant.
I built a micro-framework that is completely unorthodox in nature, but very effective for what I needed - that being a suite of tools available from an 'internet' server, available to me (and my coworkers) over port 80 or 443.
My internet server, which runs python on the backend (and uses apache to actually serve the GET / POST) literally spits out pages in 0.012 seconds. Some of the 'tools' run processes on the system, reach out to other resources, and spit the results out in under 0.03 seconds (much of that being network / internet RTT). To me, that's good enough - adding 30 or even 300 milliseconds to any of that just wouldn't matter.
I totally get that if Python wants to be a big (read bigger?) player then startup time matters more...but for my personal use cases, I'm not concerned with the current startup time one bit.
In such an event, the start-up time of the program might dominate the total run time of the application. And on my laptop or desktop with a fast SSD with good caching and a reasonably fast CPU... that still ends up being 'okay'.
But once I put that on an ARM chip with a mediocre hard drive - some python scripts spend so long initializing that they are practically unusable. Whereas the comparable Perl/BASH script runs almost instantaneously.
Often to make Python even practically usable for such systems I have to implement my own lazily loaded module system. Having some language which allowed me to say...
import(eager) some_module
import(lazy) another_module
[1] https://stackoverflow.com/questions/3095071/in-python-what-h...
The issue at https://bugs.python.org/issue32192 says the plan is to start with an easier to use system as a PyPI package.
But isn’t that the author’s point? It doesn’t seem like much time but because you’re paying it so often in so many little places it really does add up.
ruby2.5: 83ms (-e 'puts "hi"')
python3.6: 35ms (-c 'print("hi")')
python2.7: 24ms (-c 'print("hi")')
perl5.26.2: 8ms (-e 'print "hi"')
C (GCC 7.3): 2ms (int main(void) { puts("hi"); })For example, if you want a snappy command line response for a Gtk-using Python program, you probably want to handle command line arguments before even importing Gtk. Maybe it is --help or an argument that you pass on to another running instance, and you want it to be absolutely snappy and fast.
$ time ruby --disable-gems -e 'puts "hi"'
hi
real 0m0.009s
user 0m0.008s
sys 0m0.000s perl ........... 0m0.012s
siteless py27 .. 0m0.018s
gemless ruby ... 0m0.021s
siteless py36 .. 0m0.025s
siteful py27 ... 0m0.034s
siteful py36 ... 0m0.049s
gemful ruby .... 0m0.089sNot really a fair comparison given the other 3/4 have to do all their parsing and compiling. Unless in those 2ms you include compilation time. Or use tcc -run.
But the reason I included it is because it gives a baseline for the overhead of invoking any program, no matter how trivial.
There's also nailgun for Java which sounds like it works a little differently: http://martiansoftware.com/nailgun/
https://github.com/tbug/aiochannel
I also think David Baezlys curio has a wonderful way of explaining the same concept
http://curio.readthedocs.io/en/latest/reference.html#module-...
Asyncio has enabled his sort of programming a lot easier. You could do the same thing I imagine with multiprocessing and Threadpools I imagine
The fact the developers of hg went so far as to make that shows startup time is a real issue.
So why not fix the problem at the source?
1. Don't do that. Either write the driving app in Python or write the subprocesses in an ahead-of-time compiled language. Python's a great language but it's not the right tool for everything.
2. Be parsimonious with the modules you import. During development, measure the performance after adding new imports. E.g., one graph libraries I tried had all its many graph algorithm implementations separated into modules and it loaded every single one of them even if all you wanted to do was to create a data structure and do some simple operations on it. We just wrote our own minimal class.
Even if you write the driver in Python, you don't necessarily want to call the program you're testing in the same process. You might want independent launches of a command-line tool, so that you test the same behavior people get when they run the tool. Otherwise, your test suite might trip over some internal state that gets preserved from run to run in ways that command-line invocation wouldn't.
>> I believe Mercurial is, finally, slowly porting to Python 3.
I just gave up on Mercurial since it didn't let me push to BitBucket nor to an Ubuntu VPS via SSH.
For better or worse, Git just works.
I could push fine to BitBucket if I used Python 2 version locally, and same for my VPS if I used the Python 2 version both locally and remotely.
But as soon as I touched the Python 3 version of Mercurial the pull/push problems began. I don't recall the exact error and maybe it's fixed now (this happened like 6 months ago), but I don't think I'll give it another try for some time.
Python is free to tinker, and all similar interpreters are joyful to use. Anything else is probably better for heavy duty jobs environments.
Perhaps it would be possible to read in the source files, compile them, and preserve an image of the state immediately before reading input or command line.
I'm saying, use the emacs model. Start hg with a flag so it simply keeps running in the background while listening on a port. Run a bare-bones nc script to pipe commands to hg over a port and have it execute your commands.
This isn't a new problem, nor is it even a new solution. No complete re-write of the interpreter or the tool required.
Anyways, that's my 2¢
> Mercurial provides a `chg` program that essentially spins up a daemon `hg` process running a "command server" so the `chg` program [written in C - no startup overhead] can dispatch commands to an already-running Python/`hg` process and avoid paying the startup overhead cost. When you run Mercurial's test suite using `chg`, it completes minutes faster. `chg` exists mainly as a workaround for slow startup overhead.
Just like this isn't what the usual `emacs` command does (it's `emacsclient`), it isn't what the usual `hg` command does either. There are some disadvantages to this solution and some assumptions it makes, which have apparently led the Mercurial maintainers to conclude, like the Emacs maintainers, that it won't work as the default. Hence the desire for solutions that will.
in the temple of tmux
for the cult of vi
we sit and wait
for venv to activateWhat's even more annoying is that my Python code usually calls a whole lot of C libraries (OpenCV, numpy, etc.) So it's like this: app->OS process->python interpreter->my python code->C libraries. That just really feels wrong so I'd like two things:
1) better/easier path to embed python scripts into my app e.g. resident interpreter
2) some way of passing scripts to python without restarting a new process, this may exist and I'm unaware
The drawback is this may screw up your Python environment, not sure how easy it is to work around it if it does.
Optimized modules loading: maybe loading a larger 'super' module would be faster than several smaller ones? For example a python program could be analyzed to find it's dependent modules, and then pack all these into a 'super' module.
Once the python program executes, it would load the single 'super' module and hopefully bypass all the dynamic code which each module runs when imported to load up.
As mentioned previously, this is just off the top of my head and would certainly warrant more investigation/profiling to confirm my hypothesis.