Disabling npm's progress bar yields a 2x npm install speed improvement (opens in new tab)

Doing GUI calls is a notorious way to slow down apps. I sped up an app (major software from major large corporation) by a factor of 10x a few years back by removing GUI updates. What you have to do is create a thread that refreshes the GUI periodically and not let your main worker thread ever work on the GUI. This particular app was an Eclipse Plugin (a source control app) and the Eclipse SWT to update the console log window was being called after every file operation (of which there were 10s of thousands). All I did was move the GUI refreshing into it's own thread, and call it only every 250millis or so. I was hailed a hero, and deservedly so, for speeding up our app by 10X and was one of my most satisfying accomplishments in my software career.

Progress bar or not, npm has gotten extremely slow with v3, to the extent that someone is working on a performance-centered alternative called ied:

http://gugel.io/ied/

installs are literally the #1 thing npm needs to do well, and I really hope we see some improvement soon.

Some further details on the issue: https://github.com/npm/npm/issues/11283

The funky progress bar is about the coolest thing with npm. They could let it take a minute if they wanted too and i would keep watching it with amazement.

I posted this below, but I'll post again here for visibility. What the heck is up with npm install on iTerm 2? I had no idea something was weird until recently, when I did an install with Termial.

Here's a side-by-side video: https://youtu.be/odoVfHHBYVM

...has he controlled for network speed? Unless he's set up a local NPM caching proxy, he's going over the 'net for these packages. It'll take more than 1 datum before I believe this hype.

I work in China. Downloading from the outer Internet will never match the speed of writing to the console. Not too worried

I'd also like to take the time to mention that the caching system when installing / other actions is extremely inefficient (for my times I have the progress bar turned off already and this is a project with around 80 modules shared between dev / prod):

https://github.com/npm/npm/issues/10890

I started on a very (I would like to emphasize very a thousand times over) basic proof-of-concept to show how much faster it could be in the order of magnitudes:

https://gist.github.com/nijikokun/2f1f16325f8ffe14b1b3

All this does is build a json of every package you currently have installed, and utilizes that as a lookup store the next time instead of rebuilding it every install; this was targeted towards installing / uninstalling existing packages. Not fresh installs.

Fresh installs would benefit from bulk lookups via the API imo.

It's all good, Laurie fixed it - https://twitter.com/seldo/status/692192238445711360

This makes sense. It's like when I'm solving a project euler problem, I don't use verbose output because it runs much faster without having to print everything to the screen. (Python)

Upon first seeing npm 3.x I immediately added `--progress false --color false` to my npm installs and never looked back. Color and terminal graphics are the work of the devil.

Well-designed server platform.

Here's a breakdown https://github.com/npm/npm/issues/11283#issuecomment-1752468...

Someone with more UI development can comment here , but i always thought progress bars where more like a signal to the user that "something is being done" , and not a representation of how much is done and how much is left.

I don't see this at all, what's more worrying is that npm 3.* is apparently way slower than npm 2.*

I did this test on a npm library I wrote, installs some test tools and also has to compile some gyp stuff

2.14.2

progress=false real 0m26.589s user 0m12.086s sys 0m3.362s

progress=true real 0m29.553s user 0m12.455s sys 0m3.486s

3.5.2

progress=false real 0m57.489s user 0m13.298s sys 0m3.421s

progress=true real 1m1.084s user 0m15.690s sys 0m3.644s

While I don't see any significant difference between progress bars or none for 3.x, what is 3.x doing that causes almost 30 seconds more churning?

Two years ago, PHP's package manager, Composer, was sped up by disabling the garbage collector:

http://blog.ircmaxell.com/2014/12/what-about-garbage.html

(Or more accurately the cycle collector. You can't turn off reference counting.)

Could it be linked with the terminal being used?

Edit, reminds me of this, which is why I'm wondering http://stackoverflow.com/q/21947452/923847

I don't know the exact cause here, but I can say I'm thankful for my systems background that I've gleamed for years before moving into programming. Most sysadmin types know that displaying output on a filecopy slows down transfers greatly because the screen vsync holds up the operation.

When I write utilities today that do similar things, I only display output when writing and testing the program. Otherwise I'll write to memory then dump the results in a logfile at the end.

In this case since there is probably a less intensive way of providing status updates, and can probably be resolved by doing intermittent checks.

I used to joke, in the context of virus scanning, that I could give you a more accurate progress bar, but the scan would take twice as long...

sbcl's compilation output is so verbose that the compilation instructions mention that it builds faster on xterm than gnome-terminal

Yes, but, which download feels longer?

Another product bitten by the dark side of progress indicators.

One of the most often underestimated areas of product development, though certainly not among the most important underestimates.

Easy to underestimate its cost and impact. After all, it's just a progress indicator.

Screen IO is slow. A trick to speed up unzip speed, do unzip -qq. That skips all the screen output and make it flies.

That's the biggest issue I have with the new npm. I wonder how many developer-days it has wasted already.