And in this case the more complicated debugging tools didn't even explain anything. As the last page says, the answer is kind of a leap. If you already knew about this problem curl would have solved it immediately, and if you didn't you'd still be baffled even after knowing exactly why the stack traces are the way they are.
For those of us who grew up reading W.R.Stevens, that might seem absolutely back-to-front. Not only that, curl ain't gonna help you diagnose the next one, which is a Path MTU Discovery issue, or the one after that, which is an undervoltage on the DRAM refresh controller. It might, however, be helpful in nailing that HTTP/2 prefetch request connection upgrade bug.
Mechanical sympathy means feeling every part of the machine, right down in your bones.
SPOILER FOR THE GAME
I’ve seen TCP_NODELAY all over the place before, but never known why. This was a fun way to find out.
The explanation didn't mention the flushHeaders call, which is apparently the fix. I didn't run any tests, just looked at the JS and figured that sending 2 packets is worse than sending 1 w.r.t. latency.
It's also a pretty strong intuition that the client side tends to have issues, since the server is usually well-tested and standardized. Also, very often people are measuring wrong, so checking the JS to be sure the time recorded is accurate is also important.
In this case, in my judgement, you only need to know about the delayed-ack/Nagling concepts if there's a engineering requirement to keep the flushHeaders call, and the 50ms latency is a dealbreaker (i.e. not merely a curiosity). Even then, it's not clear how to fix unless you can disable one of those features.
The first thing I did was use curl to check if it's a client or server problem.
Then I looked at the code and immediately noticed the unnecessary flush call, the unnecessary loading of the whole file in memory and the unnecessary fs.stats.
One thing that was not offered as an option was to use a packet analyzer like tcpdump or Wireshark, even though that is the most reliable and systematic way to get to the bottom of many performance problems. You'd think the popularity of the network tab in Chrome's dev tools would make this less scary.
The best job interview I ever had was framed like this. The interviewer told me there was a bug in the system and had a stack of pages he'd printed out that would provide successive clues as to what caused it. I could ask them questions, in effect using the interviewer as a search engine/debugger.
It was the closest an interview has ever come to simulating the day-to-day of a web developer.
For several years now, programming has been a required course for all grade-school students.
The tech field is flooded with talent, and it is virtually impossible to get a job.
The technical interview has grown at an unbounded rate.
An aspiring engineer walks into the interview room, freshly shaven. He sits down, cracks his knuckles, takes a long sip from his branded thermos.
Waits.
And then it begins. The interview.
10, 100, 1000 logic puzzles. Faster than seems possible, he recites answers memorized from algoexpert.io.
Cut to a wide shot. Speed up time. Stubble, then a beard, appear on his face. The sun rises and sets, and yet he dare not sleep.
Eventually, he forgets language, reason, civilization, coffee. The touch of his baby son's skin, although he's not such a baby anymore.
He grunts, diagrams, and whiteboards. That's all his life is now.
"Ok, well thank you very much! You'll hear from us soon," Says the interviewer. The sounds are foreign to the engineer, but he is lead out of the office and onto the bright street. His car sits there, rusted.
The interviewer motions the next candidate inside.
He doesn't hear from them.
I wonder how many of us are able to judge how long something should take? Not me, except anecdotally.
(tl;dr Try turning off delayed ACK first, especially if you can’t update the code.)
I answered "req.flushHeaders()" but surprisingly it doesn't accept that as a cause, even though the headers would be sent with the initial packet and should improve the latency.
Delayed ACKs can be enabled on Linux kernels 3.11+ with ip(8).
ip route change ROUTE quickack 1
Delayed ACKs may be used in response to congestion.
For example, in bulk, i.e., non-interactive, transfers with large packets, delayed ACKs can be useful.
This is covered in Chapters 15 (15.3) and 16 of Stevens' TCP/IP Illustrated Vol. 1.
This draft suggests delayed ACKs are useful during TLS handshake.
https://tools.ietf.org/id/draft-stenberg-httpbis-tcp-03.html
Also, socat allows for setting TCP options via setsockopt. No need to write a new program.
ip route change ROUTE quickack 1 dev STRING``` You said: "strace -p $(ps aux| grep server.py| grep -v "grep"| awk -F ' ' '{print $2}')".
To strace the server, first you need to find its PID. You know that the program is called server.py. ```
This hints to a possible XSS and/or code injection (not completely quoted input). Input was "strace -s128 -f -p <pid>" , as an answer to "how do you strace server process"
To me this seems pretty obscure and you debug pretty deep into and outside of your application. One part of me thinks of this as Somebody Else's Problem, but definitively makes me rethink it as a SEP and something devs should know about. Specially in time critical/real time systems.
Asked what to do about it and typed 'tcpnodelay'. It replied it wasn't smart and asked me to click a button.
Feels pretty basic to anyone who's ever really touched TCP in code.