Eliminating Cold Starts 2: shard and conquer (opens in new tab)

(blog.cloudflare.com)

62 pointscmsparks6mo ago18 comments

18 comments

Surprised none of the big clouds have duplicated the handshake delay thing yet - CF is noticeably better at cold starts than the rest for small scripts. I guess the big cloud functions are aimed at bigger workloads perhaps

no_wizard5mo ago

I never understood this either, even if due to constraints it has to be a different product line.

I thought lamda@edge was going in this direction but it’s a slightly faster, more constrained version of lambdas with all the same potential downsides

smacker5mo ago

While I really appreciate Workers platform "eleminated cold starts" advertising was always bothering me.

This is a curl request from my machine right now to SSR react app hosted on CF Worker: ``` DNS lookup: 0.296826s Connect: 0.320031s Start transfer: 2.710684s Total: 2.710969s ```

Second request: ``` DNS lookup: 0.002970s Connect: 0.015917s Start transfer: 0.176399s Total: 0.176621s ```

2.5 seconds difference.

kentonv5mo ago

Does this app make any network requests that might have their own cold start or caching effects?

2.5 seconds seems way too long to be attributed to the Worker cold start alone.

smacker5mo ago

It makes requests to API server that is deployed to k8s, which doesn't have a cold start. Clearly, some caching by the runtime and framework is involved here.

My point is that "cold start" is often more than just booting VM instance.

And I noticed not everybody understands it. I used to have conversations in which people argue that there is no difference in deploying web frontend to Cloudflare vs a stateful solution because of this confusing advertisement.

1 more reply

omk5mo ago

I'm not well versed with CURL design, but curious - is your first connection handling TLS while second relying on the previously established handshake?

smacker5mo ago

I'm not very well versed with CURL design too but afaik it does reuse connections but only inside the same process (e.g. downloading 10 files with 1 command). In this case it shouldn't be re-using them as I ran 2 different commands. I should have included TLS handshake time in the output, though. You can see it here (overall time is lower because I hit preview env that is slightly different from staging/prod):

First hit: ``` DNS Lookup: 0.026284s Connect (TCP): 0.036498s Time app connect (TLS): 0.059136s Start Transfer: 1.282819s Total: 1.282928s ```

Second hit: ``` DNS Lookup: 0.003575s Connect (TCP): 0.016697s Time app connect (TLS): 0.032679s Start Transfer: 0.242647s Total: 0.242733s ```

Metrics description:

time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.

time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.

time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed.

time_starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.

swiftcoder5mo ago

TLS handshakes (outside of embedded hardware) should be measured in milliseconds, not seconds

Edit: you can kind of tell this from the connect timings listed above. TLS is faster the second time around, but not enough to make much difference to the overall speedup

1 more reply

samschooler5mo ago

I would say Cloudflare "eliminated cold starts" in the context of bringing the server online, not in the form of rendering + caching the SSR page.

bluelightning2k5mo ago

I love Cloudflare engineering writeups & the workers platform

jtbaker5mo ago

Same. Really wish they had a managed k8s offering to complement it though. Some workloads just don't fit into the workers paradigm.

arcfour5mo ago

Containers+Workers is now a thing, at least.

It's very much still maturing as an offering. But it does exist!

1 more reply

candiddevmike5mo ago

Maybe a hot take, but if you're that concerned about cold starts, you probably shouldn't use a service that scales to zero. If your service is really not used that heavily, it's like $5-10/month for an instance that can handle more than your traffic needs 24/7, hell you can even host other similarly unused services on it!

stackskipton5mo ago

Because attractiveness of Workers/Lambdas/Functions is whole "write simple amount of code and pay pennies to run it." Downside is cold starts, twisting yourself into knots you will do at scale to make them work and vendor lock in.

If you start to say "If you are using this for production, 5-10/month is real cost you need to pay + transactions" Well, now the cost is about to same to deploy fly.io shared CPU container and it does not come with cold starts, vendor lock in and can run as long as you want. Cloudflare knows that so they don't want to introduce that charge or even talk about it.

j / k navigate · click thread line to collapse

18 comments

Havoc5mo ago

no_wizard5mo ago

I never understood this either, even if due to constraints it has to be a different product line.

I thought lamda@edge was going in this direction but it’s a slightly faster, more constrained version of lambdas with all the same potential downsides

smacker5mo ago

While I really appreciate Workers platform "eleminated cold starts" advertising was always bothering me.

This is a curl request from my machine right now to SSR react app hosted on CF Worker: ``` DNS lookup: 0.296826s Connect: 0.320031s Start transfer: 2.710684s Total: 2.710969s ```

Second request: ``` DNS lookup: 0.002970s Connect: 0.015917s Start transfer: 0.176399s Total: 0.176621s ```

2.5 seconds difference.

kentonv5mo ago

Does this app make any network requests that might have their own cold start or caching effects?

2.5 seconds seems way too long to be attributed to the Worker cold start alone.

smacker5mo ago

It makes requests to API server that is deployed to k8s, which doesn't have a cold start. Clearly, some caching by the runtime and framework is involved here.

My point is that "cold start" is often more than just booting VM instance.

1 more reply

omk5mo ago

I'm not well versed with CURL design, but curious - is your first connection handling TLS while second relying on the previously established handshake?

smacker5mo ago

First hit: ``` DNS Lookup: 0.026284s Connect (TCP): 0.036498s Time app connect (TLS): 0.059136s Start Transfer: 1.282819s Total: 1.282928s ```

Second hit: ``` DNS Lookup: 0.003575s Connect (TCP): 0.016697s Time app connect (TLS): 0.032679s Start Transfer: 0.242647s Total: 0.242733s ```

Metrics description:

time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.

time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.

time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed.

swiftcoder5mo ago

TLS handshakes (outside of embedded hardware) should be measured in milliseconds, not seconds

Edit: you can kind of tell this from the connect timings listed above. TLS is faster the second time around, but not enough to make much difference to the overall speedup

1 more reply

samschooler5mo ago

I would say Cloudflare "eliminated cold starts" in the context of bringing the server online, not in the form of rendering + caching the SSR page.

bluelightning2k5mo ago

I love Cloudflare engineering writeups & the workers platform

jtbaker5mo ago

Same. Really wish they had a managed k8s offering to complement it though. Some workloads just don't fit into the workers paradigm.

arcfour5mo ago

Containers+Workers is now a thing, at least.

It's very much still maturing as an offering. But it does exist!

1 more reply

candiddevmike5mo ago

stackskipton5mo ago

j / k navigate · click thread line to collapse