Migrating Dropbox from Nginx to Envoy (opens in new tab)

In a business context, I'd definitely consider paid support for an Open Source product. But I'm not interested in a proprietary version that I can't modify or get third-party support for or otherwise work with in a pinch; I'm certainly not going to make a business dependent on it. Push the proprietary version hard enough and I'll reconsider whether I even want to use the Open Source version, or if it might be on more tenuous ground that might get undermined in the future (pushing back on improvements to the Open Source version to maintain differentiation, or worse, deciding to switch to a non-FOSS license in the future).

I think a part of this is that engineers in particular have a preference to use software which can be treated as a transferable skill if/when they move on. They would rather use the OSS version of Nginx, of Envoy, because they know they will have access to it in the future. I think there is some aversion to becoming familiar with the features, functionality, and characteristics of a piece of software that your current employer is paying a non-trivial amount for, when you know that chances are your next employer will refuse. This may not be in the best interest of the current company, but it's a bias that I think impacts a lot of engineers.

Why does it blow your mind? Various obvious and sane reasons for this, including cost. I bet many of those companies you have in mind “buy” Windows and macOS, and if they are sufficiently big, most certainly buy Oracle or SAP for their corporate operations, finances, and accounting. It’s usually only the production side of sufficiently large internet-scale companies that is biased towards building. Most of the time you can easily explain it with “it’s cheaper than the contract with supplier”. Often, it is strategic ownership over your fate and not being locked into the vendor that comes into play as well. The vendor positioning in the market and its leverage may also change over time and pose a risk down the road (getting acquired, changing focus, abandoning the product, going out of business, etc.) Many times it is just that their problem is so unique to their scale that the generic solution does not technically work for them or the pricing model is not designed to be a fit.

In the particular case of nginx, I can tell you their reputation is not great in adapting to the users’ needs.

The subject of monetizing opensource software is a tricky one. Some companies pursue the Open-Core principle, others monetize through the consulting services or cloud infrastructures.

As for investing into opensource, Dropbox is trying to do that when possible, for example we (along with Automattic) did sponsor HTTP/2 development in Nginx.

It's not necessarily about money. An engineer can burn through tens of thousands of dollars per month in cloud spend because they have access to the AWS [or] GCP console, but that same engineer may not have the first idea about how to get the CFO's sign-off to purchase a license that will facilitate a halving of that spend. And that same CFO can institute a policy against using credit cards for recurring payments that prevent that engineer from expensing the purchase through a corporate card. And the software company may not offer a bill-via-invoice option — or they may only offer it for amounts greater than the amount the engineer wants to spend.

So much of what happens in sufficiently large organizations has nothing to do with profit maximization. Think confederacy of dunces, not a conspiracy of greedy evil geniuses.

Exactly my thoughts when I read the article - a hugely successful company not contributing to an open source project which enabled them to succeed in the first place ...

Just speculation, but there are motives to not use the closed source version beyond purely profits driven point of view. One of the prime benefits of OSS is that you have the power to change it whenever necessary. If something is breaking bad- you might not have the luxury of waiting on support to track down and fix your problem. If you don’t need the features of the paid version- then using the paid version is actually limiting your options.

I don't see what's surprising -- companies that earn money selling product can make even more money by cutting costs. And if OSS software gives them an equivalent (or even better) solution, why wouldn't they use it? For any sizable production deployment, the cost of nginx licenses could be applied to hire a number of engineers to help maintain the OSS software.

I don't know what their volume licensing is like, but at $2500/server list price, costs add up quickly.

Isn't this simple economic reasoning?

If you buy something or worse you have to pay license fees on a regular base your earnings will be smaller.

We live in a world that is driven by economic growth so the ultimate goal is to maximize profit.

Of course this has a moral aspect to it as well and I see it but in this case I think it is not outraging enough to be something on the scale of a scandal.

Many businesses use ideas or products for free to start a successful enterprise that earns a lot of money.

In germany its the opposite, no free software in production! Only software with enterprise support!

We Germans are very risk adverse (i hate that sometimes)

Reminds me of private companies who profit from public resources.

Like selling tap water in bottles.

This also rubbed me the wrong way. As an individual I think that shows selfish and opportunistic behaviour and it raises a red flag about that organisation in my mind.

However, for profit companies are not here to do what’s “correct” they’re here to make money for its investors. If I had decision making abilities at Nginx I’d be conducting a comprehensive review of the free OSS offering and redacting the features and overall value with extreme prejudice.

Dropbox never paid because it COULD not pay. If you have an enterprise, paid version of your OSS product it has to be impossible for an enterprise to use it for free.

This ain't mind-blowing by any means IMO.

If the said company has unknown track record, then doing business with them is risky.

What if the company goes out of business in near future? Or get acquired (actually I think A lot of infra companies's end goal is to get acquired)? What if they raise the price out of sudden? How extensible/customizable their solution is?

The trust is the key here. If I am in the position to buy software from somewhere and cost isn't the primary concern, the money would goes to a known/stable figure in the industry.

In the case of buying from a small company this can make sense. If they fold it is good to know that the software will still be around.

I’m increasingly concerned about being screwed by non-OSS vendors. Imagine a use case like Slack. Say you have an employee that goes to visit a family member in Venezuela & connects to the company Slack. Slack has been given a mandate to terminate accounts for people in Venezuela by the Trump administration, and now your key employee is cut off from communication, or perhaps your Slack account gets flagged.

Business is motivated to avoid anything which is a tax. Said another way, they are motivated to avoid or escape from anything that grows in line with earnings. If their infrastructure grows, their bill from nginx will grow, modulo the skills and efficiency of their infrastructure teams and the speed of whatever servers they are buying.

In my company thousands of CentOS servers were running, we still had the support license though.

You have a good problem. What sucks is when you sell a foss solution and they want paid support and SLA but the foss maker does not want free money in form of closing out issues/bugs/features they might anyways workon without getting paid for it.

I fully agree. One other good point with paid software is that is more long term. It will be supported as long as there is money involved.

Just look at the JS ecosystem. Everything is for free. But also shitloads of crap. A lot of libraries left unmaintained.

Not sure what nginx is like, but in my experience, the developer/operator experience of commercial software tends to be subpar. For instance, when I worked at a shop that used a ton of Red Hat software (millions of $$ per year in licensing), the commercially-supported versions often were a pain, with requirements like phone-home (that didn't play well with the mandatory corporate proxy), documentation behind a paywall and hard to discover (yes, we had login accounts, but Google couldn't index it), and other disadvantages. The OSS equivalents were easier to access, had better (or at least better-indexed) documentation, and we didn't need to worry about per-seat licensing (again, we were paying for it, but we still had to track it).

If you're going to sell software that has an OSS variant, make sure the commercial experience actually outshines the free one.

Makes sense actually. Your motives are conflicted so you can't see it.

Also if I can ask, is your product also closed source (in any nature at all), but made with open source components?

I think the best slice of who's migrating to Envoy can be observed via EnvoyCon talks[1][2]:

* Lyft (of course)

* Spotify

* Stripe

* Square

* eBay

* Yelp

* Pinterest

Plus the support from major cloud providers: Google, Microsoft, and Amazon.

[1] https://envoyconna18.sched.com/ [2] https://envoycon2019.sched.com/

It may actually become a trend. For well known reasons:

- Community

- Nginx served us well for almost a decade. But it didn’t adapt to current development best-practices

- Operationally Nginx was quite expensive to maintain

- C++

- Observability and monitoring

etc...

It's a drop in the water compared to Nginx usage.

Slack announced they were going to switch.

The sibling comments point towards the difference in configuration if you take the "out of the box" product. But there is also a vast difference in how code is organized, in case you ever have to touch it.

From my point of view Nginx feels "old". It's a C codebase without a great amount of abstractions and interfaces, and instead having a bunch of #ifdefs here and there. Unit-tests and comments are not to be found. Build via autotools.

Envoy looks as modern as it gets for a C++ codebase - apart from maybe the lack of using coroutines which could be interesting for that use-case. It uses C++14, seems to be extremely structured and documented, has unit-tests, uses Bazel for builds, etc.

So I think the experience for anyone being interested in working on the code will be very different, and people that prefer the style of project A will have a very hard time with the style of project B and the other way around.

The operational overhead shifts to more API stuff, so people can write 100 lines of code instead of modifying 1 line of config, it feels like.

This is never going to end as more things shift towards being core APIs that allow you to write code instead of configure things. It's not even configuration-as-code, it's just code managing configuration files.

edit: I think my comment comes across maybe kinda rude. My beef with Envoy is that the documentation is _extremely_ complex, and I've repeatedly asked 'How do I get started with xDS?' and been pointed to the spec, which I think took some time to read through and when I asked others about how to setup LDS/RDS/CDS/SDS was met with a like 'what are these things...? just use xDS,' which led me to a lot of frustration. This has been my experience each time trying to approach Envoy, and xDS.

Yep, gRPC is the new toy for distributed computing, after everyone realised that DCOM, CORBA, RMI, Remoting actually made sense instead of parsing XML and JSON text formats all the time.

Totally get it! The team (@veshji and @euroelessar) struggled a bit in convincing me that the new Envoy way is a simpler one. I do not regret giving in.

Operationally, there are many differences (esp. around Observability) but if I were to distill it down to one thing it is a clean separation between data- and control-plane. This basically means that it was designed to be automated and the automation layer (xDS) itself runs just like any other normal service in production.

This is just the software industry. Maybe it’s because we’re so young. Maybe it’s because software is relatively easy to change and experiment with.

Who knows. All I know is, it’s exhausting, and ultimately it’s terrible for the end user. We have no idea what we’re doing when we pull in a new dependency like this. There’s tiny corner cases we don’t think about, and those get passed on to the user.

Innovating is fun, but exhausting in aggregate.

Envoy is lot more configurable and rivals nginx on performance (especially throughput). Codebase is a lot more manageable (but that’s my personal preference). Runs circles around nginx on observability features.

Several companies use Caddy for exactly this purpose. Fathom Analytics for example uses it for their custom domains feature. Caddy can even reactively provision certs during TLS handshakes. It's a native feature. Why does it require lots of workarounds?

You can definitely “lazy load” TLS certs into Envoy.

The SDS (Secrets Discovery Service) supports this, and is touched on in TFA: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

You provide a gRPC service that can return the keypair needed for any host, with host config also being dynamic.

https://www.envoyproxy.io/docs/envoy/latest/configuration/se...

We are using OpenResty with lua-auto-ssl for exactly this purpose, and it works like a charm.

Lighttpd seems to have solutions for that. Did you have a look at it?

https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL...

> A traditional problem with SSL in combination with name based virtual hosting has been that the SSL connection setup happens before the HTTP request. So at the moment lighttpd needs to send its certificate to the client, it does not know yet which domain the client will be requesting. This means it can only supply the default certificate (and use the corresponding key for encryption) and effectively, SSL can only be enabled for that default domain. There are a number of solutions to this problem, with varying levels of support by clients.

Then, the best approach seems to be the following:

> Server Name Indication (SNI) is a TLS extension to the TLS handshake that allows the client to send the name of the host it wants to contact. The server can then use this information to select the correct certificate.

Traefik makes it fairly easy (inasmuch as it makes anything easy). But it's just a proxy, not a web server.

I would think the way to do this would be to run a separate TLS daemon that handles the certificates (including acme challenges, presumably) and then pass the socket to your http server, either by proxying it (preferably to a unix socket), or like actually pass the FD with the session keys.

I don't think hitch (formerly stud) supports acme challenges, but that's where I'd start.

Apache can do automatic TLS with mod_md.

I noticed on social media a lot of negativity toward Dropbox and sometimes even on HN. The negative sentiment appears to come from tech circles who feel One Drive offers a better price point or iCloud works great for them, so Dropbox shouldn't exist.

Personally, I prefer Dropbox. I found problems with One Drive. Google Drive client was always hit and miss and I could not rely on it. iCloud is not cross platform (afaik). Dropbox has worked where ever I needed it.

Dropbox is more expensive but I prefer to have my files in Dropbox (as a separation of concerns) rather than have a single tech company control every aspect of my life.

My experience with the 'average Joe' is that Dropbox is easy and it works. Yes, they might save a couple dollars switching to OneDrive but Dropbox still offer a good product. Will Dropbox survive long term? I certainly hope so. I have no affiliation, aside from being a customer.

I've admired Drew and the early Dropbox team for getting things done and shipped even when compiled python GUIs was edgy as the initial Rails version of Twitter was. But they shipped and validated the market. Now adding all those fancy and cool tech mentioned in the blog post will increase the complexity by a lot but it's not clear what are the real benefits. Does a decreased number of machines running really justify the migration and addition of complexity? Maybe they have some new products in the pipeline that built upon the new stack. Or they waste their time. We will see.

Google and Apple, to start.

Sadly, it would probably be as hard to maintain as an opensource version. We really want to have access to the code to make sure we can fix, troubleshoot it, understand it fast...

Things that may've help:

-- Configuration definition (e.g. protobufs.)

-- More focus on observability: error metrics (instead of logs), tracing, etc.

-- gRPC control plane.

-- C++ module development SDK.

-- (ideally) bazel.

Some dataplane features like gRPC JSON transcoding, gRPC-Web, and http/2 to backends.

The price is really insane for Nginx commercial.

We've made a note about how inefficient our solution was and what was the plan to fix it. Sadly, to get proper stats in nginx we needed two things:

* C interface for stats, so we can would have access to from C code.

* Instrument all `ngx_log_error` calls so we would have access not only to per-request stats but also various internal error conditions (w/o parsing logs.)

That said, we could indeed just improve our current stat collection in the short term (e.g. like you suggested with a per-worker collection and periodic lua_shared_dict sync.) But that would not solve the longterm problem of lacking internal stats. We could even go further and pour all the resources that were used for Envoy migration into nginx customizations but that would be a road with no clear destination because we would unlikely to succeed in upstreaming any of that work.

> The `a factor of 3` is quite large to me. Maybe you put all your stats in lua_shared_dict? You don't need to synchronize the stats every time. Since the collection regularly happens in per-minute frequency, you can put the stats as Lua table, and synchronize them once per 5/10 seconds.

Any pointers on how to achieve this for someone just starting out with lua and openresty? I have the exact same thing (lua_shared_dict) for stats collection, would love to learn a better way.

nginx had cold (for American standards) and conservative community to begin with, commercial version and F5 ownership likely "closed" it even more

it's a pity that community never evolved with nginx growth and success

To tell you the truth, we didn't consider it. From what I can get from the architecture docs[1], it can be a decent platform for apps, but might not be the best choice for a general purpose ingress/egress proxy (at least for now.)

[1] https://caddyserver.com/docs/architecture

Found some related discussion here: https://caddy.community/t/caddy-to-replace-envoy-for-ha-setu...

Nginx configuration is bound to its workers. When you reload nginx new workers are created(and start responding to new connections) and the old ones are drained. The draining finishes when the last connection is finished or a timeout is reached. In OSS nginx every upstream change requires a configuration reload. If you have lots of upstream changes and don't want to terminate connections prematurely, this can quickly require lots of RAM as you have many workers. Stock nginx worker is around 150Mb, but issues with openresty integration(they mention lua usage) can bloat this to > 1GB.

It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.)

On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.

Well, a single server doesn't really need to do more than 10Gbps or 100k connections. Going above is a "simple" matter of managing horizontal scaling.

What I wonder about is how do you distribute the traffic on the higher level? I imagine there are separate clusters of envoys to serve different configurations/applications/locations? How many datacenters does dropbox have?

I was running a comparable setup in a large company, all based on HAProxy, there was a significant amount of complexity in routing requests to applications that might ultimately be in any of 30 datacenters.

We did require some of them[1]. Esp. painful were Transfer-Encoding quirks, and some dances around old HTTP/1.0 backends and request buffering.

Compared to NGINX though, it was relatively easy to push these fixes upstream. Community is very welcoming to outside contributions.

[1] https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...

We have a mix of static and dynamic configuration. We started with almost everything defined in the configuration and implemented our control plane only for endpoint discovery service. Over the time we implemented more and more features there (certificates, tls tickets, route and vhost configuration, etc). We decided to write own implementation on control plane - actually the core part is pretty simple and easily expandable.

We have built our own control plane in golang tightly integrated with an existing infrastructure (service discovery, secrets/certificates management, configs delivery, feature gating, and so on).

Can you describe your use-case?

If you are talking about the ability to select a certificate on the fly via `ssl_certificate_by_lua_block`[1] we are not aware of such functionality. If you are missing something, I would highly encourage you discuss it with the community on a github!

From Oleg Guba, Traffic Team TL, co-author, and person driving the deployment:

* ListenerFilters + NetworkFilters are flexible enough, that some of the custom logic could be just moved to the config.

From Ruslan Nigmatullin, our head Envoy developer:

If you are talking more about a custom verification code there is already couple of ways to do that:

* Client TLS auth Network Filter: https://www.envoyproxy.io/docs/envoy/latest/configuration/li...

* Alternatively, if you are writing C++ extension you can use Network::ReadFilter, Network::ConnectionCallbacks.

[1] https://github.com/openresty/lua-nginx-module#ssl_certificat... [2] https://github.com/openresty/lua-resty-core/blob/master/lib/...

She’s on the board of directors: https://www.dropbox.com/about.

She has never worked for Dropbox.

Dropbox works for her :D

We've actually started experimenting with converting our static file serving to "proxying to S3 + caching." This is simpler from deployment and development perspectives (for companies that do not have a distributed filesystem, like Google with its GFS):

* for deployment we do not need to maintain a pool of stateful boxes with files on them and keep these files in sync.

* for development, engineers now have a programatic interface for managing your static assets.

Our technology stack is very gRPC friendly, so developer experience is actually better with it, than without (though this is very subjective.)

As for the middleboxes, using gRPC-WEB[1] allowed us to switch Desktop Client App to gRPC even behind firewalls/IDSes that do not speak HTTP/2 yet.

As for the HAProxy, Dropbox used to use (circa 2013) it specifically for loadbalancing, but we eventually replaced it with our Golang proxy. That said, recent HAProxy improvements (v2.0+) make it quite an awesome dataplane and an excellent loadbalancer!

[1] https://github.com/grpc/grpc-web

Seems like they were already using openresty. Having used openresty professionally, I appreciate that it provides ways to write code to solve a lot of the problems outlined in TFA, but solving the problems out of the box is significantly better.