Startup: use <aws|azure|gcp> because it's quick to get started. It's easy to scale it up 100X in 5 minutes for when you make it big. And it provides a long list of easy to use services that are overpriced per use, but which free you to focus on your business rather than complex DevOps.
Then once you scale to the point that you're comparing your salaries and your ops cost, you find smaller services to handle the expensive part of your business. Ones that do exactly what you need, not much else, and charge accordingly. Maybe they're self-operated, or maybe they're other companies with a narrow focus.
Then you write a blog post about how everyone is crazy to use <cloud> since it's so expensive, and you don't know why you ever thought they were a good idea.
I mean it makes sense in a way and I'm not saying it is a bad thing, just a pattern I've noticed.
You can ask the cloud to deploy a container to Kubernetes. You can rent a VM with Hetzner. You can rent rack space in a data center. You can build your own data center. You can build your own server hardware.
Depending on the scale you operate, you'll draw the line somewhere where it makes sense to you (which may change over time).
Dev teams will still operate best when they can deploy containers to Kubernetes and without having to design their own server hardware. You now need to build abstractions in house. Maybe you can sell these to other people? Congratulations you are now a cloud provider.
As others have pointed out it is relatively easy to deploy small k3s clusters on Hetzner Cloud.
Jeez really? We have like, binaries deployed to machines over here.
The machines were there fully paid. But w aws I’d be up in 10 minutes w a lot less arguing . I’m convinced this might be part of AWS edge in big players
Setup one that doesn't eat your wallet is what makes it hard.
Let's also be realistic here: CloudFront is a good solution if you need a file to be served over CDN with low latency in multiple regions. Raw s3 or just a web server is much cheaper if you don't need low latency over multiple regions.
Using Cloud services that are obviously more expensive than self-hosted options reduces both numbers- less time to launch stuff, but less money. The point is that in early-stage startups, the costs are so low that paying an order of magnitude more for Operations may not move the second number by very much at all, since most of the money is paying for developers.
How many companies actually shift their infra vendor?
Granted, thinking about this up front is ideal, but it’s better to start an mvp and build from there than to be paralized by perfection at the outset and not build anything at all.
If I'm targeting AWS, Azure, etc. Then I'm going to leverage their services in a lot of cases. Wether it's AWS Dynamo DB, or hosted PostgreSQL, there are places you will save in either resources or personnel.
A set of Dokku servers behind a load balancer can go a long way. Kubernetes further still. Taking the issue of DB management, static assets or backups goes a long way to freeing up time to make solutions without buying into lockin too much.
It’s totally overkill for the article’s use case, but it enables many cool things that need sustained bandwidth, and has been pretty fun.
I went to their home page to try and find more information about colo pricing, but it's all hidden behind a contact form.
Their site design seems very 1998, makes me feel really nostalgic.
> For example, with a 3-year term for a 42U cabinet, it would be $400 a month for 3 years, and that includes 1G already. After that, for another 3 years, would it be $340/mo for 1G, plus $500/mo for the 42U cabinet, which is $840/mo (more than double)? In both cases, is a /27 always included with the 1G port (pending justification)?
> Yes, you are understanding what our standard rates are without the promotion. Yes, we include the /27 with service. Our promotion is currently our most cost effective solution. Let me know if you have any questions.
Do you really want to build a whole 42U rack full of big heavy servers and have your price doubled after your contract expires?
It is certainly possible that the rate increases at some point, but I don't see why either HE or you would want to commit to anything beyond 3 years from now?
Wonder what if Netflix turns around, and blue oceans [0] its video delivery infrastructure [1][2], like Cloudflare did for text/html. We do know that video has all but eaten up the attention that text/html once had on the Internet [3]
[0] https://en.wikipedia.org/wiki/Blue_Ocean_Strategy#Concept
[1] https://news.ycombinator.com/item?id=28584738
Storing a large number of "cold" or "long tail" bytes is a different physical problem when it comes to storage and IO. And that's also the exact problem you have when hosting lots of third party content (or apps, or lambdas, or anything else). There you're also optimizing for bytes stored:served, but have to place it on cheaper media and with less duplication to make the physical space/power/dollar constraints work.
They certainly could change their business model and infra. But then they end up looking a lot more like a commercial CDN with hierarchal stores, more centralized POPs at IXPs, etc.
Disclaimer: Principal at AWS. Used to work on Amazon CloudFront and a bit of time in AWS Elemental, served a lot of video & streaming bits. Everything above is public domain knowledge.
We used the extra space for spare parts.
Also agreed with sibling comment that 1Gbps is pretty slow nowadays. Do you know if that can be customized?
(And yeah, 42U.)
Is this so different from the cloud? Even an EC2 instance should be assumed to be a single point of failure and you also need multiple (preferably in different AZs or regions) if uptime is important.
If you want to save some money write the scripts yourself and run them in a cron job. Took me a couple hours and I'm no sysadmin.
To me the best thing to do is move all your services into k8s and automate you DB (use open source). Then you can move to any provider when you realise the cloud pricing is shit.
Old and boring things, DNS, load balancer, yada-yada.
If you save thousand(s) it is still could be cheaper.
Another awesome thing about Hetzner is that bandwidth internally is free and automatically negotiated (i.e if you send traffic to a Hetzner IP it will flow internally).
One of the hardest things is being able to tell the bandwidth usage and when it’s inside/outside Hetzner.
[0]: https://nimbusws.com
Is there a faster way to mount storage boxes? I suppose SSHFS would be even slower.
Traditional object storage scales automatically and is very easy to integrate with (most people write apps to interface with S3 these days, and less SFTP), so there's just that small edge! That "sort of" is what I want to get rid of.
Other smaller clouds have services that are a better fit (OVH has object storage, Scaleway, etc), but Hetzner doesn't yet.
I'd be super interested in hearing from someone who has set it up whether it's something an experienced but generalized sys admin could build, or if you need an expert in the matter.
Ceph has much more public resources available (SwiftStack recently got acquired, seemed like they had most of the knowledge in the space), and I've actually set it up a few times now thanks to the excellent Rook[0].
It's definitely a lot easier to set up with Kubernetes (the tradeoff being you need to understand Kubernetes), but it's definitely manageable for a generalized sys admin (albeit one with a bit more experience). I've written about the process:
https://www.google.com/search?q=site%3Avadosware.io+ceph
[0]: https://rook.io
Get an expert, or a consultant. You'd need a really experienced storage admin at the least, and possibly more.
(Yes, I googled it, but I'm looking for a more practical example)
You would use block storage more for a file system mount as a raw device (virtual).
There's overlap and many platforms support both close to interchangeably.
- S3
- VPC
- Compute instances (types don't have to bee too fine grained)
- SQS, SNS
- Some sort of Dynamo and/or RDS functionality.
- Some basic API coordinators, I guess Teraform has providers for lots of stuff these days.
Lambda- and fargate-like things would be a plus, but not strictly necessary.
I feel like 90% of the projects I've ever built could be easily made and scaled with nothing else. Further features hit diminishing returns really fast, and serve to muddy the waters up around what tooling is best, or even what exists.
Also it doesn't matter. Cloud vs vps/colo/rented bare metal basically comes down to surplus of cash vs surplus of tech talent.
Any small tech focused company is gonna have more of the latter then the former, and in that situation, cloud doesn't make sense when ansible, k8s, and the like exists.
Tune in next month for their article on how much fun manually scaling everything is.
In fact, before the cloud craze came around, internet services used to run just fine despite autoscaling not being commonplace. For a lot of use-cases, autoscaling is a problem created by the cloud's high prices and is unnecessary if hardware is cheap enough to run continuously.
Or maybe being 38 makes me really old in the way I approach things...
All that can be done by a competent admin/dev/devops but it takes nontrivial amount of time to setup and maintain. Having solutions integrated into your cloud of choice is a huge time saver.
People who manage huge "cloud" installations feels more like what the traditional sysadmin role was. Everything is so different than your personal setup, you have no use of the same skills outside of that particular provider, and there are many concepts that you have to deal with that you never deal with locally.
https://pellegrino.link/2021/05/31/scaleway-object-storage-s...
Similarly, I saved about x100. For sure, the services do not offer the same level of guarantees and speed but depending on my case it was worth a migration.
Looking forward to trying Cloudflare R2 soon!
[0] https://news.ycombinator.com/item?id=28747798
[1] https://blog.cloudflare.com/workers-optimization-reduces-you...
Why? Azure egress bandwidth is too expensive for a Tor relay, even for a well-paid software engineer like me. I have ~1.5 Gbps Tor bandwidth. If I were to put that on Azure, I would be paying a mortgage worth of bandwidth fees. Same can apply to AWS and Google Cloud.
However, Big Clouds charge more for bandwidth because (a) they're a leader and (b) they have paid peering deals with Big Telecom (e.g. AT&T, Deutsche Telekom, Telefonica, etc.). Smaller hosts like Hetzner don't peer with those ISPs and rely on "transit", but that also means slightly worse performance to Big Telecom.
In short, big clouds can use their market power and advanced technology to command a premium, whereas smaller clouds have to win over customers with lower prices to make up.
Tor relays are ill-suited for Big Clouds, but then your 5000-node Kubernetes cluster is ill-suited for OVH or Hetzner.
I do use the free Azure credit for Tor Bridges, however.
Wait, this seems backwards to me. I'd expect companies that can peer with Big Telecom to charge you less for bandwidth than companies that have to pay for transit. What am I missing?
If they receive any kind of abuse report, they instantly put you on a 24h timer, but they won't show you contents of that message. If you do not respond in time, they will simply block the IP. What's strange is that I was using Cloudflare in front of Hetzner, but it seems that Cloudflare is ratting their users out and forwarding any abuse mail to the owners of "cloud protected" IPs, all without any notification or warning. Hetzner then sends you an email like "Please remove <sitename> from our network within the next 24 hours. This site violates 6.2 of Hetzner's terms and conditions.", where site in question is an obvious mirror/search engine of another social network.
What's perhaps more irritating is that Hetzner Cloud has a tiny LIFO queue of ipv4 addresses unique per customer, so you will get the same IP as soon as you release one. You can imagine what happens to a Rancher-managed k8s cluster with Hetzner node driver when one of those IPs gets a no-route: it will get stuck trying to re-deploy a failed node. Mine eventually fell apart though because I did not attend to it for a couple of years.
I'm now hosting from a proverbial garage (NixOS and cron) but proxying traffic through a cheap cloud VM, which saves me at least $1500/mo.
Whatever you (or your customer) were doing likely hit the point where they assumed the police would be by for a chat.
For this kind of response, you're probably looking at (1) extremist material or (2) CSAM.
Is that a typical way to describe cost savings? I would have said 1/2 the cost of AWS. It almost sounds like cloudflare would pay them money (e.g. you go to the store and see something marked 200% off, so you get paid to buy it, like when oil prices went negative).
Related: I've always found the 2x, 10x, etc notation confusing. Does a 2x improvement mean a 100% increase (e.g. 10 -> 20) or a 200% increase (e.g. 10 -> 30)? Would someone ever say a 1x improvement or a 0.5x improvement?
2x savings means you need to double the new price to get the old
10x savings means multiple new price by 10x to get old
59x savings...multiply by 59.
And it still makes me feel nervous that 3x improvement = 200% increase. It seems to me like a big opportunity for misunderstanding.
I think the interest in this comes from the idea of identifying the needs and optimizing appropriately.
although I hate to shill for Cloudflare, as I believe they are evil, but anyway there you go
What efforts against censorship? Like how they (actively) cancelled 8chan and Daily Stormer?
I won't go into anti-surveillance much because it's too contentious. Very few recognize or want to acknowledge the downsides of 1.1.1.1.
I honestly don't want to get into it more than that in part because I've forgotten more than I used to know about their various business practices. (I used to work near to this space.) I have a vague idea that they have made centralized (power concentrating) choices in some cases, where they could have made decentralized choices, but I can't recall specific products ATM and, well, I'm only willing to invest so much time in this post. By their literal mission "to build a better internet", we can see from those choices that better is in the eye of the beholder. Just as evil has always been as defined by Google.
I don't mean that Cloudflare is willfully evil. At the time that I was intensely interested in them and formed this opinion I could find no evidence of willfulness, and their products and their operations have all the good protections. But they do get to define 'better' in their own image and in favor of their own interests.
I'm sorry to be so vague.
My understanding was that they dropped the per-TB billing a few years ago and now it's just uncapped?
1) Website assets are generally optimized for size (images, etc.) and so reduce Cloudflare's total cost to service.
2) Cloudflare believes that free tier users who use it for their website assets are more likely to want the other Cloudflare services and therefore convert to paying customers.
Meanwhile on my Hetzner, I log in, tweak stuff directly, and it works. My static site wound up being much easier and more comprehensible when I dropped all the AWS stuff. I don't miss AWS one bit.
“Why doesn’t everyone do this?”
“I did the other thing so I can do … other things.”
The linked blog post mentions October 2021, while in December 2021 AWS lowered pricing for CloudFront where the first terabyte out to internet is now completely free:
https://aws.amazon.com/about-aws/whats-new/2021/11/aws-price...
For larger use cases most cloud providers have custom or private pricing options available. For example in CloudFront pricing page AWS publicly states that "Custom discounted pricing is available for customers willing to commit to a minimum of 10 TB of data transfer per month for 12 months or longer."
I have noted that most of the blog posts like the one linked above talk about migrating workloads around for savings of tens or hundreds of dollars while focusing on low level IaaS (VMs), and completely ignore the benefits of leveraging cloud services, APIs and related software ecosystems instead of DIY.
E.g., if you’re paying someone $5-7000/month (no idea what an SRE costs) to manage a fleet of say 10 subsystems, their salary should factor into the cost calculations, right?
And the need to use-only-as-needed creates additional burden that doesn't exist with cheaper monthly-paid hardware.
The answer might very well be "it will take less time to manage AWS", but it's not clear. Some might need a pay-stack to handle the cloud's pricing.
Last week I got an email from them saying my server cost would increase from 30eur/month to 48eur/month. I can take afford the increase, but I wonder how many companies with infra bills of tens or hundreds of thousands will be able to afford a 60% increase in hosting costs.
On this topic as its relevant, at Vantage we launched an automated cost recommendation that keeps an eye on egress out of Cloudfront and S3 and recommends when it makes financial sense to switch to another provider (in the linked blog post, Cloudflare) https://www.vantage.sh/blog/cloudflare-specific-cost-recomme... -- this has been very popular with current customers/users.
We're actually in process of adding a bunch of new infrastructure and service providers and should include Hetzner in our list of recommended providers but thought I'd mention for anyone wanting an automated approach to knowing when you hit the financial tipping point, this is one option.
I blame level3 as that’s where it seems to go pear-shaped.
Full on self hosting your own cloud comes usually with a lot more services than "the" cloud does because thats usually why the hoster is considered "small".
In Hetzners case you will for sure at least have to self host a decent RDBMS and it better be replicated, which means more maintenance and servers.
But you do not have to put all your eggs into one basket! There are more than enough ways to split the load and you can even add dedicated servers in the mix too!
It should not be "yep we are on azure, or yup we are on hetzner" but "my job as an architect is to combine the two for maximum value" ;)
It seems like you could build out a CDN on OVH using 1-5Gbps unmetered+guaranteed bandwidth and place servers in the US, Europe, and a few PoPs in Asia for relatively cheap, then use GeoDNS for balancing traffic. But it seems like OVH's pricing offerings have been shifting over the last year to remove the "guaranteed bandwidth" in favor of "unmetered bandwidth" (potentially throttled 50%) on more machine types.
It would still require monitoring and managing bandwidth saturation per host, and it's unclear how much extra hassle OVH adds. But in theory it seems easier than setting up and managing colos in many countries?
