Maybe I'm weird, but I'd consider colo to be a closer cooperation than just renting some virtual servers from wherever. And just getting told "Yupp, your null-routed. No, we can't give you access for specific sources over a different path. Get fucked" - or, in fact, not getting told that - is ... one of our ex-hosters was like that.
And as a service provider, I have strong feelings about the customer service there.
Maybe I don't know big infrastructures, but this just leaves me with a weird feeling in my guts.
But hell. Make sure to give your engineers - and their family - something. After some hell-weeks, we've given people some budget to do something fun with their family, because the company had to take so much private time during those weeks.
I had a ten year colo relationship with AT&T, mostly but not entirely datacenters, where the salescritter would be changed every single time I tried to talk to them. 3-4x a year, they either left or were re-org'd elsewhere or our account was moved to a different group. (Datacenter techs: competent. Datacenter manager: competent. Sales staff: chaotic.) Eventually I learned to open a ticket to find out who the salescritter would be.
On the other hand, there was a competent server salesperson at company X who I used to buy lots of stuff from; company X started raising prices and flubbing tech support returns, so on my next order I tried company Y. I was rather surprised to get the same server salesperson -- who had just taken a job at Y.
Good coverage of the event: Security Now! Podcast https://www.youtube.com/watch?v=ehfV7cRLkFE
But, even if you only have a single action available, isn't communication with the customer key? We've pulled customer systems offline, and parts of the customer were glad because we could provide information and help with handling harsh situations they had.
And even if this isn't a 1% revenue customer, if they are receiving a DDOS-whopper you cannot happen, that would probably be a customer you want communication with. Even if the goal was just to figure out if there was glory in hosting them or to just fire them.
Just null-routing and being unavailable seems unprofessional, though I know quite well how .. well-structured these acquire-everything hosting providers are internally. Doesn't change that this is bad customer service.
I'm not a big fan of some of his hot takes, but I still respect him and trust him with my data.
Social/collaboration features are explicitly deprioritized by design; I think this is a natural consequence of srht being built by and for lone wolf developers. GitHub and Gitea (which is basically a github clone) seem much more geared toward collaboration by groups, something most small-time f/oss developers don’t need.
Also, the emphasis on email and irc is bad, imo. The web won because it is better. A lot of the anti-web stuff is just tradition.
> We also did our best with hg.sr.ht, but it is community maintained
It looks like git.sr.ht is hosted on OVH in France, while hg.sr.ht is hosted on High5! in the Netherlands.
It's not entirely clear to me how this affects their product roadmap or support, but definitely good to know.
> restoring service was delayed until we could get the community maintainer, Ludovic Chabant, online to help
Maintainer, singular!
The only reason i use Sourcehut, and the main reason i pay for it, is because i stubbornly still use Mercurial, and want first-class support for it. With the utmost of respect to M. Chabant, that is not exactly first-class.
It would appear that Ludovic Chabant is working full-time at Epic Games. He is unlikely to have the capacity to be on call for Sourcehut.
They explain it here:
> However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS
Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with an anti-establishment manifesto than any practicality. Then again, I might be wrong.
I appreciate the service though so I hope the differences between maintained and operated doesn't mean anything in the long term.
That said I'm very happy to use Sourcehut and I think they'll overcome these challenges over time. They seem to have the staying power.
> However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.
1) Test/demonstration of a DDOS against a random target.
2) Attack against a project hosted on sourcehut to make it unavailable (there was even the speculation of disabling a master repository so an end-user could not check that his own local version was the correct one, thus using it with a security hole or a trojan)
3) Attack against a page hosted on sourcehut (I joke that someone wrote "Putin = Fag" on his sourcehut hosted blog).
4) What else ?
I'm actually serious there - school/university holidays invariably bring with them a shitload of fairly random DDoS attacks aimed at ... whoever the attacker feels like.
I don't understand and I don't know where they get that volume of packets from, but I've seen it happen a quite sufficient number of times over the years to consider it as a possibility here.
(excuse the vagueness; if you're ever at the same conference bar as me you're welcome to follow me outside when I go for a smoke and ask for the full story then)
> Following our initial quote from CloudFlare, we understand that some CloudFlare employees undertook a grassroots effort internally to convince the leadership to sponsor our needs, and eventually CloudFlare came back to us with an offer to sponsor our services for us free of charge. This was a very generous offer for which we are very appreciative; in the end we did not take them up on it as we had made substantial inroads towards an alternative solution by that time. I have had my reservations about CloudFlare in the past, but they were there for us in a time of need and I am grateful for that.
That said, what will happen when more companies publish their experiences with "enterprise sales"? There's an article from HEY[1] about how broken the sales process is. To get a quote, you normally have to endure 2 or 3 zoom calls before the price is unveiled.
There's probably room for an innovator to fix all of this.
1: https://world.hey.com/dhh/the-only-thing-worse-than-cloud-pr...
The article mentions how setting up SourceHut from scratch is a complex undertaking with hundreds of small tasks. Some of those are understandable, especially given the amount of data they surely handle, but setting up a complete environment and restoring from a backup should be a simple and mostly automated procedure, not a gargantuan undertaking that needs to involve the entire team several days. There are always difficulties when production is down, and you're trying to restore a full system while undergoing a DDoS attack, I get that, but the reason we have modern cloud providers and tooling is to make creating new environments as painless as possible. It seems foolish not to take advantage of that.
I'm not a fan of Kubernetes either, but it's good that they're experimenting with it. Hopefully it leads to quicker deployments if this happens again.
This sounds like a case of treating your infra as pets and getting stuck when it suddenly needs to be replaced.
This right here is invaluable and something you only get from experience. Planning and theory only get you so far.
I extend this thinking to deploying large infrastructure changes you've never done before - you can only plan so much before pulling the trigger and just doing it and seeing what happens.
No it wasn't. The outcome is due to major networks being shite. Not accommodating newer technologies and gate keeping services to resolve DDoS attacks.
All major network upstreams could do so much more to make the net more reliable and resilient to small ISP. Myself included.
peer neutral networking, not having tons upon tons of e-waste prone to botnet behaviour, it wouldn't be like this.
