Before you buy a domain name, first check to see if it's haunted (opens in new tab)

I purchased a valuable premium domain to host a personal art collection (of anime cels). For some bizarre reason, the site was inaccessible from my work computer and it was de-listed from Google even if I typed the url itself into search.

I hired a square space specialist to figure out why, to no avail. I then begged our company’s CISO to investigate and it turns out we had some firewall setting on UniFi that blocked the domain because it appeared on a list. Once I checked way back, it turns out that it was as an anime porn aggregator years back. I personally reached out to all the web filters out there (Google, Symantec, bing) and one by one filed tickets for them to mark it as art instead of pornography and it worked. I am now properly crawled on Google but still MIA on Bing, search console is giving me some BS error that’s incomprehensible, typical of MSFT.

Sadly, I think this would be instantly gamed by abusers. They would release the domain name and attempt to register as a new owner or start repeatedly doing handoffs. It's difficult to tell who the owner is changing between and whether or not the new one is a better actor than the former.

So I checked the Bing Webmaster Tools. URL Inspection says "Discovered but not crawled - The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you to follow Bing Webmaster Guidelines to increase your chances of indexation."

That's quite unhelpful. What's more, when I open the "Live URL" tab, it says, in green: "URL can be indexed by Bing."

It's a simple static Hugo site hosted on Cloudflare R2 (DNS mapped directly to bucket). https://pagespeed.web.dev gives it a score of 100 in every category.

Anyone else had something like this happen?

HSTS (which forces browsers to validate HTTPS when connecting) asks browsers to cache the configuration for a set "max-age". Some sites set huge values here, like Twitter's 20 year max-age[1]. There's also the preload lists [2] to consider. This creates a problem if you want to serve non-HTTPS/unencrypted HTTP on your new domain and the previous owner didn't.

MTA-STS [3] is another variant that's becoming more popular. It limits which mail servers your domain uses and enforces TLS certificate verification. "max_age" is capped to a year by the RFC. If you don't set your own policy, then the previous domain owners policy would impact any senders who previously cached the policy.

Thankfully HPKP (key pinning) is obsolete, otherwise you'd also need to worry about old pinned keys too. That RFC recommended, but did not enforce, a 60 day max-age limit.

These are especially tricky as the old security policy only lives in the caches of any end-user devices that previously connected to the domain. Double haunted.

[1] https://alexsci.com/blog/hsts-adoption/

[2] https://hstspreload.org/

[3] https://alexsci.com/blog/smtp-downgrade-attacks-and-mta-sts/

Just one more place where the web gets screwed by a company too big to have to do basic customer service.

Some practical advice here: do not change your canonical domain[1] name unless you really really have to.

If he had just set his fun new domain to redirect to the existing domain, instead of making the new domain the canonical, it likely would have had no negative effect.

I’m not saying this is how things should work. But the practical reality is that your domain name is like a Social Security number: it’s the basis for assigning a type of reputation score, even though it was not intended to do that originally.

[1] The domain at which your web pages finally load, after all redirects have completed.

I don't think it's possible to fix this problem without also helping bad actors. Maybe it's a problem that just isn't worth fixing. Just don't buy preexisting domains unless it's a project big enough to justify the necessary cost of due diligence.

Checking web archive is a basic operation to test if site was hosting anything fishy - not only pirated stuff or porn - often websites has been hacked and changed into link farms or simply were bought on aftermarket simply to use it's SEO value to pass the strength to other domains.

Anyways good point regarding email filters.

I think the mistake here is the redirect old to new. That is always risky so only do it if deseprate. In this case I would have done the redirect from new to old. Then just use the new as a vanity url.

Even automated queries are likely to spill the beans. Someone else could snag the purchase before you, or bid up the price. But it's a risk you may need to calculate.

Wayback machine would've saved me there, had I done my due diligence!

Here in the UK with EE/BT that correctly redirects to automattic.com, but it might not for you depending on your ISP.

The wayback machine shows adult content links prior to the domain being put on sale, hence the blocking.

But it does require manually reporting false positives to each vendor

I am admittedly a bit distant from SEO. The above is not true and hasn't been true for a long time.

Managed to get a takedown notice thanks to that idiotic "feature" while not even aware the domain is serving anything

It’s like when cars took over the streets, and instead of blaming cars for being dangerous for regular people using the streets for walking, the concept of “jaywalking” was invented by car companies to place the blame on people for daring to obstruct cars. Or the concept of “personal carbon footprint”, commonly used to move blame from companies to individuals, when in reality whatever individuals, even in aggregate, could do is utterly insignificant compared to what companies and legislation could accomplish.