If you have a wide low-traffic website then bots of all sorts will make a majority of your traffic and subsequently a majority of your AWS costs.
If you see money spent on search engine X indexing and very few users incoming from that search engine it's a rational decision to block it. Or ask for money (that actually happens).
Overall it's a systematic problem with building a search competitor: 1) Your costs are largely proportional to the size of your index 2) Your income is proportional to your userbase 3) You need a huge index to be competitive even if you don't have any users yet
So, very hard to bootstrap even when you exclude all other advantages of the existing monopoly like browser-based distribution.
Microsoft has the money to beat the indexing problem so they argue about distribution in court but all the small players can't even get to that level of failure.
