But cloudflare references robots.txt a lot ( which i mentioned before => to respect robots.txt)

Additionally, they solve the authentication problem here. As a website owner that got bad crawlers ( that copied user agents), i just whitelisted Google's IP's and blocked all the other crawlers.

It seems that cloudflare is actually fixing this problem and making competition for Google possible here.

I recall an article that Bing actually circumvented robots.txt a bit, because site owners were only allowing Google and blocked all the rest => Bing. Which gave Google an unfair advantage ( searched for it, couldn't find it)

Similar article to highlight the issue: https://www.fastcompany.com/90709672/the-little-known-reason...

Our opinions about this seem to differ severely. Cloudflare actually enables good bots to start competing ( while respecting robots.txt).