If you make sure that the response is cacheable, then Cloudflare will cache it at the edge and so only the first check for any particular IP will be slow.

What makes a response cacheable is a little complicated. There's cache headers, but also some heuristics involved. However, you can override all of that from a Worker by passing an explicit cache TTL to fetch():

    fetch(url, {cf: {cacheTtl: 86400}})

This will force Cloudflare to cache the response at the edge for one day regardless of anything else. (Note: The documentation currently claims this option is available to enterprise customers only, but as of this week, it actually works for everyone. Docs to be updated soon.)