1
I wanted to share a simple RL environment I built that gave frontier LLMs a set of tools that lets it zoom and pan across a digitized pathology slide to find the relevant regions to make a diagnosis. Here are some videos of the LLM performing diagnosis on a few slides:
(https://www.youtube.com/watch?v=k7ixTWswT5c): traces of an LLM choosing different regions to view before making a diagnosis on a case of small-cell carcinoma of the lung
(https://youtube.com/watch?v=0cMbqLnKkGU): traces of an LLM choosing different regions to view before making a diagnosis on a case of benign fibroadenoma of the breast
Why I built this:
Pathology slides are the backbone of modern cancer diagnosis. Tissue from a biopsy is sliced, stained, and mounted on glass for a pathologist to examine abnormalities.
Today, many of these slides are digitized into whole-slide images (WSIs)in TIF or SVS format and are several gigabytes in size.
While there exists several pathology-focused AI models, I was curious to test whether frontier LLMs can perform well on pathology-based tasks. The main challenge is that WSIs are too large to fit into an LLM’s context window. The standard workaround, splitting them into thousands of smaller tiles, is inefficient for large frontier LLMs.
Inspired by how pathologists zoom and pan under a microscope, I built a set of tools that let LLMs control magnification and coordinates, viewing small regions at a time and deciding where to look next.
This ended up resulting in some interesting behaviors, and actually seemed to yield pretty good results with prompt engineering:
- GPT 5: explored up to ~30 regions before deciding (concurred with an expert pathologist on 4 out of 6 cancer subtyping tasks and 3 out of 5 IHC scoring tasks)
- Claude 4.5: Typically used 10–15 views but similar accuracy as GPT-5 (concurred with the pathologist on 3 out of 6 cancer subtyping tasks and 4 out of 5 IHC scoring tasks)
- Smaller models (GPT 4o, Claude 3.5 Haiku): examined ~8 frames and were less accurate overall (1 out of 6 cancer subtytping tasks and 1 out of 5 IHC scoring tasks)
Obviously, this was a small sample set, so we are working on creating a larger benchmark suite with more cases and types of tasks, but I thought this was cool that it even worked so I wanted to share with HN!
There’s a demo here: https://www.loom.com/share/761cac3090ba4db8b2ce9d713873333a?..., as well as a walkthrough of our Python SDK: https://www.loom.com/share/c4b3f95235e24d99a7ea571c4564602b
YC initially funded us for AI web-scraping, but early in the batch we realized we wanted to pivot to something in data privacy - in some ways the opposite of web-scraping…
From talking with SaaS developers, we learned tenant isolation (making sure one customer’s data is not shown to another) is often considered in development but rarely a core competency. Many developers struggled setting up Row-Level-Security (RLS) correctly on Postgres, some are enforcing application level access controls with none at the database-level at all, and many didn’t know which multi-tenant db architecture to start with and just decided to deal with it later. However, with increasing data sensitivity and compliance requirements, they’ve seen growing customer demand for stricter data requirements, including more demands for dedicated database instances or even databases deployed on the customer’s private cloud.
We also learned that SaaS developers prefer to have databases on their own cloud, and many who start on a 3rd party DBaaS eventually move infrastructure to their own cloud later. Having things on your own cloud makes it easier for SaaS to offer on-prem/full-siloed deployments and meet compliance requirements for larger customers. So we pivoted into being a BYOC platform and made Fortress integratable with cloud-native databases.
Our goal with Fortress is to give SaaS developers the ease of use of a managed DBaaS, native isolation for tenants, and the ability to programmatically provision and access database instances on any cloud.
Fortress provides SaaS developers an abstraction at the tenant level, allowing them to enforce tenant isolation through a function without having to set up RLS themselves or use WHERE statements in every query. Currently, on the Fortress platform, this is simple as every tenant in a shared database is given a logical db in a Postgres Cluster and we handle routing (We are currently working on a solution that provides native tenant isolation for tenants stored in shared tables).
Through our SDKs, developers only need to handle one connection to the Fortress client and we'll route requests and handle connection caching to ensure minimal latency to the right tenant’s data. We are working on creating more SDKs and simplifying existing ones. We currently support Postgres via AWS Aurora with plans to support other cloud-native databases and open-sourced databases via Kubernetes.
If you want to try it out, we’ve opened up self-serve to spin up new databases on your AWS cloud. You will need to create an account with Fortress (https://fortress.build/auth/sign-up) to connect it to your own AWS account - that’s what BYOC (Bring Your Own Cloud) means! But if you are uncomfortable with granting us IAM permissions, we are offering a free db on the managed Fortress cloud for you to test out our SDKs (just cancel out of the Integrations page, create a database on the Database page, and select managed as the option, you will have a limit of 1). Note: AWS takes a while to spin up new clusters, adding tenants to existing clusters should be fairly instantaneous.
To provide you something basic to play with, we created a simple python flask application where you can easily test out our python SDK (https://github.com/fortress-build/python-example). We created a video of us walking through the example with Fortress (https://www.loom.com/share/0245bec78d9d4dbeba8836c4112aa5da?...)
We hope you’ll test it out! We wanted to launch on HN to hear honest criticism, ideas, and pain points in this space. We are a super early stage database product and recognize that we have a ton of room for improvements. Looking forward to your input!