This was our launch: https://news.ycombinator.com/item?id=36032081
We recently decided to revive and rebrand the project after seeing a sudden spike in interest from people who wanted to connect LLMs to data - but specifically through browsers. It's also a problem we've experienced firsthand, having built scraping features into Psychic and previously working on bot detection at Robinhood.
If you haven’t built a web scraper or browser automation before, you might assume it’s very straightforward. People have been building scrapers for as long as the internet has existed, so there must be many tools for the job.
The truth is that web scraping strategies need to constantly adapt as web standard change, and as companies that don’t want to be scraped adopt new technologies to try and block it. The old standards never completely go away, so the longer the internet exists, the more edge cases you’ll need to account for. This adds up to a LOT of infrastructure that needs to be set up and a lot of schlep developers have to go through to get up and running.
Scraping is no easier today than it was 10 years ago - the problems are just different.
Finic is an open source platform for building and deploying browser agents. Browser agents are bots deployed to the cloud that mimic the behaviour of humans, like web scrapers or remote process automation (RPA) jobs. Simple examples include scripts that scrape static websites like the SEC's EDGAR database. More complex use cases include integrating with legacy applications that don’t have public APIs, where the best way to automate data entry is to just manipulate HTML selectors (EHRs for example).
Our goal is to make Finic the easiest way to deploy a Playwright-based browser automation. With this launch, you can already do so in just 4 steps. Check out our docs for more info: https://docs.finic.io/quickstart
This actually creates an evergreen problem that companies need to overcome, and our paid version will probably involve helping companies overcome these barriers.
Also I should clarify that we're explicitly not trying to build a playwright abstraction - we're trying to remain as unopinionated as possible about how developers code the bot, and just help with the network-level infrastructure they'll need to make it reliable and make it scale.
It's good feedback for us, we'll make that point more clear!
While this might be true in theory, it doesn't stop them from trying! And believe me, it's getting to a point where the WAF settings on some websites are even annoying the majority of the real users! Some of the issues I am hinting at however are fundemental issues you run into when automating the web using any mainstream browser that hasn't had some source code patches, I'm curious to see if a solution to that will be part of your service if you decide to tackle it.
I wish your company the kind of success it deserves.
If I remember correctly, Skyvern also has an implementation of scaling these browser tasks built in.
ps. Is it not called Robotic Process Automation? First time I'm hearing it as Remote process Automation.
Cloning into 'tarsier'...
remote: Enumerating objects: 15238, done.
remote: Counting objects: 100% (1613/1613), done.
remote: Compressing objects: 100% (929/929), done.
Receiving objects: 100% (15238/15238), 3.01 GiB | 14.82 MiB/s, done.Based on the feedback in this thread we're going to be releasing an updated version that focuses more around tooling for the browser agents themselves as opposed to scaling/scheduling, so stay tuned for that!
If you don't already have this feature for your system, I would recommend it.
What does this check look like for you? Do you just diff the html to see if there are any changes?
Thanks for the feedback! I just updated the repo to make it more clear that it's Playwright based. Once my cofounder wakes up I'll see if he can re-record the video as well.
I have never, ever understood anyone who goes to the trouble of booting up a browser, and then uses a python library to do static HTML parsing
Anyway, I was surfing around the repo trying to find what, exactly "Safely store and access credentials using Finic’s built-in secret manager" means
I am open to the fact that html5lib strives to parse correctly, and good for them, but that would be the case where one wished to use python for parsing to avoid the pitfalls of dragging a native binary around with you
- connect to it remotely
- ghost cursor and friends
- save cookies and friends to data dir
- run from residential ip
- if get served captcha or cloudflare, direct to solver and to then route back.
- mobile ip if possible
…can’t go into anymore specifics than that
…I forget the site right now, but there a guy that gives a good rundown of this stuff. I’ll see id I can find it.
It seems that some sites can determine when using headless or web-driver enabled profile.
Sometimes I'm through a VPN.
The automation is the easy part.
One thing I’ve also been doing recently when I find a site that I just want an api is just use python and execute a curl via python. I populate the curl from chrome’s network tab. I also have a purpose built extension I have in my browser that saves cookies to a lan Postgres DB and then the use those values for the script.
Can even probably do more by automating the browser to navigate there on failure.
This is not always possible, but if the product in question has a mobile app or a wearable talking to a server, you might be able to utilize the same API it's using:
- intercept requests from the device - find relevant auth headers/cookies/params - use that auth to access the API
I can see a few years from now almost all web traffic is agents.
I don't think the dead internet theory is true today, but I think it will be true soon. IMO that's actually a good thing, more agents representing us online = more time spent in the real world.
Your sign up flow might be broken. I tried creating an account (with my own email), received the confirmation email, but couldn't get my account to be verified. I get "Email not confirmed" when I try to log in.
Also, the verification email was sent from accounts@godealwise.com, which is a bit confusing.
If you're trying to build an agent for a long-running job like that, you run into different problems: - Failures are magnified as a workflow has multiple upstream dependencies and most scraping jobs don't. - You have to account for different auth schemes (Oauth, password, magic link, etc) - You have to implement token refresh logic for when sessions expire, unless you want to manually login several times per day
We don't have most of these features yet, but it's where we plan to focus.
And finally, we've licensed Finic under Apache 2.0 whereas Browserless is only available under a commercial license.
I think this needs more elaboration on what the Finic wrapper is adding to stock Playwright that can't just be achieved through more effective use of stock Playwright.
Also, curious why your unstructured idea did not pan out?
Our approach is a bit different. With finic you just write the script. We handle the entire job deployment and scaling on our end.
1. Developer tooling should be open source by default 2. Open source doesn't meaningfully affect revenue/scaling because developers that would use your self-hosted version would build in-house anyway.
So to have some respect for the open source way on top of which you are building all this please stop advertising it as "open source infrastructure" in bold and sell it like a normal software product with "source available" on the footer.
If you do plan to go open source and actually follow its ethos, remove the funded by VC label and have self hosting front and center in the docs with the hosted bit somewhere in the footer.
Also, if my mental model is correct, the more browsing and mouse-movement telemetry those cloudflare/akamai/etc gizmos encounter, the more likely they are to think the browser is for real, versus encountering a "fresh" one is almost certainly red-alert. Not a panacea, for sure, but I'd guess every little bit helps