Launch HN: Trellis (YC W24) – AI-powered workflows for unstructured data

Make sure you have an on-premise option for this type of customer. I've worked at two software companies in Europe with tangentially similar products related to document analysis. On premise is a key requirement.

Even though it's 2024, banks, financial institutions like insurance companies etc. tend to be _very_ cautious with valuable documents involving customers. There are also regional regulations that prevent things like patient data being shared with _any_ 3rd parties. Even one of the big 4 oil companies that I've dealt with as prospective customer - very strict rules requiring on premise solutions.

The good news is many are using things like Kubernetes and OpenShift internally, so it should be possible to port what you do on AWS to on-premise.

I have just been working through the same problem (though just PDFs). Google DocAI helped enormously after a bit of initial input.

Who is liable when the ML model hallucinates™ while parsing some critical data?

Better still if it can then become a source of truth for further departures from reality.

Great to hear that you worked saw similar use cases. Doing this before LLMs seem like a big challenge.

We face similar challenges you listed and handle all of the above. 1. Out of the box OCR doesn't perform as well for complex documents (with tables, images, etc.). We use vision model to help process that documents. 2. Recall (for longer documents) and accuracy are also a major problem. We built in validation systems and references to help users validate the results. 3. Maintain this systems in production, integrate with the data sources and refresh when new data comes in are quite annoying. We manage that for the end users. 4. For non-technical users, we allow them to iterate through different business logic and have a one unify place to manage data workflows.

Would recommend using the updated guide here! That link is from the v0.1 docs. https://python.langchain.com/v0.2/docs/how_to/structured_out...

OOC which openai model were you using? Would recommend trying 4o as well as Anthropic claude 3.5 sonnet if ya haven't played around with those yet

FWIW I've seen noticeably better results on (1) and (4) extracting JSON from images via Claude, although (2) and (3) still take effort.

> 80% of enterprise data is unstructured

I've seen quotes like this many times. It's silly. I worked at a big bank for over a decade. 95% of the data we cared about was already in a SQL database. Maybe ~80% of our data was "unstructured", but it wasn't stuff we cared about for risk management or other critical functions.

> people are not willing to pay more money for less risk

I'd disagree here. Banks are willing to pay money to reduce risk, it's just unlikely to come from scraping data out of PDFs with an LLM because they've already done this if it's worth it.

Thanks for the feedback. We built Trellis based on our experience with ingesting and analyzing unstructured customer calls and chats in a reliable way. We couldn’t find a good solution apart from developing a dedicated ML pipeline, which is quite difficult to maintain.

There are some elements that might resemble Dagster, but I believe the challenging part is constructing validation systems that ensure high accuracy and correct schemas while processing all kinds of complex PDFs and document edge cases. Over the past few weeks, our engineering team has spent a lot of time developing a vision model robust enough to extract nested tables from documents

Nope! From yesterday:

Launch HN: Synnax (YC S24) – Unified hardware control and sensor data streaming - https://news.ycombinator.com/item?id=41227369 - Aug 2024 (23 comments)

also recent:

Launch HN: Stack Auth (YC S24) – An Open-Source Auth0/Clerk Alternative - https://news.ycombinator.com/item?id=41194673 - Aug 2024 (140 comments)

Launch HN: Firezone (YC W22) – Zero-trust access platform built on WireGuard - https://news.ycombinator.com/item?id=41173330 - Aug 2024 (88 comments)

Launch HN: Airhart Aeronautics (YC S22) – A modern personal airplane - https://news.ycombinator.com/item?id=41163382 - Aug 2024 (618 comments)

That's 4 of the 8 most recent Launch HNs btw. But it's true that there are reams of AI startups nowadays.

The problem for me is that all of them look more of the same. I have a feeling of dejavu every time I see a Show HN of an AI generator, AI nocode, AI supabase, AI PDF scanner, AI monitoring startup.

I'm developing an "AI wrapper" myself and I know how difficult it is to create a reliable system using LLM integration and I guess these many similar projects are competing on being the one to create something that won't risk ruining their customers reputation. But I see no differentiation, no eye-catching tech, algorithm, invention.

YC and HN used to be the bastion of innovation in tech.

This year, nearly yes in "some way":

> This year, we’ll fund more than 500 companies out of 50,000 applications, and almost all of them are related to AI in some way.

Source: https://www.ycombinator.com/blog/why-yc-went-to-dc/

(Edited to be more precise.)

Thanks! There are still a lot of amazing hardware companies and vertical applications in our YC batch.

We believe that AI is only one part of our product. A significant amount of value comes from building robust integrations with different data sources and managing the business logic that operates on top of this unstructured data.

Thanks for sharing! It handled the emails very well.

That is very cool, thank you for sharing.

Appreciate the note.

1. I completely agree. Last-mile accuracy is crucial for enterprise buyers, and the challenge isn't just the AI. It's about mapping their business logic and workflows to the product in a way that demonstrates fast time to value.

2. Thanks for the feedback. We're still refining the messaging and don't want to be overly focused on just the extraction aspect. Do you think positioning it as ETL for unstructured data or high-accuracy extraction for enterprises might work better?"

We've been building something similar with https://vlm.run/: we're starting out with documents, but feel like the real killer app will involve agentic workflows grounded in visual inputs like websites. The challenge is that even the best foundation models still struggle a lot with hallucination and rate limits, which means that you have to chain together both OCR and LLMs to get a good result. Platforms like Tesseract work fine for simple, dense documents, but don't help with more complex visual media like charts and graphs. LLMs are great, but even the release of JSON schemas by OpenAI hasn't really fixed 'making things up' or 'giving up halfway through'.

Good question and we have seen this extraction workflow a lot in financial services. We just added table mode to the product (select table in transformation parameters) where we extract table structure in the documents that match that schema. So 1 file map to N rows where N is all the row in the table.

Just did an extraction and table mode targets this rly well :)

Is that the chat thing that pops up in the bottom right corner? It is the most annoying thing in the world. Because it pops up uninvited, and obscures the page content I am trying to read. So annoying.

Thanks! We got quite a few good enterprise leads from Intercom chats.

Maintaining the right level of accuracy across different domain is quite hard and something we spend a lot of time on. The accuracy bar tends to be quite high for financial services so we're adding some validation steps and checking to make sure any errors get caught beforehand.

Not OP but based on their writeup it sounds like you do need to provide at least a target schema, so what data you need or expect to extract from the unstructured input.

I assume that in the validation step if you don't get all those data points, then that routes to an error state for further review or something.

The users specify the schema and output format and a validation rule and we make sure the system adheres to that.

I would say the opposite. We want to make sure that we build our systems in a way that it get better as foundational model becomes better.

Our thesis is that foundational models will become good and affordable enough to be used in almost all data processing pipelines. We build systems on top of that to manage workflows, integrations, and data applications that people may want to develop.

That is only part of the problem; the others include:

1. writing connectors for various sources

2. writing connectors for destination

3. supporting multiple models, embeddings, vector database, text extractors

3. workflow automation engine(cron jobs)

4. performance tuning for speed and costs

5. security and compliance

We use a combination of fine-tune LLMs models that're specialized at extraction, data validation and parsing and large foundational models for more general reasoning tasks.

Model routing architecture has been quite interesting to explore.

NER is good for really simple things (like getting names, addresses, etc.).

A lot of the use cases that we see, like extracting data from nested tables in 100-page-long private credit documents or flagging transactions and emails that contain a specific compliance violation, are impossible to do with NER.

NER is good for really simple things (like getting names, addresses, etc.).

A lot of the use cases that we see, like extracting data from nested tables in 100-page-long private credit documents or flagging transactions and emails that contain a specific compliance violation, are impossible to do with NER.

With Trellis, the idea is taht you can write any mappings and transformations (no matter how complex the tasks or the source data are).

Good question—NER and entity normalization work well for documents that have been standardized (e.g., IRS 1040a tax forms). However, the moment something slightly changes about the form, such as the structure of the table, the accuracy of NER drops dramatically.

This is why logistics companies, financial services, and insurance firms have in-house teams to process these documents (e.g., insurance claims adjusters) or outsource them to BPOs. These documents can vary significantly from one to another.

With LLMs fine-tuned on your data, the accuracy is much higher, more robust, and more generalizable. We have a human in the loop for flagged results to ensure we maintain the highest accuracy standards in critical settings like financial services.

...which also autocompletes if the domain does not start with the company name :-)

Exact accuracy depends on the domain and tasks. Processing emails will naturally have higher accuracy than 150+ pages of private credit documents. Generally, we see 95%+ accuracy out of the box and can go up to 99%+ with fine-tuning, and human in the loop validation.

+1 to this

This was the top comment for quite a while but suddenly dropped to the bottom. Was it automatically downranked for mentioning an OS alternative?

How many upvotes does your comment have?

In many use cases, like flagging documents for compliance issues or processing customer emails, it's challenging to manage this at the vendor level because end customers want the ability to apply business logic and run different analyses.

For data ingestion and mapping, I agree that in an ideal world, we would all have first-party API integrations. However, many industries still rely on PDFs and CSV files to transfer data.

Super cool! This aligns with our experiences. These tools are great and can get to near 100% of accuracy but it's quite a lot of work on the Eng side to get it there reliably.

Thanks! Always surprised to see how many dodgy PDFs and scans there is in enterprises.

Jason and Richard from Roe AI are amazing people! We were in the same YC batch and section. Excited for what Roe AI is building and their focus on building a new type of data warehouse.

At Trellis, we're focused on building the AI tool that supports document-heavy workflows (this includes building the dashboard for teams to review, update, and approved results that were flagged, reading and writing directly to your system of record like Salesforce, and allowing customers to create their own validations around the documents).

Glad to see people experiencing similar problems! We previously spend way too much time building and maintaining document processing pipeline that doesn't really scale.

Thanks! We also start to see the patterns where search systems are being improved with filters and hierarchy level metadata. Another use case that people use Trellis for is ingesting data into their downstream LLMs applications.

Yes, HIPAA compliance is on the roadmap and should be out in a few weeks. We spent a lot of time on healthcare/sensitive data use cases.

Google Document AI and Watson SDU seem to be an afterthought for IBM/Google. The accuracy and configurability often fall short when you want to use them in a production setting.

Comparing to other legacy document processing companies, I think there are a few areas where we differentiate:

1. We handle end-to-end workflows from integrating with data sources, defining the transformation, and automatically triggering new runs when there’s an update to the data. 2. We built our entire stack on LLM and Vision transformers and use OCR/parser to check the results. This allows the mapping and tasks to be a lot more robust and flexible. 3. We have validations, reference checking, and confidence score metrics that enable fast human-in-the-loop iteration.

Please let me know the issues and happy to get it set up correctly for you. I'm at mac@runtrellis.com

For them and all other 50 AI PDF scanning wrappers that were featured on Show HN in the past month.

Yes, we combine LLMs as a judge with traditional checks like reverse search in original data sources, defining your own post-processing logic, and simple classifier for confidence score.

Unstructured seems to be focusing a lot on the document chunking and data ingestion into RAGs part. Trellis handles the process end-to-end from extraction to transforming the data into the schema that you need for downstream applications.

The way unstructured built their parsing and extraction are mostly based on traditional OCR and rule based extraction. We built all preprocessing pipeline in an LLM and vision model first way that allows us to be flexible when the data is quite complex (like tables and images within documents).

That’s awesome. Curious to hear your use cases and also happy to share notes on what we see work well. Feel free to ping me (email in Bio)