Anyone working on LLM tools for enhancing data quality?

5 pointscstanley2y ago7 comments

Big problem, let's break it down...

1. Data issue identification 2. Solution and implementation

Most issues are discovered in the data warehouse. Entity matching customer data across different systems, some business process results in duplicate, or null data. I know there are existing, non-LLM, products that do this. I'm curious to compare those with new LLM first products.

On solution/implementation. Ideally you're able to fix this in the source system, either in your SaaS tool or in the way you write production data. You can also fix this in the data warehouse, munging/ETL'ing the data. Seems like LLMs could help to 1) identify and recommend a change in an external system, 2) submit a PR to solve this in the data warehouse.

Anyone know anyone working on these problems?

7 comments

hxypqr2y ago

Obviously, the cost of using LLM to solve this problem is relatively high. At most, it can only treat LLM as one of the signals instead of processing all the information. Combining LLM with tree models or graph models and adding NER technology may achieve an approximate solution to 2. Can you really trust the answers provided by LLM to be used in a production environment?

johannesboyne2y ago

The combination is truly key, how we do it:

- Combining LLMs, gradient boosting, and multiple statistical/heuristic based approaches - For LLMs: rely on (quite) large context length + fine-tuning - The (ML) models are used to extract mathematically provable quality checks

latentpot2y ago

1 is already a solved problem. My employer had originally put a ML based system to find DQ issues (already working) and is looking at pocs to add LLMs in the model mix. Hearsay is that our lakehouse vendor will have their own solution to this question via a acquisition.

2 is interesting, possible to do via LLM but I worry about data privacy and hallucinations making data more believable but not real.

johannesboyne2y ago

reg. 1) Which one has been used?

spdustin2y ago

I'm using LLMs to help build dbt-flavored Great Expectations tests. Pretty low-hanging fruit, but it's a start.

johannesboyne2y ago

We (https://dqc.ai) are doing something in the space, yes. Next to a mixture of ML, and heuristic based approaches, plus link & integrations into source systems. Happy to talk about it, feel free to reach out.

j / k navigate · click thread line to collapse

Anyone working on LLM tools for enhancing data quality?

5 pointscstanley2y ago7 comments

Big problem, let's break it down...

1. Data issue identification 2. Solution and implementation

Anyone know anyone working on these problems?

7 comments

hxypqr2y ago

johannesboyne2y ago

The combination is truly key, how we do it:

latentpot2y ago

2 is interesting, possible to do via LLM but I worry about data privacy and hallucinations making data more believable but not real.

johannesboyne2y ago

reg. 1) Which one has been used?

spdustin2y ago

I'm using LLMs to help build dbt-flavored Great Expectations tests. Pretty low-hanging fruit, but it's a start.

johannesboyne2y ago

j / k navigate · click thread line to collapse