1. What feature extractor is used to derive code embeddings?
2. Would support for more complex queries be useful inside the app?
--- Retrieve a subset of code snippets
SELECT name
FROM snippets
WHERE file_name LIKE "%py" AND author_name LIKE "John%"
ORDER BY
Similarity(
CodeFeatureExtractor(Open(query)),
CodeFeatureExtractor(data)
)
LIMIT 5;support for more complex queries could be useful, but probably not using a query language since that would make it more difficult to use free-form text input.
You can already use it using an API: https://kantord.github.io/SeaGOAT/0.27.x/server/#understandi... so probably the best way to add support for more complex queries would be to have additional query parameters, and also to expose those flags/options/features through the CLI
[0] https://docs.trychroma.com/embeddings#default-all-minilm-l6-...
so that could provide a nicer interactive experience for more complex queries
I had this problem trying to learn a library and figuring out what all the functionalities are. I ended up making a non-ai solution (an emacs pkg), but this seems just a step or two away from your current project imho.
There is still a lot of Perl code around. Something like this would be super useful.
It is to avoid wasting time processing files that cannot lead to good results. If you want to try it for a different programming language, please fork the repo and try adding your file formats and test if it gives meaningful results, and if it does please submit a pull request.
Other than that one limitation is that it uses a model under the hood that is trained on a specific dataset which is filtered for a specific list of programming languages. So without changing the model as well, the support for other languages could be subpar. At the moment the model is all-MiniLM-L6-v2, here's a detailed summary of the dataset: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v...
Can the generated database be easily shared within the team so not everyone has to run the initial processing of the repo which seems that it will take a couple of hours on my laptop?
Edit: processing chunks; see you tomorrow...
But yours has a more permissive license!
I also had to modify it a bit to allow for the line endings I needed and it frustratingly doesn't allow specifying a path, and often returns tests instead of code
Thank you for sharing.
will it generate code comments for indexing using a language model? will that be expensive (assuming using GPT3)?
What would it take to support other programming languages?
I've done the same but was very disappointed with the stock sentence embedding results. You can get any arbitrary embedding, but then the cosine similarity used for nearest neighbor lookup gives a lot of false pos/negs.
*There are 2 reasons:*
1. All embeddings from these models occupy a narrow cone of the total embedding space. Check out the cos sim of any 2 arbitrary strings. It'll be incredibly high! Even for gibberish and sensical sentences.
2. The dataset these SentenceTransformers are trained on don't include much code, and certainly not intentionally. At least I haven't found a code focused one yet.
*There are solutions I've tried with mixed results:*
1. embedding "whitening" forces all the embeddings to be nearly orthogonal, meaning decorrelated. If you truncate the whitened embeddings, and keep just the top n eigenvalues, you get a sort of semantic compression that improves results.
2. train a super light neural net on your codebase's embeddings (takes seconds to train with a few layers) to improve nearest neighbor results. I suspect this helps because it rebiases learning to distinguish just among your codebase's embeddings.
*There are solutions from the literature I am working on next that I find conceptually more promising:*
1. Chunk the codebase, and ask an LLM on each chunk to "generate a question to which this code is the answer". Then do natural language lookup on the question, and return the code for it.
2. You have your code lookup query. Ask an LLM to "generate a fabricated answer to this question". Then embed it's answer, and use that to do your lookup.
3. We use the AST of the code to further inform embeddings.
I have this in my project UniteAI [1] and would love if you cared to collab on improving it (either directly, or via your repo and then building a dependency to it into UniteAI). I'm actually trying to collab more, so, this offer goes to anyone! I think for the future of AI to be owned by us, we do that through these local-first projects and building strong communities.