Just wanted to share an interesting experiment I ran to see what kind of performance gains can be achieved by fine-tuning a model to code from a single repo.
Results: The fine-tuned model achieves a 47% improvement in the code completion task (tab autocomplete). Accuracy goes from 25% to 36% (exact match against ground truth) after a short training run of only 500 iterations on a single RTX 4090 GPU.
This is an interesting result because it shows that there are significant gains to be had by fine-tuning to your own code.
Full details of the experiment: https://prvn.sh/build-your-own-github-copilot/
Highlights of the experiment: - Model: qwen2.5-coder 14b, 4-bit quantized - Training data: Svelte source files from this repo: https://github.com/hcengineering/platform - Unsloth for LoRA training with rank 16, 4096 sequence length - GPU: single RTX 4090 - 500 iterations with effective batch size 8
A fine-tuned open-source model could be a real alternative to commercial coding assistants like GitHub Copilot, Cursor, etc., especially for organizations with a ton of legacy code that these LLMs have not seen, and also for those that are wary of exposing their code to external systems.
No comments yet.
