It’s a little bit Swiss Army knife and a little bit skynet:
https://github.com/ferrislucas/promptr
From the README: Promptr is a CLI tool for operating on your codebase using GPT. Promptr dynamically includes one or more files into your GPT prompts, and it can optionally parse and apply the changes that GPT suggests to your codebase. Several prompt templates are included for various purposes, and users can create their own templates.
Jumping onto a new technology with unknown risks and getting burned is shortsighted.
He didn't say anything about ignoring AI. Pretty reasonable if you ask me.
OrgA will be 'all in' with the convenience of Copilot; OrgB will flee due to legality/litigious concerns.
The relevant code is here https://github.com/ferrislucas/promptr/blob/3ae09d1cffbb6b93...
And here: https://github.com/ferrislucas/promptr/blob/3ae09d1cffbb6b93...
"Romani ite domum"
After catching you writing:
"Romanes eunt domus"
How much are you going to pay for example to send a 1kb, 2kb, 4kb, 8kb, 16kb source file for analysis? If you are paying per "token", a 3kb file I have with like 70 lines of code might have 2-3 tokens per line if I understand what a token is, 210 tokens? Each token costs...
Ok, not as bad as I thought.
Optionally send other specific code alongside it.
This should be similar in quality and dramatically cheaper. It's quite doable with langchain, and you can use Chroma + DuckDB to avoid having to pay for Pinecone.
You could even maintain there vector store using git diffs to only update chunks that have changed.
Using the "give it everything" method will cause it to forget most of what you're feeding it if you have a large repo anyway, right?
- implement a function, providing structural context (e.g. model, service, controller) with stubbed modules, but leave out utilities/libraries and database model - drop the structural context, and ask it to expand the stub implementation with some additional context about the database model - drop the database model context, and ask it to refactor the solution with some additional context about utilities/libraries
I think this is doable, if we can build up some utilities for context management and iterative development, GPT should start to be usable on large code bases. It could work similarly to how one person wrote an entire novel using GPT.
I've tried a few things:
1. Summarize/collapse the code.
- Use GPT to summarize code blocks into a 1 line comment.
- Collapse all the top level code blocks into their summary line. This turns the entire codebase into a much smaller "top level map".
- Use a ReAct pattern via langchain to have the LLM itself try and determine which collapsed blocks need to be recursively "expanded" to understand the code with respect to the user requested feature/bugfix/change/modification.
- Feed GPT this partially expanded code base along with the user requested change.
- Have it spit back the modified version of that partially expanded code.
- Apply the GPT changes back to the original source files.
3. Again using ReAct, give GPT tools to "grep" and "cat" the code so that it can explore the codebase to find and understand relevant chunks. I've even armed it with bash in a sandbox. It started importing and running parts of the python code I was operating over.
None of these approaches have panned out fully yet. But there are promising signs.
[1] This is the small webapp that GPT wrote for me. I'm working on it mainly as a forcing function to explore these sorts of "GPT as junior developer / coding collaborator" workflows.
https://github.com/paul-gauthier/easy-chat#created-by-chatgp...
It's more work, but maybe language specific tooling as a first pass? I'm wondering how far you'd get by feeding it all the type information first (from lets say rustdoc as a specific example), and then asking the LLM to understand the structure of the program.
Then taking that output (which you could cache) + any source file local context information + the users request for a change.
I have been thinking a lot about the time when I can start jamming with GPT inside my dev environment and code base and this is a step closer.
Use case 3, where you define tests and it tries to give you a passing implementation is the dream.
Given the utter absence of solid test coverage, a much more likely approach is massive coverage generation via ChatGPT test generation & human audits, and then a gradual rewite by humans assisted by ChatGPT.
It's a really good tool, it's nowhere good enough to do automated rewrites. (And as long as the results matter, you'll continue to have humans in the loop for more than 5 years - if for no other reason than cleanly assigning legal blame)
What I don't see is, unique complex applications with a great UI that don't overlap with the applications like office with copilot, or copilotX.
promptr gets rid of the copy paste. That's a much better developer ergonomic (IMO)
The code is a prompt, a foreach file execute prompt and return whatever, it can be used as a template build something useful and not redundant, but definitely not code. Don't reinvent the wheel.
It's not something I'd be using with any regularity (on the same project), but I can see where it'd be worth it, even at a dollars-per-project cost.
:'<,'>!promptr -m gpt4 -t refactor -p "Cleanup the code"If that's not enough for you, projects like Auto-GPT give GPT-4 full autonomy to figure out how to do complex multi-step tasks beyond modifying text all on its own with only a vague goal provided. https://github.com/Torantulino/Auto-GPT
I've been using chatblade, it has the nice feature of having a cost estimate call:
https://news.ycombinator.com/item?id=35223759
As to how to use GPT to help with coding, I wanted to be able to store the chatblade output to a file with a simple one-word call, so I wrote a wrapper that does that, and the first step was to submit this question to GPT:
> "I want to use Python's subprocess module in a script (that takes command-line arguments) to manage a call to an OpenAI API (that takes a variable amount of time to complete), and which prints output to the terminal, and I also want to tee that output to a specified file for storage. What do you recommend, from the perspective of an expert Python programmer?"
It laid out a nice template using several python modules, and I was able to write it in a single morning with a little bit of additional reference to pydocs and a few more questions about specifics, so now all the command-line queries get appended to a file with an up-to-date cost estimate attached to each one. And I'm pretty junior as far as programming goes.
I have also been exploring a similar pattern for using GPT as a coding collaborator:
- Send all the (relevant) code to GPT along with a change request
- Have it reply with all the code, modified to include the requested change
- Automatically replace the original files with the GPT edited versions
- Use git diff, etc to review and either accept/reject the changes.
I have tried quite a large number of alternative workflows. Outside the "all code in/out" pattern, GPT gets confused, makes mistakes, implements the requested change in different ways in different sections of the code, or just plain fails.
If you're asking for self contained modifications to a single function, that's all the code that needs to go in/out. On the other side of the spectrum, I had GPT build an entire small webapp using this pattern by repeatedly feeding it all the html/css/js along with a series of feature requests. Many feature requests required coordinated changes across html/css/js.
https://github.com/paul-gauthier/easy-chat#created-by-chatgp...
Another HN user has also released a command line tool along these lines called gish:
Having GPT's response in json was useful to be able to easily apply the changes GPT wants to the filesystem. GPT4 is significantly more consistent with only responding with json.
Gish looks cool!