I am looking for a service or effective technique using which I can get quick and reliable translation of our Node SDK (https://github.com/supertokens/supertokens-node) applied to our Python (https://github.com/supertokens/supertokens-python) and Golang (https://github.com/supertokens/supertokens-golang) SDK.
Our SDKs are not just wrappers around an OpenAPI spec, so we can’t use existing tools to auto generate our backend SDK. Or even if we did, that would only generate a very tiny percentage of our SDK. Other than being API wrappers, our SDKs:
- Provide several overridable functions for users to hook into.
- Manage reading from the request and writing to the response objects of various web frameworks of these languages.
- Expose APIs to via a middleware (that again integrates into various web frameworks). Each API has a lot of logic that includes input checking, business logic of that API, calling various other APIs to do actions like send emails, and finally writing an output JSON.
Currently, we are hand writing each of these SDKs, and as you may imagine, it is very expensive. As a result, we have the Node SDK (which is our most used one) far ahead in terms of features compared to our other two SDKs.
As an example of the set of changes that need to be replicated in the other SDKs, have a look at this PR: https://github.com/supertokens/supertokens-node/pull/670/files (many of the files are build files which can be ignored, but even then, it’s 200+ files changed, a large chunk of which are adding tests).
What I have already tried
I have primarily played around with GPT-4 with different types of prompts for simpler PR changes (for example this PR: https://github.com/supertokens/supertokens-node/pull/782/files).
For the TS code changes, I tried a prompt that gave the raw git diff of the node PR and the contents of the python function in which the changes are to be applied, and then asked to generate python snippet with the changes, and the output was quite accurate.
For test case changes, I gave the existing python test file and added it to ask new test cases based on the diff, and that went quite well too (though not at all perfect - i had a bunch of false imports, or made up function names, but 90% of it was good).
This, of course, is a very simple PR, and it would have taken me lesser time to do it by hand than try with the prompts, but I wonder if there is a way to scale this to make it work (even if it’s 50-60% correct).
My ask
If anyone has experience with working with LLMs for this or a similar purpose, or if there is a service out there which can help me with this, I would love to be connected. You can email me on rishabh@supertokens.com.
Thank you!
