No comments yet.
Happy Monday HN,
I've long been a believer in automatic tests and static analysis to help LLM agents give us better output. pytest combined with ruff is a baseline for any AGENT.md file I'm working on these days.
But neither helps in situations where agents write code that's technically valid but just doesn't meet my standards. Overly vague typehinting, importing a library I know I don't want to use, using dataclasses instead of pydantic, or straight up cheating and using a try/except to catch cases within test files that aren't passing but should be.
Thankfully LLMs are able to generate AST parsing code pretty accurately. Could we just use LLMs to create deterministic guardrails for themselves in the future? Here's basically how determystic works:
``` uvx determystic new-validator
1. Enter the bad Python code that your Agent generated: var: Optional[int] = None
2. Describe the issues Don't use Optional - use A | None
3. Name your validator Modern typehinting ```
After this you can run `uvx determystic validate` and get a full validation of your project based on your new rule:
``` Custom Validator main.py:6: Use 'T | None' instead of 'Optional[T]' for type hints 4 | 5 | class MyModel(BaseModel): >>> 6 | name: Optional[str] = None 7 | age: int 8 | ```
As LLMs get better and we put more compute behind RL environments, I think a lot of this will be solved at the model layer. But even then the system prompt makes us rely on natural language to describe what is otherwise a more precise problem.
I'm still in the very early days on this approach. But I'd love some feedback about ergonomics and the unusual coding conventions I'm sure people are going to throw at it. Raise an issue if you have examples of things that aren't working and I'll try to get them fixed. Thanks!
