hansonkin

I’m playing around with using LLMs for voice-controlled app actions and trying to learn what approaches actually work well in practice.

I became curious after seeing Todoist’s voice ramble feature. I wanted to try building a simple version of it:

- creating tasks

- creating subtasks under a task

- marking tasks or subtasks as done

- moving or deleting items

For example, a user might say: “I need to go to the grocery store. While I’m there, buy milk, cheese, and bread.”

Ideally this would result in a json list of ordered actions:

- create task “Grocery Store”

- create subtasks “Buy milk”, “Buy cheese”, “Buy bread” under that task

The speech-to-text part I have covered but I'm having trouble with the action extraction and output part. I am using gpt-5-mini with structured output in text format mode (not tool calling mode). The schema is discriminated, meaning each action type (create_task, move_task, mark_task_done, etc.) has its own shape and required parameters and the output must conform exactly.

I’m running into action selection reliability issues. Sometimes the model chooses the wrong action entirely, for example emitting a move action when the transcript clearly implies creation. And for the earlier example, sometimes I get subtask creation actions but the initial grocery store task creation is missing.

Chatgpt suggests adding more and more ad-hoc rules to the system prompt for guidance but it feels so fragile and doesn't actually improve the results during testing.

Curious how people approach a problem like this. Also how does the setup scale to when there are 20+ possible system actions? What are the system parts that meaningfully impact robustness?

- Crafting a better system prompt?

- Having two passes where the first pass extracts the intended actions in some intermediate format and then the next pass to output the structured format?

- Using tool calling mode?

It's the new year and many of us have resolutions to start new hobbies or focus more on existing ones. Having someone you can meet with to do or talk about a shared activity helps us stay motivated and more importantly, makes it more fun. Finding an accountability buddy can be tricky because a good match involves several factors. Ideally, you want someone nearby so it's convenient to meet up, someone at a similar skill and commitment level and someone who shares your interests/styles/inspirations within the hobby.

In my case, I would consider myself an intermediate guitar player and have always wanted to find a practice partner. Living in NYC, there is no shortage of musicians, but many have more serious goals and aspirations than I do. As a hobbyist, my main focus isn't playing gigs or being in a band. I’m not trying to practice for hours every day to become an expert. I just want to find someone to play some songs with casually for fun. I’m sure this person exists in my neighborhood, but I had no easy way to find them.

There are apps for specific activities like music collaborations, tennis, or basketball, but I found they often cater to a more serious crowd. Beginners and casual dabblers are usually left on the sideline. That’s why I built Kindred, a way for anyone to connect with others through hobbies.

Here’s a quick overview of how it works:

- Set preferences for age, gender, and location to help you find compatible buddies nearby. - Create a profile focused on your experience level and specific interests within your chosen hobbies. Unlike typical social media, superficial details like looks, education, or career aren’t emphasized. - Meet people through plans. Plans are like traditional events but can have more flexible time and date. You can choose "sometime this week" or even "whenever." Plans with small groups of 4-5 people are encouraged but you can also choose one-on-ones or larger groups. It should feel like you're just making a post about what you want to do to neighbors who fit your preferences instead of it feeling like you're formally hosting a public event.

Check it out here: https://kindredsocialapp.com. The app is currently available for iOS only.

I’d love to hear your feedback and answer any questions!