Durable Objects for agent state is the correct move for low-latency memory. I've been building these exact patterns as a TypeScript first class memory primitive for AI agents:
https://github.com/ereztdev/langmem-ts
it handles the extraction/recall loop without the Python-heavy overhead of LangMem or Mem0. Pure TS, no wrappers