Most implementations of toasts-with-actions that I've seen don't have the problem OP described. I more often find myself manually dismissing them than wishing they'd have stuck around longer.