You can type `cd ~` and press CTRL+r to immediately fuzzy match commands you've run with `cd ~`. fzf naturally ranks paths to cd into on top. If you find that too noisy you can just hit CTRL+r with an empty prompt and then search for `^cd ~` to only find cd commands.
I've written about filtering related history with zsh here: https://nickjanetakis.com/blog/hooking-up-fzf-with-zsh-tab-c...
If you want to go into ultra lazy mode you can also type `cd ` and spam the up / down arrows to only show commands from your history where you cd'd into a directory. That use case is also covered in the above post. I normally don't use this for changing directories but it can be done.