You can use vitest like a standalone testing tool, it will install vite as dev-dependency but you don't need to build your whole project around vite.
In my case it was literally a drop-in-replacement for jest, done in two minutes and without a config file. A simple config file only was required a bit later when I added test- and coverage-reporting as junit.xml and cobertura format for Gitlab CI.
Didn't even have to change a single line of testing code (I didn't use Jest's mocking magic in that project though).