Preact diffs against the real dom and is faster than Svelte in benchmarks while the library is only around 3.5kb (the minified code fits easily on a single screen). I'd note that authors from most of the frameworks represented have submitted their own optimizations, so performance isn't strictly based on the author's familiarity.
InfernoJS is massively faster than Svelte or Preact (the benchmark author had to do some rather convoluted re-writes of vanillaJS over the years to keep ahead) and it uses a virtual DOM, but has a few optimizations that React doesn't have or can't use due to its external API.
stdweb is the real thing to keep your eye on as it seems to show that WASM can be every fast. They still haven't gotten the vanilla benchmark up to the same speed.
https://stefankrause.net/js-frameworks-benchmark8/table.html
React's Fibers do some cool batching behind the scenes which means that large updates (especially constant ones like animations) don't have large negative impacts on user interaction. I doubt this would be possible without a vdom or something very similar to track all changes before deciding batches and update priority.
Also, remember that vdoms have gotten a lot more efficient too. Newer ones can do things like re-use existing sub-trees or recycle vdom objects (they can also recycle the associated real DOM nodes too). Preventing all that extra garbage goes a huge way toward keeping those vdoms efficient.
As to svelte in particular, you aren't "just using an equals". It actually compiles behind the scenes into a completely different set of reactive code. This gets at my biggest issue with such frameworks. Their abstractions aren't free. I have to learn their proprietary syntax and markup. I then have to learn how their compiler works behind the scenes when its time to debug the code I wrote. When I compile a React-style component into ES7 and run it in the brower, I have to deal with webpack for imports and I have to understand that JSX is really just JS functions. Otherwise, what I wrote is what I'll see.
https://cdnjs.cloudflare.com/ajax/libs/preact/8.5.2/preact.m...
