Ultimately I just moved to Godot to get stuff done, which is not pretty C++ at all, but it's about as lightweight as you can get for a fully featured game engine, as opposed to a framework like SFML.
Mostly though, I didn't enjoy using SFML as much as SDL, even with no C experience. A C++ game project I contributed to switched from SFML to SDL, and while SFML's API is more batteries-included in places (particularly input and rendering), it's a tradeoff. If you want more control over how rendering works, or want more flexibility in input, SDL is better. You can get 80% of the way faster with SFML, but the last 20% you wind up fighting more with SFML than the problems you're trying to solve. SDL tries to do less and stays more out of the way in the process.
Multi-monitor support, a significant feature request for the project I worked on, was very difficult or just outright unsupported in SFML. Non-HID hardware interface support (DMX, MIDI) was a requirement and SFML got in the way more than it helped. SFML 2.4 in particular was very painful, with some significant input bugs or designed limitations that the switch to SDL made much easier to improve. Stuff like GL blend modes and render textures never really worked on SFML and Android devices, and kept breaking on macOS, all of which went away with SDL; touchscreen devices and SFML were hacky to work with, but they were a requirement for the project.
SFML didn't really have the resources or community to try to fix problems with platforms or devices the further you get from PC targets. API docs were often incomplete with terse examples, making it harder to learn how the APIs were designed to interact with each other unless you go spelunking in their forums. It was often easier to fork SFML and fix bugs our project was blocked on and build SFML ourselves than to try to use the libs as packaged or wait for patches to be accepted upstream.
SFML also has some steeper competition now — Godot is obvious, but Raylib scratches a similar itch.
