It's true that GPU is itself not busy during a lot of this because it's waiting on pixels, but whatever is preparing the pixels (copying memory) is super busy.
Downscaling is a win not just for the blurring, but primarily the compositing. KDE describes the primary constraint as the number of windows and how many of them need to be blended:
The performance impact of the blur effect depends on the number of open and translucent windows
https://userbase.kde.org/Desktop_Effects_Performance#Blur_Ef...