I always though it was the 3rd-party but today I removed the Bluetooth PAN from the Network settings and almost immediately everything was working as new.
PS: I'm reading on some (6 years old) articles that the Bluetooth PAN was causing issues like this. Can't believe how Apple sometimes can be this unreliable, I almost gave up on using my equipment thinking it was faulty.
-- For people having same issues, hardware references :
* Mac : Pro 15 (2018) / Air 2020 both on Catalina * Keyboard : Apple Magic Keyboard * Mouse : Logitech MX Master 2S * Speakers : Harman/Kardon Onyx Studio 4