I think it
could catch this, but it could just as easily miss that there was a huge degridation on HDD vs SSD; high ram vs. low ram; fast vs slow CPU; GPU vs no GPU.
Obviously you can build test suites for each of these scenarios, but I think it would be expensive to run all of them. That's all I'm really saying, it's by no means a bad idea, I think it's a great idea, just going to require some upfront thought of what type of environment the software is going to run in.