The nature of Second Life makes optimization a challenge. The geometry is fully dynamic so you can't pre-compute line of sight or use a lot of other performance saving tricks. The people making things are not skilled with efficient 3D design, so even if there was a way to do things like remove hidden triangles from models most people would not use it. Textures get uploaded in ridiculous sizes for things that should be using low res sheeted textures.

And even if you manage to build yourself a nice efficient well designed island other people will show up with avatars wearing massively over-detailed jeweler with 4k textures on every surface and with a few buggy CPU-killing addons that fight each other to try and fix a few of Second Life's design problems, and suddenly your server is on it's knees begging for more CPU cycles while your grapics card is busy choking every time their avatar is near.

The short version is when people can build anything you'll get some amazing works but you'll also get a lot of poor quality, performance killing crap. I don't see VR changing that, but there is a place for a Second-Life life competitor that limits what people can do to hit a nice balance of performance vs. freedom.