I guess it depends on how much of a success you consider Laserdisc to be. It wasn't a blowout success like DVD, but it was a healthy niche.
I got the strong impression that DVDs were sell-through, a market I think Disney pioneered. So maybe video rental wasn't so important for DVDs, but I don't know.
You are of course right that recording was a killer app for VHS, lots of time shifting ... although, hasn't VHS died without most people replacing it with recording to hard disks? I'm not sure TiVo got big enough, but I don't know.
As brudgers points out in his reply in this subthread, recording got big enough to worry the content holders such that they imposed DRM on DVDs and Blu-ray, but it's not clear their fears are realistic. Certainly the cracking of DVD DRM didn't stop production of them.