This is not remotely true.
First of all, life isn't something that I must squeeze all "productivity" out of possible, humans need rest and downtime. They also need socialization even though that's not "productive."
Secondly, podcasts help one accomplish productive work as they turn boring mundane (yet productive) tasks like cooking, cleaning, working out, gardening, folding laundry, woodworking, etc. into slight entertainment. For example, I feel much more motivated to clean the house when I have a podcasts I want to listen to vs before I started listening to podcasts (I don't really enjoy listening to music much). Almost nobody sits on the couch idly listening to podcasts.