Heavy editing can drive me crazy. The really bad ones are the podcasts/videos that jump in between each sentence. It's really disconcerting to hear somebody talk with no pauses in between sentences. I like the format of Joe's podcast because it feels like a real conversation and I'm participating in it, even if I'm not. Editing would ruin that feeling and make it something else. There's plenty of other podcasts that service that.
Joe's issues are that can be long winded at times, and he fails to press people on some issues which occasionally sucks (the first episode with Jack Dorsey, the last episode with Alex Jones, or the recent episode with Bob Lazaar). Most of the time, however, that's exactly what I want. I don't want two blowhards shouting at each other, I want two people trying to understand each other's ideas and to learn something new from it.