I think 5 videos per channel is an appropriate limit. I understand what you want to achieve with this. But suppose I want to watch an older video from the same channel. A simple 'Expand' button would do the trick. Otherwise I'd have to go to YouTube which I wanted to avoid in the first place.
And I think you don't have to stick to a certain number of videos as the only type of limit. How about showing as many videos as would fit within the width of the window?