How is something consensual unfair? If a person places a low value on their time or a high value on the experience that will be gained, it seems like it might be a reasonable trade. This seems especially true if the person giving up their time isn't in dire straits... i.e. If you offer to pay pennies and breadcrumbs to a starving man, that might not be cool. If you offer a good unpiad experience to a hacker/designer who is otherwise darned employable, it doesn't seem evil at all.
If I lived in the Valley and had an opportunity to hang out at YC dinners in exchange for a bit of free work, there was probably a time in my life when I would've taken that opportunity.
Random other note: cool that you play the banjo! I've recently begun teaching myself the fiddle.