Why prematurely optimize for differences that may not (and in practice aren't really likely to) happen?
Keep in mind that the tradeoff with generics is usually not 3 lines of clear or 1 line of hard to read, it's one line of clear code (T -> T), one line of unclear code (Interface{} -> Interface{} + casts), or n lines of complex code (concretely reimplementing the function for your particular case).