That's a fair point. Still seems off to me in terms of sensitivity, and once again putting aside the discussion of what the goal of districting should be.
Simplifying the algebra in the calculation, though, the efficiency gap is actually independent of the inefficiencies of any particular district, but dependent only on the population of the districts -- the numerator simplifies to
D - R + sum_i { I(D_i < R_i) (D_i + R_i) / 2 }
Where D and R are the total Democratic/Republican votes in the state (across all districts) and the I(x) is an indicator function, +1 if true, -1 if false.
If you hold the size of districts to be about even, then this just says that the ratio of the total population of districts that vote R to the total population of districts that vote D should be approximately equal to the ratio of the total state population of R voters to the total state population of D voters.
That seems reasonable, just seems like a roundabout way to get there, and is sensitive to unequal population distributions.