From my (albeit limited understanding) the damage from alcohol primarily lies in the 'backup' metabolic pathways getting used when the liver is saturated and cannot keep up with the primary pathway. The backup metabolic pathways produce acutely toxic byproducts - acetaldehyde most significantly - which are significantly more harmful than ethanol itself.

Some people may generate significant amounts of acetaldehyde from a single drink, but I don't think that's the case for the majority of healthy people.