It's the same problem as those portable AC units: the exhaust (chimney in this case) draws large amounts of air in from outside which is at the wrong temperature (cold in this case).