At higher temperature, a given mass of gas occupies more volume as per Charles’ law and is thus less dense. Cold air being denser than hot air, raises the balloon to a lesser height.