Well, either they'd have to contain the less-dense air in some way, or present a large surface-area inversely proportional to the amount they heat the air above them.
-Said surface needs to be heat/flame resistant, and lightwieght, for obvious reasons.

Would be kinda nifty to see a bunsen-burner powered flying index-card...

...

I totally am not building one right now.

*Soaks an index card [paper airplane] in lighter fluid, lights it on fire, throws it.*
The fluid keeps the paper from burning to ash for about 15 seconds.

Best example of this: if you ever do a paper/rubbish fire, the paper ash has a tendancy to float... propotional to how much it is on-fire.

Alternately, they could take just about any stovepipe and turn it into a jet-engine.