You're right, was thinking quart.
On second thought, I would say that a character who is doused with oil then lit up should take 2d6 per round (that's like being super-on-fire), plus the 1d6 per round if the floor under them is on fire too.
If you aren't covered in oil but the floor under you is, then yea, standard "extreme heat" applies. 1d6 per round, with a save each round to avoid catching fire (which would be another 1d6/round).
So, assuming they got off the first part of their plan and the monsters were burning, would they have taken some of this damage if the monsters climbed out and started grappling them, horror-movie style? I'm figuring the success of the plan would hinge on how long it would take to execute it. If they lost the element of surprise(weren't sneaky enough, were expected, or just got flamed by the trapped trapdoor), could one player alone execute the plan if he won initiative? Is one round long enough?