I don't view hit points a physical damage rather pool of endurance active avoiding taking damage. When you do take damage you are in negative hit points. Exceed you con and you die. For some people taking any physical damage of significance means they are down and out. For others they can fight on some better than other depending on feats and racial abilities. So shooting a flying dragon isn't harming them yet but they lose altitude when they take damage they are spending hit points to avoid death by say flying in manner that could cause them to lose altitude.
I look at it like this because it doesn't make any sense that you can get stabbed once at first level but you can get stabbed a hundred time at level 20. It just makes more sense that you avoid the stab by spending hit points. Not mechanical change just way of looking at it.