I have been GMing for a while but I been struggling with enemies. My usual group size is two but I have a five person group and about to get another one. When their just two people I don't have any troubles but when it comes to the five person group, I always throw too weak of encounters at them. Is there a equation for HP and damage, that I can use?
P.S. I don't use the bestiary because I like to be able to throw any thing at them at any time. I once threw a level 1 dragon at my players and it was so fun. In the same campaign I used goblins that was scaled for level 5.
Also keep in mind that one or two monsters will generally be much weaker than four or more monsters of equal total CR. Action economy is a huge thing, but typically doesn't improve that much over the first 10 levels or so. In short, I recommend you increase the number monsters before you increase the power of the monsters.
The only really noteworthy exceptions to this are IME highly optimized melee reach control builds specializing in free action combat maneuvers and AoOs. Such a build can actually match the action economy of maybe three or even four PCs, often by virtue of getting a lot of actions outside of its own turn. IME, these kinds of builds can combine exceptionally well with larger monsters (for reach and numbers), and can offer a different form of tactical challenge for your players without necessarily increasing the risks of unintended PC kills much, since damage is typically a low priority.
And of course, terrain features and traps can be great equalizers and make fights more interesting.