I have a party of seven level 3 PCs, many of whom are inexperienced in the ways of D20 combat.
The problem is, that the ones who DO know what they're doing are power gaming, and anything I throw at them goes down within 1-2 rounds.
I worry that I won't be able to build an encounter that can last more than 3 rounds, without risking someone being 1 shotted.
So far, Not a single one of them has been hit in combat, and even after I threw a Babau at them, with 2 party members missing, they still wrecked it.
I'm having a great deal of trouble with this, any advice would be appreciated.
About the Babau: single enemy's often fail.
About the power gaming, are they really power gaming or are they just better in the game? If they have minmax builds that make them shine above the rest you could talk to them, advicing on nerfing the pc for table enjoyment. If they took suboptimal builds and managed to nail it I don't see much wrong with it, as long as they involve the other pc's in the game.
So, APL is 4, for standard CR calculations, and they annihilated an APL+2 encounter without getting hit, with only 5 of the party hitting it (and therefore presumably managing to deliver over 123 damage, since at that level a babau's DR should be a factor), and it couldn't hit them?
I'm mildly (not a lot) concerned about the power-gamers. Might be worth a character sheet audit. Also, rolls or point buy for stats? It helps us help you to know roughly what their stats look like.
As for encounter design help: multiple creatures. Preferably more than the number of PCs. I suspect you could give this party a problem with, say, a dozen orcs. Though since they can manage APL+2, you could go up to 18 if you want to put pressure on.
However, please remember that encounter design favours the PCs, anyway.
Also, multiple encounters per adventuring day. Don't let them rest between every fight. The opportunity to rest should occur every 4-5 encounters.
More monsters. Even if they are weak its still better to have more even numbers. An "even fight" would be 7 CR 3's, which could kill lots of your party. But you could have your babau with say 10 lemures and some quasits. Those little guys provide flanking, cover, and can do Combat Maneuvers like grapple and bull rush or disrupt spell casters. Red shirt monsters (goblins, orcs, elves, kobolds and any humanoids with 1 npc/pc levels, vegepygmies, mites, etc.) are good for letting players accomplish a lot of kills without earning much XP, but can still be a big threat when neglected. They use up resources, whihc will make later fights harder. If the party leaves tracks, send enemies tracking them to attack once while resting on occasion.
The battles are going to favor the PCs in general. The "hard" battles often occur with swingy dice, e.g., multiple crits to a PC, or with unforeseen complications in environment (area under a silence spell, wet floors, on a cliff, etc.).
Obviously more mooks and monsters means longer battles, particularly with a newbish group. Unfortunately, you have a large party which comes with the territory. Also, evil outsiders can summon other evil outsiders - use that to your advantage.
'Tucker's Kobolds',
This. This. 1,000 times this.
Also, equip your humanoids with alchemical items (tanglefoot bag, thunderstone, acid, alchemist fire, oil bottles lit of fire, bottled lightning, pellet grenades) and nets. NEts, even with a -4 penalty without exotic proficiency, only need a touch attack to entangle, and they are cheap.
Have a few Sorcerer snipers with light crossbows. Give them True Strike, and have them cast it every other round. On the rounds they attack, have them do a called shot. A called shot to the eye has a -10 to hit, but with a +20 to hit from true strike, a human sorcerer 1 with a +1 dex bonus has a +11 to hit. In the eye.
Shots to the throat can disrupt spell casting. Deafening a target with a called shot to the ear or with thunderstones disrupts spell casting.
First, there are seven PCs. Your action economy is really messed up. APL is more like 5. Try splitting the party as often as possible, but not by much. Divide and conquer. Optimally, split them into two nonfunctional parties. Traps, magic, and just tactics can make this happen.
Next assess their weaknesses. Apply foil liberally.
Are they fighters? Improved invisibility + Dominate Person. Now you've reduced their number by 1. Then create a pit and drop a few of the players in with the buffed out fighter. They are a high enough level party that a 7th level spellcaster isn't too far from the realm of possibility for a challenge. Attack those Will saves frequently.
Are they casters? Swarm them with low level baddies. I mean swarms of goblins or kobolds or whatever each with class levels and mean tricks. 12 Goblin rangers with Favored Enemy(Human) and Sneak Attack dice should be able to kill a couple party members. EL 8 or 'Epic' Encounter(but not really for this group, the RAW rules for encounter building are silly). Or actual swarms. Casters are good against swarms - until they are inside one.
Use lots of low level casters with area control spells like Color Spray, Sleep, Grease, etc, to force them to fail saves and put members on the ground.
For lots of fun, create a shadow duplicate of the party and pit them against it. Take everything the players have and throw it back at them.
And finally, don't worry about someone being one-shotted. With the size of your party, they should have 3-4 healing capable classes. Put the hurt on their resources by downing weak members consistently.
I assume the goal is not to kill, so I think outfitting a bunch of goblins may bet the bad idea from your party( that your are an evil maniac when aren't), so avoid mass favored enemy's. Coming up with things that require tactics , like sneak attacks may be more well recieved by your players
Also, it's generally nice to have your boss, and a lower lieutenant that does the opposite ( big fighter + caster), and then some minions. That way there is a logical leader or two, so that when one or two of them die, the minions might start surrendering
Tucker's Kobolds played straight is fine.
Schrodinger's Kobolds where whatever the players do suddenly the kobolds have a counter for is crap.
(no claims about anyone suggesting schrodinger's Kobolds -- just a warning to not fall into that trap)
**********************************************************
Unless you are going for a "perfect kill" encounter always leave some gaps -- something the PCs can figure out and use to their advantage.
It should be exceptionally rare the PCs have all the advantages or all the disadvantages in a fight.
Also remember that in pathfinder (and honestly most of 'd&d') a challenge 'equivalent' fight is one that should be fairly easily won by the PCs. If you want to challenge them you want to increase the CR.
If you want to challenge them regularly with combat without them leveling up faster use the slow exp track.