Dilemma about PC death

In a homebrew game, I have given a lot of freedom to the player to create their PC. They, for the most part, went for min/max + optimize. I have no problem with that, I have the whole monster manual at my disposition.

The thing is, in the past 5 session, 3 PC have died. The players are veteran, having played over 8 years of dnd and pathfinder.
My encounter are not too hard, belive me.

Example : (level 8 group) They know they are going to fight a vampire sorceress with weak vampire spawn, they have 1 week to prepare for the fight. Nevertheless, they went bursting thru front door with little preparation, no potion, no scroll ...

So you might say : Well, they died by their own fault. And you are right.
The thing is, I need the party to stay alive dammit. I've ran out of idea on how to bring these new PC to the group. And I want my story to progress but it get weird to bring new PC that have no clue on what happened before.

And I can't tone down the battle because than, it's going to be over in 2 round.

Any idea?

What exactly kills them? Failed saves? Lots of HP damage from big attacks? Steady attrition of HP from a lot of minions? Gross stupidity? Whatever it is (except the last), just tone down the attacks a little bit.

If they're optimizing a lot, it's surprising that they aren't prepping a bit more.

Ultimately, you can't save the players from themselves and their own decisions. I'd strongly recommend talking to them and letting them know how you're feeling - if they're having fun, they're having fun. You have to have fun too, though.

Your example needs more story. As much as I would like to believe you. With what you've written I can't tell if your encounters are too hard or not.

Have you talked with your players about this issue?

Alright, here's the full fight CR :

Vampire sorceress from the d20pfsrd + 4 vampire spawn.
I have removed their ability to Energy drain.

The party is :
Ranger , Bard , Anti-Paladin , Sorcerer.

They enter the crypt, trigger an alarm, kill some skelleton.
At this point they are at full HP with almost all of their spell/ability.

They did not prepare for anything (They had 1 week in town to prepare)

So of course as soon as they open the door leading to the lair, I launch a fireball at them. They enter the room on their turn and I cast greater invisibility. Nothing happen in the next round aside from positionning.
Than I launch my 4 vampire spawn and they get immediately annihilated.
Still have 4 round of greater invisibility so I cast dominated mind, it fail, I cast it again next round it fail, I cast it 1 more time I got one.
Then I go away from the party and leave the ranger kill the bard while I cast another fireball to end the bard once and for all.

The next part of the fight I just gave up and made them kill the sorceress otherwise I think I would have kill 1 more member.

So they're dying due to failed Will saves and inability to deal with possession. This is something a simple protection from evil could have countered, and if they didn't have that -- or its big brother, magic circle against evil -- up when entering a vampire lair.

Also, no way to detect invisible creatures -- by 8th level, someone should have see invisible or its equivalent available! Plus there are always the old standbys like "cloud of flour"... The vampire goes invisible and they just do "positioning"?

Yeah, "gross stupidity" is the word for it.

Admittedly, your players are just entering the levels where greater invisibility might be a thing, but even regular invisibility can be a problem from much lower levels, and mind-affecting spells can be a problem too (charm, suggestion). I think you need to have a talk with them about being prepared for various sorts of things.

So they knew that they were going against a Vampire and didn't think of Prot Evil/Good as a counter for mind control?

That's Darwinism in action.

You don't force their hand, but you can alter the fights as to push them to better prepare. There is nothing frustrating even for the GM than to see some players rushing head front in a miscalculation that cost a couple of deaths. They know what may happen, they have time to prepare but decide not to. They fight in unfavourable terrain but don't adapt to offset that.

Removing the energy drain was nice from you, but keeping it would not have changed a lot of things. Vampires are highly intelligent so playing them as you did doesn't sound abusive. You used the abilities to good effect. Don't feel guilty to not holding back.

At that level, speaking of gross stupidity might be an understatement. At lower levels or for less experienced players it could be excused, but not there.

Alight, I also had the feeling of talking to my player about better preparation, but is there any other thing I can tell them other than :
Have better preparation?

Tell them to learn the common threats you're going to have to deal with at this level, and the counters to them.

Invisibility can be countered by See Invisibility, Invisibility Purge, Glitterdust, etc.

Grapple-related threats (like Swallow Whole) can be countered by Freedom of Movement, or teleporting your ally out of the grapple, or (to some extent) Grease.

Energy Drain attacks can be countered by Death Ward and Restoration.

Mind control can be countered by Protection from Evil.

Darkness can be countered by Daylight, etc.

Flight can be countered by archery or by giving your own guys Fly / Air Walk.

Fireballs can be countered by Resist Energy.

Paralysis can be countered by Remove Paralysis or Freedom of Movement.

Nausea can be countered by Remove Sickness.

Excessively dangerous enemies can be countered by teleporting or dimension dooring away.

The party should be ready for all these things, with scrolls if nothing else.

Alternatively: give them some scrolls of Raise Dead. Or an NPC ally who can raise them.

And I can't tone down the battle because than, it's going to be over in 2 round.

You probably can if you put your mind to it. There are ways to make battles last longer without being much more dangerous. Large numbers of low-grade foes, tanky enemies with good defences but limited damage output, terrain that makes it hard to close with the enemy, etc.

Terrain obstacles are, I think, one key in making battles longer and more interesting. Make the PCs work to get to their enemies -- or make them have to get to their enemies (e.g., a blindsensing foe in a smoke-filled room, who can throw AOE stuff at them but can't be sighted in return, unless someone is right next to them already).

Multiple enemies are another one -- and having the vamp sorceress have some minions was a really good idea (kudos to you). Looks like the PCs are quite able to quickly slaughter lots of minions, but not all battles are Achilles/the PCs mowing down mooks/the Trojan army.

Have the PCs come back as minion vampires might be interesting, if you think your party would be up for that (the anti-paladin, maybe...)

Hmmmmm. I agree with the statement that it is their fault for their terrible terrible tactics. To solve this I would nerf the enemy tactics. Some players just don't want to plan and strategize against enemies. They just want to run in and beat up the villain. If the numbers are even, make the tactics even.

Or try and make it so the enemy's victory doesn't mean the PCs' death. Maybe the sorceress needs living minions for some task... captives can be bargained with, or inveigled, or maybe even rescued by someone sneaking into the villain's lair. ;)

After they all die: "You wake up! It was all a collective dream sent by some helpful higher power. You have the eerie feeling that it was 100% accurate... and that you'd better prepare for the damn encounter now that you've been warned. You've got a week."

How did the combat end?
You could have a mini scenario where the Vampire had revived them to feed off of, but have not turned, then allow the remaining party members to rescue them. This way it stays in story, and they get another crack at the vampiress. They may even think about researching vampires before they go rushing in next time :P