Planning a horror one-shot for spooky season at my FLGS and I need a good horror monster to build the adventure around for a party of around 4-6 level 3 PCs. I'm shooting for a pretty severe encounter to really drive home the scariness. Bonus points for if it has some abilities I can use to make red herrings for them as they try to investigate the monster before fighting it.
I'm unsure whether this will mesh with your goals, but it's something I heard recently that resonates with my sense of horror: "If you're giving Cthulhu a health bar, you're playing Cthulhu wrong." In RPG terms, this could mean that one encounter might be "Cthulhu's Aura" where you simply try to survive madness (maybe with a solution like acquiring a protective trinket/ritual/meditative phrase). Another might be dodging its flailing tentacle as you attempt to cross a room. Another where you try to avoid its notice. There are some video games that replicate similar situations, like fighting a giant monster in stages. There's a recent AP where a kaiju makes an appearance and unleashes several high-level effects on the party, framed as Hazards/Traps (and I'm unsure the kaiju even notices the PCs!). I did this with an evil god's avatar long ago, with each round it existed on the battlefield being a "trap" w/ its own CR/level.
Another facet of horror is that survival is "winning". Whether you actually kill the monster, rescue a victim, or gain treasure is secondary (if even possible) compared to getting your ass to safety. PF doesn't have much in the way of losing body parts or sanity, but you might adapt something from other systems. It is a one-shot after all. Various Conditions could work for attrition. Since many are status penalties they don't stack and make things insurmountable. In PF2, with Medicine and time it becomes hard to impact PCs much via hit points so think of "stickier" effects they have to persevere through or outright gauntlets where they have no time for Medicine (and thus appreciate what time they might earn along the way). Hmm, rest time as a reward...
In a horror campaign I ran, one of the scenarios that stuck most with my players and myself revolved around a haunted house which was the enemy. The house had resources, i.e. paintings w/ peculiar powers, animated items, horrific flashbacks, reenacting a death scene in real time w/ a PC as the target, etc. that it would throw at the PCs whenever able. All come from the house's backstory. The party had to drain the house's energy by facing these hazards, a fact they only learned through investigation. Other ways to help were to rectify a wrong, like reassuring the ghost hiding under a bed, a child whose mother had been beheaded in front of him (the beheading being part of a Haunt's effect) in the same room. The part would remain stuck until they faced enough. I forget how I played the time limit, whether it was survive until dawn or escape before dawn (and you couldn't do that while the house was still active). Part of my brainstorming process was imagining various visually impactful ways to die, i.e. hanging, mauling, etc. and then integrate them, ex. a painting of a jungle made noises if one listened, then one might feel the predator stalking, until eventually claws swiped out from it (and it was in a passage PCs would have to pass through at least twice).
Though more difficult to implement, one aspect of the house was when PCs lost line of sight with each other, they "lost" each other. This involved dimensional shenanigans more than sensory penalties and I wouldn't be able to provide a rigorous way to execute this in PF. If the party's big enough consider temporary separations (with encounters balanced around sub-groups even if it doesn't necessarily feel that way to the players).
One last touch was players had to roll to even remember their experiences. "What happened to me?!" is quite scary, as is having a teammate want to check out that old house like they'd planned. "NO!"
Also feel more free than usual to wing encounter pacing since to really terrorize one's party requires finesse that the dice might ruin even if one were able to plan every threat's balance meticulously.
So yeah, what I'm suggesting is that the best horror monster is one the party can't defeat, only survive. Otherwise it's just a tough fight, even if tougher than usual.
Just now remembering an old superhero RPG scenario (so a genre where one expects to punch their way to glory). Except this one had a truly unstoppable horror that could shred the toughest PCs. First, the GM had to demonstrate that! Then the players knew to run. Destroy an unbreakable wall or a throwaway "tough guy NPC", i.e. a fierce monster the party's already afraid of getting one-shot in front of them. Second, the creature was slow so PCs could get away (albeit temporarily since they were trapped in a lab complex with it). Third, the complex was full of clues (in the original re: experimentation), or in PF one might have traps & minor monsters...which draw the attention of the unstoppable one! These clues led to fixing a machine (or in PF a magical apparatus) which in turn helped depower the monster (or it could assuage the pain that's making it hostile). One important trait was its ability to recover so the PCs couldn't kite it; no appreciable damage until they'd solved the problem! (In the original it was a strong force field on top of its already solid defenses, so the scratches were already minor and quickly recovered from).
Some sort of Charnel Creation or Carrion Golem variant, maybe something Clockwork or Fleshwarp in nature. I like the idea of its stench preceding it so PCs smell it before seeing it directly. Add masks or spices for their noses to help with that, a form of Regeneration that they learn the counter to, maybe a magical tool they fix that lowers some damage aura/force field, etc. All the debuffs culminating in a CR 5-6 creature they can tackle (5 if there's been attrition, 6 if they get to face it at full, perhaps w/ even more resources they've gathered).
Hope that helped. Cheers.
Human. Nothing scarier. And given the right (wrong?) impetus, there’s almost nothing they won’t do.