Halving the visibility range effects things like the maximum distance for perception checks to notice something. Considering your sea critter will be swimming up on them, that isn't going to come into play.
Keelboats are decent sized. I'm not sure how much distance you have between the deck and the ocean, but it might be far enough that a 10' reach can't reach the deck, or at least not well.
There would be cover from the deck, probably both ways. think of it like fighting around a hard corner.
You may also want to consider what effects this creature will be having on the stability of the ship. If the boat is sailing on choppy water or the sea creature will be rocking the boat, you may think about if you'll need to impose a concentration check for casters. Maybe the "Vigorous Motion" option. For concentration check stuff, see here.
I figured half visual range meant things like torches shed light to 10 feet instead of 20 feet.
Could you rephrase the Perception question? I can't quite parse where you are finding two -4 penalties.
This is gonna be a lot like defending a castle.
Assuming the deck of the keelboat is 10 feet above the water line, and there's a decent railing along the ship's edge, It'll be hard for the sea monster to hit anyone on deck.
Assuming it's large, it'll have to lift half its body out of the water in order to reach. The PCs will have soft cover from the railing, and will be able to make ranged attacks with no cover penalties and without provoking. It'll take reach weapons on board in order to hit it in melee, though. Again, with cover penalties only applying to the monster's attacks.
If its a grabby sort of beast, you can remind the party that ships often have tie lines. Pulling someone overboard might be a death sentence; I presume they'll want to mitigate that.
Depending on how aggressive it is, it might just attack the boat directly. Rather than expose itself, just punch a hole in the bottom and eat the tasties after they drown. You'll want to have the ship's hardness/hp on hand, and have mechanics in mind for "how to make the ship stop sinking"
When it fades, won't it be submerging? If you want to be kind to the party and have it stick close to the surface as it fades, then the perception check will come into play. I'd give it concealment, let it make stealth checks. Of course, it'll have equal difficulty figuring out where folks are on deck.
Ships should have lanterns capable of withstanding waves crashing over the side, and would have both standard and bullseye lanterns aplenty. Sailors take visibility seriously, yup yup. The bullseye lanterns might come into play to find the critter after it fades.
I figured half visual range meant things like torches shed light to 10 feet instead of 20 feet.
*facepalm*
Of course. And halving the range of darkvision and lowlight.Do y'all think it also means that perception takes a -1 penalty every 5 feet instead of every 10? I was thinking that was covered by the blanket -4 penalty.
Ok, I see what your Perception question is. Bummer that I don't know the answer. Seems to me that there's a lot of confusing verbage going on right there. "Rain reduces visibility ranges by half, resulting in a –4 penalty on Perception checks." sounds like you don't actually reduce visibility, and that the -4 takes care of it for you. Maybe?
We can FAQ this, but frankly we shouldn't need to.
These are situational modifiers, and as a DM you have absolute control over them, RAW be damned. You are inventing the situation, and can decide what the Perception penalties are based on how heavy you think the rain is.
Ok, so if your boat is effectively 5' above the water with some railing added on, then your beastie can probably remain submerged and attack things on the edge of the deck. That should probably get the beastie concealment (from water), and the party cover (from railing).
It can heave half its bulk out of the water and lose its concealment in order to negate the party's cover, and be able to reach farther onto the deck. (Again, assuming its large)
I figured half visual range meant things like torches shed light to 10 feet instead of 20 feet.
It is more likely referring to the "Stealth and Detection" entries in the Environment chapter.
http://paizo.com/pathfinderRPG/prd/environment.html
Stealth and Detection in Plains: In plains terrain, the maximum distance at which a Perception check for detecting the nearby presence of others can succeed is 6d6 × 40 feet, although the specifics of your map might restrict line of sight. Cover and concealment are not uncommon, so a good place of refuge is often nearby, if not right at hand.
Stealth and Detection Underwater: How far you can see underwater depends on the water's clarity. As a guideline, creatures can see 4d8 × 10 feet if the water is clear, and 1d8 × 10 feet if it's murky. Moving water is always murky, unless it's in a particularly large, slow-moving river.