I think keel hauling would likely do ability score damage more so than hit point damage. Maybe 1d6+1 Charisma damage to reflect scarification and then 1d8 Constitution to reflect broken bones. Reflex save for half, DC 20.
and some 3d6 for the damage ... and a bit depending on the speed, maybe drowning. It might easily be 5 min below water ...
well, in a fantasy game this could possibly do phenominal damage and it clearly depends on how large the ship involved is and how long it has been in the water. As keelhauling pulls an object under the boat from larboard (nowadays called port) side to starboard side, the object is underwater for quite some time; not only that; is dragged and scrapped across all the barnacles and whatnot that is attached to the bottom of the ship; in a fantasy setting; this might be quite a collection of pretators and vicous whatnots; the bottom of a ship was much like any coral bed as all this stuff grows on the bottom of a ship and begins its own microverse or colony or whatnot. This also tends to slow down the ships speed quite a bit over the years which is why the more profitable owners or government vessels where pulled up out of the water and the laborious task of scraping the ship clean was begun; some ships lowered sailors to scrap in short dive sequences to slow down the production of these colonies which is where the whole idea of keelhauling originated. In a fantasy setting you could make up all kinds of shelled creatures and whatnot to do just about any damage that you like to kill just about any character short of epic level and make cleaning of these ships very interesting; hehe there is just about an infinite number of dark, creapy, wiggly things with spines, teeth, sharp shells that can float around in a fantasy water setting.
also keep in mind; that if rope is used; the sharp stuff on the bottom can tear up the rope also and cause it to break; thereby losing the object being keelhauled at sea; of course; lots of sea creatures like sharks and whatnot follow ships due to the refuse that is always dumped and thrown overboard; yep; just like in our world; people leave lots of trash around and the sea is no different; this could also make for some interesting times.
I remember looking at pictures of the bottom of my ship that divers took after we were in the Persian Gulf for about five months. You could hardly tell that it wasn't a coral reef. In warm water the marine growth really mushrooms. These organisms create lots of drag--our top speed dropped about 25% during this time.
Well, my PCs will have to watch out if they careen the ship (lay it up on the beach) to clean the hull so they can keep up with the fast-sailing sloops of the Crimson Fleet. Dire hermit crab attack!
A fantasy angle for hull cleaning--its actually a documented concept in the Ierendi Gazetteer for Mystara rather then my original idea--the Ierendi navy is allied with merrow (merfolk) that both patrol the waters and assist with in-water ship maintenance. One of the reasons the Ierendi navy is the most effective in the Known World...
Fantasy worlds have fantasy solutions as well as problems; I'm sure any good sea-mage worth his salt knows the spell barnacles to sand!
In a fantasy setting you could make up all kinds of shelled creatures and whatnot
I'm now fighting off the urge to stat up the Dire Barnacle. ;)
EDIT: It just occurred to me that Universal Solvent might be the ideal barnacle-removal treatment for the richer merchants' ships in a D&D setting.