I'd say your ruling seemed fair - this is way too specific for there to be an existing rule unless this exact circumstance happened in an AP
If you need to make a determination on the fly, remember that side elevations exist. Draw a picture of the scene from the side and see what it looks like.
Turn it on it's side and see how it works horizontally.
If I understand you right,
How much of the fighter can the cleric see or, more relevantly, if there was a character positioned in the bottom left square of the ogre.
Hi, In the last game I was GMing I had my PCs assault an NPC character who was standing on top of and in the centre of a 3x3x3 (15ft) cube. The rogue was standing at the very base of the cube and therefore argued that he was hidden from view of the NPC, which I allowed. This then brought up the question of how far away from the cube could the rogue move before coming into line of sight/effect of the NPC spellcaster? At the time I ruled that it was only the direct 5ft at the base of the cube, until I could find out the exact rule, but I can't find anywhere discussing this? From a real world perspective I would say for every 5ft away from the edge that the NPC spellcaster was, that is how many feet below he is unable to see, up to a maximum of the height of the cube? But obviously if there is an already existing rule that would be better.
Assumptions: Both NPC and PC are of the same height, using 5' to ease calculations.
If you consider that the eyesight of the NPC is 5' above the height of the cube, and you have a 5' x 12.5' right triangle to determine the angle of LOS to the edge of the cube. Anything below this angle is out of LOS for the NPC. This hypotenuse is roughly 13' long.
The actual height above ground the eyesight of the NPC is 20' from the floor, which is a scale factor of 4 as compared to his height above the cube. Using this scale factor, the hypotenuse of the larger right triangle is roughly 54'.
Which makes the base of the larger right triangle to be 50' (starting from the centerpoint of the cube at ground level).
In pathfinder terms, this is 9 squares of occupiable space - 2 of which are inside the cube). If the PC is also medium (5' height), you would subtract 2 squares from the length of the base of the larger triangle at it's furthest end from the cube. Anything between these two points puts the head of PC below LOS for the NPC.
This means that the PC can stand anywhere between the base of the cube (square 3 from the center square) and 5 squares out (square 7) from the base of the cube and still be fully concealed from the sight of the NPC. Any further out and he would progressively lose cover.
And everyone says you won't use geometry out of high school.
Alternative calculations: this uses the ranged weapon rules (starting from the highest "corner" of the 5' cube the NPC is in) as well as the assumptions for my previous post as base assumptions.
Small right triangle is 5' high, 10' length. Large right triangle is 20' high, 40' length. Occupiable space is 8 squares, two of which are inside the cube.
Subtracting 2 squares from the outer lenght of the base, puts a total of 4 squares of space that is in the shadow of the cube from the perspective of the NPC.
Squares 3-> 6 are all inside the shadow of the cube and give 100% cover to the npc from straight line ranged attacks. square 7 (from the center of the cube being square 1) is improved cover, square 8 is partial cover.
The shadow of the cube only gives concealment vs. Lobbing or curved ranged attacks.