At home, I organize my Pathfinder books according to general type (ex. the "Ultimates"), then alphabetically within that. I put the types in order according to how often I use them. The Core Rulebook, "Ultimates," "Advanced Guides," etc. are to the left, while the few adventures I own are to the far right. Most things on the far right will get sold at some point.
In the libraries where I have worked (and the one I ran), we occasionally developed small sections that just weren't worth cataloging with Dewey. These include some types of thin paper children's books (the types that would fall apart quickly), "honor books" (the patrons could keep them or bring them back), and things that didn't last long (physically) like Harlequin romances. When we needed a rough organization system for them, we used colored dots. So, for example, in the "honor books" we used different colored dots to indicate the genre, and (in some cases) the subgenre. That made it easy to throw them on the shelves quickly and still give the patrons the ability to zoom in on the areas that interested them.