Honestly, that sounds painfully hard in a pointless, suck the fun sort of way, I can't imagine how anyone would have fun doing that, kind of like playing WotR without killing demons or insects :-)
I've actually had the thought of trying that with an enchantment-specialized arcanist who wants to follow in the Thassilonian tradition of enslaving giants.
Could be interesting - if I try it I'll report back and let everyone know how it goes.