I like the idea of trimming the spacers down to have the bearings more flush with the hub ridge. I am going to open it up again tomorrow and see how it looks after a 40mi ride I did on it yesterday. By the end of the ride I had a hard time moving the bike from a stop, which has been an affect of the bearings coming out.
I too am thinking that the hub is warped in some way, whether it be from age, use, or abuse. I'm going to end up sourcing some new wheels soon i think.