MOPED ARMY

Ideally, you only want contact with the inner race. The seal spins with the outer race, so if there's any contact with the spacer, you'll get some drag, and probably wear out the seal.

You could remove the inner seal completely, but what I would do is just bevel the edge of the spacer on a grinder, so it only hits the inner race.