Snug sliding fit is no good, if you ever want to get those bearings set correctly. You need to "size" the shaft, so the bearing slides easily, but not so much that there's slop. You can do this easily in a lathe with abrasive cloth - you probably only need to take it down a tenth or two.
Regards,
Ray L.