I think you're right about just needing to increase the diameter by a couple thousandths. I'll slightly reduce the diameter of the tool in my CAM software. So far I've been able to use a hand tap to clean the whole hole in one pass (I don't have to clean out the tap) so it's gotta be pretty close.