4-40 and 2-56 have pretty darned small holes - are there thread mills small enough that can fit into the drilled holes?

Mike