Have you ever used a compression fitting on copper tube? Collets work on the same idea.

The type that uses a drawbar (long bolt from the top of the spindle) to pull a tapered collet into a matching taper inside the spindle can only grip the tool size they are made for. The collet has splits so it can open slightly to allow the tool to be pushed in but because the splits are only at one end as the collet closes up in the taper the sides of the hole will not be parallel if the tool is smaller than the collet is designed for.

The other type that are pushed into a taper by a nut (very similar to a compression tube fitting) often have a taper inside the nut as well as the body of the collet and there are two sets of splits from each end of the collet. This means the sides of the collet gripping the tool are compressed in more or less parallel so these collets can have a range of grip that can be as much as .039".