Originally Posted by
blinkenlight
A 0.1mm error on 0.6mm is a rather large one. A half-decent mill should not need any calibration - I'm saying this as a practical statement, not as some form of philosophical snobbery. It simply doesn't have any room for imprecision - in the typical case the number of steps per revolution of your steppers and the lead per revolution of the leadscrew unambiguously set the distance you move, without possibility of mismatch; all you have to do is set the right (calculated) number of steps/mm in your driver software. Assuming that number is correct, if significant error does still exist, you're either losing steps somehow, you have a spindle with too much runout or you have significant backlash - either condition unacceptable for any sort of fairly high precision work, like PCB milling. Calibration won't help much with either, they really need to be sorted out in hardware. In theory it is of course possible that your leadscrew isn't quite precise, but in practice I've never heard of significant long-distance pitch mismatch, and local variances are usually negligible for all practical purposes.
If the error is not originating in hardware, there's something really fishy going on there; all forms of layout (gerbers, g-code etc.) is precise to a rather high degree - even if they display something like 9.999 instead of 10 due to different units being used internally, that should never become a problem, it should always be way more precise than the physical mill could ever hope to be - the single exception to this being the use of a irresponsibly imprecise setting for the gerbers / gcode decimals, like "x.xx" (4:2 etc.), 2 digit precision with inch units. That sort of thing should be configurable in the generating software, and is easily verifiable with a text editor.
Similarly, since all these formats are text-based, it should be easy to pick two known coordinates and follow them through the chain trying to see where, if anywhere, is the error being introduced. Good luck!