I'll stick a reply here in case someone else wonders whether the encoder resolution change affected the operation of my mill.

In short, changing the resolution made the encoders work better in my setup because the combination of the PID loop being closed in the DG2S drives, the resolution of the encoders, and the physical characteristics of my mill's system made it hard to tune to any reasonable level of performance and not have oscillations.

Eventually, I ended up replacing the DG2S drives with Aerotech analog servo amps that accept a +/-10V control signal and closing the PID loop in LinuxCNC. This works much better performance wise in my setup and was easier to tune than the DG2S drives... not because the DG2S software is bad, but because enough other people have used LinuxCNC this way that there are a number of walk-throughs available for it.

After switching servo amps I changed the encoders back to max resolution and they continue to work well. I may eventually switch servos to brushless with digital amps, but for now my mill is working quite well.

Erik