To a great extent, incremental encoders are very reliable accurate source, especially when using differential drivers. There are some older controllers I have seen that has done a cyclical check of the marker against the total per rev. pulses. But the current use is mainly for accurate homing as you mentioned.
The typical explanation of the reading of the graticule head is mostly explained in a simplistic fashion. It is impossible for a LED source and sensor on their own to differentiate between the individual slots in a reading head that has a resolution of more than around 100p/rev. The technique to read very high resolutions is done using a method called the Moire effect.
This is acheived by a the reading head carrying a graticule that is slightly skewed to the main scale which results in the quadrature 'shutter' being much wider than the individual graduations. The rotation of the optical 'shutter' is at right angles to the direction of motion.
Al.
CNC, Mechatronics Integration and Custom Machine Design
“Logic will get you from A to B. Imagination will take you everywhere.”
Albert E.