That's a better description of your issue and definitely not a good thing.
For what you are describing now, I would doubt that the belt will be the root cause as I have never witnessed a belt do something like that. A worn belt/pulley will not help things granted, but it's quite possible that the encoder is bad as you suspect.
When the machine is at a commanded rpm, does it hold the commanded speed well or does it make big jumps; a few jumps in rpm is fairly normal? Any issues with orientation changing?