It does make me a little nervous that the thing might suddenly seize one day, but I suppose your belt will just slip before it tears anything apart (depending on the spindle momentum when the seize occurs). A dry "bearing" has a way of filling up with dust or grit that could provoke an "event".
I hope your spindle is rolling bearings and not babbet.
There's nothing stopping you from drilling a lubrication hole in your brake block. Even oiled, the clamping will be sufficient for gear cutting. Lots of machine spindles run with multidisk clutches that have oil flowing between the plates and they still clamp.
Another way would have been to make a split bearing with a cap (like a connecting rod in an engine). Remove two bolts and take half of the surround away.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)