A drill press spindle is the sh!ts for milling with. They have entirely too much overhang and improper bearing support. They are designed for axial thrust, not radial thrust of milling.
Like Steele says, scrounge around for something with a real mill spindle on it. Get the XY table for free. Put it out in the backyard if you have to. I've always wanted to machine outside
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)