CA will work just fine, just make sure you don't over-do it, as it will make a mess of ANY surface it touches. Gorilla brand "Super Glue" is, by FAR, the best I've found for gluing to Acrylic and PolyCarbonate, and is thick enough to fill any small gaps. Many others will not stick at all. Applied to a machined surface, you will never be able to get the magnets out, without destroying the Poly.

Regards,
Ray L.