The allen key size.
You probably needed a 4.0 mm allen key. If the key is not a good tight fit get the correct key. The Syil machines use METRIC HARDWARE.
No doubt you got a 5/32" which is 3.968mm. Only a little bit under size. Buy some good quality METRIC KEYS, and don't use the 'BALL END TYPE' for getting things really tight. They only contact in ONE SPOT, and can damage the hex socket.
But once you've damaged the screw that makes it difficult.
If it is the earlier coupler with multiple straight slots then the screws tightens against flats on the stepper shaft and the ballscrew end which also should have flats. IMHO dimples would have been better than flats.
With the POWER ON so that the stepper is held from moving, jog Z using page-up/down keys until the screw is accessible.
Now gripping the ballscrew firmly but without causing it any damage turn it back and forth. You can grip with pliers and some aluminum right NEAR THE BOTTOM as the head will never go down more than 380 mm from the top without hitting the bed.
There should be NO BACKLASH.
Make sure the screws line up with the center of the flats and then tighten.
I had the same problem. You should be able to get it better than 0.001" if loose gibs are not creating subtle errors with head tilt.
After getting the coupling tightened adequately, you can check the backlash in the nut.
TURN OFF the power, temporarily LOCK the GIB with the locking lever, then attempt to turn the coupling and hence the screw. Any movement there will be backlash in the nut AND/OR the thrust bearings.
Don't forget to unlock the gib.
Super X3. 3600rpm. Sheridan 6"x24" Lathe + more. Three ways to fix things: The right way, the other way, and maybe your way, which is possibly a faster wrong way.