3/8 diameter cutter is probably on the big side for a 770 in high belt. Have you tried low belt? In high belt you are going to sacrifice torque for rpm. 3/8 is the largest endmill I run on my 1100mx.

Might be worth checking out provencut since the feeds and speeds have been successfully run on tormach. (free trial) If you can't make a cut with one of those recipes then you would have a clear indicator that the motor isn't producing as much as it should.

It's tough getting going and you aren't the first to go have these frustrations and won't be the last, so hang in there.