If you are using a ordinary meter it may not be fast enough to catch the actual low value.
But it does sound like either the supply is dipping or there a supply sensitive to spikes.
Is there a 120v source that is not supplying the rest and has a relatively unloaded connection direct back to the panel?
Al.