In a factored model W = UV^T, the loss is invariant to rotations (U,V) → (UQ, VQ). GD respects that. Adam's per-coordinate second moment doesn't, because it depends on which basis you happen to write the factors in. The claim is that this one property is what sorts optimizers into keeping or losing
