Latency alsmot always comes down to… Either resources or networking.
So, determine which of those first.
Then, if resources memory and disk (remember that by default a backing file is created that represents the Guest system RAM).
If networking, then determine if the same problem is seen in any other Guests or the HostOS.
There are various fine tuning opbimizations like disconnecting memory use from the backing file, but IMO those are mainly for <fine> tuning. What you’re describing should have a much more obvious cause.