"Received a packet with an invalid Magic Value" due to huge message

[scs-b]

Description

This bug has only happened a few times in the past 6 months, and we're not sure what's causing it.

We are letting the game run on repeat for hours or days, and this happened after successfully running the game for over 24 hours.

The log says that the client received a seemingly huge message, but the server is not complaining about trying to write too much to the FastBufferWriter. This is how we send the message;

public void SendInitState(InitState message) {
    using var writer = new FastBufferWriter(4096, Allocator.Temp);
    writer.WriteValueSafe(message);
    _sharedData.NetworkManager.CustomMessagingManager.SendNamedMessageToAll(Constants.InitStateMessage, writer, NetworkDelivery.ReliableFragmentedSequenced);
}

Error message

[Netcode] Received a packet with an invalid Magic Value. Please report this to the Netcode for GameObjects team at https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/issues and include the following data: Offset: 4, Size: 6357069, Full receive array: (25mb of data here)
[Netcode] Received a packet too small to contain a BatchHeader. Ignoring it.
[Netcode] Received a packet too small to contain a BatchHeader. Ignoring it.
ArgumentOutOfRangeException: ArgumentOutOfRange_NeedNonNegNum
Parameter name: count
  at System.ArraySegment`1[T]..ctor (System.Byte[] array, System.Int32 offset, System.Int32 count) [0x00011] in <467a840a914a47078e4ae9b0b1e8779e>:0 
  at Unity.Netcode.Transports.UTP.BatchedReceiveQueue.PopMessage () [0x0003a] in <cb73a45f761942c68e8e5de2a8859e77>:0 
  at Unity.Netcode.Transports.UTP.UnityTransport.ReceiveMessages (System.UInt64 clientId, Unity.Networking.Transport.NetworkPipeline pipeline, Unity.Networking.Transport.DataStreamReader dataReader) [0x00046] in <cb73a45f761942c68e8e5de2a8859e77>:0 
  at Unity.Netcode.Transports.UTP.UnityTransport.ProcessEvent () [0x000c4] in <cb73a45f761942c68e8e5de2a8859e77>:0 
  at Unity.Netcode.Transports.UTP.UnityTransport.Update () [0x000c3] in <cb73a45f761942c68e8e5de2a8859e77>:0 

Environment

• OS: Windows 11
• Unity Version: 2022.3.30f1
• Netcode Version: 1.11.0
• Unity Transport Version: 1.4.1

[NoelStephensUnity]

Are you using the default UnityTransport or a 3rd party transport?

[scs-b]

We're using the default UnityTransport, installed automatically as a NGO 1.11.0 dependency.

[NoelStephensUnity]

@scs-b
I have the same question as #3153 about using relay and whether it is a pure server or host you are using. This will help me narrow down where the issue is coming from.

For both this issue and #3153, if you are using Unity relay then my next question is whether you have you run the same kind of burn in tests locally without the use of relay or any services and if so do you get the same issue ?

[scs-b]

@NoelStephensUnity
We were able to fix this by replacing one of our network message calls with an RPC.

Old implementation:

// Server
private void SynchronizeState(AppState message) {
    SetLocalState(message).Forget();
    using var writer = new FastBufferWriter(1024, Allocator.Temp);
    writer.WriteValueSafe(message);
    NetworkManager.CustomMessagingManager.SendNamedMessageToAll(Constants.AppStateMessage, writer, NetworkDelivery.Reliable);
}

// Client
private void ReceiveMessage(ulong senderClientId, FastBufferReader messagePayload) {
    messagePayload.ReadValueSafe(out AppState newState);
    SetLocalState(newState).Forget();
}

New implementation:

[Rpc(SendTo.Everyone)]
private void SynchronizeStateRpc(AppState message) {
    SetLocalState(message).Forget();
}

We haven't had issues with larger network messages that contain the entire game state, so I thought the issue might be that this message was too small. I changed this method to an RPC as well just to be sure.

private void SendMessage() {
    using (var writer = new FastBufferWriter(1024, Allocator.Temp)) {
        _appManager.NetworkManager.CustomMessagingManager?.SendNamedMessageToAll(Constants.RecentreMessage, writer, NetworkDelivery.Reliable);
    }
}

We successfully ran our burn in test for up to 3 days without any issues.

We are not using relay, and we are using a pure server - it does not double as a client.

Hope this helps!