UNET Player shakes back and forth as it moves over the network

Hello. I am working on a space experience where the user can fly around and a player can join in and follow them via a camera. I am using Network Transform to send the players position across the network. The values are:
Send Rate : 29 (Currently) have tried various values smaller and higher here
Sync Transform
Movement Threshold : 0.001
Snap Threshold: 5
Interpolate Movement Factor: 1
Rotation Axis: XYZ
Interpolate Rotation Factor: 1
Compress Rotation: none
Sync Angular Velocity: Checked (currently) have tried both.

The camera when it connects, has a script that sets it to find and follow the player. Note nothing else in the scene shakes. The player when they start moving, on the follow camera client, shakes very fast side to side. I have tried putting the camera script in Update, FixedUpdate and LateUpdate (it is currently in LateUpdate) and the players scripts are currently in FixedUpdate, though I have tried each of the three mentioned update methods for the player as well.

Does anyone know why the shaking is happening? I can replicate it by replacing the character with a cube and a simple movement script as well, so that rules out the character controller.

Any help would be greatly appreciated!

It is also worth noting that the player moves smoothly on the local client, it only shakes on the client that spawns the camera to view it.