MatchMaking : OnServerReadyToBeginMessage invalid playerControllerId 0 and weird behavior

So here's a strange error i've no clue how to solve.
I have my my lobby manager expanding from the network lobby manager already in Unet. When somebody joins the lobby, if he is not the server, it sends a ready to begin message. This works fine when I connect to the lobby from my pc in two instances. I also tried it in 2 different pcs but same connection and worked fine as well.

So now I'm testing it out with a colleague who lives in Canada (I'm in Greece) I forced it to always connect to the eu1 relay servers

So when I create match and he connects I get this warning:

NetworkLobbyManager no more players for this connection

and this error

NetworkLobbyManager.OnServerReadyToBeginMessage invalid playerControllerId 0

But, and here's the weird part, if he creates the match and I join, then everything works fine and we can actually play and move past the lobby. I tried both with a build and from the editor. So now i'm at a complete loss as to what to do.

I've already tried a few stuff i found that seemed relevant from a google search but nothing seems to fix the issue.

Any help is appreciated