From what you say it is very hard to tell.
If you look at my article: Writing a Multiplayer Game (in WPF)
] I talk about possible game developments and I did it as server centric (and I think that's what you want to do).
An important thing to note is that if your communication is not like:
Client send data, client receive data, client send data, client receive data.
You will probably need to use the TcpClient.NoDelay set to true.
Such NoDelay is false by default and its purpose is to avoid many small packages to be sent. That is, the first send goes directly, but the second, third, etc will wait either until the tcp buffer is full (usually 4kb) or 200 milliseconds. That justifies having only 5 frames per second.
Also, by this same trait, you should never serialize/write data directly to the tcp/ip stack. If you need more than one write call, then you should use your own buffer and, when you finish, you must copy the entire buffer (write it to the tcpclient/stream) in a single call. This is to avoid the opposite problem if you use tcpdelay to false, a WriteByte call is enough to start sending data.