I have a question for all those security boffins out there.
I am developing a Tcp/Ip based client/server system that needs to be VERY secure.
I'd like to know if there are any obvious vulnerabilities in the way the system works.
Each communication is initiated by the client, the server sends a response and the connection is closed. Much like HTTP, - The connection must be re opened for each transaction. Session state is handled via session-id parameters in the messages from the client.
The client and server create and exchange public keys, - these keys are particular to the client and server (ie the server keeps the public key sent from the client, and a private key for that client (of which it sends the public part))
Once public keys have been exchanged, it uses the RSA asymmetrical algorithm to exchange a pair of AES key + IV byte arrays (generated for/from the Rijndael Algorithm). The server sends data using one key, and receives using another.
Each message is signed using RSA and checked against the IP address recorded for that session, to prevent a third party attempting to imitate either the client or server. Although, if one party was fake from the start, I don't think this helps.
All further communication is encrypted using the Rijndael algorithm.
Authentication occurs at this point, and the client sends username/password information to the server, which generates a hash from the password and checks it against the hash stored against the username.
All messages are exchanged by serializing a class through a compression stream into a network stream. The formatter is given a custom binder that scrambles the assembly and type names.
The serialized bytes are not encrypted, instead the contents of the message are (the class being serialized has a Message property, that will contain the encrypted data)
Apart from being easier, (correct me if I am wrong here) I think it is more secure. By encrypting the contents of the message, and not the whole serialization, it means that commonly known, easily guessable serialization headers are not encrypted, so they cannot be used to assist with breaking the key.
The only thing I can think that might go wrong, is if someone was able to re-route packets to their own copy of the server, and therefore sniff username/password data from a client attempting to authenticate.
Assuming a malicious third party is able to redirect Tcp/Ip packets, How would you go about validating the identity of a server? it that even possible?