|
finally i confirmed that vc6 ifstream::open has bug and this has been fixed in vs2010. So i modified the code to match with existing behaviour.
|
|
|
|
|
I wrote a scalable TCP server using I/O completion ports. The basic design is that I kick off an accept thread that waits forever for the FD_ACCEPT network event and then accepts the connection and ties it to the I/O completion port. There are 4 to 8 worker threads that process events coming off the I/O completion port. Everything works great.
Now I'm trying to add scalable UDP support. I am new to UDP, so I'm not sure how I should go about this.
1) if I have 4 to 8 worker threads to process TCP events coming off the I/O completion ports, should I do the same for UDP? Some links I found said you only need one thread for UDP and there is no need for IOCPs??? This seems wrong as say I get 1000 UDP events at the same time, I'd like to be able to multi-thread process them.
2) I don't seem to get any network events when a UDP client "connects", just get the FD_OOB at "start up".
Was kind of expecting this to behave like the TCP side in which I'd get a network event when the client "connects" and then another event when there is data to read (via WSAEnumNetworkEvents())...
I know UDP is connectionless, but I'm not getting any network events when the server gets data.
|
|
|
|
|
m_hevWSA = ::WSACreateEvent();
m_sockRecv = ::WSASocket(AF_INET, SOCK_DGRAM, IPPROTO_UDP, NULL, 0, WSA_FLAG_OVERLAPPED);
::bind(m_sockRecv, reinterpret_cast<sockaddr*>(&sinRecv), sizeof(struct sockaddr));
UINT CMyClass::WorkerThread(void)
{
int nRet = -1;
bool bPending = false;
DWORD dwWSARead = 0;
DWORD dwWSAFlags = 0;receive flags
char NetBuf[DGRAM_BUF_SIZE];
WSABUF WSABuf = { DGRAM_BUF_SIZE, NetBuf };
HANDLE ahWait[2] = { m_evKill.m_hObject, m_hevWSA };
WSAOVERLAPPED ovWSA;
struct sockaddr_in SenderAddr;
::ZeroMemory(&SenderAddr, sizeof(SenderAddr));
::ZeroMemory(&ovWSA, sizeof(WSAOVERLAPPED));
ovWSA.hEvent = m_hevWSA;
do
{
int nWSAErr = 0;
unsigned nNetRead = 0;
if (!bPending)
{
dwWSARead = 0;
dwWSAFlags = 0;
WSABuf.len = DGRAM_BUF_SIZE;
WSABuf.buf = NetBuf;
int nSenderAddrSize = sizeof(SenderAddr);
if (::WSARecvFrom(m_sockRecv, &WSABuf, 1, &dwWSARead, &dwWSAFlags,
(SOCKADDR *)&SenderAddr, &nSenderAddrSize, &ovWSA, NULL) == 0)
{
nNetRead = static_cast<unsigned>(dwWSARead);
}
else
{
nWSAErr = ::WSAGetLastError();
if (nWSAErr == WSA_IO_PENDING)
bPending = true;
else
nRet = 1;
}
} switch (::WaitForMultipleObjects(bPending ? 2 : 1,
ahWait, FALSE, bPending ? INFINITE : 0))
{
case WAIT_OBJECT_0 : nRet = 0; break;
case WAIT_OBJECT_0 + 1 :
if (::WSAGetOverlappedResult(m_sockRecv, &ovWSA, &dwWSARead, FALSE, &dwWSAFlags))
{
nNetRead += static_cast<unsigned>(dwWSARead);
bPending = false;
}
else
{
nWSAErr = ::WSAGetLastError();
if (nWSAErr != WSA_IO_INCOMPLETE)
bPending = false;
}
::ResetEvent(m_hevWSA);
break;
}
if (nNetRead)
{
}
}
while (nRet == -1);
return (nRet == STILL_ACTIVE) ? 1 : static_cast<UINT>(nRet);
}
|
|
|
|
|
Thanks Jochen. A follow up question:
Jochen Arndt wrote: Using one receiving thread is enough even with many clients when the UDP packets
are small.
What specifics are we talking here? Seems like common convention on the internet is telling me to restrict UDP packet sizes to 256 bytes, correct?
Assuming my UDP server is receiving AND sending to clients, how many clients can I handle in a single thread? 100? 1,000? 10,000? 100,000?
Basically trying to do something that could scale to millions of clients. Not even sure if thats possible without multiple servers behind a load balancer though  .
|
|
|
|
|
SledgeHammer01 wrote: What specifics are we talking here? Seems like common convention on the internet is telling me to restrict UDP packet sizes to 256 bytes, correct?
Yes. Bigger packets may be used when only a few packets are expected (clients * rate).
SledgeHammer01 wrote: Assuming my UDP server is receiving AND sending to clients, how many clients can I handle in a single thread? 100? 1,000? 10,000? 100,000?
It depends on many things. At least the server should be able to process all packets according to the available network bandwidth. If the server has low resources, additional threads would not help. If data processing after receiving is time consuming, using multiple threads for processing may be an option. With TCP connections, threads are used to unlock the port for new connections. This is not necessary with UDP.
SledgeHammer01 wrote: Basically trying to do something that could scale to millions of clients. Not even sure if thats possible without multiple servers behind a load balancer though
Using a load balancer is common practice if a single server could not handle all requests.
|
|
|
|
|
Thank you very much.
One final question . Seems like a common UDP loop is pretty similiar to the common TCP loop:
while (1)
{
int i = recvFrom(...);
// do stuff
}
so, I may be misunderstanding, but say I have 10,000 clients in say a game perhaps, or whatever. They might send me a "here is my new position" UDP packet and then I'd have to send out a "here is user x's new position" UDP packet.
So with a single thread, wouldn't the loop be something like:
while (1)
{
int i = recvFrom(...);
UpdatePosition(recvBuff);
for (int j = 0; j < 10000; j++)
SendPositionUpdate();
}
if I got 100 position updates, I'd have to send out 10000 * 100 position updates to the clients (I know there is multi-cast , but I'm trying to figure out the basic concept at this point ).
point is... I'd have to finish sending out recv #1's 10000 notification packets before I ever got to servicing recv #2??
Maybe I need to get all available recv's in each iteration and then just send out 10000 notifications once?
Just trying to figure out how this stuff is implemented in the real world .
|
|
|
|
|
The loop is similar. I don't know how it is done with game servers. For sending packets, you should use non-blocking calls. These return immediately and you did not have to wait.
You may also use an own thread for sending. Then you must only call a signaling function from within the receive thread. If the sending thread has not finished sending all data when new data should be send, you can skip the new data, terminate the actual transmission loop and restart it with the new data, or just use the new data for the outstanding transmissions. I think the last option would be the best solution for positions with a game server.
|
|
|
|
|
Gotcha... thanks! 5's all around .
|
|
|
|
|
Thank you too.
Did you find the problem? When I tried to send an answer, the message was gone. 
|
|
|
|
|
Jochen Arndt wrote:
Thank you too. Did you find the
problem? When I tried to send an answer, the message was gone.
Yeah... as I indicated originally, I was adding UDP support to my working TCP IOCP server. I neglected to mention that it was IP agnostic (Ipv4 & Ipv6) . So passing in a sockaddr_in was too small a buffer for the Ipv6 addresses, so it kicked it out. Switched it over to the correct SOCKADDR_STORAGE and its working now.
Thanks again!
|
|
|
|
|
Hmm... seems like its not working after all  This is what I do:
1) WSAStartup() and m_hIOCompletionPort = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
2) getaddrinfo() to get the IPv4 & IPv6 interfaces
3) for each on items returned from getaddrinfo() WSASocket(...WSA_FLAG_OVERLAPPED) and bind()) and CreateIoCompletionPort((HANDLE)socket, m_hIOCompletionPort, (DWORD)pClientContext, 0); foreach worker thread (I have 8), I post a WSARecvFrom()
step #3 is executed twice... once for the ipv4 socket and once for the ipv6 socket
ISSUE #1: the WSARecvFrom always returns -1 and WSAGetLastError is 997 (IO pending), I don't think thats quite right??
ISSUE #2: in my workerthread, I do:
BOOL bReturn = GetQueuedCompletionStatus(m_hIOCompletionPort, &dwBytesTransferred,
(LPDWORD)&pClientContext, &pOverlapped, INFINITE);
I get past this blocking call and dwBytesTransferred = 3 which is correct since my client just sent "0\r\n".
so I call WSARecvFrom() and again, I get -1 and IO pending? 
|
|
|
|
|
Error 997 / IO_PENDING is not an error. It is the normal return value when no data are available (non-blocking call, WSARecvFrom() returns always immediately). When returned, you have to wait until data has been received and copied to the buffer which is signaled by the socket handle passed to the overlapped struct. After calling WSAGetOverlappedResult() you can call WSARecvFrom() again. Don't call WSARecvFrom() again while the previous call returned the pending state! Before calling it again, the read count and flag parameter variables must be cleared, because they contain values from the previous receive.
That's the reason why I posted a rather big code block. All lines are necesarry to make it work. When using the example code without any processing of the received data, the time consumed by the thread is insignificant. Receiving occurs in the background and the thread is only activated when data has been read into the buffer.
|
|
|
|
|
I think that is what I'm doing ... I am using the network notifications w/ IOCP though. So in my start up I do:
m_hIOCompletionPort = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
when I create the socket, I do:
SOCKET socket = WSASocket(pAddrInfo->ai_family, pAddrInfo->ai_socktype, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
if (bind(socket, pAddrInfo->ai_addr, pAddrInfo->ai_addrlen) == SOCKET_ERROR)
{
...
}
CreateIoCompletionPort((HANDLE)socket, m_hIOCompletionPort, (DWORD)pClientContext, 0);
for (int nIndex = 0; nIndex < m_nThreadCount; nIndex++)
{
SOCKADDR_STORAGE addr;
addr.ss_family = AF_UNSPEC;
addr.__ss_align = 0;
int nAddrLen = sizeof(SOCKADDR_STORAGE);
DWORD dwBytesSent = 0;
DWORD dwFlags = 0;
char szBuffer[MAX_BUFFER_SIZE];
ZeroMemory(szBuffer, sizeof(szBuffer));
WSABUF wsaBuf;
wsaBuf.buf = szBuffer;
wsaBuf.len = sizeof(szBuffer);
OVERLAPPED overlapped;
ZeroMemory(&overlapped, sizeof(OVERLAPPED));
int nResult = ::WSARecvFrom(socket, &wsaBuf, 1, &dwBytesSent, &dwFlags, (sockaddr*)&addr,
&nAddrLen, &overlapped, NULL);
int nError = WSAGetLastError();
}
The worker thread func looks like:
while (WaitForSingleObject(m_hShutdownEvent, 0) != WAIT_OBJECT_0)
{
CClientContext* pClientContext = NULL;
OVERLAPPED* pOverlapped = NULL;
DWORD dwBytesTransferred = 0;
DWORD dwFlags = 0;
WSABUF* pWSABUF = NULL;
BOOL bReturn = GetQueuedCompletionStatus(m_hIOCompletionPort, &dwBytesTransferred,
(LPDWORD)&pClientContext, &pOverlapped, INFINITE);
SOCKADDR_STORAGE from;
int nFromLen = sizeof(SOCKADDR_STORAGE);
memset(&from, 0, sizeof(SOCKADDR_STORAGE));
int nBytesRecv = -1;
nBytesRecv = WSARecvFrom(pClientContext->GetSocket(), pWSABUF, 1, &dwBytesTransferred,
&dwFlags, (sockaddr*)&from, &nFromLen, pOverlapped, NULL);
int j = WSAGetLastError();
int k = 0;
}
}
|
|
|
|
|
Try my code without using GetQueuedCompletionStatus() and waiting for data using WaitForMultipleObjects(). The code has been copied from an existing application.
I have never used GetQueuedCompletionStatus(), because blocking functions may cause problems. E.g. your shut down event will be only processed after the blocking function returns. If the network connection of the system drops, this will never occur.
To test your code, you should check the return value of GetQueuedCompletionStatus(), and test if bytes has been read (dwBytesTransferred).
You should also test your code with only one thread first to ensure that problems are not sourced by using multiple threads. In your code you are starting 8 receives in a loop where each uses the same variables. I'm not sure if that is working.
Another point:
int nBytesRecv = -1;
nBytesRecv = WSARecvFrom(pClientContext->GetSocket(), pWSABUF, 1, &dwBytesTransferred, &dwFlags, (sockaddr*)&from, &nFromLen, pOverlapped, NULL);
int j = WSAGetLastError();
WSARecvFrom() does not return the number of bytes reads. It returns 0 upon success. Upon success you should not call GetLastError() because it returns the most recent error which may be from any previous called function that fails. Alternatively, you can call SetLastError(0) before executing a function that may set an error value.
|
|
|
|
|
Jochen Arndt wrote: I have never used GetQueuedCompletionStatus(), because blocking
functions may cause problems. E.g. your shut down event will be only processed
after the blocking function returns. If the network connection of the system
drops, this will never occur.
That case is actually handled, although I did not post that snippet . The method that sets the shutdown event also posts an I/O completion message so that it exits the blocking function (copied from MSFT sample code).
Jochen Arndt wrote: To test your code, you should check the return value of
GetQueuedCompletionStatus(), and test if bytes has been read
(dwBytesTransferred).
Yes, GetQueuedCompletionStatus() returns true and says there are 3 bytes transferred which is correct since the client is sending "0\r\n".
Jochen Arndt wrote: You should also test your code with only one thread first to ensure that
problems are not sourced by using multiple threads. In your code you are
starting 8 receives in a loop where each uses the same variables. I'm not sure
if that is working.
Yes, for the TCP side of things, this is not an issue because each client is going to new up a CClientContext and get its own buffer.
For the UDP side of things, yes, I understand this will be an issue and that code needs to be refactored / fixed since it will create a CClientContext for the SERVER socket (I just did this to get the code fundamentally going through the same logic flow). Although for now, just testing with a single client sending a single packet .
Jochen Arndt wrote: WSARecvFrom() does not return the number of bytes reads. It returns
0 upon success. Upon success you should not call GetLastError()
because it returns the most recent error which may be from any previous called
function that fails. Alternatively, you can call SetLastError(0)
before executing a function that may set an error value.
Understood. Just for testing since WSARecvFrom is returning -1. .
For testing again, I set m_nThreadCount to 1, so only 1 worker thread is used. Same issue. dwTransferredBytes = 3. BufferLen = 256, Data is empty. Return value is -1. from is not filled. pOverlapped has everything 0 except the Internal flag which is set to 259.
WSAGetLastError = 997.
I will experiment with your code some more unless you have other ideas?
|
|
|
|
|
One simple question: Is data empty before you call the next RecvFrom()? If not, all is working. The next RecvFrom() call returns the pending state when there are no new data (the previous packet has been already received).
|
|
|
|
|
Yes, the WSABuf is initialized and empty before each WSARecvFrom call. It always returns -1 / 997.
|
|
|
|
|
Made an interesting discovery!! Changed the code to look like:
pOverlapped->hEvent = WSACreateEvent();
int nBytesRecv = WSARecvFrom(pClientContext->GetSocket(), pWSABUF, 1, &dwBytesTransferred,
&dwFlags, (sockaddr*)&from, &nFromLen, pOverlapped, NULL);
WaitForSingleObject(pOverlapped->hEvent, INFINITE);
Now, when I send it "0\r\n" ONE time, it blocks on the WaitForSingleObject forever. If I send it a SECOND "0\r\n", it now passes the WaitForSingleObject() and has the 0\r\n in the buffer! what the heck??? Its like one behind or something.
|
|
|
|
|
Hmm.. another discovery . It is not one packet behind. It is just not getting the first packet for some reason. If I send it "0\r\n", "1\r\n", "2\r\n", 1 and 2 work, but 0 does not.
|
|
|
|
|
ARGH!! found the problem. I had a fundamental misunderstanding of the logic flow when using I/O completion ports. The TCP side was working, but not in the way I thought it was . The UDP side is also working now.
Basically with I/O completion ports, you ALWAYS get a IO pending on sends & recvs. The GetQueuedCompletionStatus() function operates like a wait. On the TCP side, I created a new CClientContext for each connection which is correct and thus the WSABUF was passed around correctly. On the UDP side, I can't have the shared buffer obviously because it all comes from multiple clients at the same time. So I need to derive a new struct from OVERLAPPED and add the shared members there for UDP and pass that around.
Basically once you get the GetQueuedCompletionStatus() notification, the buffer is already full (assuming you still have access to it -- in my current design, I didn't). Then you call wsarecvfrom AGAIN and it'll start a new overlapped call until new data is available, then GetQueuedCompletionStatus() will return again and so on.
Thanks again for your help.
|
|
|
|
|
Fine that you find the problem.
SledgeHammer01 wrote: So I need to derive a new struct from OVERLAPPED and add the shared members there for UDP and pass that around.
I expected something like that.
|
|
|
|
|
I have ported a DLL project from VC6 -> VS2010. It gives the following link error.
LINK : error LNK2001: unresolved external symbol __DllMainCRTStartup@12
How to resolve this error?
|
|
|
|
|
|
it compiles in vc6. Only in vs2010 link error.
|
|
|
|
|
so, check your vs2010 settings. confirm that it's linking with the MS runtimes.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
