Asynchronous overlapped I/O - C / C++ / MFC Discussion Boards

Hakan Bulut wrote:
Sorry

how does that help?

Hakan Bulut wrote:
I could not remember

Then go find out first so you can use the correct terms when posting questions.

Assingment.. I just to mean assing a variable to an integer pointer itself where in the sNumber structure and to objective oriented these elements to using easy?.. Have you got an idea OOP method?

Was is das¿

sNumber& DrawNumber() //pointer itself.

What does "pointer itself" mean? Your function returns a reference

{

sNumber* pNum = new sNumber;
pNum->i1 = edi1;
pNum->i2 = edi2;

You need to return a value from the function.

}
sNumber& sn = DrawNumber();
for(int i=0; i<2; i++)

The DrawNumber function (supposedly) returns a reference to a sNumber object.
How can you iterate through an array of two objects??

{
int x1 = sn.pNum[i];
int x2 = sn.pNum[i];

There's no such thing as "pNum" here.

}

Pretty much none of this makes sense Smile | :)

Mark

Mark Salsbery
Microsoft MVP - Visual C++
Java | [Coffee]

Yeah how can i iterate thorough an array of two object. I just to mean that. So "sn" what has been refered to? What should i it to do? Can you help me?

"The DrawNumber function (supposedly) returns a reference to a sNumber object." I can see it. Anyway I asked this question myself. I don't asking about this function does returns a reference to a sNumber object. [I just ask how can i appoint an integer variable to an integer pointer sn referred object.?]
What do you say? it's no required?

-- modified at 13:52 Wednesday 14th November, 2007

Was is das¿

Hakan Bulut wrote:
how can i iterate thorough an array of two object.

You need to have that array to begin with.

I'm not sure where to start here....this is really C/C++ fundamentals that I'm not sure if you know or not.

Mark

Mark Salsbery
Microsoft MVP - Visual C++
Java | [Coffee]

A project which took 2.5 mins to compile is now taking more like 12 mins, and I've noticed that the PCH file is 22MB where it used to be 11MB. Could that be part of the slowdown? How much of an effect does the size of a PCH file have on compilation? Does it get loaded when each individual file is compiled or just once at the beginning? I'm being driven nuts waiting for it to compile when I've only made a small change to one header file!

TIA.

PS - it's about a 280kloc project in VC6.

delete it and start a clean + rebuild all. it's a known bug that its size a getting bigger and bigger sometimes.

[VisualCalc][Binary Guide][CommDialogs] | [Forums Guidelines]

Hi C++ coders,

Im developping two softwares that need to communicate with each other over a network. The IPC mechanism we chose is the named pipes. Both softwares are being developped in C++.

When I did my preliminary tests, everything went fine but performance was not that good. I read that I needed to send and receive data asynchronously, which made sense to me.

So I changed everything to incorporate asynchronous overlapped I/O.

Here is what my softwares do:

Software A creates about a dozen named pipes.

Software B connects to those pipes.

Now, software A sends data through pipe #1. Pipe #1 was designed to send and receive blocks of 112 bytes. This data is received correctly by software B.

Up to here, everything is ok.

Then, software A sends data through pipe #2. Pipe #2 was designed to send and receive blocks of 153,268 bytes. This data is NOT received correctly by software B.

When I do my tests locally (on one pc running both softwares), this block of 153,268 bytes is sent and received correctly.

So my assumption is that there is something wrong with sending and receiving large blocks of data through named pipes over a network.

I read on different forums that there is a limit on the size of data that can be sent through named pipes over a network.

I tried everything I could find on the Internet, but didnt find anything that would work.

My questions are:

If a ReadFile or WriteFile call returns false, and that GetLastError() returns ERROR_IO_PENDING, what should I do to make sure the data is written or read entirely?

Should I call GetOverlappedResult()?

Does a call to GetOverlappedResult() keeps on reading or writing or should I call WriteFile or ReadFile again and again until its done?

Should I wait for the event that was specified to the Overlapped structure?

If someone asks, I can post the code I use.

Thanks.

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
The IPC mechanism we chose is the named pipes.

racette wrote:
If a ReadFile or WriteFile call returns false, and that GetLastError() returns ERROR_IO_PENDING, what should I do to make sure the data is written or read entirely?

Should I call GetOverlappedResult()?

Does a call to GetOverlappedResult() keeps on reading or writing or should I call WriteFile or ReadFile again and again until its done?

Should I wait for the event that was specified to the Overlapped structure?

So you chose Named pipes before you knew how they worked? Interesting. There should be numerous articles and tutorials on using Overlapped IO on MSDN and probably here in the articles on CodeProject. Look around sometime, you might find you get what you need.

led mike wrote:
So you chose Named pipes before you knew how they worked? Interesting.

I said WE chose since I wasn't the only one deciding.

Also, the choice of named pipe was made for performance issue rather than our knowledge of the technology.

led mike wrote:
There should be numerous articles and tutorials on using Overlapped IO on MSDN and probably here in the articles on CodeProject. Look around sometime, you might find you get what you need.

I have read a lot on forums and on MSDN but it seems like there is conflicting information; what I mean is that different sources offer different solutions, but none of them seems to do the trick for me. Go to MSDN and look at the GetOverlappedResult() function documentation. When I read it, it seems like that function doesn't do anything except return the result of the last overlapped operation that used the overlapped structure, but on other sources, from what I understand, it seems that GetOverlappedResult() does continue to read or write. I just want accurate information thats all.

Thanks!

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
I said WE chose since I wasn't the only one deciding.

Also, the choice of named pipe was made for performance issue rather than our knowledge of the technology.

I said you as opposed to you all but that is what I meant. However I find your statement confusing. You (all) choose it form performance reasons without knowledge of the technology? So I guess that means you read some claims about performance and so you (all) decided? Or what?

racette wrote:
I just want accurate information thats all.

Then I suggest you read the documentation

On the page for GetOverlappedResult, at the bottom, there is section titled "See Also" and in there is a link titled "Overlapped Input and Output". At the bottom of that page it says:

For examples that illustrate the use of overlapped operations, completion routines, and the GetOverlappedResult function, see Using Pipes.

where "Using Pipes" is a link to a page that has multiple example links. There are several including one for using Overlapped IO and another for using Completion routines. If you haven't figured this out yet, you should read both.

Please keep in mind these examples illustrate and comment on how the API's work. They are NOT an example of a good software design using the API's.

led mike wrote:
So I guess that means you read some claims about performance and so you (all) decided? Or what?

Yes. We found a benchmark showing that named pipes could accomplish what we want with the performance we want.

led mike wrote:
Then I suggest you read the documentation

As I said already, I have read a lot before posting here. Of course I read those articles you mention, a couple of times each, but I still can't figure out how to do it. What I can see is that overlapped IO is used A LOT in the win32 API, not only with named pipes, so the examples rarely apply to what I am doing.

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
Of course I read those articles you mention, a couple of times each

Good that gives us some reference.

racette wrote:
but I still can't figure out how to do it.

Ok let's work off the examples and you point out what you don't understand.

Last modified: 3mins after originally posted -- ooops... clicked the wrong button

led mike wrote:
Ok let's work off the examples and you point out what you don't understand

ok.

First, I created my named pipes with the FILE_FLAG_OVERLAPPED and PIPE_WAIT parameters. Then, for each call to ReadFile and WriteFile, I specify a distinct OVERLAPPED structure in which I specified a distinct event.

After each call to WriteFile and ReadFile, I verify the return value. If its TRUE, then the data has been sent/received correctly. If it's FALSE, then I verify the value of GetLastError(). If GetLastError() returns ERROR_IO_PENDING, Im not sure what to do. (note that if it's any other kind of error I just close the app)

I want to make sure all the data has been sent/received.

I read that I can either call GetOverlappedResult() or wait for the event I specified to the overlapped structure with a call to WaitForSingleObject().

I tried both, but it does not seem to work. It does work locally (both softwares running on the same pc) but it does not work over the network. I seem to get only 4212 bytes out of my 153,268 bytes.

It is so weird!

Also, I have the perfect example of conflicting information on the MSDN website:

On this web page
http://msdn2.microsoft.com/en-us/library/ms686358.aspx

you can read in the fourth paragraph that "To call a function to perform an overlapped operation, the calling thread must specify a pointer to an OVERLAPPED structure that has had all of its members initialized to zero."

Then on this web page
http://msdn2.microsoft.com/en-us/library/ms684342.aspx

look at the description of the 'Pointer' parameter. It says "Reserved for system use; do not use."

Now what should I do? set it to zero? it's a pointer!!!!!!

The same goes for hEvent.

I just can't seem to find accurate information.

Anyway, thanks for your reply.

Benjamin Racette
CAE software developer
racette@cae.com

Ok well that's a lot of stuff. Perhaps you might like this idea, I suggest you take the example "named pipe server using overlapped IO" and the matching "named pipe client" and build them and execute them and watch. Watching with the debugger might prove difficult due to the async operations but if you add some logging to the code you should be able to get a better understanding of what happens and therefore how this all works.

Thanks for the suggestion.

I saw that example a while ago and though it did not apply to me since I use a thread for every pipe I create.

I like the idea of debugging it (stepping in the code). I will do just that and come back to you.

Thanks!

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
since I use a thread for every pipe I create.

Ah, that is at least part of your problem. You have not yet understood the fundamental concept of using overlapped IO. You "were" using a thread for each pipe. When you switch to using overlapped IO, you drop the "thread per pipe" design. You have a single thread and use overlapped IO to read/write from all the pipes in that single thread. This aspect is part of the optimization that makes overlapped IO more efficient.

First of all, I realize I had a lack of knowledge on named pipes and asynchronous IO when I started coding both softwares.

I designed a solution where I would use one thread per pipe and use overlapped asynchronous IO.

I now know that it is not the way to do it.

Here is my new question:

Is there a difference, in terms of performance, to using one thread that does all the IO with overlapped structures compared to using one thread per pipe without using overlapped structures?

Note that I say 'overlapped structures' but I also mean creating the named pipes with or without FILE_FLAG_OVERLAPPED.

Anyway, I have rearranged my code so that I can use the overlapped structures or not use them by changing the value of a parameter I put in my config (.ini) file.

That being said, it looks like using or not using overlapped structures does not solve my problem.

My softwares can communicate with each other without any problem when I run them locally, but when I try to do it over a network, WriteFile returns with TRUE and says that 153268 bytes have been written, but on the other side, ReadFile returns TRUE and says it has read 4292 bytes. And its not like ReadFile reads what it can for the remainder of the timeslice so that sometimes it reads 2000 bytes, sometimes 3000 and sometimes 4292, no, its always 4292 bytes exactly.

Im baffled by this but I will continue to try to make it work.

Thanks to all you who are helping me.

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
using or not using overlapped structures does not solve my problem.

Benjamin, what previous experience do you have with networking? Your previous post indicated there was a team, you said "we". There is no indication in your posts of other team members assisting in your effort, why is that?

racette wrote:
And its not like ReadFile reads what it can for the remainder of the timeslice

"timeslice"? What are you talking about?

racette wrote:
My softwares

racette wrote:
ReadFile returns TRUE and says it has read 4292 bytes.

Is that the behavior of the MSDN example applications when you run them?

led mike wrote:
racette wrote:
using or not using overlapped structures does not solve my problem.

Benjamin, what previous experience do you have with networking?

I don't have much experience with networking. I did some simple network programs using Java and VB but this is my first C++ software that deals with networking. But I don't believe my experience with networking has anything to do with the question I asked. I'm telling you using overlapped or not using it does not solve my problem: what I mean by that is that now that I can turn it on or off, I still see the problem of 4292 bytes when its on and when its off. Ovelapped IO does not change anything to my problem.

led mike wrote:
Your previous post indicated there was a team, you said "we". There is no indication in your posts of other team members assisting in your effort, why is that?

There is in fact a team. We agreed on the design as a team, but I'm the one doing the implementation of the IPC while the other team members focus on the actual processing inside the softwares. Also, my team does assist me in my effort, but I feel like everyone has it's own idea on the subject. I've heard plenty of different things and now I'm kinda lost. This thread is a way for me to try and clarify IPC with named pipes over network.

led mike wrote:
racette wrote:
And its not like ReadFile reads what it can for the remainder of the timeslice

"timeslice"? What are you talking about?

I'm talking about threads. I have one thread for evey pipe I have. So when I try to send 153268 bytes through a pipe, a thread does the actual work and I know that a thread might not do all it's work in one timeslice. Maybe the thread will start it's work then the OS will put it to sleep while some other thread is wake to perform its own task. What I meant in my other reply was: the function ReadFile returns TRUE and says that 4292 bytes have been read. And it's not due to the fact that the thread calling the ReadFile function is put to sleep by the OS and ReadFile has had time to read only 4292 bytes when it is put to sleep, NO, it's something else. And ReadFile ALWAYS says that it has read 4292 bytes, ALWAYS.

led mike wrote:
Is that the behavior of the MSDN example applications when you run them?

I'll take a look at that and come back to you.

Thanks for your reply.

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
And it's not due to the fact that the thread calling the ReadFile function is put to sleep by the OS and ReadFile has had time to read only 4292 bytes when it is put to sleep, NO, it's something else.

Yes it's called networking and therefore....

racette wrote:
But I don't believe my experience with networking has anything to do with the question I asked.

.... that statement is dead wrong. I don't know how to put this any simpler. Networking and threading requires you actually know stuff. You can't just drag and drop some VB controls on a form and get it to work, period.

As I stated earlier, you are lacking understanding of some fundamental issues and unless you gain that understanding you will continue to struggle. I strongly urge you to stop working on your project and do some studying of network programming issues. This would include reading and then writing, executing and observing networking code in research projects. Unless of course someone on your team already has the knowledge and experience you require, if so just get them to work with you.

If indeed your team does not have said experience, it seems that the decision to use named pipes is premature.

led mike wrote:
I strongly urge you to stop working on your project and do some studying of network programming issues. This would include reading and then writing, executing and observing networking code in research projects.

That's what I'm doing right now. I'm doing tests.

Now, there must be tons of material to read on network programming issues, don't you have a particular one that could be of interest to me?

Benjamin Racette
CAE software developer
racette@cae.com

I just wanted to add a couple of things.

led mike wrote:
Networking and threading requires you actually know stuff. You can't just drag and drop some VB controls on a form and get it to work, period.

Well, I do know a lot about threading and the different synchronisation mechanism in the win32 API. So this part I know. My inexperience relates only to the network part.

led mike wrote:
If indeed your team does not have said experience, it seems that the decision to use named pipes is premature.

Yes I agree with you. I'm an intern here and the decision to use named pipes was made more or less without my agreement. It seemed like everyone here wanted to use them, so I have to work with that even though I never used them.

Thanks.

Benjamin Racette
CAE software developer
racette@cae.com

led mike wrote:
I strongly urge you to stop working on your project and do some studying of network programming issues. This would include reading and then writing, executing and observing networking code in research projects.

I did some tests and discovered that I don't get the problem of 4292 bytes being sent and received when I switched to PIPE_TYPE_MESSAGE instead of PIPE_TYPE_BYTE. However, I think I went back to synchronous I/O with PIPE_TYPE_MESSAGE since I read somewhere that this type of pipe enables a mode known as 'write-through' (FILE_FLAG_WRITE_THROUGH option).

Can someone confirm this?

Also, is there a way to achieve asynchronous overlapped I/O with PIPE_TYPE_MESSAGE pipes?

Thanks!

Benjamin Racette
CAE software developer
racette@cae.com

racette wrote:
However, I think I went back to synchronous I/O with PIPE_TYPE_MESSAGE since I read somewhere that this type of pipe enables a mode known as 'write-through' (FILE_FLAG_WRITE_THROUGH option).

Have you read the documentation for CreateNamedPipe? That statement seems to contradict the documentation.