Ah, that is at least part of your problem. You have not yet understood the fundamental concept of using overlapped IO. You "were" using a thread for each pipe. When you switch to using overlapped IO, you drop the "thread per pipe" design. You have a single thread and use overlapped IO to read/write from all the pipes in that single thread. This aspect is part of the optimization that makes overlapped IO more efficient.
First of all, I realize I had a lack of knowledge on named pipes and asynchronous IO when I started coding both softwares.
I designed a solution where I would use one thread per pipe and use overlapped asynchronous IO.
I now know that it is not the way to do it.
Here is my new question:
Is there a difference, in terms of performance, to using one thread that does all the IO with overlapped structures compared to using one thread per pipe without using overlapped structures?
Note that I say 'overlapped structures' but I also mean creating the named pipes with or without FILE_FLAG_OVERLAPPED.
Anyway, I have rearranged my code so that I can use the overlapped structures or not use them by changing the value of a parameter I put in my config (.ini) file.
That being said, it looks like using or not using overlapped structures does not solve my problem.
My softwares can communicate with each other without any problem when I run them locally, but when I try to do it over a network, WriteFile returns with TRUE and says that 153268 bytes have been written, but on the other side, ReadFile returns TRUE and says it has read 4292 bytes. And its not like ReadFile reads what it can for the remainder of the timeslice so that sometimes it reads 2000 bytes, sometimes 3000 and sometimes 4292, no, its always 4292 bytes exactly.
Im baffled by this but I will continue to try to make it work.
Thanks to all you who are helping me.
Benjamin Racette
CAE software developer
racette@cae.com
using or not using overlapped structures does not solve my problem.
Benjamin, what previous experience do you have with networking? Your previous post indicated there was a team, you said "we". There is no indication in your posts of other team members assisting in your effort, why is that?
racette wrote:
And its not like ReadFile reads what it can for the remainder of the timeslice
"timeslice"? What are you talking about?
racette wrote:
My softwares
racette wrote:
ReadFile returns TRUE and says it has read 4292 bytes.
Is that the behavior of the MSDN example applications when you run them?
racette wrote:
using or not using overlapped structures does not solve my problem.
Benjamin, what previous experience do you have with networking?
I don't have much experience with networking. I did some simple network programs using Java and VB but this is my first C++ software that deals with networking. But I don't believe my experience with networking has anything to do with the question I asked. I'm telling you using overlapped or not using it does not solve my problem: what I mean by that is that now that I can turn it on or off, I still see the problem of 4292 bytes when its on and when its off. Ovelapped IO does not change anything to my problem.
led mike wrote:
Your previous post indicated there was a team, you said "we". There is no indication in your posts of other team members assisting in your effort, why is that?
There is in fact a team. We agreed on the design as a team, but I'm the one doing the implementation of the IPC while the other team members focus on the actual processing inside the softwares. Also, my team does assist me in my effort, but I feel like everyone has it's own idea on the subject. I've heard plenty of different things and now I'm kinda lost. This thread is a way for me to try and clarify IPC with named pipes over network.
led mike wrote:
racette wrote:
And its not like ReadFile reads what it can for the remainder of the timeslice
"timeslice"? What are you talking about?
I'm talking about threads. I have one thread for evey pipe I have. So when I try to send 153268 bytes through a pipe, a thread does the actual work and I know that a thread might not do all it's work in one timeslice. Maybe the thread will start it's work then the OS will put it to sleep while some other thread is wake to perform its own task. What I meant in my other reply was: the function ReadFile returns TRUE and says that 4292 bytes have been read. And it's not due to the fact that the thread calling the ReadFile function is put to sleep by the OS and ReadFile has had time to read only 4292 bytes when it is put to sleep, NO, it's something else. And ReadFile ALWAYS says that it has read 4292 bytes, ALWAYS.
led mike wrote:
Is that the behavior of the MSDN example applications when you run them?
I'll take a look at that and come back to you.
Thanks for your reply.
Benjamin Racette
CAE software developer
racette@cae.com
And it's not due to the fact that the thread calling the ReadFile function is put to sleep by the OS and ReadFile has had time to read only 4292 bytes when it is put to sleep, NO, it's something else.
Yes it's called networking and therefore....
racette wrote:
But I don't believe my experience with networking has anything to do with the question I asked.
.... that statement is dead wrong. I don't know how to put this any simpler. Networking and threading requires you actually know stuff. You can't just drag and drop some VB controls on a form and get it to work, period.
As I stated earlier, you are lacking understanding of some fundamental issues and unless you gain that understanding you will continue to struggle. I strongly urge you to stop working on your project and do some studying of network programming issues. This would include reading and then writing, executing and observing networking code in research projects. Unless of course someone on your team already has the knowledge and experience you require, if so just get them to work with you.
If indeed your team does not have said experience, it seems that the decision to use named pipes is premature.
I strongly urge you to stop working on your project and do some studying of network programming issues. This would include reading and then writing, executing and observing networking code in research projects.
That's what I'm doing right now. I'm doing tests.
Now, there must be tons of material to read on network programming issues, don't you have a particular one that could be of interest to me?
Benjamin Racette
CAE software developer
racette@cae.com
Networking and threading requires you actually know stuff. You can't just drag and drop some VB controls on a form and get it to work, period.
Well, I do know a lot about threading and the different synchronisation mechanism in the win32 API. So this part I know. My inexperience relates only to the network part.
led mike wrote:
If indeed your team does not have said experience, it seems that the decision to use named pipes is premature.
Yes I agree with you. I'm an intern here and the decision to use named pipes was made more or less without my agreement. It seemed like everyone here wanted to use them, so I have to work with that even though I never used them.
Thanks.
Benjamin Racette
CAE software developer
racette@cae.com
I strongly urge you to stop working on your project and do some studying of network programming issues. This would include reading and then writing, executing and observing networking code in research projects.
I did some tests and discovered that I don't get the problem of 4292 bytes being sent and received when I switched to PIPE_TYPE_MESSAGE instead of PIPE_TYPE_BYTE. However, I think I went back to synchronous I/O with PIPE_TYPE_MESSAGE since I read somewhere that this type of pipe enables a mode known as 'write-through' (FILE_FLAG_WRITE_THROUGH option).
Can someone confirm this?
Also, is there a way to achieve asynchronous overlapped I/O with PIPE_TYPE_MESSAGE pipes?
Thanks!
Benjamin Racette
CAE software developer
racette@cae.com
However, I think I went back to synchronous I/O with PIPE_TYPE_MESSAGE since I read somewhere that this type of pipe enables a mode known as 'write-through' (FILE_FLAG_WRITE_THROUGH option).
Have you read the documentation for CreateNamedPipe? That statement seems to contradict the documentation.
Have you read the documentation for CreateNamedPipe?
Yes, of course. As I said already, I read all those MSDN pages a lot of times each.
I found the article where it is mentionned that read and write operations on a message-type named pipes are treated as if WRITE_THROUGH mode was enabled.
I found the article where it is mentionned that read and write operations on a message-type named pipes are treated as if WRITE_THROUGH mode was enabled.
The description of the FILE_FLAG_WRITE_THROUGH option goes like this:
Write-through mode is enabled. This mode affects only write operations on byte-type pipes and, then, only when the client and server processes are on different computers.
No matter what the answer is, is it possible to achieve asynchronous I/O with message-type named pipes?
Yes for read operations. For write operations I don't know if will block or not. However what seems more significant is that the efficiency will not occur:
If this mode is not enabled, the system enhances the efficiency of network operations by buffering data until a minimum number of bytes accumulate or until a maximum time elapses.
That information also speaks to the different ways data transmission libraries use buffering that may or may not effect user code that is reading/writing using the libraries which is what you have been seeing.
Now in your early posts you indicated that your team decided to use named pipes for efficiency reasons. Following that decision I would not think the write through option is appropriate for your solution so your concern over overlapped operations in that mode seems inappropriate.
Same principles apply to pipes (which very well may be using sockets underneath).
Like I stated in my other reply to you here, you need to check if all the bytes you request to send and receive
actually get sent and received. If not, then you need to continue sending and receiving until you do.
You are correct of course but Benjamin has some "fundamentals" problems that he needs to get straight before any of that matters. See my last post, it's a common theme (struggling to grasp the concepts) that occurs with an introduction into both subjects, threading and asynchronous operations. Actually for me, anytime I have a gap in time of doing either, I go through the struggle each time I have to do it again.
it's a common theme (struggling to grasp the concepts) that occurs with an introduction into both subjects, threading and asynchronous operations.
Indeed
led mike wrote:
Actually for me, anytime I have a gap in time of doing either, I go through the struggle each time I have to do it again.
Heh it's like that for me with anything. That's probably the biggest reason I respond to questions here - so I can keep
a little sharp on stuff I don't use often. Graphics, asynchronous I/O, and threading, however, I do every day, all day.
I tried something like that but it would not work. I put the ReadFile in a while loop and I changed the offset of the buffer and the number of bytes to read by an amount I got from a call to GetOverlappedResult but it gave me weird results.
I guess led mike is right when he says that there is fundamental things I need to understand before getting things right.
Thanks for your comment and suggestion.
Benjamin Racette
CAE software developer
racette@cae.com
Does a call to GetOverlappedResult() keeps on reading or writing or should I call WriteFile or ReadFile again and again until its done?
You are responsible for checking the overlapped result and sending/receiving any remaining bytes.
FWIW, if you are calling GetOverlappedResult() right after a function call fails with ERROR_IO_PENDING,
then you're essentially doing synchronous I/O, and all you've done is added complexity to the code.
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
![Java | [Coffee]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/coffee.gif)
![Beer | [beer]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/beer.gif)
