|
I don't know what your requirements for the project are, but here are some suggestions:
1) If you can, try to take a sampling of the data. Instead of pulling in all of it, pull in 3 or 4 sets per second.
2) Buffer the data into a smaller section. Pull in some small set of data (say 13 MB) and write it to a file when it is full. During the write process, block all incoming data until you are done
3) Similar to #2, if you can spawn multiple threads, create a buffer, fill it up, and then pass it to a thread to write it out to a file. While it is writing, you create a new buffer (or keep a pool of buffers around for this -- which would be more efficient) to write to.
4) If none of those will do, and you absolutely must allocate all the necessary memory up front (which, I would question those requirements if that is the case), you will need to create your own heap management (meaning, you will need to override the new and delete operators to suit your needs). This is NOT recommended and again, if you really think this is what you need, then I would question your requirements.
Where are you receiving these samples from? That is, are they coming over a serial port? ethernet port? USB/firewire? hard drive?
Keep in mind that when you declare large blocks of memory (such as 1.1 GB), you will be leaving RAM and using virtual memory (meaning that you will be reading/writing to the hard drive continuously for this). From your description of the problem, you will not be able to make use of that since the hard drive write speed can't keep up with your sampling rate.
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week
Zac
|
|
|
|
|
From a long time I am thinking on the same lines as your 3rd suggestion. But, I doubt that at somepoint the buffer will over run or we may skip data (as the HDD speeds are very very low compared to acquiring speeds). Actually the data of the 3 channels comes from a National Instruments Data Acquisition PCI board, the 3 channels (VIdeo signal, Hsync and Vsync)are connected to the board. I have seen that SCSI drives can write at around 150MB/s speeds. If I can get my SCSI drive to drive at that speeds, I think my problem will be solved for the most part. Maybe now I am thinking more onto hardware side, do you think I can improve my drive speeds. I dont understand why it has such slow speeds while it is an U320 SCSI interface (ofcourse its not in any RAID array configuration, its the only drive the system is having) and being a workstation system (dell PWS 650).
wil keep trying your other suggestions too.
thanks,
-Pavan
|
|
|
|
|
So this information is coming from an PCI board? I'm surprised you get that high of a data rate using PCI ... but anyway. Try this:
1) Create a pool of 10-20 byte buffers to recieve your data.
2) Pull data from your device and load it into the first buffer.
3) When the buffer is full, pass it off to a thread to write out the data
4.1) Your main thread will select the next available buffer and continue to place the received data in it.
4.2) Your write theads will write the data to a file (presumably locking the file so you don't get sync errors) and then release the buffer back to the available pool.
5) Repeate 2-4 until complete.
Your buffers will need to be small enough that you can easily allocate the memory for them, but large enough so that your write operations will be optimal (that is, if your drives write at 13 MB/sec, 13 MB might be a good size for you). If you want to get really daring, you might try to create a set of thread pools as well (just to gain some efficiency by not creating and closing threads constantly).
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week
Zac
|
|
|
|
|
Hi,
I tried both ways, but still either I am not able to keep up with the read speed or the porgram hangs by taking up 100% of processor. Here are the 2 ways I tried for-
Method1-
1) Created 5 different same size buffers for each of the 3 channels (so 3x5 = 15 different buffers) where the buffers for channel one are of size 100000*6bytes each and that of channels 2&3 are of 100000*1byte each.
2) Created 15 threads to process each of those 15 buffers whenever they are ready (initiated by the main acquisition thread as soon as a buffer becomes ready).
3) Created different binary files to write the corresponding buffer data (to overcome the file access issue).
-In this case the program hangs as soon as I run it and locks the processor at 100% (I think its due to the pool of threads ruuning at the same time).
Method2-
1) Created only 3 threads (for each channel data) to write any of the 5 buffers (allocated in the same fashion as before) depending upon their corresponding buffer flags (again initiated by the main acquisition thread, same as before).
2) Created only 3 files to write the data into (since at any given point of time, only 1 buffer is being accessed by the corresponding threads).
-In this case too, my porcessor is being locked up for a while and at some point the data acquisition board gives out buffer overrun error.
Here is a sample of how I am writing my thread-
DWORD WINAPI StartAThread0(LPVOID param)
{
wwrite_ASignal0(); //for buffer[0] of channel1
return 0;
}
void write_ASignal0()
{
int i, dataA, dataA1 = -8191;
do
{
if (m_bUpdateA[0])
{
for (i=0; i<100000; i++)
{
if (dataAI[0][i] < 0)
fwrite(&dataA1, sizeof(int), 1, streamA[0] ;
else
{
dataA = (int)((dataAI[i][0]-1.25)*6552.8);
fwrite(&dataA, sizeof(int), 1, streamA[0] ); }
}
m_bUpdateA[0] = false;
}
}while(m_bInitialize);
}
Where m_bInitialize keeps the thread running waiting for m_bUpdate[0] to activate. As soon as it does it writes the buffer to the corresponding file and deactivates the flag. In the sam fashion I am having for all the other 14 threads.
Here is my application to be exact-
The board is a Nation Instruments Data Acquisition board which can acquire data at a max of 10MHz. The data is acquired using their own native commands in C. The minimum buffer size I can use to acquire data is 10000 and then 100000. Any number other than these will give out buffer overrun error. So if I use 100000 samples, it means that the board acquires data at 10000000 samples per sec in small chucnks of 100000samples/channel, while syncronizing b/n the board memory and the memory allocated by us. At last the baord acquires 100 chucnks of these data packets to finish off 10MSamples/channel. So, in 1000msec it is acquiring 100 sets of 100000data samples/channel = 10msec/set/channel. When I test my present program for the time it takes to write 100000samples of data on to HDD it shows 14-16msec (so I am having a lag of atleast 4msec, which does have a large effect I think).
This is my observation till now. Plz. do lemme know if you come up with a different idea for implementing this thing.
thanks,
-Pavan
|
|
|
|
|
Well, here's your problem:
fwrite(&dataA, sizeof(int), 1, streamA[0] );
You need to reorganize this into a single write - have your buffers as a coninuous block of bytes and do the single operation in a single WriteFile call.
Also, try using CreateFile, Readfile, and WriteFile and avoid the 'C' functions, which have to get decoded into one of the other three calls anyways.
I usually have two layers in a case like this:
Aquisition -> Buffer Writes
Buffer Read -> Disk Writes
You just need to synchronize the access t your 'buffer pointers' between the threads.
Only two threads required, one doing IO and one doing disk writes.
Also, defragment hte hard drive frequently and defragment the paging file.
Turn off the Virus Scnaner if possible, at least on the folder doing the disk IO.
Any sufficiently gross incompetence is nearly indistinguishable from malice.
|
|
|
|
|
Hi,
thanks for your reply. I tried to use WriteFile, but how can I write non-text data using this command. It writes text data fine, but not any other datatypes. Also where can I set the mode to binary while creating the file using CreateFile (I have seen lot of flags, but none relating to binary mode).
One more thing I would like to point out is, my porgram does I/O operations in a single function (it acquires the 2 digital channels data at the same time it acquires analog channel data). So I need to do the disk write operations for the 3 channels data at the same time within the time delay of next I/O or atleast within a span of 5 buffers. So, atlast my read-in rate is (100000x6bytes + 2x100000x1byte) data every 10msec. So I need to write that much data to 3 different files at the same time.
I am using fwrite(&dataA, sizeof(int), 1, streamA[0] );, instead of writing the buffer in a single function, as I need to convert the datatype from 6bytes to 2bytes (using some scaling function) and from 1byte to 2bytes data. But I can do that even after I do the acquisition, if it can resolve my problem.
thanks,
-Pavan.
|
|
|
|
|
The low level File IO does not 'care' about text versus binary. Createfile always makes a binary data file. You only make it a 'text' file by choosing to add CRLF or LF or LFCR or whatever to the 'end' of a line.
Youc an write out an entire block of memory with a single functionc all to Writefile. You are killing your performance with the thousands of individual fwrite calls.
Any sufficiently gross incompetence is nearly indistinguishable from malice.
|
|
|
|
|
I did time comparison between the two functions fwrite and writefile for writing 800000 bytes of data and they both take same amount of time (among 20 trails).
I tried using 1 thread for writing all the 3 channels data using 10 buffers for each channel (once the IO records data into 1 buffer, it updates the corresponding buffer flag and as soon as it updates the write thread writes the corresponding 3 channels buffers to file and updates the correspoding flags to false. I tried this for 10 times for acquiring data about 5sec, in those 3 times it gave out files of same size and 3 times it gave out different size files (so I think it is missing some data in between while writing). Will check and try different methods of implementation and see if I can acheive it.
thanks,
-Pavan
|
|
|
|
|
hi, i am trying to write an IPC program using a named pipe. The basic premise is that there should be two "chat" console applications (basically a client and a server) and they should be able to pass messages until the client types "goodbye" so far i can only get the client to chat to the server and not the other way around.The code i have is listed below. Please remember im only new to this and if i annoy anyone with my stupidity im sorry but i would really appreciate some help and guidance on how i can make both applications send and recieve messages to one another
**********************************************************<br />
<br />
#include <windows.h><br />
#include <iostream><br />
#include <assert.h><br />
#define BUFFER_SIZE 1024 //1K<br />
<br />
using namespace std;<br />
<br />
void main(){<br />
cout << "Server Process Running" <<endl;<br />
<br />
<br />
<br />
HANDLE hNamedPipe = CreateNamedPipe("\\\\.\\pipe\\josephpipe",
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_MESSAGE |
PIPE_READMODE_MESSAGE |
PIPE_WAIT,
PIPE_UNLIMITED_INSTANCES,
BUFFER_SIZE,
BUFFER_SIZE,
NMPWAIT_USE_DEFAULT_WAIT,
NULL);
assert(hNamedPipe !=INVALID_HANDLE_VALUE);<br />
<br />
SetConsoleTitle ("Chat_1");
<br />
<br />
<br />
BOOL bStatus = ConnectNamedPipe(hNamedPipe, 0);<br />
assert(bStatus !=0);<br />
cout << "Client Has Connected" << endl;<br />
<br />
<br />
<br />
DWORD dwNumberOfBytesRead;<br />
char clientMessage[80];<br />
do{<br />
<br />
bStatus = ReadFile( hNamedPipe,<br />
clientMessage,<br />
79,<br />
&dwNumberOfBytesRead,<br />
0);<br />
assert(bStatus!=0);<br />
<br />
<br />
clientMessage[dwNumberOfBytesRead] = '\0';<br />
<br />
<br />
cout << "Client Message is: " <<<br />
clientMessage <<<br />
endl;<br />
}<br />
<br />
DWORD dwNumberOfBytesWritten;<br />
BOOL bStatus;<br />
char serverMessage[81]; <br />
do{<br />
cin.getline (serverMessage, 81, '\n');<br />
bStatus = WriteFile(hNamedPipe,<br />
serverMessage,<br />
strlen(serverMessage),<br />
&dwNumberOfBytesWritten,<br />
0);<br />
assert(bStatus !=0);<br />
}<br />
while (strcmp("goodbye", clientMessage) !=0);<br />
<br />
<br />
<br />
bStatus = DisconnectNamedPipe(hNamedPipe);<br />
assert(bStatus !=0);<br />
CloseHandle(hNamedPipe);<br />
<br />
cout << "Goodbye" << endl;<br />
}
<br />
<br />
<br />
********************************************************** <br />
<br />
<br />
#include <windows.h><br />
#include <iostream><br />
#include <assert.h><br />
#pragma warning (disable:4267) /*originally i had issues involving a warning of type '4267' <br />
which was dealing with t_size and DWORD issues, since t_size<br />
is for win64 and DWORD is win32 so the two types do not work well together<br />
however simply disabling the error was enough to allow me to run<br />
the program for the mean time however i eventually resolved the issue<br />
without the need for the disable command however left the code in<br />
as a means of proof as to my builds */<br />
<br />
using namespace std;<br />
<br />
void main(){<br />
cout << "Client Process Running" << endl;<br />
<br />
<br />
HANDLE hNamedPipe = CreateFile("\\\\.\\pipe\\josephpipe",
PIPE_ACCESS_DUPLEX,
1,
NULL,
OPEN_EXISTING,
0,
NULL);
assert(hNamedPipe !=INVALID_HANDLE_VALUE);<br />
<br />
SetConsoleTitle ("Chat_2");
<br />
<br />
<br />
DWORD dwNumberOfBytesWritten;<br />
BOOL bStatus;<br />
char clientMessage[80];<br />
do{<br />
cin.getline (clientMessage, 80, '\n');<br />
bStatus = WriteFile(hNamedPipe,<br />
clientMessage,<br />
strlen(clientMessage),<br />
&dwNumberOfBytesWritten,<br />
0);<br />
assert(bStatus !=0);<br />
}<br />
while (strcmp("goodbye", clientMessage) !=0);<br />
cout << "goodbye" <<endl;<br />
}
If i'm not dead enough for life, am i alive enough for death?
|
|
|
|
|
Hi,
I want to set my dialog window background with JPG image. I want to set that image to fit in the entire window size.
I tried to use an ACTIVE-X control( Microsoft Picture Clip control)
But I cannot draw that image.
Is there any other way to make it?
Or can anyone provide info. abt Microsoft Picture Clip control & its usage?
Thanks in advance....
vinsankar
|
|
|
|
|
See here
<br />
#include AtlImage.h<br />
m_Image.Load("c:\\nature\\1.jpg");<br />
<pre>
CRect Rect;
GetWindowRect(Rect);
ScreenToClient(Rect);
m_Image.BitBlt(pDC.m_hDC,Rect.left,Rect.top,Rect.right,Rect.bottom,0,0);
</pre> <br />
<br />
<div class="ForumSig"><font size="2" color="#99ff66"><hr></font><div style="padding: 2px; text-align: left; color: rgba(255, 255, 255, 1); width: 80px"><font size="2" color="#99ff66">WhiteSky<br />
</font></div><hr></div>
|
|
|
|
|
Hi,
thanks .....
But one more doubt...
how can I create this "m_Image" object ?
Whether I need to insert any ATL object?
help me.....
vinsankar
|
|
|
|
|
Did you see MSDN
From the MSDN
"You can use CImage from either MFC or ATL.
Note When you create a project using CImage, you must define CString before you include atlimage.h. If your project uses ATL without MFC, include atlstr.h before you include atlimage.h. If your project uses MFC (or if it is an ATL project with MFC support), include afxstr.h before you include atlimage.h.
Likewise, you must include atlimage.h before you include atlimpl.cpp. To accomplish this easily, include atlimage.h in your stdafx.h."
|
|
|
|
|
Dear all,
I have created a window with three button control and 3 static text control in a win32 application.
Now I need to display the tooltips when i place the mouse over this button...So how to display the tooltips in win32 appication...
thanking you...
Manjunath S
GESL
Bangalore
|
|
|
|
|
You'll need to respond to the TTN_NEEDTEXT (or TTN_GETDISPINFO) notification.
"Money talks. When my money starts to talk, I get a bill to shut it up." - Frank
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
See TTM_ADDTOOL and seethis[^] and OnButton() maybe it is some helpful to you
|
|
|
|
|
Formerly, in the tc environment, i used the inport/outport functions to acheive my goal.But now, when i move to the vc++ 6.0 environment, how can i read/write a word from/to a physical address?
Please help me!
|
|
|
|
|
zhoufanking wrote: how can i read/write a word from/to a physical address?
Are you sure that the HAL will allow you to do this? What is it that you are trying to accomplish? Have you looked into _inp() and _outp()? There's also the WinIO library.
"Money talks. When my money starts to talk, I get a bill to shut it up." - Frank
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Hi:
Thank you for your help! In my program i need to read/write data from/to a digital I/O, so i need this function.
|
|
|
|
|
Under Windows NT/2000/XP/2003, user applications are not allowed to access I/O ports directly. I/O ports are allowed to be accessed only by device drivers.
If you have purchased a piece of hardware that you wish to control, then you need to contact the manufacturer for a driver and interface software for it.
If you are building custom hardware, then you will need to write a device driver using the Windows Driver Kit[^] or WDK, formerly called the DDK.
Software Zen: delete this;
|
|
|
|
|
you should use driver and call its functions .
|
|
|
|
|
Hi ,
I want to write a VPN client in C++. I am wondering if someone can tell me what is involved in this or maybe point me at some whitepaper on how to do this. I know that VPN servers work on several protocals ie. L2PP, IPSEC, PPP etc. but I am not sure how to connect to a server using these protocal or what is required in terms of sending and recieving.(ie, handshaking) in VC++.
I have looked at the web and couldn't really find an open source Windows VPN program to study.
Thanks in Advance
S
|
|
|
|
|
_mss wrote: I am wondering if someone can tell me what is involved in this or maybe point me at some whitepaper on how to do this.
There are several RFCs available, each covering a specific topic.
"Money talks. When my money starts to talk, I get a bill to shut it up." - Frank
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Dear Sir;
I am using Visual c++ 6.0 with sql server 2000 as database.
I have one view on wich there are 3 buttons : Update button, clear and search button.
One time, when I search some thing, i can change other fields of my screen and I click on Update button to update my database . But there is some thing strange.
even if I update my database, when I search for second time the same record, I don't see those modifications but when I open manually the database I see that the record is changed. Then where comes that from the program it show me the first record(without changes) ?.
What to do?
|
|
|
|
|
hi i want to make tha packet filtering application using packet filtering api's
i m using code below
if(::PfCreateInterface(0,PF_ACTION_DROP,PF_ACTION_FORWARD,false,true,&ihandle)==NO_ERROR)
{
AfxMessageBox("Step 1 Done");
if(::PfAddGlobalFilterToInterface(ihandle,GF_FRAGMENTS)==NO_ERROR)
{
AfxMessageBox("Step 2 done");
if(::PfAddGlobalFilterToInterface(ihandle,GF_STRONGHOST)==NO_ERROR)
{
AfxMessageBox("Step 2 done");
if(::PfAddGlobalFilterToInterface(ihandle,GF_FRAGCACHE)==NO_ERROR)
{
int i[]={202,165,250,50};
AfxMessageBox("Step 2 done");
PBYTE ip=(PBYTE)i;
if(::PfBindInterfaceToIPAddress(ihandle,PF_IPV4,ip)==NO_ERROR)//here it fails all other works fine i dont know what the problem is.
{
AfxMessageBox("IP Adress Binded");
}
}
}
}
}}
thanks in advance.
Tasleem Arif
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
