C / C++ / MFC

18-Jun-09 21:00

We must write file very fast ,so writing speed is no less than 150MB/s.RAID0 is used for raise writing speed.I should create the file with the flag FILE_FLAG_NO_BUFFERING on the beginnging sectors of the disk to ensure so fast writing speed.I use WriteFile to write data.But the writing speed is not stable.Sometimes the speed will reduce to 50MB/s.It seems that when this occurs,the system is busy.
Can anyone can tell me something about this?Any information is appreciated.Thank you.
PS:I use VC8 and the OS is WinXP SP3.

chandu00418-Jun-09 21:07

18-Jun-09 21:07

what is the write speed supported by the RAID controller?
try with other flags also.
what is the size of the buffer you are using in your code?

denghp18-Jun-09 21:27

18-Jun-09 21:27

Thank you for your reply.Well，the RAID controller supports the write speed is no less than 300MB/s.The buffer is 200MB,but each time I write 731136 bytes.Other flags are not suitable for our project,so I can't use other flags.

chandu00418-Jun-09 21:36

18-Jun-09 21:36

denghp wrote:
I write 731136 bytes

any special significance of this figure?

--------------------------------------------
Suggestion to the members:
Please prefix your main thread subject with [SOLVED] if it is solved.
thanks.
chandu.

denghp18-Jun-09 21:38

18-Jun-09 21:38

This num is multi-time of 512 bytes and it is the size of data I want to write.

Stuart Dootson18-Jun-09 22:32

Stuart Dootson

18-Jun-09 22:32

denghp wrote:
But the writing speed is not stable.Sometimes the speed will reduce to 50MB/s.

Welcome to the world of general purpose operating systems....

Your task isn't the only one in the system. You're coming across issues similar to those encountered by people trying to implement real-time functionality on Windows. You'll either need to move to a real-time OS, where you have greater control over what happens, or just find a workaround.

I mean - you want to write 731136 bytes @ 150MB/s? That's over in 4ms? What's the deal? Why do you need to write so fast? Maybe if you share a little more detail, we could suggest a workaround?

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

denghp18-Jun-09 22:46

18-Jun-09 22:46

To Stuart Dootson:
Thank you for your information!
Because we get the data continually from a PCI card and if I can't write the file in time, the buffer is not big enough to hold all the data.Mostly the API WriteFile works fine,but if the system is busy the write speed will slow.
Changing OS is almost impossible,I just want a solution to fix this problem.

chandu00418-Jun-09 22:54

18-Jun-09 22:54

we faced the same problem long ago and we solved it by taking a circular queue.
btw, are you doing all theze in user mode or in kernel mode.
the thread which reads the data from a pci card puts it into a queue and another thread takes the data from the queue and writes it to a file.
since RAID supports upto 300mbps, when the system is busy, the data will get accumulated inthe queue and when the system has come out of the busy mode, the data will get vacated from the queue.
but be careful.
you should use proper synchronization mechanisms between threads.

--------------------------------------------
Suggestion to the members:
Please prefix your main thread subject with [SOLVED] if it is solved.
thanks.
chandu.

Stuart Dootson18-Jun-09 22:58

Stuart Dootson

18-Jun-09 22:58

It's not WriteFile. It's the fact that you do not have 100% of the system to yourself all the time.

BTW - what's the real write speed of your disks? Your write speed will fill the buffer in 1.3 seconds - how long does it take to flush the buffer to disk?

Othe ravenues to investigate - scatter/gather IO[^], asynchronous IO[^]. Also - what happens if you let the system buffer your IO? It's possible that Windows could help you achieve your requirement of averaging of 150MB/s.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

denghp18-Jun-09 23:12

18-Jun-09 23:12

Q:what's the real write speed of your disks?
A:The speed is no less than 150MB/S.

Q:how long does it take to flush the buffer to disk?
A:When I receive data,I will write them to file at once.The buffer is visited by two threads:one reads and one writes.

Asynchronous IO may lose data.

Stuart Dootson18-Jun-09 23:27

Stuart Dootson

18-Jun-09 23:27

denghp wrote:
Q:what's the real write speed of your disks?
A:The speed is no less than 150MB/S.

That may be true on average, when it's able to write sequentially. However, when some other process needs to read or write with the disk, you are going to get a lot slower performance. The other read/write will likely require the head to seek to somewhere else on the disk, which will take ~10ms, then seek back to your file (another 10ms) before starting to write again.

Is the RAIDed partition used purely for writing your data? If not, I'd alter your system so it does work that way (i.e. get a separate disk for Windows, your programs, data etc).

denghp wrote:
Q:how long does it take to flush the buffer to disk?

I meant the buffer(s) in hte RAID controller/hard drive controllers. Never mind.

denghp wrote:
Asynchronous IO may lose data

Really. Got proof for that assertion?

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

creating own gps map in vc++

Michael Schubert19-Jun-09 0:03

Michael Schubert

19-Jun-09 0:03

Try to alter the buffer size, bigger is not always better! It depends on various factors like the controller, hard drive, etc.

Also, for copying large files the flag FILE_FLAG_SEQUENTIAL_SCAN can make quite a difference.

gpsmercato18-Jun-09 20:17

18-Jun-09 20:17

HI,,, i got a garmin 18x usb device and using the sdk I am able to get the gps data lat/lon. now i want to plot this gps data on to my own gps map i dont have any gps map, so is it possible to create so that i can plot the data in it.from some tutorials i got the codes for plotting it in google map which i did already. i just want to make it work on my own map of my town..plz help.

chirag_chauhan18-Jun-09 20:47

chirag_chauhan

18-Jun-09 20:47

Are you having maps in form for pure images(JPG, BMP)?
There are some formulas available on NET to convert lat/long to x/y co-ordinates on basis of width/height & lat/long of ((0,0), (width, height) of map.

Here is the link: http://www.actionscript.org/forums/showthread.php3?t=164956[^]

chandu00418-Jun-09 20:51

18-Jun-09 20:51

sure you can do that.
but answer these.
1.is your town visible on googleearth?
2.are you using visual c++?.
3.if yes, are you aware of device context and drawing routines? like moveto and line to.

if you can answer these, then i can help you out how to achieve your task.
FYI: i have worked on the same.

--------------------------------------------
Suggestion to the members:
Please prefix your main thread subject with [SOLVED] if it is solved.
thanks.
chandu.

modified on Friday, June 19, 2009 2:58 AM

gpsmercato18-Jun-09 21:13

18-Jun-09 21:13

thankyou for your response... the answers
1. yes my town is visible in googleearth
2.yes i am suing vc++
3. yes i am aware of these stuff but didnt worked on it, but i am sure i can work on it with your help..
thank you once again...

chandu00418-Jun-09 21:23

18-Jun-09 21:23

cool,
now take a sample of longitude and latitude.
for example, 78d32m24.88s 17d33m24.21s
now convert these into intregers like
78*60*60*100+32*60*100+24*100+21
--------------------
on the other side,
now your aim is to draw your town map on your screen right?
now you should try to get the digital data related to your town.
this can be obtained in several ways. we will decide which would be easier for you at your circumstances.
first go to google earth and note the longitude and latitudes of the bounding rectangle of your town.
ootherwise you assume them approximately.
let me know them and we shall proceed further
if any doubt in what i tried to explain, please be comfortable.
ans also let me know how much time you have with you. i mean dead line?

--------------------------------------------
Suggestion to the members:
Please prefix your main thread subject with [SOLVED] if it is solved.
thanks.
chandu.

gpsmercato18-Jun-09 22:22

18-Jun-09 22:22

ok hope u had a great lunch...
here is my latitude and longitude from google earth.
25d16min16.10secN and 55d18mi26.95secE
so when i convert it into integers
25*60*60*100+16*60+100+16*100+10= 9097610
55*60*60*100+18*60*100+26*100+95= 19910695
my deadline is on july second week.

chandu00418-Jun-09 22:37

18-Jun-09 22:37

fine then.
could u not find the bounding rectangle of your town.
to find it, just draw a virtual rectangle around ur town in GE (ggl erth)
take the longi and lati of the top left corner of the rect and bottom right corner of the rect.
parallelly, experiment drawing a rectangle(0,0,700,700) on your screen and lines 0,0 to 100,100 to 100,150 to 150,200. if you are stuck up, let me know i shall help u.

gpsmercato wrote:
my deadline is on july second week.

far more than sufficient.

[Message Deleted]

gpsmercato19-Jun-09 0:01

19-Jun-09 0:01

[Message Deleted]

chandu00419-Jun-09 0:06

19-Jun-09 0:06

did it take 1 hr 25 mins for you to create a rectangle ? Big Grin | :-D

donot take such bigger rectangles. Poke tongue | ;-P

take one that you can see on ur screen Wink | ;)

or areu trying to paint it on the ground around ur town? Sniff | :^)

have a sumptuous lunch. i will have a nap till then.

--------------------------------------------
Suggestion to the members:
Please prefix your main thread subject with [SOLVED] if it is solved.
thanks.
chandu.

gpsmercato19-Jun-09 0:34

19-Jun-09 0:34

well ...ya it took long time..actually i went for friday prayer in between......
ok so rectangle with the lines done.
in GE the values are 25d17m28.90sc & 55d18m31.07sec
25d11m32.57sec & 55d24m17.68sec

chandu00419-Jun-09 0:37