C / C++ / MFC

20-Nov-02 10:52

Hi people,
Could anyone point me to some code that loads any file in 10% pieces. i.e. load first 25% of file content, then the next 25% and so on till 100%?

I'm sure this has been done before......

TIA.

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

James R. Twine20-Nov-02 10:59

20-Nov-02 10:59

You mean something like:

<br />
    DWORD dwFileSize = ::GetFileSize( hTheFile, NULL );<br />
    DWORD dw10Percent = ( dwFileSize *.10 );<br />
    BYTE *pBuffer = new BYTE[ dw10Percent ];<br />
<br />
    if( !pBuffer )<br />
    {<br />
        return( ERROR_NOT_ENOUGH_MEMORY );<br />
    }<br />
    DWORD dwStatus = 0;<br />
    DWORD dwRead = 0;<br />
<br />
    dwStatus = ::ReadFile( hTheFile, pBuffer, dw10Percent, &dwRead, NULL );<br />
    if( dwStatus != ERROR_SUCCESS )<br />
    // <br />
    // ...<br />
    // <br />

Peace!

-=- James (Sonork:100.21837)
"There is nothing worse than being oblivious to the fact that you do not know what you are doing."
[Get Check Favorites 1.5 Now!]

Amit Dey20-Nov-02 11:10

20-Nov-02 11:10

Thanks,James.

But for large files, what if enough memory cannot be allocated? Can a smaller buffer(say 5000 chars) be reused?

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

User 988520-Nov-02 11:23

20-Nov-02 11:23

If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files.

My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers

modified 29-Aug-18 21:01pm.

Amit Dey20-Nov-02 11:30

20-Nov-02 11:30

Thanks.

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

James R. Twine20-Nov-02 13:20

20-Nov-02 13:20

Thomas George wrote:
If you do not want the whole file in emeory at one time, then you can go to any small size you want. But, if you actually want the whole file in memory and you are trying to limit memory usage, so that you do not get an 'out of memory' error, try using memory mapped files.

   A small note: using memory mapped files alone does not guarantee that the entire file will (or will not) be mapped into your address space: the amount of the file mapped in can be specified by the user, so that you can map 10KB chunks into your address space as needed, if desired.

   If you try to map a large enough file (or part of a file), you can still encounter a out-of-memory-type situation, because you need a contigous range of addresses large enough for the size of the mapping. Paging, however, will still occur as-needed.

   Peace!

-=- James (Sonork:100.21837)
"There is nothing worse than being oblivious to the fact that you do not know what you are doing."
[Get Check Favorites 1.5 Now!]

User 988520-Nov-02 13:31

My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers

20-Nov-02 13:31

As I understand it, if the mapping is unnamed, it is mapped into the local address space of the application and is not in the global space. So, the chances of running out of memory is remote - unless you have a 2 GB file. In that case, you are not in a place where generic solutions work.

I am not sure whether I am 100% correct on this. Please feel free to correct me if I am wrong. Smile | :)

modified 29-Aug-18 21:01pm.

James R. Twine20-Nov-02 13:59

20-Nov-02 13:59

   [Edit: this had an undesired aggressive tone] I do not believe so: specifying a name only makes the name of the file mapping object (a kernel object) available for others processes to open.  Regardless of it bring named or unnamed, it is always mapped into the address space of the calling process.

   However, in the case of a named object, the two (or more) different processes share the same physical memory (pages), and physical disk space (the file).

   You are correct in saying that (in general) you need a pretty large file to fail, unless your physical address range is fragmented, or you specified a starting address that does not have the contigous range available.  BTW: a process' address space is normally 2GB (unless you booted with /3GB).

   Peace!

-=- James (Sonork:100.21837)
"There is nothing worse than being oblivious to the fact that you do not know what you are doing."
[Get Check Favorites 1.5 Now!]

User 988521-Nov-02 5:26

21-Nov-02 5:26

you are right.

The file is always mapped into the process address space, which is 2 GB. But since the mapping is to a logical process address space, i do not think that there will be a failure to map a view, if the physical address space is fragmented. AFAIK, It will just cause more page faults, because each logical page can be mapped to a different physical page.

I am saying this because I have mapped 512 MB on a machine with 512 MB RAM, obviously, there was not enough physical memory to do that. In any case, the idea is to be able to use more memory than available physically.

Thomas

My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers

modified 29-Aug-18 21:01pm.

James R. Twine21-Nov-02 9:17

21-Nov-02 9:17

Thomas George wrote:
[...] if the physical address space is fragmented. AFAIK, It will just cause more page faults, because each logical page can be mapped to a different physical page

Correct. I said "physical address", when I should have said "virtual address", because you need a contigious range of addresses in your virtual address range to satisfy the size of the view.

Peace!

-=- James (Sonork:100.21837)
[Tip for SUV winter driving survival: "Professional Driver on Closed Course" does not mean "your Dumb Ass on a Public Road"!]
[Get Check Favorites 1.5 Now!]

Amit Dey21-Nov-02 10:34

21-Nov-02 10:34

Supposing our discussion was only limited to text files, moderately large to small text files, would such buffer allocation at one shot be of some advantage? what are the general rules for handling text files?

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

User 988521-Nov-02 10:45

21-Nov-02 10:45

I would think that you can handle small files by allocating memory in the heap. You will need some representation of the file in memory, if the program is like a text editor, which will be written to the file, when saved.

My article on a reference-counted smart pointer that supports polymorphic objects and raw pointers

modified 29-Aug-18 21:01pm.

Amit Dey21-Nov-02 12:03

21-Nov-02 12:03

What if my files were small to large emails? I would guess such allocation would be fine way of doing things?

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

Amit Dey21-Nov-02 10:35

21-Nov-02 10:35

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

James R. Twine21-Nov-02 17:37

21-Nov-02 17:37

IMHO, only if you were making changes to the file.

Peace!

-=- James (Sonork:100.21837)
[Tip for SUV winter driving survival: "Professional Driver on Closed Course" does not mean "your Dumb Ass on a Public Road"!]
[Get Check Favorites 1.5 Now!]

Amit Dey22-Nov-02 4:24

22-Nov-02 4:24

Thanks,James.

'My capacity for happiness', he added, 'you could fit into a matchbox without taking out the matches first'.

- Marvin, the robot.

Amit Dey

sonork: 100:18407
msn: visualcdev

UI Thread question..

RobJones20-Nov-02 10:41

RobJones

20-Nov-02 10:41

Howdy,
Is it possible for a UI thread to do a SendMessage to it's self? If so SendMessage requires HWND.. does a UI thread have a HWND? I guess I'm a little confused.. Anyone have any suggestions?

Thanks!!
Rob

James R. Twine20-Nov-02 10:43

20-Nov-02 10:43

Like PostThreadMessage?

Peace!

-=- James (Sonork:100.21837)
"There is nothing worse than being oblivious to the fact that you do not know what you are doing."
[Get Check Favorites 1.5 Now!]

RobJones20-Nov-02 10:48

RobJones

20-Nov-02 10:48

Thanks..

I'm really using WSASelect() which requires a HWND.. I posted SendMessage (Dumb of me Big Grin | :-D

)because I wanted to see if someone would say it's possible to get a HWND from a thread.. Have you ever used WSAAsyncSelect() inside a UI Thread?

James R. Twine20-Nov-02 13:26

20-Nov-02 13:26

RobJones wrote:
Have you ever used WSAAsyncSelect() inside a UI Thread?

Maybe, some time ago. Now, I would just use WSAEventSelect(...) and use Event objects to indicate whenever something is ready for me. (This works just like normal Win32 events.)

IIRC, UI Threads usually manage their own GUI objects (windows), so they should already have their own message pumps. Worker threads are the ones that do not normally have message pumps. And a thread that occassionally "touches" a GUI object it did not create is not a GUI thread. Smile | :)

Peace!

-=- James (Sonork:100.21837)
"There is nothing worse than being oblivious to the fact that you do not know what you are doing."
[Get Check Favorites 1.5 Now!]

RobJones21-Nov-02 2:57

RobJones

21-Nov-02 2:57

Thanks.. I'll look into event objects..