Re: File Size Limits - C / C++ / MFC Discussion Boards

carks31-Aug-05 6:20

31-Aug-05 6:20

Im trying to read a large text file with a CStdioFile object (the file's around 2.3 Gb, OS Windows XP, File System NTFS).

After reading for a while an exception is thrown during a CStdioFile::ReadString call which says something like "Controler Invalid"

I simplified the code so that its doing nothing more than loopìng around the ReadString call, and the error persists.

Could there be an inherent limit on the size of a file readable using CStdioFile (I dont belive that 2.3 Gb is too big for NTFS ?!?)

Thanks,

Dave

Re: File Size Limits

JWood31-Aug-05 6:41

JWood

31-Aug-05 6:41

You may be running into some unexpected 32bit dword limitations, not necessarily a file os problem, but a file pointer limitation.

Re: File Size Limits

carks31-Aug-05 6:54

carks

31-Aug-05 6:54

That sounds something like whats happening, but does anybody know of any settable limmitations of this type that exist?

Re: File Size Limits

PJ Arends31-Aug-05 7:05

PJ Arends

31-Aug-05 7:05

Are you using VC6? if so then the file pointer used is a 32 bit value, with a limit of 2GB. AFAIK if you are using VC7+ then the file pointer is a 64 bit value.

If you want to get around this limit in VC6 then do not use the MFC classes (which wrap the fopen, fread, etc CRT functions) and use the SDK functions CreateFile, ReadFile etc.

"You're obviously a superstar." - Christian Graus about me - 12 Feb '03

"Obviously ??? You're definitely a superstar!!!" - mYkel - 21 Jun '04

"There's not enough blatant self-congratulatory backslapping in the world today..." - HumblePie - 21 Jun '05

Within you lies the power for good - Use it!

Re: File Size Limits

carks31-Aug-05 7:21

carks

31-Aug-05 7:21

Yes thats it PJ!

I am using VC6.

(I guess its time to shell out for an upgrade!)

Thanks so much for the help Smile | :)

Re: File Size Limits

Bob Stanneveld31-Aug-05 7:54

Bob Stanneveld

31-Aug-05 7:54

PJ Arends wrote:
do not use the MFC classes (which wrap the fopen, fread, etc CRT functions)

Seriously? I always though that MFC didn't wrap CRT functions, but rather the WIN API directly! The CRT file API's (I don't know about the others) are just a layer on top of the OS's API's, which are CreateFile, ReadFile, etc..

I don't know about the MFC shipped with VC6, but the newer MFC version 8, does wrap the window's API's...

Behind every great black man...
... is the police. - Conspiracy brother

Blog[^]

Re: File Size Limits

Blake Miller31-Aug-05 8:37

Blake Miller

31-Aug-05 8:37

From VC 6.0 MFC Source (FILECORE.CPP Line 111)
Yes it wraps Win32 API, not standard C runtime calls...

BOOL CFile::Open(LPCTSTR lpszFileName, UINT nOpenFlags,<br />
	CFileException* pException)<br />
{<br />
<br />
...<br />
<br />
	// attempt file creation<br />
	HANDLE hFile = ::CreateFile(lpszFileName, dwAccess, dwShareMode, &sa,<br />
		dwCreateFlag, FILE_ATTRIBUTE_NORMAL, NULL);<br />
	if (hFile == INVALID_HANDLE_VALUE)<br />
	{<br />
		if (pException != NULL)<br />
		{<br />
			pException->m_lOsError = ::GetLastError();<br />
			pException->m_cause =<br />
				CFileException::OsErrorToException(pException->m_lOsError);<br />
<br />
			// use passed file name (not expanded vesion) when reporting<br />
			// an error while opening<br />
<br />
			pException->m_strFileName = lpszFileName;<br />
		}<br />
		return FALSE;<br />
	}<br />
	m_hFile = (HFILE)hFile;<br />
	m_bCloseOnDelete = TRUE;

Re: File Size Limits

Bob Stanneveld31-Aug-05 8:47

Bob Stanneveld

31-Aug-05 8:47

Just as I expected. It would be just silly for MS to use some portable API that is a layer on top of their own API, instead of their own API.

Thanks for the info! Big Grin | :-D

Behind every great black man...
... is the police. - Conspiracy brother

Blog[^]

Re: File Size Limits

PJ Arends31-Aug-05 10:10

PJ Arends

31-Aug-05 10:10

I guess I should read the MFC source instead of MSDN[^]

[quote]
A CStdioFile object represents a C run-time stream file as opened by the run-time function fopen.
[/quote]

Re: File Size Limits

Bob Stanneveld31-Aug-05 11:00

Bob Stanneveld

31-Aug-05 11:00

So much for the accoutibility of MSDN...

Behind every great black man...
... is the police. - Conspiracy brother

Blog[^]

Re: File Size Limits

Blake Miller31-Aug-05 13:00

Blake Miller

31-Aug-05 13:00

Well, I DID look at the CFile code. Who knows, maybe the CStdioFile object does something different? Not likely, but this IS MFC ... OMG | :OMG:

Re: File Size Limits

Toby Opferman31-Aug-05 14:35

Toby Opferman

31-Aug-05 14:35

Generally what I think that this is referring to is that this object *IS* the C run time and behaves the same. In the open it does call CFile::Open:

0:000> u MFC42!CStdioFile::Open
MFC42!CStdioFile::Open:
73e2c96c 8bff             mov     edi,edi
73e2c96e 55               push    ebp
73e2c96f 8bec             mov     ebp,esp
73e2c971 53               push    ebx
73e2c972 8b5d0c           mov     ebx,[ebp+0xc]
73e2c975 56               push    esi
73e2c976 57               push    edi
73e2c977 ff7510           push    dword ptr [ebp+0x10]
0:000> u
MFC42!CStdioFile::Open+0xe:
73e2c97a 8bc3             mov     eax,ebx
73e2c97c 25ffbfffff       and     eax,0xffffbfff
73e2c981 50               push    eax
73e2c982 ff7508           push    dword ptr [ebp+0x8]
73e2c985 8bf9             mov     edi,ecx
73e2c987 33f6             xor     esi,esi
73e2c989 897710           mov     [edi+0x10],esi
73e2c98c e8e043fbff       call    MFC42!CFile::Open (73de0d71)

And that calls CreateFile, however right afterwards they do create a standard handle:

73e2c9db 75d2             jnz     MFC42!CStdioFile::Open+0x43 (73e2c9af)
73e2c9dd ebd7             jmp     MFC42!CStdioFile::Open+0x4a (73e2c9b6)
73e2c9df c644050c74       mov     byte ptr [ebp+eax+0xc],0x74
73e2c9e4 40               inc     eax
73e2c9e5 51               push    ecx
0:000>
MFC42!CStdioFile::Open+0x7a:
73e2c9e6 ff7704           push    dword ptr [edi+0x4]
73e2c9e9 c644050c00       mov     byte ptr [ebp+eax+0xc],0x0
73e2c9ee ff150467e773 call dword ptr [MFC42!_imp___open_osfhandle (73e76704)]
73e2c9f4 83f8ff           cmp     eax,0xffffffff
73e2c9f7 59               pop     ecx
73e2c9f8 59               pop     ecx

Look, most other functions are wrappers around C runtime, as an example, look here:

0:000>  u  MFC42!CStdioFile::WriteString
MFC42!CStdioFile::WriteString:
73e2cb01 8bff             mov     edi,edi
73e2cb03 56               push    esi
73e2cb04 8bf1             mov     esi,ecx
73e2cb06 ff7610           push    dword ptr [esi+0x10]  <-- FILE *
73e2cb09 ff74240c         push    dword ptr [esp+0xc]
73e2cb0d ff151867e773     call    dword ptr [MFC42!_imp__fputs (73e76718)]
73e2cb13 83f8ff           cmp     eax,0xffffffff

In WriteString, fputs is called and it gets the FILE * from the "this" pointer at offset 16. So, CStdioFile calls some standard APIs, some C runtime APIs and it in itself essentially re-implements some of the functionality that would have been done had you really called fopen() in order to create a compatible FILE *.

Read calls fread:

MFC42!CStdioFile::Read:
73e2ca4b 837c240800       cmp     dword ptr [esp+0x8],0x0
73e2ca50 56               push    esi
73e2ca51 8bf1             mov     esi,ecx
73e2ca53 7504             jnz     MFC42!CStdioFile::Read+0xe (73e2ca59)
73e2ca55 33c0             xor     eax,eax
73e2ca57 eb62             jmp     MFC42!CStdioFile::Read+0x70 (73e2cabb)
73e2ca59 55               push    ebp
73e2ca5a 57               push    edi
0:000> u
MFC42!CStdioFile::Read+0x10:
73e2ca5b ff7610           push    dword ptr [esi+0x10]  <-- Again, FILE *
73e2ca5e ff742418         push    dword ptr [esp+0x18]
73e2ca62 6a01             push    0x1
73e2ca64 ff74241c         push    dword ptr [esp+0x1c]
73e2ca68 ff151067e773     call    dword ptr [MFC42!_imp__fread (73e76710)]

Notice:

CStdioFile::m_pStream
Remarks

The m_pStream data member is the pointer to an open file as returned by the C run-time function fopen. It is NULL if the file has never been opened or has been closed.

In the documentation. While "fopen" may not have been called, they basically implemented fopen themsevles on top of CFile::Open (that calls CreateFile). Some of their APIs then call standard C Runtime and this public data member (most likely the same one I showed in the dissassembly) is exported as a C runtime compatible pointer. So, whether they use the C Runtime internally or not is not the point, the object itself exports the appropriate data structures and behaves idential to the c runtime implementation.

8bc7c0ec02c0e404c0cc0680f7018827ebee

Re: File Size Limits

John R. Shaw31-Aug-05 21:09

John R. Shaw

31-Aug-05 21:09

Yes there is a limit. We can normaly ignore those, but we can not ignore the amount of memory we have for storage.

Where are you copying the data to?

INTP
"The more help VB provides VB programmers, the more miserable your life as a C++ programmer becomes."
Andrew W. Troelsen

xmldocument

picazo31-Aug-05 5:35

picazo

31-Aug-05 5:35

Hello,

I am using xmldocument to read/edit an xml file. Along with it, I am using an XmlValidatingReader to validate the data. I have two questions on this subject:

1) Assuming that a document is opened to be edited (and it's valid initially), is it possible to validate changes that are made to it without having to rewrite the current tree to file and then reading it in again?

2) When initially validating a document: if an error occurs, is it possible to get the type of error that occurred in the handler? I mean, for example if an element is missing, or if an element is invalid, or if an attribute is missing... Of course, I would like a way to do this without having to parse and interpret the error message. The reason I would like this feature is so that I could try to 'fix' invalid xml files if I find them.

Question 2 is a bit more important than 1.

Thanks a lot,

-----------------
Genaro

\\\|
_ _
@ @
_\\
--|
_/

RegCreateKeyEx returns 87 "ERROR_INVALID_PARAMETER"

win_newbie31-Aug-05 5:15

win_newbie

31-Aug-05 5:15

Hi,

I'm new to windows programming, and am having problems logging to the application log. In fact I can't seem to create/open a key. I picked up some example code from MSDN and tried to use it, always returns error code 87 - ERROR_INVALID_PARAMETER. I've looked at and tried other examples from the web, with the same results. The call below is the first call in main(), my app's name is vsreq2:

wsprintf(szBuf, (LPCWSTR)"SYSTEM\\CurrentControlSet\\Services\\EventLog\\Application\\vsreq2");
status = RegCreateKeyEx(HKEY_LOCAL_MACHINE,
szBuf,
0, NULL, REG_OPTION_NON_VOLATILE,
KEY_WRITE, &secAttr, &hk, &dwDisp);

if (status)
{
printf("Could not create the registry key. Error Code = 0x%x", status);
return FALSE;
}

Thanks for any help!

Re: RegCreateKeyEx returns 87 "ERROR_INVALID_PARAMETER"

PJ Arends31-Aug-05 6:30

PJ Arends

31-Aug-05 6:30

win_newbie wrote:
wsprintf(szBuf, (LPCWSTR)"SYSTEM\\CurrentControlSet\\Services\\EventLog\\Application\\vsreq2");

The first thing that jumps out at me is the cast of an 8-bit string to a 16-bit string.

Drop the cast, and wrap your string literals in the _T("") t-char macro.

Re: RegCreateKeyEx returns 87 "ERROR_INVALID_PARAMETER"

win_newbie31-Aug-05 9:35

win_newbie

31-Aug-05 9:35

Hey thanks, I'm with mYkel.

This has gotten me to my next problem, where the description field of the event
indicates "The description for Event ID [101] cannot be found....". The message
does have my app's name as the source and the correct category/event id numbers.
I have the msgs.dll file in the same directory as my app and made these API calls:

Is the placement of the text string after (LPBYTE) ok?

thanks again

// Set the name of the message file.
if (RegSetValueEx(hk, // subkey handle
(LPCWSTR)"EventMessageFile", // value name
0, // must be zero
REG_EXPAND_SZ, // value type
(LPBYTE)"msgs.dll", // pointer to value data
(DWORD) lstrlen(_T("msgs.dll"))+1)) // length of value data
{
printf("Could not set the event message file.");
RegCloseKey(hk);
return FALSE;
}

// Set the supported event types.
dwData = EVENTLOG_ERROR_TYPE | EVENTLOG_WARNING_TYPE |
EVENTLOG_INFORMATION_TYPE;

if (RegSetValueEx(hk, // subkey handle
(LPCWSTR)"TypesSupported", // value name
0, // must be zero
REG_DWORD, // value type
(LPBYTE) &dwData, // pointer to value data
sizeof(DWORD))) // length of value data
{
printf("Could not set the supported types.");
RegCloseKey(hk);
return FALSE;
}

// Set the category message file and number of categories.
if (RegSetValueEx(hk, // subkey handle
(LPCWSTR)"CategoryMessageFile", // value name
0, // must be zero
REG_EXPAND_SZ, // value type
(LPBYTE)"msgs.dll", // pointer to value data
(DWORD) lstrlen(_T("msgs.dll"))+1)) // length of value data
{
printf("Could not set the category message file.");
RegCloseKey(hk);
return FALSE;
}

if (RegSetValueEx(hk, // subkey handle
(LPCWSTR)"CategoryCount", // value name
0, // must be zero
REG_DWORD, // value type
(LPBYTE) &dwNum, // pointer to value data
sizeof(DWORD))) // length of value data
{
printf("Could not set the category count.");
RegCloseKey(hk);
return FALSE;
}

Re: RegCreateKeyEx returns 87 "ERROR_INVALID_PARAMETER"

PJ Arends31-Aug-05 10:20

PJ Arends

31-Aug-05 10:20

Same problem as before, this time with the value names.

I think you should read some of the articles here on CP about string handling. I strongly recommend http://www.codeproject.com/string/cppstringguide1.asp[^].

carriage return line feed not working in a string

LCI31-Aug-05 4:48

LCI

31-Aug-05 4:48

Anyone know why when i place a carriage return line feed in a string to be displayed in a control, that the \r\n don't get recognized?

I do something like lpzText = "CODE1 NORMAL\r\n""CODE2 HIGH\r\n"

But when i display this in an edit control or even a tooltip, the \r\n is not recognized and there for i do not get the desired line feed.

Re: carriage return line feed not working in a string

toxcct31-Aug-05 4:51

toxcct

31-Aug-05 4:51

did you set your edit box to multiline ??

TOXCCT >>> GEII power
[toxcct][VisualCalc]

Re: carriage return line feed not working in a string

LCI31-Aug-05 4:55

LCI

31-Aug-05 4:55

How is that done... Is that done from the control variable ?

Re: carriage return line feed not working in a string

C^hris L^osinger31-Aug-05 5:11

C^hris L^osinger

31-Aug-05 5:11

it's a property you can set via the dialog/resource editor (or as a flag, if you're creating the control dynamically)

Cleek | Image Toolkits | Thumbnail maker

Re: carriage return line feed not working in a string [edited]

toxcct31-Aug-05 5:21

toxcct

31-Aug-05 5:21

in design mode, click on the edit control to have its properties, and searche for one called something like "multiline"...
to accept more than one line, your edit control must have that property engaged...

see here and here[^]

TOXCCT >>> GEII power
[toxcct][VisualCalc]

-- modified at 11:40 Wednesday 31st August, 2005

Re: carriage return line feed not working in a string

Blake Miller31-Aug-05 4:52

Blake Miller

31-Aug-05 4:52

I think it is possible they only want the \n to be in there.
Also, if you are trying to create a single string, I am not sure why you have two strings there?
Try this instead:
lpzText = "CODE1 NORMAL\nCODE2 HIGH\n";
and see if your results are better.

Re: carriage return line feed not working in a string

LCI31-Aug-05 4:59

LCI

31-Aug-05 4:59

It still does not recognize
lpzText = "CODE1 NORMAL\nCODE2 HIGH\n";

I get a square box in place of the \n. I am also using this string in a tool tip and get the same result.

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.