Re: Converting char* to unicode big-endian - C / C++ / MFC Discussion Boards

Converting to unicode and converting to big-endian are 2 completely separate things.

To convert to unicode use mbstowcs from the C runtime library or the Windows API call MultiByteToWideChar.

Do you reall need it in big-endian? Are you sending the data over TCP or something? Anyway to convert to big-endian loop over each character in the unicode string buffer and either call htonl from the winsock library or manually swap the bytes yourself like:

wchar_t bigendchar = ((littleendwchar & 0xFF) << 8) | ((littleendwchar & 0xFF00) >> 8);

Actually I want to convert the string pointed by a char* to big-endian encoding, nothing deal with TCP/IP.

Like the following managed C++ code but work in unmanaged C++:

#include "stdafx.h"

#using <mscorlib.dll>
using namespace System;
using namespace System::Text;

int main()
{
String* unicodeString = S"This string contains the unicode character Pi(中)";

// Create two different encodings.
Encoding * unicode = Encoding::Unicode;
Encoding * bigendian = Encoding::BigEndianUnicode;

// Convert the string into a Byte->Item[].
Byte unicodeBytes[] = unicode -> GetBytes(unicodeString);

// Perform the conversion from one encoding to the other.
Byte bigendianBytes[] = Encoding::Convert(unicode, bigendian, unicodeBytes);

// Convert the new Byte into[] a char and[] then into a string.
// This is a slightly different approach to converting to illustrate
// the use of GetCharCount/GetChars.
Char bigendianChars[] = new Char[bigendian ->GetCharCount(bigendianBytes, 0, bigendianBytes -> Length)];
bigendian -> GetChars(bigendianBytes, 0, bigendianBytes->Length, bigendianChars, 0);
String* bigendianString = new String(bigendianChars);

// Display the strings created before and after the conversion.
Console::WriteLine(S"Original String*: {0}", unicodeString);
Console::WriteLine(S"bigendian converted String*: {0}", bigendianString);
}

That code isn't actually converting from char* (ANSI string) to big-endian. Rather, it's converting a String object, which internally holds a unicode string encoded as UTF-16) to another String object, which internally will hold a unicode string, again encoded as UTF-16 but this time using big-endian.

From end to end it's just swapping pairs of bytes.

The entire process it's doing is as follows:

1. Get the internal buffer of the original string as a byte array
2. Convert the byte array to big-endian (swapping each pair of bytes)
3. Copy the byte array into an array of Unicode UTF-16 chars
4. Create a new String object from the array of Unicode UTF-16 big endian chars

So, an equivalent in unmanaged C++ would be converting a wide char string to big-endian. I'm not aware of any standard C++ or Win32 API function that can be used to swap pairs of bytes, but something like the function below should work:

// Note nBufLen is the count of wide chars, not a byte count.
// nBufLen represents the buffer size and must include space
// for the NULL terminator
void ConvertToBigEndian(const wchar_t* pw, wchar_t* pwBuffer, int nBufLen)
{
   int i = 0;
   for( ;i < nBufLen && pw[i] != 0; i++)
   {
      const char* p = (const char*)&pw[i];
      char* q = (char*) &pwBuffer[i];
 
      q[0] = p[1];
      q[1] = p[0];
   }
 
   // terminate destination string
   if (i < nBufLen)
      pwBuffer[i] = 0;
   else
      pwBuffer[nBufLen-1] = 0;
}

Besides that, if you really need to start from a char* (MBCS/ANSI string), first convert it to wide chars (Unicode UTF-16) using MultiByteToWideChar.

Hope that helps,

--
jlr
http://jlamas.blogspot.com/[^]

Thanks for everybody.

I have constructed a method:

char* UnicodeCharToBigEndianConverter(char* message)
{
wchar_t input[2000];
wchar_t output[2000];

MultiByteToWideChar(
CP_ACP, // code page
MB_COMPOSITE, // character-type options
message, // string to map
-1, // number of bytes in string
input, // wide-character buffer
strlen(message)+2 // size of buffer
);

// escape character for unicode
output[0]='\xFE\xFF';

MessageBoxW(NULL,input,L"Input",0);

for(int i=1;i<=wcslen(input);i++)
{
output[i] = ((input[i-1] & 0xFF) << 8) | ((input[i-1] & 0xFF00) >> 8);
}

wcscat(output, (wchar_t*)"\x00\x00");

return (char*) output;
}

It works fine if the char* that I got from my application is a string with all unicode characters, e.g. all chinese characters. However, what if a user input a string with some chinese characters and some ASCII characters, e.g. A-Z, a-z?

Is that whole string can be converted to big-endian using:
output[i] = ((input[i-1] & 0xFF) << 8) | ((input[i-1] & 0xFF00) >> 8);
Or we need to handle those ASCII characters?

I hope you don't mind my comments Smile | :)

scchan1984 wrote:
wchar_t input[2000];
wchar_t output[2000];

MultiByteToWideChar(
CP_ACP, // code page
MB_COMPOSITE, // character-type options
message, // string to map
-1, // number of bytes in string
input, // wide-character buffer
strlen(message)+2 // size of buffer
);

Those magic 2000 don't look well. If you are going to specify a maximum length of strings you support, you should at least check that the string you receive isn't longer than what you support.

The last parameter to MultiByteToWideChar is wrong. There you are expected to pass the size of your buffer, which in this case is 2000. strlen() returns a char count of the string (not counting the null terminator), so for example, if you receive a message of 2100 chinese characters, strlen will return 4200. When you call to MultiByteToWideChar will try to write 2100 wide chars in a buffer that can only hold 2000, based in the erroneus information you gave it (you would be telling it that your buffer has space for 4202 wide chars instead of the actual 2000). The most likely result is a crash of your application.

You should call MultiByteToWideChar first using 0 as the buffer size. That will return the required size for the buffer in wide chars. With that info, you can allocate a buffer in the heap, and then call MultiByteToWideChar again to do the conversion.

scchan1984 wrote:
// escape character for unicode
output[0]='\xFE\xFF';

You defined both buffers with the same size, but if output will hold a byte oder marker, then it should be at least one wide char bigger, or you might not have enough space.

scchan1984 wrote:
for(int i=1;i<=wcslen(input);i++)

You are calling wcslen in each iteration, making it traverse the entire string in the search for the NULL terminator. It would make more sense to call wcslen outside of the loop, store its value in a variable, and use the variable in the loop.

Then again, you don't even need to call wcslen, as the length of input is what MultiByteToWideChar returns; you'd just need to receive it in a variable.

scchan1984 wrote:
output[i] = ((input[i-1] & 0xFF) << 8) | ((input[i-1] & 0xFF00) >> 8);

This seems to be a more readable version of the byte swap I wrote in my previous post, right? Smile | :)

scchan1984 wrote:
wcscat(output, (wchar_t*)"\x00\x00");

Here you are again traversing the entire string just to add a terminator at the end. You already know in advance that the length of output would be exactly one wide char more (the byte order marker) than the lenght of input, which in turn was returned by MultiByteToWideChar. So you could simply write

int nInputLen = MultiByteToWideChar(...);
.
.
.
output[nInputLen+1] = 0;

scchan1984 wrote:
return (char*) output;

Why are you returning a pointer to a wide char string as if it were a char string? It doesn't make sense. The return type should be wchar_t, and you shouldn't be casting it to anything else.

Besides, you are returning a pointer to a variable allocated in the stack in the context of the function. As soon as the function returns, the variable will be destroyed!

scchan1984 wrote:
It works fine

I think you haven't done enough testing Smile | :)

scchan1984 wrote:
However, what if a user input a string with some chinese characters and some ASCII characters, e.g. A-Z, a-z?

That wouldn't be a problem. For example, in a MBCS string, which is what your function is supposed to receive as the char* message, an 'a' will occupy just one byte, while a Chinese character will probably use a lead byte followed by a second byte (3 bytes total, plus one additional byte for the null terminator = 4 bytes). After using MultiByteToWideChar to convert it to Unicode, you'll have a wide char (that's 2 bytes) for the 'a' and a second wide char for the chinese character (total = 2 wide chars plus a null wide char as terminator = 3 wide chars = 6 bytes). Then you would swap the bytes in each of those 3 wide chars while copying them to the output buffer, and everything would be fine.

scchan1984 wrote:
Or we need to handle those ASCII characters?

No need for special handling. Each ascii char will be represented by two bytes (one wide char), one with the original value and the other with a value of zero. Those 2 bytes will be swapped like every other wide char in the buffer.

--
jlr
http://jlamas.blogspot.com/[^]

Really really thank you so much.
I have to improve and test my code now Big Grin | :-D

My improved version:
WCHAR* UnicodeCharToBigEndianConverter(char* message)
{
// get the number of WCHAR from the message
int nInputLen = MultiByteToWideChar(
CP_ACP, // code page
0, // character-type options
message, // string to map
-1, // number of bytes in string
NULL, // wide-character buffer
NULL // size of buffer
);

WCHAR* input = new WCHAR[nInputLen+1];
WCHAR* output = new WCHAR[nInputLen+1];

MultiByteToWideChar(
CP_ACP, // code page
MB_COMPOSITE, // character-type options
message, // string to map
-1, // number of bytes in string
input, // wide-character buffer
nInputLen+1 // size of buffer
);

// escape character for unicode
output[0]='\xFE\xFF';

for(int i=1;i<=nInputLen;i++)
{
output[i] = ((input[i-1] & 0xFF) << 8) | ((input[i-1] & 0xFF00) >> 8);
//output[i] = input[i-1];
}

output[nInputLen+1] = 0;

return output;
}

It's much better now Smile | :)

Note that you are allocating two buffers. You are returning one of them as the result. The other should be deleted before returning, to avoid leaking memory. Add the following line before the return:

delete[] input;

You should note that the callers of this function are responsible for deleting the result...

Besides that, it seems to be already doing what it's supposed to do. Now, if you want to do it really right, keep reading...

First I reccomend you read the following article by Joel Spolsky: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
[^]

Now that you have read it (if not, go and read it, I'll wait right here), you surely understand why a name like UnicodeCharToBigEndian[...] doesn't make much sense for what this function does (i.e.: it's not receiving an Unicode string, but a multibyte string.)

A better name would be MultiByteToBigEndianWideChar...

One additional problem with your function is that it returns a buffer that must be released by the caller. This is a problematic approach because as a user of the function, I couldn't deduce that from the function signature alone.

If all I see is a function declared in a header file as

WCHAR* MultiByteToBigEndianWideChar(char* message);

I have no way to know whether I should delete the returned buffer or not, or in case I'm expected to delete it, whether I should use delete[], free, or anything else. I'd have to rely on some documentation or on having access to the implementation itself in order to find out.

To avoid this ambiguity, it's a common practice in this kind of functions to make the caller supply the buffer and its size as parameters to the function. When the buffer is supplied by the caller, the caller already knows if that buffer is allocated in the stack or in the heap, and how to release it in the latter case. That's exactly the approach followed by MultiByteToWideChar and all the API functions.

In fact, you might start with a function with exactly the same parameters as MultiByteToWideChar

int MultiByteToBigEndianWideChar(
  UINT CodePage,         // code page
  DWORD dwFlags,         // character-type options
  LPCSTR lpMultiByteStr, // string to map
  int cbMultiByte,       // number of bytes in string
  LPWSTR lpWideCharStr,  // wide-character buffer
  int cchWideChar        // size of buffer
)
{
   int nResult = MultiByteToWideChar(
                   CodePage,
                   dwFlags,
                   lpMultiByteStr,
                   cbMultiByte,
                   cchWideChar? lpWideCharStr + 1 : 0,
                   cchWideChar? cchWideChar-1 : 0
                 );
 
   if (cchWideChar != 0)
   {
      lpWideCharStr[0] = 0xFEFF;
      WideCharToBigEndian(lpWideCharStr, nResult+1);
   }
 
   return nResult + 1;
}

When cchWideChar is 0, it calls MultiByteToWideChar with 0, and returns its result plus 1, because this function will need to add a byte order marker (BOM) at the start of the buffer).

When cchWideChar is not 0, it calls MultiByteToWideChar to make the conversion to Unicode, leaving the first position available for the BOM, then puts the BOM at the first position, and calls WideCharToBigEndian, which simply does the byte swapping in the same buffer it receives.

void WideCharToBigEndian(LPWSTR lp, int cchWideChar)
{
   for(int i = 0; i < cchWideChar; i++)
      lp[i] = ((lp[i] & 0xFF) << 8) | ((lp[i] & 0xFF00) >> 8);
}

Now your function can be used exactly like MultiByteToWideChar, including the option to query a buffer size, and with the same flexibility (i.e.: all the options in code page, flags, etc.) You can refer anyone to the documentation for MultiByteToWideChar and simply state that you add a BOM and convert the unicode string to big-endian.

Now suppose you want to provide a simplified version in which some of the options are filled with default values.

int MultiByteToBigEndianWideChar(
  LPCSTR lpMultiByteStr, // string to map
  LPWSTR lpWideCharStr,  // wide-character buffer
  int cchWideChar        // size of buffer
)
{
   return MultiByteToBigEndianWideChar(
                   CP_ACP,
                   MB_COMPOSITE, // by the way, are you sure you want this flag as default?
                   lpMultiByteStr,
                   -1,
                   lpWideCharStr,
                   cchWideChar
                 );
}

There, now you only need to compile and test. Good luck!

--
jlr
http://jlamas.blogspot.com/[^]

I got the answer!!!

The reason is that when I change the ASCII character to big-endian, take 'B', with unicode \x42\x00, it become \x00\x42.

And I used strlen to check the string length of (char* UnicodeCharToBigEndianConverter(char* message)), which finds the terminating character \x00, and gives me the wrong string length!!

My code in the above post should be ok.
Anyone interested in have a look in it and test it.

MultibyteToWideChar[^]

My programming blahblahblah blog. If you ever find anything useful here, please let me know to remove it.

Hi,

Is there any application which tells the component class of the Java application which is similar to WinSPY ?

If anybody know please let me know.

thanks,
Mahender

hi,
i am trying to create a ftp application in which i want to show the progress bar. i tried using the OpenFile of CFtpConnection to open the file and then read it. but the problem i am facing is that whenever i try to get the file size by using the SeekToEnd function which returns the file size i get an exception. and i think that the file is not being opened properly. can anyone help me. i have used IIS for the ftp adn created a virtual directory. the name of the computer in IIS is : homepc. so how should i give the file path( what should be the path of the file : suppose the file to be copied is test.txt what should be the exact path). also can anyone provide any example for the same. is there any other way to use a progress bar in this application

thanks

aditya

morning,

my debug will let error like "access violation" by pass when I keep going by breakpoints, it says "first exception handles....", but I think the access violation is the window error, it will crash in release version, not sure why my debug is not picking it up as "unhandled exception"

thank you!

and what is the exception thrown ???

and what is the code that make the crash ?

TOXCCT >>> GEII power
[toxcct][VisualCalc]

the exception will be like :

"First-chance exception in Epred.exe: 0xC0000005: Access Violation."

crash because the code couldn't recorgnize and return data type

thanks

can we have a piece of the crashing code to understand what you're doing wrong ?

TOXCCT >>> GEII power
[toxcct][VisualCalc]

valerie99 wrote:
"First-chance exception in Epred.exe: 0xC0000005: Access Violation."

This mean you application try to access the memory which is actually not available.

try to debug and check is every Varible getting required memory ?

"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow

cheers,
Alok Gupta
VC Forum Q&A :- I/ IV

I am looking for a free proxy server to redirect UDP and TCP Traffic for a brief test.

A free UDP/TCP proxy server or source code for same would be cool as well. C++, VB (Gah!), C#, VB.Net, anything like that would work.

Doesn't have to be the greatest thing in the world, just something that can handle a few tests. The OS platforms would be a windows server, NT, 2000, 2003 server, any of those.

The reason I need it on windows is because the VP client to perform the tests is windows only.

Thanks!

.............................
There's nothing like the sound of incoming rifle and mortar rounds to cure the blues. No matter how down you are, you take an active and immediate interest in life.

Fiat justitia, et ruat cælum

hiho@ll

i just wanted to know if it's possible to "hook" the windows shutdown and if yes how?

i thought about a simple alarm prog, which does something if you want to shutdown windows

for example it checks if you did some task before shutdown
or it simply reminds you to do something before shuting down

anybody has an idea?

thx@ll

ThinkingPrometheus wrote:
i thought about a simple alarm prog, which does something if you want to shutdown windows

HANDLE WM_QUERYENDSESSION message !

"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow

cheers,
Alok Gupta
VC Forum Q&A :- I/ IV

maybe i'm wrong but if i handle such an event, windows will try to kill the application if it doesn't exit!?

so is it possible to abort the shutdown if i use WM_QUERYENDSESSION?

ThinkingPrometheus wrote:
so is it possible to abort the shutdown if i use WM_QUERYENDSESSION?

Yeap it possible, you just return the FALSE o this event and then you have to shutdown by your self!

"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow

cheers,
Alok Gupta
VC Forum Q&A :- I/ IV

hello to all members i have been given task to add functionality in project

on right clicking icon on taskbar it shows menu like windows messenger shows
Exit,open,signout,sign in,sign in as,my status,send an email message and my e-mail Inbox
my project icon is also showing its menu like that
but on double clicking windows messenger icon in taskbar messenger gets opened
how will i add this double clicking functionality to my project

Ankit Aneja

Ankit Aneja wrote:
windows icon in taskbar it gets opened

?

V

i mean when i double click windows messenger icon on task bar it gets opened
same functionality i want here

Ankit Aneja
"Nothing is impossible. The word itself says - I M possible"