arabic letter frequency in vb.net

Question

1.00/5 (1 vote)

See more:

I want to perform letter frequency of Arabic characters contained in a sample text, but i don't know how to read the text file containing the Arabic content and how to count the frequency of each characters

for example

البيت الكسز اللتيل الزجاج الست.

where individual letters are

ا أ إ آ ى ؤ ئ ء ب ت ة ث ج ح خ د ذ ر ز س ش ص ض ط ظ ع غ ف ق ك ل م ن ه

How do i detect the code-page a file is written or stored in?
Can it help in this program, because am opening text files created with Arabic language?

Posted 11-Feb-11 3:11am

Cool Smith

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Dave Kreskowiak · Accepted Answer · 2011-02-11T06:18:00

Solution 1

Don't repost the same question in multiple places. This has already been answered in the VB.NET forum.

Posted 11-Feb-11 6:18am

Dave Kreskowiak

Comments

Nish Nishant 11-Feb-11 12:26pm

He's given more info in this thread though. As you can see, the observed characters are actually a combination of multiple characters (or so it would seem based on what he says). So in the first Arabic sentence, it seems there are 5 words where each word seems to be one continuous and long piece of character, but a single character. But in the 2nd piece of Arabic text, he's disassembled them into the individual characters. So from this perspective, it certainly looks as if some non-trivial work needs to be done to extract the individual characters that go into making those composite characters.

Cool Smith 11-Feb-11 13:48pm

if i convert the whole string to unicode ie u+0000, or \x?? will it work? How do i convert the string to unicode or hex

Dave Kreskowiak 11-Feb-11 20:37pm

In .NET, all Strings are Unicode. Nish is correct. In order to get the individual characters, you've got a rather complicated problem. There is no way to detect the correct code-page from a text file. The information is simply not in the file.

Try opening the file with a StreamReader using the Encoding option and specify Encoding.Unicode. Read one character at a time and see what you get.

Cool Smith 12-Feb-11 6:30am

can you provide sample code on how to do this?

Dave Kreskowiak 12-Feb-11 10:08am

An example to open a file?? http://msdn.microsoft.com/en-us/library/x8xxf0x5.aspx#Y1078