Click here to Skip to main content
15,902,198 members
Home / Discussions / C#
   

C#

 
GeneralRe: Adjusting size in a picturebox Pin
missrej1-Apr-07 16:52
missrej1-Apr-07 16:52 
QuestionMeasuring total system memory Pin
chatnoir1-Apr-07 9:02
chatnoir1-Apr-07 9:02 
AnswerRe: Measuring total system memory Pin
Wayne Phipps1-Apr-07 10:01
Wayne Phipps1-Apr-07 10:01 
QuestionTo get shortcuts(icons) of documents stored in SQL server [modified] Pin
missrej1-Apr-07 6:30
missrej1-Apr-07 6:30 
AnswerRe: To get shortcuts(icons) of documents stored in SQL server Pin
joon vh.1-Apr-07 21:41
joon vh.1-Apr-07 21:41 
QuestionQuestion on deleting a file once the user closes the application? Pin
Khoramdin1-Apr-07 5:23
Khoramdin1-Apr-07 5:23 
AnswerRe: Question on deleting a file once the user closes the application? Pin
Leslie Sanford1-Apr-07 5:50
Leslie Sanford1-Apr-07 5:50 
Questionfast processing techniques required to process text files greater than 35 GB! Pin
Adeel Chaudhry1-Apr-07 2:38
Adeel Chaudhry1-Apr-07 2:38 
Hello to all!

i am new to c# and we are required to process the text files, minimum size of 35.4 GB. The tasks we want to acheive are

a. Make the file consistent - by removing unwanted 'return' chars
b. Generate the XML file
c. Load the data in XML file to database (MS SQL Server 2005)

the problem we are facing is, the process is taking massive amount of time, i.e. more than 1 hour to process just 500 MB of the file!!

currently we are just going through the file char by char, and checking if it is a data or '/r' or '/n', which definitly is not the best way to read such huge file...

so, we will definitly like to know the possible ways/techniques with which we can reduce the processing time.

the following is the code we are currently using:

while (read_from_file.EndOfStream != true)
            {
                read_from_file.ReadBlock(readfile, 0, 1);
                temp = readfile[0].ToString();
                
		if (!IsAlphaNumeric(temp) && readfile[0] != '-' && readfile[0] != '_')
                {
                    if (temp == " ")
                    {
                        if (token == match_word && first_line_check>1)
                        {
                            write_to_file.WriteLine();
                            write_to_file.Write(token);
                        }
                        else
                        {
                            write_to_file.Write(token);
                        }
                        write_to_file.Write(" ");
                        token = null;
                    }
                    else if (readfile[0] == '\n')
                    {
                        write_to_file.Write(token);
                        write_to_file.Write(" ");
                        token = null;
                    }
                }
                else
                {
                    token = token + temp;
                }
            }


in code, the variable "match_word" is the data before which we require the new line to be started, '/r' and '/n' at any other place are invalid and must be removed.

also, will "threads" and "reading text file in byte instead of char" be helpful and increase the speed of processing? and will the threaded code will be able to keep the data in same sequence??

and how much can we reduce the processing time?

Thanks in advance Smile | :)

Adeel
AnswerRe: fast processing techniques required to process text files greater than 35 GB! Pin
DavidNohejl1-Apr-07 4:21
DavidNohejl1-Apr-07 4:21 
GeneralRe: fast processing techniques required to process text files greater than 35 GB! Pin
Adeel Chaudhry1-Apr-07 5:40
Adeel Chaudhry1-Apr-07 5:40 
GeneralRe: fast processing techniques required to process text files greater than 35 GB! Pin
CPallini1-Apr-07 5:54
mveCPallini1-Apr-07 5:54 
AnswerRe: fast processing techniques required to process text files greater than 35 GB! Pin
Guffa1-Apr-07 4:25
Guffa1-Apr-07 4:25 
GeneralRe: fast processing techniques required to process text files greater than 35 GB! Pin
Adeel Chaudhry1-Apr-07 5:43
Adeel Chaudhry1-Apr-07 5:43 
GeneralRe: fast processing techniques required to process text files greater than 35 GB! Pin
Guffa1-Apr-07 20:59
Guffa1-Apr-07 20:59 
GeneralRe: fast processing techniques required to process text files greater than 35 GB! Pin
mabo422-Apr-07 1:26
mabo422-Apr-07 1:26 
GeneralRe: fast processing techniques required to process text files greater than 35 GB! Pin
mabo422-Apr-07 1:38
mabo422-Apr-07 1:38 
QuestionStarup object Pin
Tauseef A1-Apr-07 2:18
Tauseef A1-Apr-07 2:18 
AnswerRe: Starup object Pin
Mike Hankey1-Apr-07 2:46
mveMike Hankey1-Apr-07 2:46 
QuestionSimple Password Form compare using database.Tq Pin
krajah101-Apr-07 1:43
krajah101-Apr-07 1:43 
Questionthis one is more understandable:D Pin
yvzhn1-Apr-07 0:36
yvzhn1-Apr-07 0:36 
AnswerRe: this one is more understandable:D Pin
Wayne Phipps1-Apr-07 0:49
Wayne Phipps1-Apr-07 0:49 
AnswerRe: this one is more understandable:D Pin
Colin Angus Mackay1-Apr-07 1:07
Colin Angus Mackay1-Apr-07 1:07 
AnswerRe: this one is more understandable:D Pin
Professor Sharada Ulhas1-Apr-07 11:21
Professor Sharada Ulhas1-Apr-07 11:21 
Questionneed a simple solution Pin
yvzhn1-Apr-07 0:27
yvzhn1-Apr-07 0:27 
AnswerRe: need a simple solution Pin
Wayne Phipps1-Apr-07 0:54
Wayne Phipps1-Apr-07 0:54 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.