|
I think the problem may be that your input is not EBCDIC - ebcdic is an 8-bit code so only has characters 000 to 255.
Wiki on EBCDIC[^]
No trees were harmed in the sending of this message; however, a significant number of electrons were slightly inconvenienced.
This message is made of fully recyclable Zeros and Ones
|
|
|
|
|
Hi,
Yes, none of these characters in the file are in the 255 codes of EBCDIC.
As i had mentioned earlier, the file from the BLOB, contains packed decimal also.
Are you sure its not EBCDIC, when i do .ToCharArray() i get like:
[0]: 22909 '好'
[1]: 8224 '†'
[2]: 8256 '⁀'
[3]: 8592 '←'
[4]: 879 'ͯ'
[5]: 12629 'ㅕ'
[6]: 16527 '䂏'
[7]: 16448 '䁀'
[8]: 16448 '䁀'
[9]: 16448 '䁀'
[10]: 16448 '䁀'
[11]: 16448 '䁀'
[12]: 16448 '䁀'
[13]: 65533 '�'
[14]: 49605 '쇅'
[15]: 16601 '䃙'
[16]: 16448 '䁀'
[17]: 16448 '䁀'
[18]: 16448 '䁀'
[19]: 16448 '䁀'
[20]: 16448 '䁀'
[21]: 16448 '䁀'
[22]: 16448 '䁀'
[23]: 62448 ''
[24]: 63216 ''
[25]: 63220 ''
[26]: 63473 ''
[27]: 63216 ''
[28]: 62451 ''
[29]: 61689 ''
[30]: 54769 '헱'
[31]: 16448 '䁀'
[32]: 16448 '䁀'
[33]: 16448 '䁀'
[34]: 16448 '䁀'
So their character codes are greater then 255.
How to resolve this?
|
|
|
|
|
The problem is that chars in .NET are Unicode, which is generally two bytes. Using a byte array instead of a char array might solve your problem.
|
|
|
|
|
I highly recommend using Jon Skeet's EBCDIC encoding library[^]
We have been using this for years. You basically just set up Encoding objects and pass them into Stream objects.
Granted, this is VB.NET code, but it should still make sense:
Dim inputEncoding As Encoding = EbcdicEncoding.GetEncoding("EBCDIC-US")
Dim outputEncoding = Encoding.ASCII
Dim inputReader As StreamReader
Dim outputWriter As StreamWriter
inputReader = New StreamReader(inputFile, inputEncoding)
outputWriter = New StreamWriter(outputFile, False, outputEncoding)
I don't claim to be a know it all, for I know that I am not...
I usually have an answer though.
|
|
|
|
|
how to install the Jon Skeet's EBCDIC encoding library[
|
|
|
|
|
Try This
#region public static byte[] ConvertAsciiToEbcdic(byte[] asciiData)
public static byte[] ConvertAsciiToEbcdic(byte[] asciiData)
{
// Create two different encodings.
Encoding ascii = Encoding.ASCII;
Encoding ebcdic = Encoding.GetEncoding("IBM037");
//Retutn Ebcdic Data
return Encoding.Convert(ascii, ebcdic, asciiData);
}
#endregion
#region public static byte[] ConvertEbcdicToAscii(byte[] ebcdicData)
public static byte[] ConvertEbcdicToAscii(byte[] ebcdicData)
{
// Create two different encodings.
Encoding ascii = Encoding.ASCII;
Encoding ebcdic = Encoding.GetEncoding("IBM037");
//Retutn Ascii Data
return Encoding.Convert(ebcdic, ascii, ebcdicData);
}
#endregion
|
|
|
|
|
Hi...
I am using c#.net and sql2005.and in my form i am taking datetime picker for date.in sql taken datatype as datetime. it takes date correctly but it cant take time correctly,taken 12:00 default for all records.
i want to add current date and time in database.
Suggest me something.
|
|
|
|
|
DateTime.Now retrieves the current date and time 
I are troll
|
|
|
|
|
If you need the user to be able to select a date and time then set the Format to Custom and the CustomFormat to yyyy/MM/dd HH:mm:ss
DaveBTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)
Visual Basic is not used by normal people so we're not covering it here. (Uncyclopedia)
Why are you using VB6? Do you hate yourself? (Christian Graus)
|
|
|
|
|
Hi guys. I have managed to resolve a proble I had yesterday with taking a value from one file and searching for it in another file. Now, my file1 containing the values I am searching for has approximately 2000 records in it. I pick up the first id in file1 and search for it in file2 which has approx 80000 records in it. Once the value has been found I replace it with another value from file1. The below code seems to taking forever to run. Is there anyway I can make this run much faster? 
fileNot = textBox1.Text;
fileHid = textBox2.Text;
string[] lines = File.ReadAllLines(fileHid);
StreamWriter sw = new StreamWriter(outFile);
foreach (string line in lines)
{
string oldhid = line.Substring(0, 11);
string newhid = line.Substring(14, 11);
StreamReader sr = new StreamReader(fileNot);
string holdLine;
while (!sr.EndOfStream)
{
holdLine = sr.ReadLine();
string matchId = holdLine.Substring(0, 11);
string noting = holdLine.Substring(11, 650).Trim();
string dnot = holdLine.Substring(11, 2);
if (oldhid == matchId && dnot != "D1")
{
sw.WriteLine(newhid + noting);
break;
}
}
sr.Close();
}
sw.Close();
swr.Close();
Any advice woudl be greatly appreciated.
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
Have you try'd doing everything in memory?
Read file 1 in memory (like your doing)
Read file 2 in memory
Do your operations in memory (so the replacing of the id's)
Write to your outfile
As it is now you are creating (opening and closing) 2000 streamreaders, this is bound to be a performance bottleneck.
|
|
|
|
|
Rather than going through the file line by line could you not just search for the index of the word and check if it's right or not (Seeing as you know exactly where everything should be if your using substrings).
StreamReader sr = new StreamReader(fileNot);
string entireFileString = sr.ReadToEnd();
sr.Close();
sr.Dispose();
int location;
while((location = entireFileString.IndexOf(oldhid)) >= 0)
{
}
entireFileString = string.Empty;
Don't forget to Dispose() to clear up memory.
My failometer is detecting vast quantities of FAIL!
"Its SQL - hardly programming..." (Caslen)
modified on Friday, May 15, 2009 8:34 AM
|
|
|
|
|
Hi,
IMO performance can be improved dramatically, however the right tactics depend on some extra info:
- Could you show say 10 lines of each of those files?
= is the order of the output lines relevant?
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
[aside] thinking sorted data are we? [\aside]
Panic, Chaos, Destruction.
My work here is done.
|
|
|
|
|
yep.
for performance, always sort; the sort order is not that important, as long as you sort.
if need be, sort an existing collection; better yet sort while collecting.
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
While back had to fix a wee oops due to sorting:
Original logic was as items are added repaint the list; even if items are not visible.
Then sparky came along and added a sort. A resort of EVERY row as EACH row was added.
Then users came along and added a shed load of rows.
Can you guess why it took 10 minutes to get 500 items into the list...
Panic, Chaos, Destruction.
My work here is done.
|
|
|
|
|
you violated performance rule #1: don't let the users touch anything.
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
I've just signed another item of my list without doing a thing! Bug raised [low priority] 18 months ago has bubblde up for the next phase. Investigate and a fix elsewhere has already resolved it.
Problem is, I've get 2 days on the plan to fill...
[![Beer | [beer]](https://codeproject.freetls.fastly.net/script/Forums/Images/beer.gif) ]
Panic, Chaos, Destruction.
My work here is done.
|
|
|
|
|
HI Luc. I could post a few lines for you but they are pretty much what I posted in my orriginal post. What I think I want to achieve is read the first ID in FILE1 take that and search for it in FILE2, if found then replace the ID in FILE2 with the ID alongside the one in FILE1. So, File1 has the following data in it:
<br />
C0000000001 C0001000010<br />
C0000000002 C0001000011<br />
C0000000003 C0001000012<br />
FILE2 contains the following data:
<br />
C0000000001N1 SAMPLE NAME 1<br />
C0000000002N5 SAMPLE NAME 2<br />
C0000000123D2 SAMPLE NAME 3<br />
C0000000003N4 SAMPLE NAME 4<br />
Now, FILE1 has approximately 2000 lines of data and FILE2 has approximately 80000 lines of data in it. So in short, take C0000000001 from FILE1 and search through 80000 lines of data and for each instance found replace it with the Value C0001000010 + the remainder of the data in FILE2 on the line and write that to a new file.
Hope this makes sense.
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
Hi,
0.
several things aren't clear yet:
- You did not answer my last question: is the order of the output relevant to you?
- is the first input file ordered at all?
- why is there a string newhid = line.Substring(14, 11); when the replacement string seems to be located at column 12?
Anyway, here are some toughts:
1.
the job at hand consists of one or maybe two parts: sorting lines and replacing the first part of lines. both can be tackled independently.
2.
the way you have it the second file gets read (maybe half of it on average) for each line present in the first line, that makes this a quadratic algorithm, so execution time depends on filesize1 * filesize2 which is bad.
3.
you must ry to overcome this, the one way I see is by:
- keeping one file in memory
- stepping through the other file only once
4.
my prefered way, not sure it fits your needs (see 0), would be to:
- read the first file into a Dictionary<string, string=""> which would map oldhid -> newhid
- read the second file line by line, process every line in turn using the dictionary, and write the result.
That would be a linear operation, dramatically faster than what you have; however it would basically keep the data in the order it is in the second file.
5.
point 4 does not work in the same order as your current code. If order is relevant, rather than trying to keep it by quadratic code, sort the file; either the second file before you process it, or the result file, depending on your exact needs.
6.
To sort a file, either use an existing utility, or write some code; easiest again is by using File.ReadAllLines() and Array.Sort(); for 80,000 lines of say 120 chars this amounts to one million chars, so no memory problem. And a decent sort algorithm will behave much better (run much faster) than the linear search your code is representing right now.
7.
For special sort orders, there is an Array.Sort() overload that takes an IComparer, so all is possible.
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
0. The order of the file is not important at all.
Luc Pattyn wrote: why is there a string newhid = line.Substring(14, 11); when the replacement string seems to be located at column 12?
This is in FILE1 and is located at pos:14 size:11.
I think that reading the first file into a dictionary could maybe work. I'll give that a go and see how it works. I have tried reading file1 into memory and then reading file 2 line by line and that took ages as well. So resorted to reading both files into memory after suggestions received and this seems to be taking forever as well. What I want to achieve is the same effect as Micrsoft Access Simple Query. I don't need to have the output fle in any kind of sort order so sorting would probably not make much of a difference unless I sort both files?? Could help I recon.
Thanks guys for the advice but I think there should be a much simpler way to get this done but getting it right is a bit of a ^&*^*#^*...........LOL........When I get it all hanging and running fast enough I will post the code or if you guys come up with some more ideas please post!
Thanks again.
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
Kwagga wrote: This is in FILE1 and is located at pos:14 size:11
Not according to the samples you provided earlier. I can see only one space between oldhid and newhid.
Kwagga wrote: I don't need to have the output fle in any kind of sort order
OK, so my current estimate is the job is worth no more than 3 seconds on a modern PC provided it gets coded right, without showing it all on some GUI. Strategy:
1. read file 1, probably line by line, into a dictionary
2. read and process file 2 line by line
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
OK, there are 3 spaces there.
I wish it would take all of 3 seconds but what seems to get the better of me is that file1 is all of 289kb and file2 is all of 247kb, this is one of the smaller files that I want to process. When this runs it generates a file that is 800MB and I eventually have to abort it as it seems to get into an infinate loop. file2 could contain multiple instances of the searchID in file1. I have searched high and low and far and wide for any articles on this topic but can't find anything relating towhat I want to do. There are hundreds of articles on Finding and replacing single strings in files but nothing that reads the strings from an input file. NOthing that even suggests the same logic that I want to impliment.
As mentioned before, I want to impliment something similar to what is done in a Microsoft Access Database. I may just have to write this to a DB into two tables and then do the matching in there with a SQL statement and export the new data!! 
Thought this could be possible other than going the Database route.
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
It does not make any sense at all:
1.
you showed the following content of file1:
C0000000001 C0001000010
C0000000002 C0001000011
C0000000003 C0001000012
which is some 25 characters per line, times 2000 lines, equals 50KB (or 100KB when all in wide characters), not 289KB. With 289KB the average line length would be 140 bytes. Check its content!
2.
file2 holding 247KB for 80,000 lines results in some 30 characters per line, that sounds fine.
3.
the result file growing beyond 800MB is utter nonsense. For it to grow that large you would have to:
- either have 80,000 lines of 10,000 characters each (where are these coming from?)
- or have many more lines. How would you explain that?
Did you bother checking what is inside that file? is anything in there correct?
Now start looking at what you have, and stop making up stories. 
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
The way I had the code written was that for each instance of ID in file1 found in file2 it wrote the whole of file2 as I had file2 completely read into memory and for some stupidness of myself I had it coded that way. Now, I have ammended this code to the below which seems to work just fine but the performance is still "pish" as it itterates 2000* through 80000 records to find a match. There are instances where an ID in FILE1 could exist more then once in FILE2. So, on a file of 80000 records it does a 160 000 000 reads on file2 which could mean a major hit on performance.
Luc Pattyn wrote: Now start looking at what you have, and stop making up stories
I swear I'm not making up stories.
THis is what I have refined the code to now but I am sure it CAN be sped up much more.
My actual files have these amounts of records on it.
FILE1 = 10948 Lines of 25 characters per line. (289KB)
FILE2 = 360 Lines of 650 characters per line. (247KB)
FILE3 = 79341 Lines of 650 characters per line. (54,392KB)
FILE4 = 76283 Lines of 700 characters per line. (50,508KB)
To process this it takes all of 34.61 seconds to complete which is fine and the resultant file is 100% spot on.
Now, processing FILE1 with 10948 lines of data against another file, say FILE3 with 79341 lines with 650 characters per line it takes forever.
The code implimented is as follows:
string fileNot;
string fileHid;
string outPath = @"c:\Documents and Settings\Mumbleb\Desktop\Ess\";
fileNot = textBox1.Text;
fileHid = textBox2.Text;
StreamWriter sw = new StreamWriter(outPath + "ER_AU_TBSZ_SEQMHN.TXT");
string[] LineHID = File.ReadAllLines(fileHid);
string[] LineNot = File.ReadAllLines(fileNot);
foreach (string hid in LineHID)
{
string OldHid = hid.Substring(0, 11);
string NewHid = hid.Substring(14, 11);
foreach (string content in LineNot)
{
string matchid = content.Substring(0, 11);
string noting = content.Substring(11, 605).Trim();
if (OldHid == matchid)
{
string contentnew = Regex.Replace(content, matchid, NewHid);
sw.WriteLine(NewHid + noting);
}
}
}
sw.Close();
I am about to try and process the bigger files on my Dualcore Laptop and see what the performance is like. I will post if I have any timings on the performance.
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
