How Do I Load Big File Hex Parall

Question

1.00/5 (1 vote)

See more:

hi i want load quickly big file hex like this
Download hex.rar - 38.3 KB this file have(69,632 byte)
HexCounter
what fast way to load 1 million hex cod?
*-open hex file with notepad+ (is one line)
*-last test speed 1.3125S with(i3-3240 cpu @ 3.40Hz)
*-sort==false

C#

private void Load_file(string Address_file,List<String> add,bool sort)
    {
        FileStream magic = new FileStream(Address_file,FileMode.Open,FileAccess.Read);
        BinaryReader magic2 = new BinaryReader(magic);
        int _length = (int) magic.Length;
        for (int i = 0; i <= _length - 2; i++)
        {
             // 3 way i test it but?
            //add.Items.Add(cp[i].ToString() + cp[i+1].ToString());  //(char)
            // add.Items.Add(Convert.ToChar(magic2.Read()).ToString());
            //add.Add(Convert.ToChar(magic2.Read()).ToString() + Convert.ToChar(magic2.PeekChar()));
            add.Add((char)(magic2.Read()) + Convert.ToChar(magic2.PeekChar()).ToString());
              i++;
          }

        if (sort==true)
        {
            add.Sorted();
        }
    }

for load file i use this void

C#

private void button2_Click(object sender, EventArgs e)
  {
      //open folder dilog
      ofd.Title = "Hex";
      ofd.Filter = "Text Normal|*.txt|Text INF|*.inf";
      if (ofd.ShowDialog() != DialogResult.Cancel)
      {
          Benchmark.Start();
          Load_file(ofd.FileName, list, false);  //set adress
          Benchmark.End();
          double seconds = Benchmark.GetSeconds();
          MessageBox.Show(seconds.ToString());

      }
  }

i use this class for testing speed

C#

class Benchmark
{
    private static DateTime startDate = DateTime.MinValue;
    private static DateTime endDate = DateTime.MinValue;

    public static TimeSpan Span { get { return endDate.Subtract(startDate); } }

    public static void Start() { startDate = DateTime.Now; }

    public static void End() { endDate = DateTime.Now; }

    public static double GetSeconds()
    {
        if (endDate == DateTime.MinValue) return 0.0;
        else return Span.TotalSeconds;
    }
}

Posted 16-Oct-14 3:20am

majid torfi

Updated 16-Oct-14 6:46am

v5

Add a Solution

Comments

BillWoodruff 16-Oct-14 11:07am

What value would seeing a sorted list of pairs of characters provide ?

2 solutions

Solution 1

First off, you don't need to invent your own class: there is a Stopwatch class as part of the System.Diagnostics namespace which does that, but better, and with a higher precision: http://msdn.microsoft.com/en-us/library/system.diagnostics.stopwatch(v=vs.110).aspx[^]

And my system does this:

C#

byte[] data = File.ReadAllBytes(@"D:\Temp\Output.bin");

in ~15 milliseconds, and that's a 3.5 MB test file on a much lower spec PC that you are using.
The slowness you are seeing is not the read: it's the load of the data into your listbox.

Are you seriously expecting your users to read a megabyte of data in one million little list box entries? Because if I was one of your users, I'd hunt you down and remonstrate with you...:laugh:

I think you want to seriously reconsider your UI, rather than complain about it's lack of speed!

Posted 16-Oct-14 4:01am

OriginalGriff

Comments

majid torfi 16-Oct-14 10:17am

no i want read file with Parallel Programming in c#

OriginalGriff 16-Oct-14 10:28am

Parallel reading of the file in this case won't improve matters - it may even make it slightly worse.

The bottleneck is not the file IO - so nothing you do there will speed things up - it's in the updates to the ListBox, *which can only be made from the UI thread*.
So if you parallelise the IO, you will still have to feed all the data back to the one single UI thread (via Invoke) to load the control - and that's the slow bit already!

majid torfi 16-Oct-14 11:31am

in know i can get more speed with list=>string or Array but i want read 2Byte in one circle
change code

<pre lang="c#">
public List<string> list = new List<string>();
private void Load_file(string Address_file, List<string> add, bool sort)
{
FileStream magic = new FileStream(Address_file,FileMode.Open,FileAccess.Read);
BinaryReader magic2 = new BinaryReader(magic);
int _length = (int) magic.Length;
for (int i = 0; i <= _length - 2; i++)
{
//add.Items.Add(cp[i].ToString() + cp[i+1].ToString()); //(char)
// add.Items.Add(Convert.ToChar(magic2.Read()).ToString());
//add.Items.Add(Convert.ToChar(magic2.Read()).ToString() + Convert.ToChar(magic2.PeekChar()));
add.Add((char)(magic2.Read()) + Convert.ToChar(magic2.PeekChar()).ToString());//<<----this line
i++;
}

if (sort==true)
{
add.Sort();// = true;
}
}
</pre>

i want performance this line
<pre lang="c#">
add.Add((char)(magic2.Read()) + Convert.ToChar(magic2.PeekChar()).ToString());
</pre>

Dave Kreskowiak 16-Oct-14 11:53am

You're not listening.

You CAN NOT speed that line up! It's the Add() call that's slow and there's nothing you can do about it. It is NOT the "(char)(magic2.Read()) + Convert.ToChar(magic2.PeekChar()).ToString()" code that is slow.

You're an idiot for trying to stuff 1,000,000 items into a ListBox. THAT'S THE SLOW PART!

majid torfi 16-Oct-14 12:34pm

i change ListBox to list(is concept for understand) <string>
i not idiot please read last comment

OriginalGriff 16-Oct-14 12:29pm

As Dave has said: that's the slow bit, and there isn't really anything significant you can do to speed it up - it's a UI operation and there are a lot of things the system has to do for each added item.

And seriously: it's going to take a user server hours (at best!) to find the object he wants to select in a huge list box: you should never present more than a hundred or so objects at a time and give paging, filtering, and sorting options to let the user control which hundred or so he is working with.
Think about it: you have to write it once, they have to use it *every day* - so any slow user selection wastes huge amounts of time and really annoys users. Look at how Ebay and Amazon do it: limited number of products on the page, sort and filter options to control what items are shown. There are good reasons why they do that!

majid torfi 16-Oct-14 12:40pm

i not use listbox or any Gui but i want find way to read file with 10 line and ever line have(3600000 byte) Parallel Programming way else read read 2Byte in ever circle

OriginalGriff 16-Oct-14 14:29pm

Well... that's better, but you probably still won;t get what you want: you can only run as many threads in actual parallel as you have free cores, and the smaller a thread is, the more impact the overhear of setting up the thread has.
And...List is not thread safe: you can't just add items to the same list from lots of different threads and hope it will work - because it won't. There are thread safe collections you can use:

http://msdn.microsoft.com/en-us/library/dd997305(v=vs.110).aspx

but...they are more complex, and thus work slower!

Probably, you would be best reading all lines into an array of strings or similar, then kicking off a background worker task for each line which process it and produces it's own separate List. When all threads are complete, you then add the various lists together into a single list. Since the number of threads will probably exceed your core count, it'll move as fast as it can on your system.

But will it be fast? I doubt it. What are you trying to do that you think this is a good idea? there may be a better way you haven't thought of.

majid torfi 16-Oct-14 15:52pm

tanks i solve with mathematics way

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

majid torfi · Accepted Answer · 2014-10-16T09:48:00

Solution 2

C#

        private void Load_file(string Address_file, List<string> add, bool sort)
        {
            byte[] data = File.ReadAllBytes(Address_file);
            int _length = (int)data.Length;
            for (int i = 0; i <= _length - 2; i++)
            {
                add.Add((data[i]) + (data[i+1]).ToString());
                  i++;  
              }
        }
</string>

read 2,042,098 byte in 0.5s

Posted 16-Oct-14 9:48am

majid torfi

Updated 16-Oct-14 9:49am

v2

Comments

Dave Kreskowiak 16-Oct-14 16:08pm

Vote of 2: Considering you never said anything about what you're using this parsed data for, this "answer" has no real application to anyone else but yourself and this one project you're working on.