Winform | decrypt & load data in datagridview | slow

Question

1.00/5 (1 vote)

See more:

, +

Hi,

I will try and explain everything so that I do not miss out on anything important.

Winform - .NET 4.0 DataGridView
Loading 25000 rows of data which is bound to a Datagridview using BindingSource
Out of a total 50 something columns, 7 columns have encrypted data (Blowfish)
After the data load is complete - looping all rows to decrypt info
This takes a whopping 9 minutes

Help needed to:
Bring the loading time down
Exclusion of any field is not an option
Loading all 25000 is also a requirement

I think I have tried a lot of methods to reach to this point, but I think I need some to figure the last bit out. Some of my friends said they could load data using WPF collections (0.25 million records per minute). I do not know if that's true but this being a VB.NET Winform app - please suggest anything I can try to speed up the application load times.

Many thanks.

What I have tried:

1. Lazyloading - uses a Background worker to decrypt records in datatable. This currently works and helps keep the UI responsive but the user cannot export the data unless the decryption is complete. Takes appox. 8 mins to complete.

2. Hardloading - Using

VB

Parallel.ForEach

looped through all records in Datatable. This completes the job in half the time i.e. 4 mins but the UI becomes unresponsive at times. Also the CPU usage goes to 100%. Used MaxDegreeOfParallelism as a variable but using anything above 2 freezes the GUI.

3. Eagerloading - normally loop through all records in Datatable using

VB

For Each

This takes about 9 mins to complete.

Posted 10-Apr-17 6:55am

kasbaba

Updated 10-Apr-17 10:23am

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

CHill60 · Answer 1 · 2017-04-10T10:23:00

Loading 25000 items into a User Interface is just silly. That is the politest way I can put it.

WPF collections? I think your friends are talking rubbish.

How to speed up the load times?

1. You should implement some sort of paging - for example only load the first 40 items until the User presses Page Down

2. Load the data into a .NET collection - for example a List. You may need to create a class that reflects the incoming data.

3. Iterate through the collection not any UI control.

4. Use JIT thinking - only decrypt when attempting to display - so only 40 records at a time.

Best advice would be paging on a worker thread.

By the way - There is never any good reason to load that many rows into a UI component.