.NET (Core and Framework)

Luc Pattyn16-Feb-12 6:54

16-Feb-12 6:54

I suspect some of the DGV internals is still working with 16-bit quantities, maybe all the coordinates are short.

I'm not very optimistic you'll get it to work, however I would try and dumb it down, e.g. by disabling the autoformatting (set DataGridViewAutoSizeColumnsMode.None and give each column a fixed width before providing the data).

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Bernhard Hiller16-Feb-12 20:54

Bernhard Hiller

16-Feb-12 20:54

By the way, also Excel is very limited in the number of columns (255? 512?), though the number of rows can be a few thousand (65535?).
Can you locate the position of the problem more closely: is it the mere number of columns which causes the problem, or the width of all the columns together? In the latter case, you could implement a workaround by temporarily setting a very small column width for the columns out of view.

TPIRick17-Feb-12 3:12

TPIRick

17-Feb-12 3:12

Thanks, I will look for that (is it # rows or size of displayed data). What I am doing is detecting when the # Cols is > 255 and then not attaching the DataSource. Essentially, if there is going to be a problem I disable viewing of the data; at least the program can continue to run.

Setting a very small column width for those out of view might work if it is the size of the data and not the #columns. Although, if the # cols is > 1,000 I believe there still might be a problem.

I was really looking for some kind of work around. Like a different data structure. Or maybe something derived from DataGrid or DataGridView which disables are lot of the fancy feature and just displays rows & cols.

Thanks for the answer though.

Dave Kreskowiak18-Feb-12 4:08

Dave Kreskowiak

18-Feb-12 4:08

TPIRick wrote:
How does Excel display very wide tables of data?

Excel uses a custom written rendering engine. Since you seem to want to show a huge number of columns to the user (as a user I wouldn't want to scroll through it!) you have exceeded the design goals of the DataGridView. You would have to either find a grid control that supports your requirements (not likely) or create your own custom drawn data grid (not easy).

A guide to posting questions on CodeProject[^]

Dave Kreskowiak

Eddy Vluggen18-Feb-12 4:15

Eddy Vluggen

18-Feb-12 4:15

TPIRick wrote:

Can anyone help with a proposed solution? How does Excel display very wide tables of data?

Have you ever had to find something in Excel when it's crammed with that amount of data? Once the user needs to scroll, he can also use a button to browse the next page.

You could virtualize the entire thing, drawing a grid and scrolling it's data - not the grid itself.

Bastard Programmer from Hell Suspicious | :suss:

jschell20-Feb-12 9:55

jschell

20-Feb-12 9:55

TPIRick wrote:
Can anyone help with a proposed solution?

No one looks at that much data. Either they are looking for something in it or they want to manipulate it to do something else. First one means you ask the user what they want and only give them that. Second solution involves allowing the user to download the raw data (then they figure out how to manipulate it) or provide a way to manipulate it before presenting it to them.

Luc Pattyn20-Feb-12 12:18

.Net 4.0 Win Forms Caching?

20-Feb-12 12:18

Out of curiosity I performed a little experiment, with the following code inside the Form's Load handler (dgv is a DataGridView):

  1  for (i=0; i<10000; i++) {
  2      DataGridViewColumn col=new DataGridViewTextBoxColumn();
  3      if (i==0) {
  4          log("col.Width="+col.Width);
  5          log("col.FillWeight="+col.FillWeight);
  6      }
  7      col.Name=i.ToString();
  8      // col.FillWeight=3;
  9      dgv.Columns.Add(col);
 10  }

As is, it fails when i=655 with the Exception you've got, because the default FillWeight value is 100, and it somehow gets summed in an unsigned short which then overflows.

With line 8 uncommented, it runs fine. That is, if you have half an hour to spare, waiting for the DGV to get loaded with (empty) columns. After the long wait, the Form and the DGV appear, with column headers, scrollbars, and everything (except for rows and actual data, as I didn't provide any).

The conclusion can only be:
1. a DGV wasn't meant to hold that many columns;
2. your app needs another way to interact with your user.

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

WORKAROUND/SOLUTION !!!

TPIRick7-Mar-12 10:09

TPIRick

7-Mar-12 10:09

Thought I would post my workaround in case it helps others.

Once the data is in a datatable I did a check on the number of columns and if it was very large then I disabled the datagridview to disable viewing. I then added code to allow the user to view the data via HTML. I created a humongo string which was an HTML representation of the data in the datatable. Nothing fancy just kept appending to the same StringBuilder object. Started with html cmds to set up the table then looped thru #rows, #columns spitting out tr, /tr, td, /td cmds with the cell contents in the middle. I then used a StreamWriter to write the humongo string to a file. I then added a button to allow the user to view the file that was created using the default browser. Button invocation calls "Process.Start(htmlFileCreated)".

I tried Firefox and IE 8/9 and was able to view the table even though it was 750 columns wide. Hope this helps someone.

Regards,
Rick

julian@giant16-Feb-12 3:34

16-Feb-12 3:34

Hi all,

I have a scenario that is puzzling me and I'm guessing at the behavior and was wondering if anyone else knew what was happening under the skin of .Net.

I have a .Net 4.0 application on a clients' Windows Server 2008 64bit back end database is an ALPHA (not relevant).

The application is 1 form with literally hundreds of controls on it and also an image viewer. It's takes up quite a bit of screen space, and has multiple tabs (at least 8) to hold all the controls.

The puzzling scenario is this:

1. Open up a simple list
2. Select an item from the list to open up the form as described above - this takes over 1min 15secs
2. Close the form (return to list in point 1)
3. Open up the same form from the list loading the controls in exactly the same way with the same queries behind it (assuming not relevant from the logs I've looked at), and it can open in 10 seconds flat.

From the looks of the memory print in Task Manager, the memory obviously shoots up when the form is first opened, but doesn't go down too far after closing the form. On reopening the form again, the memory doesn't go up much but only takes 10 seconds to open.

I'm thinking that .Net is loading up a load of stuff (!for want of a better word!) into memory on first load of form, and subsequent forms use what has previously been loaded into memory.

If it's not this that is happening, I'm not sure what is happening, as I have been asked by our client to try and make the first load of the form as quick as each subsequent one, and not sure if this is indeed possible.

Any light shed on this would be most appreciated.

Many thanks

Julian

Pete O'Hanlon16-Feb-12 3:41

Pete O'Hanlon

16-Feb-12 3:41

My initial thought here, as you haven't said what database you are hitting, is that the database connection is being pooled. On the first invocation, the connection is being created which is a time consuming process. Further invocations are able to take the pooled database connection.

You are also correct that the .NET runtime will also grab extra memory which is not released immediately, allowing space for the application to load the form in again.

*pre-emptive celebratory nipple tassle jiggle* - Sean Ewington

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

My blog | My articles | MoXAML PowerToys | Mole 2010 - debugging made easier - my favourite utility

julian@giant16-Feb-12 4:59

16-Feb-12 4:59

Hi

The connection to the database is active before the form loads, and is not dropped for the duration of the application lifetime (it's a secure self contained network).

The database as stated is an Alpha database running on a VMS server. We have a windows dll interface to it.

I'm interested in your point about .Net not releasing memory. If I create a new Form each time (Form1 creating Form2 - Form2 being the Form in question), do you think that .Net will intelligently store stuff it needs and call on it again when I create a new Form2?

Thanks for your response

Julian

Luc Pattyn16-Feb-12 4:06

16-Feb-12 4:06

Assuming your description of the situation is somewhat accurate, I can only conclude one did a few things wrongly in the implementation. Adding some simple logging (with time stamps obviously) to the app would allow to pinpoint the major problems. And yes I expect there to be more than one. To name some candidates:
- user authorization on the network
- user authorization on the DB
- establishing the first DB connection
- building the GUI
- obtaining all the data
- stuffing the data in the GUI.

You'll need some facts before you can effectively tackle this, so start logging!
Alternatively, and maybe harder to implement and interpret, try simpler set-ups.

BTW: your app may or may not contain extra implementation features to make subsequent operations faster, think caching. The more of these you have, the more impossible it should become for the first operation to be as fast as subsequent ones.

FWIW: In Windows, dialog windows can be hidden and shown over and over, without recreating them. So even the most complex dialog could be shown again in a fraction of a second (and anyone can come up with ways to make things slower).

PS: if you have tab pages on a Form, then there is no need to fully populate them all before showing anything to the user, as he can see only one of them at a time. The right approach would be to either load the others in a lazy way, i.e. only when needed; or in a background operation, i.e. after the essential work has finished.

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

julian@giant16-Feb-12 4:54

16-Feb-12 4:54

Hi
Yep, already logging like crazy. I have 2 logs on the go,
Log1: almost every function logs it's 'In' and 'Out' times in milliseconds
Log2: Every single query performed on the database is logged to a separate file and time stamped.

I can see from the combination of these what takes time.

So I can see for instance that the Form Shown event finishes at time x (on first showing), and the form doesn't actually appear on the screen until 30 seconds later. Also, no queries occur or other functions called during that 30 seconds.

The database connection (as stated, an Alpha database running on a VMS server) is never dropped, it is continually open and open before the form is first constructed.

Untangling the data from the tabs would be too time consuming (showing data as needed), and we would be issuing queries multiple times, when it would only need to be once at the beginning. However I have taken this approach with the second tab control (there are 2 on the form - 1 with 1 image per tab, and 1 with controls on each tab). The image tab control does indeed load it's image only when necessary as image handling can take time.

I think it's too late to re-show the dialog and simply re-populate the controls on showing the second time, and the second time isn't a problem anyway.

I'm also using threads to help populate the form, attempting as best as possible to do many things at one time.

So, still not sure.

Julian

Luc Pattyn16-Feb-12 5:11

16-Feb-12 5:11

julian@giant wrote:
he form doesn't actually appear on the screen until 30 seconds later.

That is unacceptable, and I've never seen such thing.

julian@giant wrote:
The database connection (as stated, an Alpha database running on a VMS server) is never dropped

That sounds wrong. You should not keep connections open, .NET has connection pooling, meaning you open-query-close (actually use the using construct for getting a connection), and .NET will silently cache and reuse connections for you. With only one connection, you are preventing efficient multi-threading to the DB.

julian@giant wrote:
I'm also using threads to help populate the form

I know of many ways to get that wrong. Too many thread switches would be one; deadlocks another. Are you using Control.InvokeRequired/Control.Invoke? Are you overusing them for each little piece of displayable information? And please don't tell me you have touched Control.CheckForIllegalCrossThreadCalls!

Suggestion: use Task Manager and watch your client's CPU load; if it is near zero for several seconds, something is very wrong (deadlock suspected); if it is near 1/N (with N cores), a busy loop may be the culprit. Also have a look at your server's CPU load (assuming you can get your client to be the only client). Can you correlate both load curves? Any dead periods overall?

PS: no, .NET isn't caching anything in general, when it does, it is documented. DB connections is an example. Windows does cache files. And .NET is not throwing away things as long as it doesn't need the memory for something else. However a new Form would be independent of a previous instance of the same Form (except for its code to be loaded and JIT-ted just once).

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

julian@giant16-Feb-12 5:19

16-Feb-12 5:19

Hi Luc

Thanks.

Yes I agree, 30 seconds is totally unacceptable! And strange. I'm trying to find out what's causing it now. On the face of it, the program is not doing anything! But I do see the server memory going up and up during that 30 seconds.

The database connection thing has been the way the company does things for along time. I've tried a separate app that does indeed do what you're suggesting (putting queries in threads etc.), but somehow, the Alpha database (it came out of the Ark) can't handle queries being done on threads other than the main thread. So, I've stuck with one connection.

I'm pretty sure there's no threading issue during that 30 seconds, but it's something i will check.

Thanks again for your response.

Julian

Luc Pattyn16-Feb-12 5:33

16-Feb-12 5:33

if that 30 seconds is wasted on the server's side (and memory is spent more than expected?), it tells me either your query is awful, or your DB is performing some silly tasks such as say "dynamic indexing". Check your query, and the indexes that are in place.

But then you said the 30 seconds elapsed between the end of Shown and the Form appearing? There should be nothing there, in fact a Form normally is visible when the Shown handler begins.

I'm not convinced yet all your observations are correct (I learned long ago to suspect everything while debugging and/or hunting for performance).

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

julian@giant16-Feb-12 6:13

16-Feb-12 6:13

Thanks Luc,

You've helped me identify some places to look and I shall double check the Form Show event.

Don't think that the database has anything to do with it, we issue a query and it comes back, (some take longer than others) but I've never experienced a query to the DB causing a hang of sorts after it has returned. That's not to say it isn't a first. The DB as I say is old and the interface to it seems to be not overly supported.

Thinking about it, that is an area i haven't looked into. Time to investigate the Alpha logs.. oh no... save me....

Cheers

Julian

julian@giant22-Feb-12 5:04

22-Feb-12 5:04

OK, have now got this sorted.

Previously the .exe I was doing performance enhancements on was 450Mb in size and took 1m30s to load each form it needed to create (from within the app itself)!

I managed to get this down to 1m15s on the first time the form was opened and 10 seconds every subsequent time it was opened from within the app. Memory usage was down to 150Mb. I was quite pleased with myself.

The target application server is 64bit. Setting the target CPU in the project settings of the application to x86 from x64 brought the size of the .exe down to around 60Mb and there is no delay on the initial creation of the first form from within the app.

Knew I'd get there in the end.

Thanks for all your input into this guys. Some most interesting suggestions.

Cheers

Julian

Luc Pattyn22-Feb-12 5:10

22-Feb-12 5:10

Thanks for the update.
I have no idea why a target change would have such drastic effects. Do you? Or what haven't you told us??

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

julian@giant22-Feb-12 5:32

22-Feb-12 5:32

I don't know what I haven't told you .... yet.... Smile | :)

Dave Kreskowiak16-Feb-12 6:04

Dave Kreskowiak

16-Feb-12 6:04

Luc Pattyn wrote:

he form doesn't actually appear on the screen until 30
seconds later.

That is unacceptable, and I've never seen such
thing.

I have. When there's hundred of controls on the form. Creating a control, and the underying window, is expensive. Then, if he's populating list controls with hundreds of items, that isn't exactly setting an speed records either.

A guide to posting questions on CodeProject[^]

Dave Kreskowiak

julian@giant16-Feb-12 6:15