Re: Getdirectories performance problems - C# Discussion Boards

Anders Hedén20-Jan-12 3:49

20-Jan-12 3:49

I've tried just to get a directory tree and save it to a list, and that will take like a minute with the kind of dir structure i talked about in my previous post.

I can get a code snippet later with more explanation, I'm at work at the moment.

Anders Hedén20-Jan-12 8:47

20-Jan-12 8:47

Now lets see this is the basic code for populating my treeview. It's slow because i add all directories at once. This is written in vb sorry for that, i hope you don't get to mad with me Sigh | :sigh:

, my other C# code was a real mess atm.

Private Sub PopulateTreeView(ByVal setPath As String)
    Dim rootNode As TreeNode
    Dim info As New DirectoryInfo(setPath)
    rootNode = New TreeNode(info.Name)
    rootNode.Tag = info
    GetDirectories(info.GetDirectories(), rootNode)
    TreeView1.Nodes.Add(rootNode)
End Sub

Private Sub GetDirectories(ByVal subDirs() As DirectoryInfo, ByVal nodeToAddTo As TreeNode)
    Dim aNode As TreeNode
    Dim subDir As DirectoryInfo
    For Each subDir In subDirs
        aNode = New TreeNode(subDir.Name, 0, 0)
        aNode.Tag = subDir
        If ((subDir.Attributes And FileAttribute.Hidden) <> FileAttribute.Hidden) Then
            If subDir.GetDirectories().Count() > 0 Then
                GetDirectories(subDir.GetDirectories(), aNode)
            End If
            nodeToAddTo.Nodes.Add(aNode)
        End If
    Next subDir
End Sub

Here is also the code i used to add dummy-nodes to improve the speed, but the problem with this solutions is that you still check the underlying subdirectory with a count()/length. http://stackoverflow.com/questions/2135851/treeview-control-contextswitchdeadlock-workarounds

I don't really know how to solve this? I have tried the same with enumeration of directories but without greater performance. I've also tried to remove the node functions to see if they slow down the process but without any better performance.
Dead | X|

Luc Pattyn20-Jan-12 18:23

20-Jan-12 18:23

Anders Hedén wrote:
written in vb sorry for that

I can read it, no problem. Fortunately you used PRE tags!

I wrote:
You should determine what exactly is slow; is it obtaining the information? or displaying it?

you did not address that!? Mad | :mad:

I wrote:
a networked disk is always a lot slower than it is on a local disk

you did not address that!? Mad | :mad:

NEW: getting DirectoryInfo[] is much slower than just getting the file/folder names, especially over a network; a significant number of individual calls are executed for each and every file/folder!

Anders Hedén wrote:
If subDir.GetDirectories().Count() > 0 Then
GetDirectories(subDir.GetDirectories(), aNode)
End If

It is so slow that you decided to do everything twice?

WTF | :WTF:

Luc Pattyn [My Articles] Nil Volentibus Arduum

Anders Hedén21-Jan-12 23:46

21-Jan-12 23:46

Hi again and thanks for your patience!

First of let me answer your questions Blush | :O

...

Luc Pattyn wrote:
You should determine what exactly is slow; is it obtaining the information? or displaying it?

It's when I try to obtain the directory info structure and not when i try to view it.

Luc Pattyn wrote:
a networked disk is always a lot slower than it is on a local disk

It's all local disks and I'm only trying to get info from one disk at the moment.

I have also changed my code to enumerate method as you can see bellow.

var dirs = new List<string>(Directory.EnumerateDirectories(folderBrowserDialog1.SelectedPath));
GetDirs(dirs);

public void GetDirs(List<string> dirs)
{
    foreach (var dir in dirs)
    {
       try
        {

            var subdir = new List(Directory.EnumerateDirectories(dir));
            if (subdir.Count > 0)
            {
                GetDirs(subdir);
            }
        }
        catch  (UnauthorizedAccessException uax)
        {
            
        }
     }
}

I added a timer for this test, and to run this it took me 55 seconds with enumerate method and 74 seconds with getDirectories method for 7000 folders. The structure of the directories looks like this : A\folders were some have subfolders, but all in all 7000 folders.

Luc Pattyn22-Jan-12 3:40

22-Jan-12 3:40

Anders Hedén wrote:
var subdir = new List(Directory.EnumerateDirectories(dir));

That isn't any better than it was before: you are using an enumeration to fill a list, then using that list; so once again you insist on having it all in memory first and only then doing something with it. Try along these lines:

IEnumerable<string> subdir=Directory.EnumerateDirectories(dir);

and don't use Count() as that too would need to get all the info in order to count, where you don't even need the number; at most uou want to know if it is zero or not (and even that may be unnecessary).

Enumeration is a streaming operation with all the advantages involved, don't turn it into a get-all-then-process operation; just let it flow freely.

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Anders Hedén22-Jan-12 10:25

22-Jan-12 10:25

Hmm I'm not quite following sorry, but should it be something like this?

IEnumerable<string> dirs= Directory.EnumerateDirectories(folderBrowserDialog1.SelectedPath);
GetDirs(dirs);

public void GetDirs(IEnumerable<string> dirs)
{
    foreach (var dir in dirs)
    {
            IEnumerable<string> subdir = Directory.EnumerateDirectories(dir);
            if (subdir.Any())
            {
                GetDirs(subdir);
            }
    }
}

Some questions
How come the folderbrowserdialog can access the all the systemvolumes with folders and files directly?

In my application I want the user to select folders in the treeview (I want these path to the user selected folders), can I do that in some other way? Maybe should start a new thread for that Laugh | :laugh:

Luc Pattyn22-Jan-12 10:43

22-Jan-12 10:43

That looks better, yes.

Anders Hedén wrote:
How come the folderbrowserdialog can access the all the systemvolumes with folders and files directly?

It can't do that. Controls and applications either show the top-level, and then you can expand (while waiting); or they show "everything" but then they stop as soon as the visible part is filled, and then you can see more (scroll) while waiting (that is often called "virtual mode").

Experiment: insert a USB stick holding hundreds of files/folders, look at it with some tool; remove it and try to expand/scroll some more, it will fail as the device isn't available to gather more data. (BTW: some apps would close when you remove the stick, Windows Explorer does that).

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Shameel20-Jan-12 2:18

Shameel

20-Jan-12 2:18

Are you trying to load the entire file system into the treeview at application startup? If this is the case, then definitely performance would suffer. Instead, use the TreeView control's BeforeExpand event to populate only the directory the user is trying to expand.

Anders Hedén20-Jan-12 4:05

20-Jan-12 4:05

Yes that is how i do it atm. I know its not the best way. I've tried with the approach with populating the treeview with nodes BeforeExpand, but i haven't found any good option to add "nodedummies" to allow the node to be expanded. An option that doesn't use GetDirectories() to count the underlying subdirectories to add dummy or not.

I could add a dummy to every node and when the user tries to expand the last directory with no underlying directories, the dummy is removed and the expand +-sign for that node/directory is removed. But this is not really nice for the user i think. I cant think of a better way to add dummies, and would gladly receive any tips you have Big Grin | :-D

Shameel20-Jan-12 4:56

Shameel

20-Jan-12 4:56

Add the dummy node only if there are subfolders in the directory.

Another option would be to show only the first n sub-directories (where n could be something like 50 or 100 based on your preference) and add a dummy node below them called 'Show more folder'. Expanding that node removes it and adds the next n sub-directories and so on.

modified 20-Jan-12 14:36pm.

BobJanova20-Jan-12 2:32

20-Jan-12 2:32

Querying the file system, particularly if networks or non-fixed drives are involved, is a slow operation. I'm not sure .Net is really responsible for this. You might need to design your application in such a way that this can be loaded while the user is doing other things so when it is needed it can be loaded from cache.

Problem Copying Raw RGB Data to Bitmap

Pete O'Hanlon20-Jan-12 2:49

Pete O'Hanlon

20-Jan-12 2:49

What technology are you using for your UI? If you are using WPF, I would suggest that you can improve your performance quite a bit by turning on UI virtualization, and display only those items that are visible. As Luc suggests, the EnumerateDirectory method is a much better bet for speed, but displaying that much data (a lot of which the user isn't going to see initially) is a time consuming process.

Forgive your enemies - it messes with their heads

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

My blog | My articles | MoXAML PowerToys | Mole 2010 - debugging made easier - my favourite utility

Jaffer Mumtaz19-Jan-12 23:20

Jaffer Mumtaz

19-Jan-12 23:20

Hi All,

I am using the following function to copy RAW RGB byte array to Bitmap object. The problem I am facing is when I copy the byte array content through SetPixel method, Image construction is OK and color are proper on the Bitmap. But when I try to use the following function it some how swaps the Red and Blue bytes in the Bitmap causing wrong colors on the Image.

static int WriteBitmapFile(string filename, int width, int height, byte[] imageData)
{
using (Bitmap bmp = new Bitmap(width, height, PixelFormat.Format24bppRgb))
{
  BitmapData bmpData = bmp.LockBits(new Rectangle(0, 0,bmp.Width,bmp.Height),ImageLockMode.WriteOnly,bmp.PixelFormat);
  Marshal.Copy(imageData, 0, bmpData.Scan0, imageData.Length);
  bmp.UnlockBits(bmpData);
  bmp.Save(filename);
}
  return 1;
}

It seems though when I am copying the byte array in one go it swaps the Red and Blue bytes. SetPixel works fine but is too slow.

Any help in this regard is highly appreciated.

Regards,

Luc Pattyn19-Jan-12 23:50

19-Jan-12 23:50

Indeed, GetPixel/SetPixel is not the way to perform operations on an entire image. Here are 3 ideas that may help:

1. you could write a little loop that swaps the bytes in the byte array before executing the code shown.
2. you could try a Bitmap constructor that takes an IntPtr to raw data (may get the same result you have now); make sure to read MSDN's remark though.
3. you could apply a ColorMatrix transformation once the bitmap is filled.

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Jaffer Mumtaz19-Jan-12 23:55

Jaffer Mumtaz

19-Jan-12 23:55

Thanks for the timely reply Luc.

Can you give some pointers/links/example on how to apply ColorMatrix?

Thanks and Regards,

BobJanova20-Jan-12 2:17

20-Jan-12 2:17

This is because the pixel format expects the bytes in the other order from what you're providing (BGR, I think. The 4 byte one is BGRA iirc). I would transform the bytes on the way in:

byte[] newImageData = new byte[imageData.Length];
for(int i = 0; i < newImageData.Length; i += 3){
 newImageData[i] = imageData[i + 2];
 newImageData[i + 1] = imageData[i + 1];
 newImageData[i + 2] = imageData[i];
}

Unless the image is really large, that should be negligible in time and memory terms and easier than post-processing the image with a ColorFilter.

Jaffer Mumtaz21-Jan-12 7:31

Jaffer Mumtaz

21-Jan-12 7:31

You are saying that bitmap is BGR, but I tried using SetPixel with RGB pattern and it worked fine. Does SetPixel works differently than copying RGB data?

BobJanova21-Jan-12 10:59

Time taken to load a contrl

21-Jan-12 10:59

Yes. SetPixel takes a Color, copying byte data just takes raw bytes in the appropriate format.

Subin Mavunkal19-Jan-12 19:15

Subin Mavunkal

19-Jan-12 19:15

Hi whats the accurate and best method to get the time taken to load a control.?.Any idea ?.(I am writing a performance monitoring UI test.)

With thanxx and regards

BillWoodruff19-Jan-12 20:15

BillWoodruff

19-Jan-12 20:15

I think to get a good answer to this question you need to tell us exactly what you mean by "load a control."

Are you talking about how long it takes to "fill" a databound object, like a ListView, with thousands of rows from its source ?

Are you talking about how long it takes, from the time an end-user double-clicks on a compiled .NET .exe file to present a Window (WPF, WinForms ?) with many controls ... the contents/settings of which ... all (or some) ... need to be calculated when the application launches, rather than being "static," or read in from Application.Settings or other resources ?

"It is the mark of an educated mind to be able to entertain a thought without accepting it." Aristotle

V.19-Jan-12 20:27

19-Jan-12 20:27

Difficult question. The fact that you are timing the "load" might implicate the "load" itself, making it slower.

Do you need this on the live system or just for dev and/or test environment?

Luc Pattyn19-Jan-12 21:51

19-Jan-12 21:51

Use the StopWatch class, call Start() right before and Stop() right after the execution of the code you're interested in.

I don't know what you mean by "load a control"; with all the events going on in a WinForms Form when opening up a new Form, it may well be that your control is touched in the Form's constructor, and several event handlers such as Load and Shown. And that other controls also get handled at around the same time.

You might even have to deal with an extra Control which you handle all at once, using your own run-time code only.

Warning: if lots of data is loaded into the control (such as a DataGridView fed by a database), how you organize the data load will predominate, not the generating and painting of the control itself; unless you have coded a large number of operations on a single control and you've forgotten to make good use of optimizing calls such as Control.SuspendLayout() and the like.

Smile | :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

BobJanova19-Jan-12 22:13