Win10 : Examination of Disk Utilization (Includes DiscoFiles utility)

• Download DiscoFiles1040.zip (all C# source) - 64 KB
• Download DiscoFilesEXE_1040.zip (exe for easy tryout) - 46.2 KB

Update Note 1.0.4.0

A sharp-eyed user noticed and reported that when you double-click on a file / path that you don't have full access to DiscoFiles throws an ugly exception.  I wrapped the exception so now when that occurs it looks slightly cleaner and at least is a bit more user friendly.  

I've updated the version to 1.0.4.0 (from 1.0.3.0) and uploaded the latest source and Exe for your use.

The additionally interesting thing is that since the background thread handles the update to the UI as the FileSystemWatcher events come in, the MainForm UI is still updated while this modal dialog is displayed to the user.

Introduction

Do you ever wonder where a particular app is writing or reading files from on your disk?  With the DiscoUtils utility you can easily see every time a file is created, updated, deleted or renamed on you disk. 

Background

I recently posted in the lounge (The Lounge: Win10 AGain (sigh)[^] )  about my computer being completely unusable due to disk I/O.  Basically, as you will see in screenshots I couldn't do anything on my computer because the machine was completely disk bound.

I'm hoping to do four things with this article:

1. Provide some information for anyone else who may encounter this terribly frustrating problem.
2. Get a conversation started by others who've seen this and worked through it also -- anyone?
3. Encourage developers to consider disk I/O in a time when a lot of people have SSDs but still a lot of people still don't have them.
4. Provide the DiscoFiles utility I wrote which will help you track files being updated, created, etc.

First Encounter

The first thing I noticed a few days ago was when I sat down to log into my computer.  I typed the password and hit <ENTER> and the machine just sat there hitting the disk.  I waited a long while and finally powered it down.  That may have caused the additional problems.  I'm not sure.

When my computer restarted and came up the I/O still went crazy.

Chrome Web Browser?

At first it seemed that it was related to Chrome because each time I attempted to start it, it would never show up and the machine would become so I/O bound I couldn't do anything.  At one point I was able to get to TaskManager and I could see that the Chrome process was running but there was no Chrome UI.

Could Not Kill Chrome Process

However, I could not kill the Chrome process. It told me that I did not have the rights to do so even though I'm an admin on the box.  That made me think maybe I had a virus or something.  

Chkdsk ?

At this point I rebooted and ran chkdsk but it didn't show anything bad about the drive or files.

Installed Trial of Kaspersky

At this point I rebooted and installed Kaspersky Anti-Virus.  I thought maybe it was a long shot but I wanted to make sure.  I installed it and ran it over night and I am happy to report that I have no viruses on my computer.

Day Two

I still didn't have an answer and the machine was still going high on disk I/O and becoming unusable.

Here's what TaskManager was showing during these times.  You can see that these are core Win10 system processes that aren't under the user's control.

You can see Cortana doing something in there even though I wasn't utilizing that app.  I killed it and made sure it couldn't start again, even changing options so it couldn't run any more, unless I started it manually.

Once your system gets to 100% disk utilization (at least on an HDD) you will find that you cannot really do anything.  If it stays at 100% for any length of time there won't be anything you can do.  You cannot switch to TaskManager, none of your windows will update, etc.

Day Three

I finally got it stabilized by not running Chrome at all.  However, I still wanted Chrome on my machine.  Yes, it is a convenience thing.  I like the way Chrome allows me to manage my bookmarks over numerous computers and I have an Android phone too.

Uninstall Chrome

I uninstalled Chrome and attempted to install again.  As I was doing so, I saw some other apps doing things to the disk.  I couldn't get snapshots of those but they were Office365 (which I no longer have a subscription too) and VSixAutoUpdate which was Visual Studio 2015 attempting to update.  

I was beginning to believe this was a: 

Perfect Storm of Updates / Disk I/O Utilization?

The thing that seems to occur is that multiple applications all attempt to pull down update data at the same time.  A Perfect Storm of I/O Destruction!!

After killing numerous processes and setting almost everything to not start upon startup, I decided to follow the Chrome install again.

Install Chrome Again

When I started installing Chrome, I seemed to be hit by something else.  I believe this may indicate that Chrome was at the root of my problems but I also believe it shows a problem that emerges under Windows 10 (maybe this always occurred, I'm not sure).

In this next image you can see that while installing Chrome a "Profile Error" occurred and the Microsoft Telemetry item is destroying I/O.  28.6MB/s???

Just Had To Wait...and Wait

All I could do is wait.  I waited a long time and even TaskManager wasn't responding.  

Microsoft Sends Error Report (Telemetry)

Finally, it seemed that after Microsoft sent the error report the computer became responsive again.

Additional Information

After posting the first version of this article, I still ran into a bit more disk I/O problems and had to wait a long time again to get access to my system because I attempted to run Google Chrome again as a test.

At one point, I saw the following popup dialog which seems to be from one of the background services that Microsoft runs:

Finally, the machine became completely I/O bound and the Explorer shell crashed.  At that point the only thing running was TaskManager.

OneDrive Versus Google Drive

There may be some indications that this problem could be related to having both Google Drive and OneDrive running on your machine.  This was reported to me by a coworker who recently said his wife's Win10 computer was overwhelmed by disk I/O problems and that is what he found also.  Something similar was also reported by a reader of this article in the comments section. 

I uninstalled OneDrive (seems to be installed by Win10 by default) since I run Google Drive.  I am now running Google Drive again (as I had it shut down to insure it was not causing all the issues).  So far running Google Drive alone is not causing any issues.

DiscoFiles : More Information

I wrote a program I call DiscoFiles (Discover Files) quite a few years ago so I could see what was going on with I/O.  There's a lot of hidden stuff : programs that access your HDD / SSD without you knowing it.

It's not a perfect program but it uses FileSystemWatcher to let you spy on files that are being created, updated, deleted or renamed on your disk.  Here's what I saw even after the computer stabilized.

SoftLanding..???  I have no idea what that is, but obviously it's some kind of Microsoft Diagnostic tool.

Using DiscoFiles

Get the download zip and build using Visual Studio 2013 or newer.

You can then start the app and watch when files are:

• changed
• created
• deleted
• renamed

It implements the .NET FileSystemWatcher class and updates the UI any time one of those events occurred (if you've chosen the associated checkbox.

Double-Click A Path To Go There

If you see an interesting file, you can double-click the line it appears on and the app will take you to that folder location if you have rights.

It looks something like this:

You will learn a lot of things as you discover all the files that are created and written to all over your disk.  For example, see that prefetch folder in the listing above?  That is a special windows folder used by the .NET CLR when a .NET app starts.  

Update : Fritzing Software Changes LockFile Continually

Here's an example I stumbled upon while using the Fritzing (electronic circuit drawing) software.  If you have a file open in Fritzing the the software continually updates the lockfile.  If you look inside that file, it contains one character: the letter 'a' in mine.  Obviously the developers use it to determine if the file is in use.  But they check that file multiple times a second and update it.  You can see it in the image it has updated the file 1367 times in the short time I have watched it. 

I think the devs could think of a different way of determining if the file is updated.  

FileSystemWatcher Work - Background Thread

The FileSystemWatcher work can be so overwhelming to the DiscoFiles UI that I have placed the update to the UI on a background thread so it won't get overwhelmed.  You'll see that code in the MainForm when the source creates a new WorkEvent.  

private void HandleFileSystemWatcherUpdate(EventType eventType, System.IO.FileSystemEventArgs e)
  {
      string fileName = fileSystemWatcher.Path + e.Name;
      if (checkFilter(Path.GetDirectoryName(fileName)))
      {

          string eventTypeString = string.Empty;
          switch (eventType)
          {
              case EventType.changed:
                  {
                      if (!fileChangedCheckBox.Checked)
                      {
                          // return immediately, if the user doesn't want to  listen for this event
                          return;
                      }
                      eventTypeString = "changed";

                      break;
                  }
              case EventType.created:
                  {
                      if (!fileCreatedCheckBox.Checked)
                      {
                          // return immediately, if the user doesn't want to  listen for this event
                          return;
                      }

                      eventTypeString = "created";
                      break;
                  }
              case EventType.deleted:
                  {
                      if (!fileDeletedCheckBox.Checked)
                      {
                          // return immediately, if the user doesn't want to  listen for this event
                          return;
                      }
                      eventTypeString = "deleted";
                      break;
                  }
              case EventType.renamed:
                  {
                      if (!fileRenamedCheckBox.Checked)
                      {
                          // return immediately, if the user doesn't want to  listen for this event
                          return;
                      }

                      eventTypeString = "renamed";
                      break;
                  }
          }

          ListViewItem currItem = null;
          lock (infoListView)
          {
              try
              {
                  foreach (ListViewItem lvi in infoListView.Items)
                  {
                      if (lvi.Text == fileName && lvi.Tag.ToString() == eventTypeString)
                      {
                          currItem = lvi;
                          continue;
                      }

                  }
              }
              catch
              {
              }
          }

          WorkEventArg dwo = new WorkEventArg(fileName, eventTypeString, currItem);

          // DoWorkEventArgs dw = new DoWorkEventArgs(dwo);

          if (!changeEventBackgroundWorker.IsBusy)
          {
              changeEventBackgroundWorker.RunWorkerAsync(dwo);
          }
          else
          {
              WorkEvents.Instance.Add(dwo);
          }
      }
  }

The Points

1. Even in this day of extremely fast computers, we are still all I/O bound. (Yes, I need to buy an SSD.)
2. There are things running on your computer that utilize I/O that you don't know about and they can literally bring your computer to its virtual knees.
3. Please, Microsoft, think about what you are doing to my computer when you are just "compiling error information" to send back to send back to the mothership.
4. As a dev, consider how your apps utilize I/O so you don't kill my computer.
5. Whenever you are wondering where an app is writing a file try the DiscoFiles utility.  It works with UNC paths too (as long as you have read rights).

Update Note - v1.0.3.0

While running DiscoFiles while AndroidStudio v2.3 was running I discovered that the app would throw an unhandled exception that was related to PATH_TOO_LONG.  It seems that Android Studio actually creates files that are longer than MAX_PATH.  I'm not sure how this is even handled on Windows systems, but it's quite an interesting development.

Instead of popping up a dialog which interrupts DiscoFiles processing, the app will instead simply create a log file named discoFilesError.log which it will write to the location where you started the DiscoFiles.exe.  Just make sure you have rights to that path or when an exception gets thrown, if it cannot write to the location another exception will be thrown. 

I've also added the version info in the app's title text so you can know which version you are running.

History

2017-03-07 : Discovered path too long exception when Android Studio 2.3 was compiling my Android app. Added fix, explanation and version 1.0.30 of exe and source code for download.

2017-03-06 : Added DiscoFiles.exe zip so you can easily download the utility and try it. Added additional example of finding files being updated often. 

2017-03-05 : Updated introduction.  Added more details of disk I/O failures

2017-03-04 : First version of article and code release