WWW DSL using Irony - Part 2

• Download source code - 41 KB

Introduction

In my previous article I explained how to create a Domain Specific Language to automate file downloads from different websites. This time, I want to expand the solution with a WinForms application that allows to easily download an arbitrary number of files from an arbitrary number of websites.

Background

The application uses the wwwdsl assembly presented from the previous article. The download archive contains the sample application and the project with the wwwdsl assembly.

Description

The main point of the solution is to allow the possibility to easily start downloading multiple files. The user does not have to care about the progress (e.g., wait to press a button on the web page). The user has to provide the application with recipes that define the processing. How do we create a recipe file - please check the previous article. The solution contains the files for two sources: a plain file download from a web server (download.wwwdsl) and a Rapidshare recipe (rs.wwwdsl). The user can register a recipe using the "Recipes..." button.

In the dialog box that appears, the user has to select the recipe file, give the recipe a name, and mark if simultaneous processing is allowed. For the plain file download, it shall be marked as allowed; for Rapidshare, rather not (if you are a free user).

After recipes are set, the files to be downloaded can be selected using "Add entries...". Multiple lines with download locations can be pasted to a text box. Also, the recipe for files have to be selected.

The order in which files are downloaded depends on the position on the list in the main window (of course, only if download is possible; if not, the next item is chosen). To alter the order, the user can use the "up" and "down" buttons.

The last setting is the maximum number of working threads, i.e., the maximum number of concurrent downloads. After the setting is done, the user can start download using the "Start" button. If for some reason all downloads have to be aborted, the user can press the "Stop" button. Pressing "Start" again will restart the unfinished downloads. Items can be removed using the popup menu (or by pressing the "Delete" key).

The number of threads and the order can be changed during processing.

Points of interest

Multithreading

The application allows for multiple concurrent downloads, therefore it is a multi-threaded application :).

Threads are started when the user presses "Start". Threads select the items to process. Picking is done in a critical section to avoid race conditions (the same item being selected by two threads). If there are no more items to downloaded, threads wait on the monitor's conditional variable for a change or new items.

private List<Entry> m_entries; 

...

private void WorkerFun(ThreadData td)
{
   while (!td.Stop)
   {
      Entry entry = null;
      RecipeEvaluator rpc;
      WwwDslEvaluationContext evalContext;

      lock (m_entries)
      {
         rpc = null;
         foreach (var e in m_entries.Where(e => e.State == EntryState.Pending || 
                                           e.State == EntryState.Waiting))
         {
            rpc = EntryCheck(e);
            if (rpc != null)
            {
               entry = e;
               break;
            }
         }

         if (rpc == null)
         {
            Monitor.Wait(m_entries);
            if (td.Stop)
            {
               return;
            }
            continue;
         }

         rpc.Processed = true;

         entry.State = EntryState.Processing;

         evalContext = new WwwDslEvaluationContext(
                           rpc.CompilerContext.Runtime, rpc.RootNode);
         m_processedEntries.Add(evalContext, entry);

         m_threads[Thread.CurrentThread].EntryData = entry;
      }

      ...
}

GUI operations from different threads

Here is an additional remark connected with refreshing the grid control from different threads. They must not do this (only the main thread can manipulate the GUI). Therefore, the ControlRefresh method checks if the control can be manipulated (using the InvokeRequired property). In such a case, a call to BeginInvoke starts the ControlRefresh method in the main thread context.

private DateTime _lastRefresh = DateTime.Now;
private void ControlRefresh(bool forced)
{
   if (m_control.InvokeRequired)
   {
      if (forced || (DateTime.Now - _lastRefresh.AddSeconds(1)).TotalSeconds >= 1)
      {
         m_control.BeginInvoke(new MethodInvoker(delegate { ControlRefresh(forced); }));
         _lastRefresh = DateTime.Now;
      }
   }
   else
   {
      m_control.Refresh();
   }
}

Logging

The application performs logging of the operations performed during download. In the solution, logging is directed into a text file. For each download item, a log file is created. To avoid collisions, log file names are GUIDs. The folder where log files are stored can be selected by the user using the "Log dir..." button on the main form.

Additional remark - I am not a GUI guy. I do not know how to make all these WPF blinking buttons. Therefore, please be merciful and do not comment on the GUI side of the solution :)