Re: Thread, WaitSleepJoin - C# Discussion Boards

21-Jul-07 5:59

I start four threads after each other because I want them to run simultaneously, the problem is that they only run one at a time.
The other threads has the state "WaitSleepJoin".

Any idea why this could be?

starting them with the code below, dThreads is an ArrayList:

dThreads.Add(new Thread(new ParameterizedThreadStart(test)));
string s = offset + ";" + (offset + partSize) + ";" + i;
((Thread) dThreads[i]).Start(s);

Luc Pattyn21-Jul-07 6:22

21-Jul-07 6:22

Cant tell from the little you told us. Do you care to show more of the code ?

what is i ? are you doing something with priorities ? are there any Thread.Sleep ?
what's inside the test() method: is there any locking involved, explicitly or implicitly ?
etc.

if you want to solve this by yourself: add timestamps to the major actions;
this is the log method I would suggest you use:

public void log(string s) {
    int ID=Thread.CurrentThread.ManagedThreadId;
    s=DateTime.Now.ToString("mm:ss.fff")+" ["+ID.ToString("X4")+"] "+s;
    Console.WriteLine(s);
}

Now insert log statements "everywhere", at least at the start and end of the test method,
and right before your line ... .Start(s); use the string argument to describe what you are
going to do, or just did.

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson21-Jul-07 6:54

21-Jul-07 6:54

Thanks for the reply, yes I was vague, sorry.

What I'm doing is downloading a file, in four parts.
These parts should download at the same time.

I have one download-class that I call four times.
So I'm doing four threads that each starts a new thread in the download-class.

There is no sleeping or locking.

        private ArrayList downloads;<br />
        private void StartDownload()<br />
        {<br />
            downloads = new ArrayList(1);<br />
            dThreads = new ArrayList(1);<br />
            // Create a request to the file we are downloading<br />
            webRequest = (HttpWebRequest)WebRequest.Create(downloadUrl);<br />
            // Set the proxy to use if any<br />
            if (!dowloadProxy.Equals(string.Empty))<br />
                webRequest.Proxy = new WebProxy(dowloadProxy);<br />
            // Set default authentication for retrieving the file<br />
            webRequest.Credentials = CredentialCache.DefaultCredentials;<br />
            // Retrieve the response from the server<br />
            webResponse = (HttpWebResponse)webRequest.GetResponse();<br />
            //Ask the server for the file size and store it<br />
            int fileSize = (int)webResponse.ContentLength;<br />
<br />
            int partSize = fileSize / parts;<br />
            int lastPart = fileSize - partSize * parts;<br />
<br />
            int offset = 0;<br />
            for(int i=0; i<parts; i++)<br />
            {<br />
                dThreads.Add(new Thread(new ParameterizedThreadStart(test)));<br />
                string s = offset + ";" + (offset + partSize) + ";" + i;<br />
                ((Thread) dThreads[i]).Start(s);<br />
<br />
                offset += partSize;<br />
<br />
                if (i == parts - 2)<br />
                    partSize += lastPart;<br />
            }<br />
        }<br />
<br />
        private void test(object nums)<br />
        {<br />
            string[] split = ((string) nums).Split(';');<br />
            int start = Int32.Parse(split[0]);<br />
            int stop = Int32.Parse(split[1]);<br />
            //int i = Int32.Parse(split[2]);<br />
<br />
            downloads.Add(new Download());<br />
            i = downloads.Count - 1;<br />
<br />
            ((Download)downloads[i]).StartPoint = start;<br />
            ((Download)downloads[i]).EndPoint = stop;<br />
<br />
            ((Download)downloads[i]).InitDownloadFile(downloadUrl);<br />
            ((Download)downloads[i]).DownloadFile(outFile + "." + i);<br />
        }<br />

Luc Pattyn21-Jul-07 7:13

21-Jul-07 7:13

Hi,

I have a few minor comments, that AFAIK dont relate to the problem:

- you create ArrayList instances with initial capacity=1, this is not wrong but also not
very useful. I would either not specify a capacity at all, or something that is likely
to be sufficient (4 in your case).

- you have if (i == parts - 2) partSize += lastPart; at the end of the for loop;
it would look more natural to have if (i == parts - 1) partSize += lastPart; at the
beginning of the loop. (its the last iteration that is special, not the last but one).

- you have downloads.Add(new Download()); i = downloads.Count - 1;
this is a weakness, if not a bug: if another thread runs the same test method and gets
switched in, then back out, your value of i might no longer point to the download you
just created. The solution is simple, and avoids those ugly casts later on:

Download download=new Download;
downloads.Add(download);
download.StartPoint=Start;
...

But overall it all seems OK, the problem I suspect is in the Download class; for one
reason or another (assuming your observations are right) the 4 downloads are not working
concurrently. So thats where you must look for locks, bottlenecks, or insufficient
resources.

I still am inclined to add logging with timestamps to see what happens when; I would
rather get more facts than try and formulate hypotheses about what might be happening
that would match the symptoms lacking some fundamental information.

Hope this helps.

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Luc Pattyn21-Jul-07 7:28

21-Jul-07 7:28

Hi,

sorry, some more comments:

- I missed the declaration and initialization of "parts"; are you sure it holds value 4 ?

- you seem to have a special way of passing several parameters to the test method.
.NET 2.0 has a new Thread.Start overload that supports multiple parameters.
An alternative is to use a class (say DownloadJob) that has a constructor with parameters
(or some public properties) and basically does what test does; you then create a
downloadJob isntance, make sure its parameters
are set the way you want (thru constructor or properties), then launch it.
So you probably dont need the downloadJobs arraylist any longer.

- you could as well add the above parameter handling to the existing Download class itself,
hence create Download instances and set parameters before creating the threads and
causing the Download method to run.

- and the next logical step is to move the thread creation inside the Download or
DownloadJob class itself;
so in the end you just create four jobs and launch them; the jobs are smart enough to each
create their own thread and run on it.

But again, this is, as far as I can see, not really where the problem is.

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson21-Jul-07 7:49

21-Jul-07 7:49

I will look into your suggestions, they look very interesting.
The download-methods in the download class looks like this:

<br />
        public void DownloadFile(string outPath)<br />
        {<br />
            outFile = outPath;<br />
            // Create a new thread that calls the StartDownload() method<br />
            thrDownload = new Thread(StartDownload);<br />
            // Start the thread, and thus call StartDownload()<br />
            thrDownload.Start();<br />
        }<br />
<br />
private void StartDownload()<br />
        {<br />
            try<br />
            {<br />
                // Create a request to the file we are downloading<br />
                webRequest = (HttpWebRequest)WebRequest.Create(dowloadUrl);<br />
                // Set the proxy to use if any<br />
                if (!dowloadProxy.Equals(string.Empty))<br />
                    webRequest.Proxy = new WebProxy(dowloadProxy);<br />
                // Set the startingpoint of the stream if any and resume is aloud<br />
                //if (startPoint > 0 && resumeIsAloud)<br />
                //    webRequest.AddRange(startPoint);<br />
<br />
                if(endOffset>0)<br />
                 webRequest.AddRange(startOffset, endOffset);<br />
                else<br />
                 webRequest.AddRange(startOffset);<br />
<br />
                // Set default authentication for retrieving the file<br />
                webRequest.Credentials = CredentialCache.DefaultCredentials;<br />
                // Retrieve the response from the server<br />
                webResponse = (HttpWebResponse)webRequest.GetResponse();<br />
                //Ask the server for the file size and store it<br />
                fileSize = webResponse.ContentLength;<br />
                // Get the response-stream<br />
                strResponse = webResponse.GetResponseStream();<br />
<br />
                // Set the output-stream<br />
                strLocal = new FileStream(outFile, FileMode.Create, FileAccess.Write, FileShare.None);<br />
<br />
                // It will store the current number of bytes we retrieved from the server<br />
                int bytesSize;<br />
                // A buffer for storing and writing the data retrieved from the server<br />
                byte[] downBuffer = new byte[2048];<br />
<br />
                // Set the start-time<br />
                downStart = DateTime.Now;<br />
<br />
                // Loop through the buffer until the buffer is empty<br />
                while ((bytesSize = strResponse.Read(downBuffer, 0, downBuffer.Length)) > 0)<br />
                {<br />
                    // Write the data from the buffer to the local hard drive<br />
                    strLocal.Write(downBuffer, 0, bytesSize);<br />
                    CalculateDownloadingInfo();<br />
                }<br />
            }<br />
            catch (Exception e)<br />
            {<br />
                lastError = e;<br />
                RaiseDownloadError(e);<br />
            }<br />
            finally<br />
            {<br />
                if (strResponse != null) strResponse.Close();<br />
                if (strLocal != null) strLocal.Close();<br />
                if (thrDownload != null) thrDownload.Abort();<br />
            }<br />
        }<br />

Luc Pattyn21-Jul-07 8:18

21-Jul-07 8:18

Hi,

I am stll studying the download method, but have an intermediate question:

are you sure the file server is capable of concurrently serving several requests that
feed from the same file ? this would require special care about FileShare options...

I think I must insist you add timestamp logging to the client, and if the server happens
to be yours, then to the server too.

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson21-Jul-07 9:18

21-Jul-07 9:18

I'm not sure if the servers are capable of several request but I have tried different servers and the result is the same.

FileShare is no problem as I create four different files and will piece them together in the end.

As for the logging, I'm not sure what your after but this is what I came up with as a start:

<br />
13:02.421 [000A] Starting download-thread<br />
13:02.437 [000C] Start test<br />
13:02.437 [000A] Starting download-thread<br />
13:02.468 [000C] End test<br />
The thread 0xfa0 has exited with code 0 (0x0).<br />
13:02.531 [000D] Start test<br />
13:02.562 [000A] Starting download-thread<br />
13:02.562 [000D] End test<br />
The thread 0x3b8 has exited with code 0 (0x0).<br />
13:02.578 [0010] Start test<br />
13:02.593 [000A] Starting download-thread<br />
13:02.609 [0010] End test<br />
The thread 0x1454 has exited with code 0 (0x0).<br />
13:02.609 [0011] Start test<br />
13:02.625 [0011] End test<br />
The thread 0x87c has exited with code 0 (0x0).<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
The thread 0x1318 has exited with code 0 (0x0).<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
The thread '<No Name>' (0x17b8) has exited with code 0 (0x0).<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
The thread 0x150c has exited with code 0 (0x0).<br />
The program '[1220] DownloadManager.vshost.exe: Managed' has exited with code 0 (0x0).

Luc Pattyn21-Jul-07 9:54

21-Jul-07 9:54

Hi,

lazaruz wrote:
FileShare is no problem as I create four different files and will piece them together in the end.

Right, I agree the client is OK on this, but my comment applied to the server: the server
needs to be capable of reading the same file more than once, hence it
must open it every time in such a way that it does not prevent another thread from also
opening it for reading. But since you tried several servers, that should not be the issue.

I studied the DownloadFile() method, and that seems OK too.
I was surprised to see yet another thread created. That's really one thread too many
(actually, its the other one that now has no further purpose, you could remove the thread
stuff entirely from the code you have shown at the beginning, that's the line
dThreads.Add(new Thread(new ParameterizedThreadStart(test))); and following lines.

I noticed you end the download thread with a suicide
if(thrDownload != null) thrDownload.Abort();
there is absolutely no need for that; when the method reaches its end (or otherwise returns)
the thread is done automatically.

This also applies to the test() method (the one I told you does not need separate
threads any longer), I havent seen the code at the end of it, but the log shows
the thread gets aborted...

What you can also see in the log is the four threads work completely sequentially:
the second thread does not even start before the first is all done; but they all are
very short, which is normal since all they do is they each start another thread inside
DownloadFile(). So everything here is looking OK.

So I now must suggest:
- you remove the redundant threading stuff in main or whatever it is called, not in
Download class.
- you add timestamp logging to the download class (just copy the method to that class,
and insert new log statements, at least before and after GetResponseStream(); more is better.

We definitely are getting closer now !

BTW: please also show the code for CalculateDownloadingInfo() since that is part of
what runs four times.

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson21-Jul-07 10:39

21-Jul-07 10:39

Ok, I have removed the threads and use only the threads in Download-class.

From the log I got:

<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.HostingProcess.Utilities\8.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.HostingProcess.Utilities.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Windows.Forms\2.0.0.0__b77a5c561934e089\System.Windows.Forms.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.HostingProcess.Utilities.Sync\8.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.HostingProcess.Utilities.Sync.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\Documents and Settings\Johan\Desktop\DownloadManager\DownloadManager\bin\Debug\DownloadManager.vshost.exe', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Deployment\2.0.0.0__b03f5f7f11d50a3a\System.Deployment.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Xml\2.0.0.0__b77a5c561934e089\System.Xml.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
The thread 0x13b8 has exited with code 0 (0x0).<br />
The thread 0x1a0 has exited with code 0 (0x0).<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\Documents and Settings\Johan\Desktop\DownloadManager\DownloadManager\bin\Debug\DownloadManager.exe', Symbols loaded.<br />
A first chance exception of type 'System.NullReferenceException' occurred in DownloadManager.exe<br />
'DownloadManager.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Configuration\2.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.<br />
31:10.671 [000A] Start DownloadFile-Thread<br />
31:10.687 [000A] Start DownloadFile-Thread<br />
31:10.703 [000A] Start DownloadFile-Thread<br />
31:10.703 [000A] Start DownloadFile-Thread<br />
31:11.046 [000C] Get the response-stream<br />
31:11.046 [000C] After Get the response-stream<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
The thread 0x1220 has exited with code 0 (0x0).<br />
31:15.687 [000D] Get the response-stream<br />
31:15.687 [000D] After Get the response-stream<br />
31:19.125 [000F] Get the response-stream<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
31:19.140 [000F] After Get the response-stream<br />
The thread 0xf10 has exited with code 0 (0x0).<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
The thread 0x1098 has exited with code 0 (0x0).<br />
A first chance exception of type 'System.Net.WebException' occurred in System.dll<br />
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll<br />
The thread 0x13f0 has exited with code 0 (0x0).<br />
The program '[3640] DownloadManager.vshost.exe: Managed' has exited with code 0 (0x0).<br />

The other method you wanted to see:

<br />
        // Calculates information of the download, speed, percent and remaining time<br />
        private void CalculateDownloadingInfo()<br />
        {<br />
            // Calculate speed<br />
            downElapsed = DateTime.Now - downStart;<br />
            secPassed = downElapsed.TotalSeconds;<br />
            kbsec = (strLocal.Length / 1024) / secPassed;<br />
<br />
            // Calculate percent<br />
            int percent = Convert.ToInt32((strLocal.Length * 100) / fileSize);<br />
<br />
            // Calculate remaining time<br />
            long kbLeft = (fileSize - strLocal.Length) / 1024;<br />
            int timeLeft = (int)(kbLeft / kbsec);<br />
<br />
            UpdateDownloadingInfo(percent, kbsec, timeLeft);<br />
        }<br />
<br />
        // Occurs when the download-info is updated<br />
        public event DownloadingEventHandler DownloadingEvent;<br />
        // Called when the download-info is updated<br />
        private void UpdateDownloadingInfo(int percent, double speed, int timeLeft)<br />
        {<br />
            // make sure the are some delegates to call<br />
            // all delegates will be called in the order they were added<br />
            // to the invocation list<br />
            if (DownloadingEvent != null)<br />
                DownloadingEvent(new DownloadingEventArgs(fileSize, strLocal.Length, percent, speed, timeLeft));<br />
        }<br />
<br />
    // Represents the method that will handle the Downloader.DownloadingEvent event.         <br />
    public delegate void DownloadingEventHandler(DownloadingEventArgs e);<br />
    // Class to handle the DownloadingEvent event. <br />
    public class DownloadingEventArgs : EventArgs<br />
    {<br />
        // Size of the file<br />
        private Int64 size;<br />
        // Number of bytes downloaded<br />
        private long bRead;<br />
        // Download percent<br />
        private int pcent;<br />
        // Download-speed<br />
        private int spd;<br />
        // Time left in seconds<br />
        private int tLeft;<br />
<br />
        public DownloadingEventArgs(Int64 fileSize, long bytesRead, int percent, double speed, int timeLeft)<br />
        {<br />
            size = fileSize;<br />
            bRead = bytesRead;<br />
            pcent = percent;<br />
            spd = (int)speed;<br />
            tLeft = timeLeft;<br />
        }<br />
        <br />
        // Get the size of the file <br />
        public Int64 FileSize { get { return size; } }<br />
        // Get the number of bytes that has been read <br />
        public long BytesRead { get { return bRead; } }<br />
        // Get the progress of the download in percent <br />
        public int Percent { get { return pcent; } }<br />
        // Get the current speed of the download in kb/seconds <br />
        public int Speed { get { return spd; } }<br />
        // Get the time left of the download progress <br />
        public int TimeLeft { get { return tLeft; } }<br />
    }<br />

Luc Pattyn21-Jul-07 11:03

21-Jul-07 11:03

Hi,

OK, lets not dive into CalculateDownloadingInfo() for the moment (it calls an event now,
for which I dont have the code...);

this is what I suggest:

- remove CalculateDownloadingInfo from the while loop (i.e. turn it onto a comment for now).
- remove the line if(thrDownload != null) thrDownload.Abort(); it seems to still be present.
- add say 5 more log statements inside StartDownload; say before and after webRequest=...;
before and after the while loop that copies the data; and at the end of finally.
- now run again, make sure it still works and still is slow, then study the log timestamps.
if everything is as you expect, the four threads should begin their job at about the same
time, and, if it still acts rather sequentially soon 3 of them come to a stand still.
The times will tell you exactly where that happens, and then you will have a good
opportunity to figure out why that happens.
- if you decide to post the log again, please also repost the code for StartDownload()

I would not be surprised it is the progress event that is taking too much time, and for
some reason causing things to run sequentially (hence things running better as long as
you keep it out).

Come to think of it, updating after every 2KB is way too often (if you
hope to transfer 1 MB/s that would result in an update every 2 msec !)

But even if that's true I cant yet explain the sequential behavior; so if removing the
update helps, I would like to see the code actually executed in updating the GUI...

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson21-Jul-07 11:32

21-Jul-07 11:32

well, here is the new log:

<br />
17:13.796 [000A] Start DownloadFile-Thread<br />
17:13.796 [000A] Start DownloadFile-Thread<br />
17:13.796 [000C] Do the webrequest<br />
17:13.796 [000A] Start DownloadFile-Thread<br />
17:13.796 [000D] Do the webrequest<br />
17:13.796 [000D] End the webrequest<br />
17:13.828 [000D] Retrieve the response from the server<br />
17:13.828 [000A] Start DownloadFile-Thread<br />
17:13.796 [000C] End the webrequest<br />
17:13.843 [000C] Retrieve the response from the server<br />
17:13.828 [000E] Do the webrequest<br />
17:13.843 [000F] Do the webrequest<br />
17:13.859 [000E] End the webrequest<br />
17:13.859 [000E] Retrieve the response from the server<br />
17:13.859 [000F] End the webrequest<br />
17:13.859 [000F] Retrieve the response from the server<br />
17:14.203 [000D] After Retrieve the response from the server<br />
17:14.203 [000D] Get the response-stream<br />
17:14.203 [000D] After Get the response-stream<br />
17:14.218 [000D] Start while-loop<br />
17:18.687 [000D] End while-loop<br />
The thread 0x9c4 has exited with code 0 (0x0).<br />
17:18.859 [000C] After Retrieve the response from the server<br />
17:18.859 [000C] Get the response-stream<br />
17:18.859 [000C] After Get the response-stream<br />
17:18.875 [000C] Start while-loop<br />
17:22.109 [000C] End while-loop<br />
17:22.109 [000F] After Retrieve the response from the server<br />
The thread 0x9c0 has exited with code 0 (0x0).<br />
17:22.125 [000F] Get the response-stream<br />
17:22.125 [000F] After Get the response-stream<br />
17:22.125 [000F] Start while-loop<br />
17:25.265 [000F] End while-loop<br />
The thread 0xf04 has exited with code 0 (0x0).<br />
The program '[4024] DownloadManager.vshost.exe: Managed' has exited with code 0 (0x0).<br />

and StartDownload:

<br />
private void StartDownload()<br />
        {<br />
            try<br />
            {<br />
                // Create a request to the file we are downloading<br />
                webRequest = (HttpWebRequest)WebRequest.Create(dowloadUrl);<br />
                // Set the proxy to use if any<br />
                if (!dowloadProxy.Equals(string.Empty))<br />
                    webRequest.Proxy = new WebProxy(dowloadProxy);<br />
                // Set the startingpoint of the stream if any and resume is aloud<br />
                //if (startPoint > 0 && resumeIsAloud)<br />
                //    webRequest.AddRange(startPoint);<br />
<br />
                log("Do the webrequest");<br />
                if(endOffset>0)<br />
                 webRequest.AddRange(startOffset, endOffset);<br />
                else<br />
                 webRequest.AddRange(startOffset);<br />
                log("End the webrequest");<br />
<br />
                // Set default authentication for retrieving the file<br />
                webRequest.Credentials = CredentialCache.DefaultCredentials;<br />
                log("Retrieve the response from the server");<br />
                // Retrieve the response from the server<br />
                webResponse = (HttpWebResponse)webRequest.GetResponse();<br />
                log("After Retrieve the response from the server");<br />
                //Ask the server for the file size and store it<br />
                fileSize = webResponse.ContentLength;<br />
                log("Get the response-stream");<br />
                // Get the response-stream<br />
                strResponse = webResponse.GetResponseStream();<br />
                log("After Get the response-stream");<br />
<br />
                // Set the output-stream<br />
                strLocal = new FileStream(outFile, FileMode.Create, FileAccess.Write, FileShare.Write);<br />
<br />
                // It will store the current number of bytes we retrieved from the server<br />
                int bytesSize;<br />
                // A buffer for storing and writing the data retrieved from the server<br />
                byte[] downBuffer = new byte[2048];<br />
<br />
                // Set the start-time<br />
                downStart = DateTime.Now;<br />
<br />
                log("Start while-loop");<br />
                // Loop through the buffer until the buffer is empty<br />
                while ((bytesSize = strResponse.Read(downBuffer, 0, downBuffer.Length)) > 0)<br />
                {<br />
                    // Write the data from the buffer to the local hard drive<br />
                    strLocal.Write(downBuffer, 0, bytesSize);<br />
                    //CalculateDownloadingInfo();<br />
                }<br />
                log("End while-loop");<br />
            }<br />
            catch (Exception e)<br />
            {<br />
                lastError = e;<br />
                RaiseDownloadError(e);<br />
            }<br />
            finally<br />
            {<br />
                if (strResponse != null) strResponse.Close();<br />
                if (strLocal != null) strLocal.Close();<br />
            }<br />
        }               <br />

That didn't seem to make that much of a difference and the threads still runs one after the other.

You can have a look at the whole thing here: http://johanmartensson.se/downloadmanager.rar

It's quite messy, but you'll get the idea Smile | :)

Luc Pattyn21-Jul-07 11:51

21-Jul-07 11:51

Well, tytping while I read it, it is not fully sequential; for instance we get several
"Retrieve the response from the server" close to each other (that is fine), but the
while loops are not intermingling.
So its in between those two that it starts to go wrong (i.e. sequential).

I'll study it some more...

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Luc Pattyn21-Jul-07 13:08

21-Jul-07 13:08

OK, this is the relevant part of the log:

17:13.796 [000C] Do the webrequest
17:13.796 [000D] Do the webrequest
17:13.796 [000D] End the webrequest
17:13.828 [000D] Retrieve the response from the server    --------------- good
17:13.796 [000C] End the webrequest
17:13.843 [000C] Retrieve the response from the server    --------------- good
17:13.828 [000E] Do the webrequest
17:13.843 [000F] Do the webrequest
17:13.859 [000E] End the webrequest
17:13.859 [000E] Retrieve the response from the server    --------------- good
17:13.859 [000F] End the webrequest
17:13.859 [000F] Retrieve the response from the server    --------------- good
17:14.203 [000D] After Retrieve the response from the server ============ good
17:14.203 [000D] Get the response-stream
17:14.203 [000D] After Get the response-stream
17:14.218 [000D] Start while-loop
17:18.687 [000D] End while-loop
The thread 0x9c4 has exited with code 0 (0x0).
17:18.859 [000C] After Retrieve the response from the server ============ this one is late

Observations:
- the while loop takes much longer than everything else; that's good, it tells us we
could make smaller read requests without much penalty if we would choose to do so,
and/or more GetResponse calls.
- the first thing to show after one of the threads are finished, is the line
"... After Retrieve the response from the server " so that is the one we would really like
to get sooner, much sooner.
The relevant code is:

log("Retrieve the response from the server");
webResponse = (HttpWebResponse)webRequest.GetResponse();
log("After Retrieve the response from the server");

so it is the GetResponse that fails to return on time. Why would that be ?
It seems like one strResponse must be closed before another one can return from GetResponse.
That is to say the other threads also issued their GetResponse() and are just waiting;
I would accept they must wait until the first GetResponse returns; but seeing them wait
until the while loop and close are done puzzles me (I am not a network expert tho).

I noticed the existence of BeginGetResponse and EndGetResponse, that support asynchronous
operation. Now asynchronous means the thread can continue and go on do other things;
what we dont know is, if four threads issue BeginGetResponse, will they intermingle any
better than what we have now. I think you will have to read up or Google on these.

Current summary:
- GUI update is not the cause of the problem
- we established 4 threads are starting download concurrently
- but they fall in sequential mode as soon as a GetResponse gets executed

Additional reading (I just googled "HttpRequest GetResponse concurrent" and browsed some
of the hits, without really reading/studying it all):
http://msdn.microsoft.com/msdnmag/issues/04/12/NETMatters/[^] explains .NET 1.1 needs ThreadPool threads to execute GetResponse, and
may throw a no-thread-available-exception. You are running >= 2.0 are you ?

http://discuss.joelonsoftware.com/default.asp?dotnet.12.519394[^] explains the first ever GetResponse takes very long,
so you might want to do a dummy long before you really need one. You could check by
making your program trying the same download twice and watch the timestamps.

http://www.feedshow.com/show_items-feed=04ba4cfa290afeb01d0b72f898516048[^] gives bad
news about GC temporarily disabled since threads sometimes can not be suspended.

GetResponse Performance Issue (first-time call)[^]

This one implies some good news:
http://episteme.arstechnica.com/groupee/forums/a/tpc/f/6330927813/m/969001547731[^] "My understanding of HttpWebRequest is that if you use .GetResponse instead of .BeginGetResponse then it doesn't matter how many threads you're running, the calls will block each other." Although it does not sound optimistic all the way !

Possible routes from here:

- you might stick with GetResponse and try to do much larger Reads (say 128KB or 1MB),
and do a GetResponse-Read-Close cycle; loop over these in each of the four threads.
Trick is the local stuff (writing to stream, GUI update) now happen after the close,
so another thread can get results from GetResponse in the mean time.

- try with async stuff (Begin/End); warning: MSDN states its an all or nothing choice
(all sync or all async). You may want to read this first.[^]

I hope this helps you enough to rethink and come up with a new, revised class.

Hope you keep us informed (and/or maybe write an article once solved).

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson21-Jul-07 22:14

21-Jul-07 22:14

Thanks for all your help, it's been very useful.
I will now read up on the GetResponse and see what I come up with.

I'll keep you posted and I will try to write an article in the end.

Luc Pattyn22-Jul-07 0:34

22-Jul-07 0:34

You're welcome.

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson24-Jul-07 0:15

24-Jul-07 0:15

The problem was with the ConnectionLimit on the webRequest.

So adding this line did the trick, where connectionLimit is an int:

webRequest.ServicePoint.ConnectionLimit = connectionLimit;

Luc Pattyn24-Jul-07 1:01

24-Jul-07 1:01

Thanks, and good luck. How fast is your download now (in KB/s or MB/s) with 1 and with
4 threads, so was it all worth it ?

Smile | :)

Luc Pattyn

try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] }
catch { [Google] }

Johan Martensson24-Jul-07 8:21

24-Jul-07 8:21

When I download a file that I get around 300KB/s on one thread, I get around 850KB/s on three threads.

So, yes I think it was worth it and besides, since I wanted to do it and it didn't work, I just had to figure it out, we can't let us be beaten by some code, can we? Poke tongue | ;-P

Luc Pattyn24-Jul-07 11:54