Click here to Skip to main content
15,891,607 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
i have this code which i run with queued 10,000, but the threadpool counts goes to high.

can anyone help me fish out the fault



Public Sub DoWork(ByVal objItem As Object)
		static  _sem as Semaphore  = new Semaphore(5,10)    '// Capacity of 10
		Dim objUrl as String = DirectCast(objItem, string)
		Try
			If objUrl Is Nothing Then
				exit sub
			End If
			
			Dim URL as New Uri(objUrl)
			AddItem(URL.AbsoluteUri,gethtml(URL))
		Catch ex As Exception
			debug.Print("Error " & ex.StackTrace )
		End Try
		
	End Sub
	
	Private Sub btnStart_Click(ByVal sender As System.Object, _
		ByVal e As System.EventArgs) Handles btnStart.Click
		btnStart.Enabled =False
		ThreadPool.SetMaxThreads(50, 50)
		System.Threading.Thread.Sleep(1000)
		listView1.Items.Clear
		
		For Each sItem As String In txturls.Lines
			if sItem.Trim <>string.empty
				' Queue a task
				If sItem.Contains("http://")=False Then
					sItem="http://" & sItem
				End If
				
				ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf DoWork), sItem)
				TotalItems+=1
				tsslTotal.Text =String.Format ("Total Tasks: {0}",TotalItems)
				
			end if
		Next
		
	End Sub


thanks
Posted
Updated 26-Jun-11 10:16am
v2

1 solution

The whole idea of spawning a new thread on some UI event is wrong even if you use the thread pool. You can easily deplete your system without any use. You need thread if you have logically different procedures to run in parallel. Think about it: what parallelism do you gain if you simply do the same things at once? Maybe, this is just experimenting, then I would understand. In real life you usually get get around just using some fixed amount of threads, one per each logically distinct and relatively independent activity. You can also work at the same task in more then one thread passing data from one thread to another. What you do does not seem to have any practical sense.

If you explain the goals of your application you might have an advice on what to do. Just posting a code dump does not help much.

—SA
 
Share this answer
 
Comments
Cool Smith 27-Jun-11 4:04am    
Thanks for answering, am trying to process about 100,1000 websites, looking for a particular keywords, so first i need to GetHtml of the website then AddItem of the result to listview. what is the bestway of doing this, am using .net 2.0
Sergey Alexandrovich Kryukov 29-Jun-11 5:42am    
A crowler? Fixed number of threads, perhaps configurable, no less then the number of cores/CPUs, but not much more. Each thread if fed with tasks processed processing sequentially. On thread to schedule and control all that, perhaps. Please understand: you gain no performance running that many threads, only loose.

--SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900