Click here to Skip to main content
15,892,746 members
Please Sign up or sign in to vote.
3.40/5 (2 votes)
See more:
Hi,

I have 100 xml files, I want assign 100 files into 100 threads and these threads parallel execution.

when I read the 100 files using parallel.foreach loop I pass my file into somemethod(). The file inserted data in this method.

I want
100 files-> 100 somemethod()

this 100 methods under threads executed parallely.

Please help me and suggest me, this way correct or not.

Than you......
Posted
Updated 8-May-14 20:28pm
v2
Comments
Bernhard Hiller 9-May-14 2:50am    
Are you sure that that is useful?
Consider:
- How many threads can really run in parallel on your computer?
- How many hard disk accesses can be done at the same time?
- How long does in memory-processing take in comparison to hard disk access?
NagaRaju Pesarlanka 9-May-14 2:56am    
yes this is very useful for me, I have to give between 100 to 1000 files at a time in a folder.
My server configurations is 2TB harddisk and 32GB RAM and 64 bit OS.
agent_kruger 9-May-14 10:10am    
sir, does your thread means "background processes"?
NagaRaju Pesarlanka 9-May-14 2:57am    
this is correct way or not please suggest me.
CPallini 9-May-14 4:34am    
As far as I know Parallel.ForEach already executes the loop using multiple threads. Why do you think you need another approach?

Seeing the comments, what could help improving your performance is to launch a "pipeline".

Read in XML and pass it to a thread that will process it to database. While that one is running you can start reading in the second file. There are many articles out there that handle multi-threading. I would run through an article and try a small prototype before tackling your main application.

hope this helps.
 
Share this answer
 
Comments
CHill60 9-May-14 8:42am    
5'd for the idea! I would probably expand on this to point out that the whole point of the pipeline is that you keep filling it at one end and the process at the other end just keeps emptying it as fast as it can. It's reminiscent of the Theory of Contraints. M$ even have a Pipelines article[^]
V. 9-May-14 8:47am    
thanks :-)
Matt T Heffron 9-May-14 12:59pm    
+5
You have to define the scope of the problem you're trying to solve. In other words, where is the bottleneck holding up your processing of those files.

The higher the number of threads you have does NOT mean your processing occurs faster.

The benefit you get depends on the scope of the problem. How big are these XML files? Is there a lot of processing that needs to take place on the data? How are you doing the inserts into the database? Bulk insert or individual record inserts?

In other words, is the scope of the problem I/O bound, meaning is the time being wasted waiting for disk I/O to complete or is it waiting for a network access or an SQL query?

Is the problem compute bound? Is there a lot of processing of data that needs to be done before storing the results in your database?

If the problem is compute bound, the number of cores you have in all of your CPU's on the machine running your code determines the maximum number of threads you can expect to execute at the same time. This is NO WAY MENAS THAT YOU WILL HAVE x number of threads running at the exact same time. Remember, Windows already has a few hundred threads running in various processes besides yours. You share the CPU with every other process running on the machine.

You can have more threads created, waiting to run, than there are cores on the CPU, but swamping the CPU with work will only make your threads wait to run more than they are actually running.

As for I/O bound problems, it really depends on where the bottleneck is. Remember, a single disk can only ever read one bit at a time, for one thread at a time. So if your problem is disk I/O bound, throwing a couple hundred threads at the problem will do you no good at all.

If the I/O problem is waiting for network or SQL processing to finish, again, throwing all kinds of threads at the problem will only result in hundreds of threads waiting around for something to complete.
 
Share this answer
 
v2
Comments
Matt T Heffron 9-May-14 12:59pm    
+5

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900