Click here to Skip to main content
15,879,535 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more:
Hi
I have been struggling to try and split a list of approx 200,000 List(Of Strings) into evenly split arrays for thread processing.

The user is specifying the number of threads (capped obviously) to determine best processing speed for their machine.

If i output the records in the main list: (left out surrounding code)
VB
Console.Write(SQLStrings.Count)

It Gives me
Preperation of SQL Now Complete: 183096 Records to Process

Yet When i run the below code it gives me the following:

Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18297


The Total of which is 183097 which is 1 records over, and for the life of me i cannot figure out why i am 1 over (tried several different loops and i keep getting the same result.

Is there a more structure way of splitting a List(Of String) into Array(10) for example to evenly balance it out as these will go into a thread pool for processing.

The reason for this is that is it currently processing much slower than it should be so i am tweaking the insertion of the records which could be up to 1mil records (data gathering from several sources) takes quite some time on a single connection.


VB
'Instantiate Pools
        Dim ThreadPoolSql(MaxThreadPoolSize - 1) As List(Of String)
        For tmp As Integer = 0 To MaxThreadPoolSize - 1
            ThreadPoolSql(tmp) = New List(Of String)
        Next

        Dim PerPoolCount As Integer = SQLStrings.Count / MaxThreadPoolSize + 1
        Dim RunningLoop As Integer = 0, RunningCount As Integer = 0
        For ListItm As Integer = 0 To SQLStrings.Count - 1
            'Add SqlString to ThreadPool List
            ThreadPoolSql(RunningLoop).Add(SQLStrings(ListItm))

            'Incrememnt Run Counter
            RunningCount += 1

            If (RunningCount = PerPoolCount) Then
                RunningLoop += 1
                RunningCount = 0
            End If
        Next
Posted
Comments
Matt T Heffron 17-Sep-15 13:15pm    
When I add up the shown Pool values I get 183096 which is exactly as it should be! Right?
Dev O'Connor 17-Sep-15 16:23pm    
Lol.... i checked back over the outputs, it used to be 8 and i didnt realise it had changed!! sorry about that
Patrice T 17-Sep-15 13:48pm    
Run this code with a debugger and a list of 97, and you will see what it does.

1 solution

Value changed after the last code change, it used to be 8 and i didnt realise it had changed.
 
Share this answer
 
Comments
Patrice T 17-Sep-15 18:33pm    
Don't use answer to add details to a question, use Improve question instead.
Dev O'Connor 18-Sep-15 4:14am    
Sorry, didnt realise that was what you was meant to do. I hate having 'unanswer' questions on my profile, irritates me. Ill make sure to improve solution next time.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900