|
Guys, I need an idea about implementing multithreading in vb.net.
Here's the scenario:
I've to bulk insert multiple (100's) of files into SQL Server. The list of files are stored in a table and there's an SSIS package that loops through the resultset and bulkloads each file. The downside to this approach is at any given time only one file is being loaded. I'd like to introduce parallelism into this approach. ( I can have multiple loops, but it breaks the workflow)
I redesigned the package to load multiple files into the db, but the structure is rigid. For example, if in a batch of 5 files loading simultaneously, if the first file were smaller and the rest very big, one connection would be idle until all have finished to exit out of the task.
To alleviate this, I thought of an approach that uses threading.
What I need:
An idea that uses a multithreading wherein there's a parent thread that loops through the list of files and assigns each file to child threads which do the bulk load. As soon as a child thread is done, it must notify the parent to give it the next file to load. The number if child threads must be user configurable.
Here are the questions:
1) Is the idea of any worth, if not, is there a better way of doing it?
2) I'm trying to use the ThreadPool class and callback delegates, is this correct? (I read from Marc's article that ThreadPool is not worth it's salt)
3) Would the Async callback method of the SQLCommand work?(I kinda figured that this is not feasible, since the callback delegate does not notify the success or failure of the query)
4) Any good articles that can goad me in the correct direction.
Requirement:
The VB.NET module must execute from within an SSIS Script task.
And it has to be in VB.NET, C# is not supported in SSIS scripts.
Hope I've been clear enough.
Thanks.
SG
Cause is effect concealed. Effect is cause revealed.
modified on Wednesday, January 02, 2008 9:16:45 AM
|
|
|
|
|
SimulationofSai wrote: I'd like to introduce parallelism into this approach.
First question: Why? What's the point of doing multiple threads? One of the biggest misconceptions about multithreading is that is make the process run faster. In general, no, it doesn't. Depending on where the processing bottlenecks are, you might see a small performance benefit, you might not.
|
|
|
|
|
Because in a server with multiple hard drives and multiple CPU's, and with the SQL Server data files located in different volumes, the I/O might get divided across different processors.
I particularly refer to I/O because the most intensive operation in a BULKLOAD is I/O.
Executing 5 BULK LOAD commands in a single SQL connection uses only one processor, but splitting it across multiple connections actually fan out to different processors and cut short the load time. I know that it can only run as fast as the fastest I/O thats taking place.
Dave Kreskowiak wrote: Depending on where the processing bottlenecks are, you might see a small performance benefit, you might not.
When I'm loading more than 250 files of 500MB each, I'd be willing to take any time gain thats available. A difference of 2 minutes gain for a couple of files might mean 2-4 hours gain for all my files.
SG
Cause is effect concealed. Effect is cause revealed.
|
|
|
|
|
Fair enough.
It's possible to do with threads, but I'd probably think about managing my own thread pool for this, keeping the number of threads down to the number of processors in the box. I'd even consider "numProcessors - 1" given the length of time this operation takes. This would be to keep the SQL server free to do its work for other clients.
Now, I'm no expert in SSIS for sure. I have precious little experience with it. (I think this question might be geared more for the SQL forum and not the VB.NET forum.) But, from what I've read, I'd probably create a package that works with it's own connection and file, then terminates. A second, "master" package would then launch your VB.NET code, which organizes the files to be imported, manages it's own thread pool, and launches instances of a class that executes the first SSIS package.
|
|
|
|
|
Hi Everyone,
I am writing a vb.net window application. I want to add some rows into datatable.
The problem I am having is when i first add a new row to the datatable, I get very
slow performance (20-30 second delay). The add to datatable works but I
get slow performance. Any record adds or deletes after this first add
are fast. I get slow performance on the first add. I have no idea what
is causing this slow performance. I cannot figure out what is
delaying the add. Any thoughts on where I might be going wrong?
Thanks in advanced.
Here is my codes:-
Private Sub btnAdd_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnAdd.Click
If myTable Is Nothing Then InitGrid()
Dim Row As DataRow
Try
For Each Row In myTable.Rows
If Row.Item("HBarCode").ToString = Me.txtHBarCode.Text Then
MessageBox.Show("Housing Serial Number already scanned.Duplicated.")
txtHBarCode.Focus()
Exit Sub
End If
If Row.Item("EMIBarCode").ToString = Me.txtEMIBarCode.Text Then
MessageBox.Show("EMI Serial Number already scanned.Duplicated.")
txtEMIBarCode.Focus()
Exit Sub
End If
Next
Row = myTable.NewRow()
Me.txtCounter.Text = myTable.Rows.Count + 1
Row.Item("HBarCode") = Me.txtHBarCode.Text
Row.Item("EMIBarCode") = Me.txtEMIBarCode.Text
'to check any colume without data, then only add row at myTable
If Me.txtHBarCode.Text.Trim.Length = 0 Then
MessageBox.Show("Please Scan in Housing BarCode.")
Me.txtHBarCode.Focus()
Exit Sub
End If
If Me.txtEMIBarCode.Text.Trim.Length = 0 Then
MessageBox.Show("Please Scan in EMI BarCode.")
Me.txtEMIBarCode.Focus()
Exit Sub
End If
myTable.Rows.Add(Row)
Me.txtHBarCode.Text = ""
Me.txtEMIBarCode.Text = ""
Me.txtHBarCode.Focus()
If Me.txtCounter.Text = Me.txtQty.Text Then
txtQty.Enabled = False
btnComplete.Enabled = True
txtHBarCode.Enabled = False
txtEMIBarCode.Enabled = False
End If
Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
dgResult.Update()
End Sub
Public Sub InitGrid()
myTable = New DataTable("LotNumber")
Dim HBarCode As DataColumn = New DataColumn("HBarCode")
HBarCode.DataType = System.Type.GetType("System.String")
myTable.Columns.Add(HBarCode)
Dim EMIBarCode As DataColumn = New DataColumn("EMIBarCode")
EMIBarCode.DataType = System.Type.GetType("System.String")
myTable.Columns.Add(EMIBarCode)
ds = New DataSet
ds.Tables.Add(myTable)
dgResult.SetDataBinding(ds, "LotNumber")
End Sub
thanks,eunice
|
|
|
|
|
There's nothing in the code you've posted that will do what you describe. You're going to have to step through the code in the Debugger, line-by-line, and step into all methods and functions (F11) to see on what line the code is hanging up. Set a breakpoint on the first line of the button Click event handler code and start from there.
|
|
|
|
|
hi guys
my application is not running under win.2000 though i have made intranet zone fully trusted , though i am not getting any exception on win xp, i am running exe from network path,,
what could be the cause, ?
thansk in advance
....
and second one i have win2000 virtual machine installed on my pc, but in the admin tools,
it doesnt show framework 2 configuration ,, thoug framework 2 is installed also on virtual machine ,,,
thanks again
hello
|
|
|
|
|
hi
First is your application a .net APPLICATION? (MSI)FILE NOT eXE?
And what is the exception that you get when you try to run it in win 2000?
thanks
Vuyiswa Maseko,
Sorrow is Better than Laughter, it may Sadden your Face, but It sharpens your Understanding
VB.NET/SQL7/2000/2005
http://vuyiswamb.007ihost.com
http://Ecadre.007ihost.com
vuyiswam@tshwane.gov.za
|
|
|
|
|
hello frnds,
i want to append one word file to another to anoter how can i do that ?
thankx in advance
koolprasad2003 
Be a good listener...Because Opprtunity knoughts softly...N-Joy
|
|
|
|
|
|
hai all,
can you give me an idea to compress the 'folder' in vb.net.
I wish it is better to not import external dll.
Thank You in Advance
Senthil S
Software Engineer
|
|
|
|
|
DO you want to specifically compress a folder?? Compressing all files in the folder will not work??
[I think you want to zip the folder? Am i right?  ]
modified on Wednesday, January 02, 2008 6:05:35 AM
|
|
|
|
|
Hi, Now i create a setup project to deploy my application. Now I know how to create the setup project but I get stuck with deploy a custom folder. I need to place a folder name DbFile in the root drive C:\ but I don't know which properties that I need to set in the properties windows in order to create this folder in the root drive C:\ of the target client when I install my application.
Thank in advance!
|
|
|
|
|
This is a really bad practice so I discourage you from doing it. Any application specific files should be under the app's Program Files folder or under Common Files or the User's Application Data Folder.
But, I also realzie that there are few (very few) circumstances where this is a must. All you have to do is right-click your setup project, click View, then File System. In the File System editor window, right-click the "File System on Target Machine" line and pick Add Special Folder. The in that menu, click on Custom Folder. Rename the folder to whatever you want and in it's DefaultLocation property, change it to "%SYSTEMDRIVE%\"
|
|
|
|
|
|
Dear Sirs,
in our vb.net application we need to show on the form, an e-mail that comes from outlook and it is stored in .msg extension, like the way outlook does. Any idea how to do that? Maybe there is a component that does this ?
ninkk
|
|
|
|
|
You can either use Outlook, through it's automation interface (Outlook Express does NOT have one!), or start Googling around for a third party component to do it. I started searching with "vb.net component .msg file[^]".
|
|
|
|
|
HI all
I have an application in which I am reading a image from a file & loading it into picture box.
I wanted the picture to be stretched in as per the picture box…so set the sizemode property of picture box to stretch
But the image is not stretching..
I have following doubts
- I am using file stream to read the picture. Is that has any thing to do with it?
- I am using that Picture box to allow the user to use as MsPaint and have a bitmap and graphic object associated with it. So am I supposed to close both or any of them before I can load and stretch the image???
Regards
Nishkarsh
|
|
|
|
|
The streams have nothing to do with streching the image.
nishkarsh_k wrote: I wanted the picture to be stretched in as per the picture box…so set the sizemode property of picture box to stretch
Are you expecting the image to strech with a resizing PictureBox?? If the PictureBox isn't anchored to the sides of the form, the image won't resize because the PictureBox isn't resizing.
All you have to do to get it to work is set the Image property to the image you want and change the SizeMode to StrechImage. That's it. If you want something else, you're going to have to describe what you want it to do.
|
|
|
|
|
I had done all that u told but it dose not works only at that place.
At all other places it dose.
The only difference at this place in from other places is that the picture box is placed inside a panel
|
|
|
|
|
hi
can any one tell me how to store & retrive images from Access.
poonam
|
|
|
|
|
|
hi,
i m developing a text editor. I m using Dhtml edit control rather than richtext box control. My problem is, i wana limit the number of lines in dhtmal edit control. that is i want that a user can enter only 20 lines in dhtml edit control. and after than he has to go on the other page for writing further. so is it possible to limit the number of lines in dhtml edit control. Thx in advance...
King
|
|
|
|
|
Nope. There is no property that exposes this. The DHTMLEdit control is design to edit web pages, not Rich or normal Text.
|
|
|
|
|
thx for reply me.
Can u tell me how i can do the same thing in richtext box control that is if the user writes 20 lines then how can i forces him to further write on the next page. plz reply me..... n thx in advance
King
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
