|
Hi
I have a text file which has thousands of records, I am opening it and populating it in dataset and then binding to a binding source. Then I am supposed to find specific item in specific column. In my scenario, I have to search thousands of strings within thousands of records in one routine. Currently I am using:
BindingSource1.Find("ColName","String")
Basically it is not taking much time but when it comes to thousands of records then it is taking approx 1 hour for one complete routine. (Some inside processing is also there but if records are lesser, I mean around 4,000 then it doesn't take much time but when it comes to 20,000 records then it just take me to the HELL and the PC is also at its highest, nothing else can be done however I am using multi-threading.)
What I basically want to know is that is there any way to search specific strings in a more powerful way to allow me better processing.
Any help will be highly appreciated.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
So first you loop through all the strings without looking for the string, then you do it again just to find the string? Doesn't that sound like a waste of time?
Skip the DataSet and the BindingSource, just read the file and look for the string.
Despite everything, the person most likely to be fooling you next is yourself.
|
|
|
|
|
Well, thanks for the reply.
But what I have to do is find a 20 char max string in a 20 char max field and then replace another field's data if matched and you are suggesting me to search 20 char max string in a 2000 char or more per record.
Does that sound waste of time or mine?
PS please first look at my original question, I think I was clear enough to give you the idea that why I need to search 20K records for one string although I have to search 20K strings within those 20K records continuously.
BTW I hope you got the idea on which I have to work so hopefully you can help me to solve the problem.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
I got the impression that you only did search once, that's why I said that you should search the data directly.
If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items.
If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value.
If you need to search different columns, you would need one dictionary per column.
Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
Next
Once the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
|
|
|
|
|
Thanks for the reply. I will check it now.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
Hi
Thanks a lot for the great trick. It is going like a fire now.
Warm Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
If your bindingsource is a datatable then use a dataview and dynamically filter the view. I find this works OK but I would NEVER try and display 20k records.
Why are you trying to locate 1 record in 20k in the UI. Isn't that what databases are for?
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Thanks for the reply. I will try using dataview though I have no idea about it.
BTW all process is in Background routine which doesn't have any UI, I have to search 20K records individually within another 20K records and apply some functions where data found. So, it needs to be much faster than currently is.
Hope someone might have more dynamic idea.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
If your data is stored in a database then that is where you need to do the processing. You should not be manipulating 20k records outside a database. If you are not using a database you have a design issue!
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Well dear, it is not a design issue but it is the customer demand, I am supposed to process various files which are generated by several other programs (of course I didn't develop them) so in processing those files, I have to keep the format intact without any changes.
Personally I also prefer database or XML files, it is so easy to manage XML files if data is not that huge.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
Then I would look at some other options. Just because you have to retain the format does not mean you can't deal with it smarter in your app, you should be able to return the end result to the format required, does not limit your choices of processing. It would of course depend on your requirements but I would of thought using a bindingsource as your processing vehicle to be one of the slowest options!
Can't you read the file into a decent container (table or xml  ) do your processing and then write it out.
The customer is NOT always right, and if they insist then they must be aware of the cost, then they can make the decision to ware the (time) cost or allow you to suggest a better option.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Well, I would suggest you to plz review my first post where I have mentioned already that I m using BindingSource through populating the text data into table/dataset. As per my thinking I found the binding source object more useful than working through the usual datasets. But to populate xml file from text file and then process the records and then again transfer all the records to the format seems ugly to me.
Well, no personal offense plz, but I personally like the BindingSource component, in some ways it makes the life easier however everyone has there own viewpoint. May be later on I think the same about BindingSource as you think or it might be vice-versa.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
No no no, you missed the point, your bindingsource is taking 1 hr+ to complete the process when there are a large number of records. My original suggestion was to use a dataview. I have never used the bindingsource so have no opinion of it as a tool. The only issue I have is that it is another layer of abstraction over your data and you have a speed issue.
MAP Tiger wrote: seems ugly to me
Got to agree with you there, I would have thought a datatable/view would be faster than XML. Have you use a Dataview with rowfilter and what is the performance like.
As for personal, how can a technical discussion be personal (except for preferences), the goal is to get the best solution, I don't give a rats who suggests it and I may not be right but I am happy to supply them (hopefully useful).
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Well, actually in the current scenario (in which I am currently working) and my formal experience with BindingSource component were matching but of course before the current project I have never come across with this heavy searching so I never knew the slowness of BindingSource.Find method.
Although now I am using dictionary object for it and believe me I am so happy, the same work is now processed in just 6 to 8 mins instead 1.30 hrs. So I am much happy now 
As far as DataView component is concerned, I never came across it. I would be checking it though to let it be considered in future projects.
Regards,
MAP Tiger
Tiger Softwares
Software Designer and Developer
VB.NET, ASP.NET, VFP
|
|
|
|
|
Dealing with threading always gets my raveled in trying to unravel the logic of threading.
Here is what I am doing. I am accessing a database all throughout my code. At the beginning I save my connection string in a variable.
When I get data from a Com port it creates a separate thread. That I want to initiate a database lookup but don't find anything in the variable, probably because it was created on a different thread. I think I need to invoke but am not clear how.
Here is another example that I solved with an invole
I wanted to call: TextChange(stringCom) in this same new thread.
To solve it I wrote (or got help to write from CP) this code and it worked.
Sub TextChange(ByVal Str As String)
If batch.txtboxWeight.InvokeRequired Then
Dim d As New TextChangeDelegate(AddressOf TextChange)
Me.Invoke(d, Str)
Else
batch.txtboxWeight.Text = Str
End If
End Sub
But, this time I just want to call a sub and get the saved variable.
When I code
Dim connection As New SqlConnection(Splash.cust.getconnstr)
Spash.cust.getconnstr is null.
How can this be fixed?
|
|
|
|
|
Hi Cory,
first a remark:
the subject of InvokeRequired and Invoke() both should equal the one Control
you are going to touch in your Else part, hence it is not Me.Invoke
but batch.txtboxWeight.Invoke() that would be correct.
Now the problem: I am assuming that Splash is a Form, hence you again are
trying to do something (ask for its cust property) to a Control; if this is not
the GUI thread asking, then you need again the InvokeRequired, Delegate, Invoke stuff.
You need it every time you touch a Control (i.e. read a property, write a property, call a
method, whatever) while you are not sure the code is always going to be executed
by the GUI thread.
And as you know, some devices (most timers, serial ports, etc) fire their asynchronous
events (Elapsed, DataReceived, ...) on some other thread.
Hope this helps.
|
|
|
|
|
Hi!
I hope this is the right forum for this. I'm sorry if it isn't, please recommend a different one.
I have a .wsf written in VBScript. It gathers information about the computer it is run on and prints that out to a text document. It currently works from my computer and gathers information about my computer. However, I need to be able to run this script from any machine on the local network and retrieve the information about a server computer, which is probably just a tower in some room somewhere in another state.
SSH is out because as far as I know, SSH doesn't come preinstalled on windows and I can't assume that the computers have them; the program needs to be completely independent so I can just install it on any computer and go.
I'm running the script from a .bat file, but I think I have to change the script, not the bat file, since all the bat file does is port it to a text file.
So basically the wsf is the gist of (I just inherited this; i think it's right though because it works on my own computer):
strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set CompInfo = objWMIService.ExecQuery ("Select * from Win32_ComputerSystem")
For Each objComputer In CompInfo
PCName = objComputer.Name
PCDomain = objComputer.Domain
UserName = objComputer.UserName
Next
Wscript.Echo "ComputerName: " & PCName
Wscript.Echo "Domain: " & PCDomain
Wscript.Echo "UserName: " & UserName
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\CIMV2")
Set colCSes = objWMIService.ExecQuery("SELECT * FROM Win32_ComputerSystem")
For Each objCS In colCSes
WScript.Echo "Computer Name: " & objCS.Name
WScript.Echo "System Type: " & objCS.SystemType
WScript.Echo "Number Of Processors: " & objCS.NumberOfProcessors
Next
Set colProcessors = objWMIService.ExecQuery("Select * from Win32_Processor")
For Each objProcessor in colProcessors
WScript.Echo "Manufacturer: " & objProcessor.Manufacturer
WScript.Echo "Name: " & objProcessor.Name
WScript.Echo "Description: " & objProcessor.Description
WScript.Echo "Processor ID: " & objProcessor.ProcessorID
'WScript.Echo "Address Width: " & objProcessor.AddressWidth
'WScript.Echo "Data Width: " & objProcessor.DataWidth
WScript.Echo "Family: " & objProcessor.Family
WScript.Echo "Maximum Clock Speed: " & objProcessor.MaxClockSpeed
Next
I'm sure that the first line, "strComputer = "."", needs to change from a . to the name of the computer, but how do I provide authentication? Can I just change the name to the name of the server, like hscs-gsgs or whatever, or do I need the network - networkname\hscs-gsgs? Sorry, I'm really lost.
Any help you could give would be great. Thanks in advance!
|
|
|
|
|
You can use the remote computer NetBIOS name or IP address for 'strComputer'.
If you want to run the code under different credentials, you need to use SWbemLocator:
strComputer = "192.168.1.12"
strUser = "username"
strPassword = "password"
Set objLocator = CreateObject("WBemScripting.SWbemLocator")
Set objWMIService = objLocator.ConnectServer(strComputer, "Root\Cimv2", strUser, strPassword)
Set CompInfo = objWMIService.ExecQuery ("Select * from Win32_ComputerSystem")
...
strUser needs to be an administrator account on strComputer. Firewalls can also be an issue.
|
|
|
|
|
Probably an easy question for the gang here.
I have a group box that I want to create a button at runtime. When I create the button right now, it places it 'behind' the group box instead of inside of it. How do I get the button to show up inside of the box?
Thanks in advance.
|
|
|
|
|
Nevermind, I found the answer: Me.grpQuadrant.Controls.Add(btn)
|
|
|
|
|
I have a custom control where I'd like to group a set of closely related properties. For example, look at a button control and click on the Application settings or databinding properties. There is a plus button for expanding and setting the subset properties.
How can I implement that functionality in an extended control?
Thanks!
|
|
|
|
|
|
If I am correct you will have to create a property that returns a custom made object with the properties you require. That way you can emulate the property inside a property that you describe.
Regards,
Thomas Stockwell
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning.
Visit my homepage Oracle Studios
Discounted or Free Software for Students:
DreamSpark - downloads.channel8.msdn.com
MSDN Academic Alliance - www.msdnaa.com
|
|
|
|
|
I think I found the answer in inheriting from ExpandableObjectConverter.
Thanks for the help guys.
|
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
