|
thankx
All I ever wanted is what others have....
CrazySanker
|
|
|
|
|
I am opening about 70 web pages and pulling about 1000 strings from each. My arrays tend to be 60,000 to 100,000 string arrays.
The problem I am having is this is very slow. I am using RegEx to get the 1000 names from each web page, and then putting them into a SQL database.
How do I speed this process up, it takes around 20 minutes.
I think one major slow down is I am inserting them into the database one by one. Is there a way to fix this piece? Will stored procedures help?
Is there a faster way to implement a RegEx like solution to speed up the string collection?
Is the fact that I am using an ArrayList the huge slow down?
Any help, suggestions, articles, anything, would be greatly appriciated.
Thanks!
The best way to accelerate a Macintosh is at 9.8m/sec² - Marcus Dolengo
|
|
|
|
|
I have some suggestions for you.
#1. It might be slow down because of Regular expression. Using regular expression is good but it is totally depend on how you write.. I have seen a lot of regular expressions that makes the application slows.. so, my advice is that maybe you can double check the regx that you are using ... but yeah. if you are a master in regx then this is not the fact..
Expert Coming wrote: I think one major slow down is I am inserting them into the database one by one. Is there a way to fix this piece?
#2. You should not open and close the connection each time you are inserting the data. You can open one connection and then, just keep on using the connection that you have opened. after insertion done, you can close this connection..
Expert Coming wrote: Is the fact that I am using an ArrayList the huge slow down?
I think ArrayList should be okay. I'm not sure what else we should use instead of ArrayList. you are using Generic ArrayList.. right??
Thanks and Regards,
Michael Sync ( Blog: http://michaelsync.net)
If you want to thank me for my help, please vote my message by clicking one of numbers beside "Rate this message". Why vote? Plz Read it here. Thank you. 
|
|
|
|
|
Expert Coming wrote: I think one major slow down is
There's one saying: To think means to know sh*t There's also one about optimalizations: measure, measure, measure. Get a profiler, or even use the Stopwatch to find out what really slows the code down. It could be regex, it could be database, it could be ArrayList, it could be because it's weekend.
btw, use StringCollection or generic collection (like List<String> etc), because good ol' ArrayList works with objects, so it does all the slow things like casts and boxing.
[ My Blog] "Visual studio desperately needs some performance improvements. It is sometimes almost as slow as eclipse." - Rüdiger Klaehn "Real men use mspaint for writing code and notepad for designing graphics." - Anna-Jayne Metcalfe
|
|
|
|
|
If things are slow the culprit is either regex or the database access (or both),
certainly not the ArrayList. One more possibility is you are exhausting memory
causing lots of disk accesses (I trust you would notice that).
Now 20 minutes for 100K strings is way too much, so maybe you wasted lots of time
everywhere. It may help to do the following:
- check for linear behavior: does doing half the job (half as many strings) take half
the time ? if not, something is deteriorating, maybe you selected a non-linear
algorithm somewhere, and maybe you created some memory leaks.
- get accurate time measurements on all the steps for one of the strings (but not
the very first one), so add log statements everywhere and look for the biggest
gap in time. A log statement could be:
if (stringCount==100) log("going to create regex");
and then you need to add the method to every class concerned:
public static void log(string s) {
Console.WriteLine(DateTime.Now.ToString("hh:mm:ss.ff")+" "+s);
}
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips:
- make Visual display line numbers: Tools/Options/TextEditor/...
- show exceptions with ToString() to see all information
- before you ask a question here, search CodeProject, then Google
|
|
|
|
|
Expert Coming wrote: I am opening about 70 web pages and pulling about 1000 strings from each. My arrays tend to be 60,000 to 100,000 string arrays.
Do you open all the web pages and pull the strings out before you start processing the data? If so you may wish to start processing before you have got all the data. That way you can clear out some of the data before processing the next batch which keeps memory requirements down which means you don't have to keep requesting more from the OS which can take time.
So, if you can process one page at a time at 1000 strings, rather than all 70 pages at 70,000 strings you might find that your process runs a bit quicker.
Upcoming events:
* Glasgow: Mock Objects, SQL Server CLR Integration, Reporting Services, db4o, Dependency Injection with Spring ...
"I wouldn't say boo to a goose. I'm not a coward, I just realise that it would be largely pointless."
My website
|
|
|
|
|
Please provide code for convertion of tif image file to binary format using C#
parthiban
|
|
|
|
|
Search google and codeproject. You will find plenty of articles on how to do this.
The best way to accelerate a Macintosh is at 9.8m/sec² - Marcus Dolengo
|
|
|
|
|
I have created a windows service application.
Now i need to create a .msi file to be delivered to the client.
|
|
|
|
|
Hi there,
I've got an URGENT need to get hold of a bit of coding to programmatically add directory security, while creating directories through a software which I am working on.
It could be to hide or restrict access. That doesn't matter as long as access is controlled.It's better if a toggle security functionality is also there. But, however, let's focus on getting the security par first.
Got any codings or references (relevant for C#)?
Regards,
A
|
|
|
|
|
|
I'm a novice in c# programing,I wanna to devolop a windows application implement communication between many clients and one server,and I don't know how to do it first.
|
|
|
|
|
liucaihua wrote: a windows application implement communication between many clients and one server,
That's a pretty vague description. See this[^] for several examples. I highly recommend using WCF[^].
/ravi
|
|
|
|
|
hi
i want using thread in my app to show Waiting form, then my app load it's data,i close Waiting form by threadObj.Abort() method, in my computer, app work correctly, but when i copy my app on another machine and run it, the ThreadAbortException has occured (sometimes occured and sometimes not occured), i was used try-catch block to catch this exception but was not successful, here is my code :
private void Frm_Land_Load(object sender, EventArgs e)<br />
{<br />
System.Threading.Thread th = new System.Threading.Thread(gu.Show_WaitingForm);<br />
th.Start("Loading Data ...");<br />
Application.DoEvents();<br />
<br />
this.Initialize_Data();<br />
this.txt_date.Text = this.sh.Show_Hijri_System();<br />
this.toolStripStatusLabel1.Text = "Ready";<br />
<br />
Application.DoEvents();<br />
th.Abort();
how to solve my problem ?
|
|
|
|
|
hdv212 wrote: how to solve my problem ?
Don't use the Abort method.
You could use the BackgroundWorker[^] class instead for loading your app's data. It can report the progress to your user as it loads data, you can use a progress bar on a dialog box or just display the results in tool strip.
|
|
|
|
|
Why would you want to abort the thread? Why not make it exit normally using some event signalling mechanism?
That apart, what exactly does the thread do? If all it does is show a form, you're much better off doing it in the UI thread (the thread that Frm_Land_Load runs on) and doing this.Initialize_Data on a different thread. Something like
private void Frm_Land_Load(object sender, EventArgs e)
{
Thread th = new Thread(DoWork);
th.Start();
gu.Show_WaitingForm();
}
private void DoWork()
{
this.Initialize_Data();
this.BeginInvoke(new MethodInvoker(CloseWaitingForm), null);
}
private void CloseWaitingForm()
{
gu.Hide_WaitingForm();
}
This way, when Initialize_Data completes, CloseWaitingForm will be called on the main UI thread, which will hide the waiting form.
This solution assumes that Initialize_Data does not access any controls in the form. If it does, then you'll have to run the waiting form on a different thread. Something like
private void Frm_Land_Load(object sender, EventArgs e)
{
Thread th = new Thread(gu.Show_WaitingForm);
th.Start();
this.Initialize_Data();
gu.Hide_WaitingForm();
}
class GU
{
Form f;
void Show_WaitingForm()
{
f = new Form();
f.ShowDialog();
}
void Hide_WaitingForm()
{
if (this.InvokeRequired)
{
f.BeginInvoke(new MethodInvoker(Hide_WaitingForm), null);
}
else
{
f.Close();
}
}
}
Hope this helps.
|
|
|
|
|
Dear All,
I am clueless about how to create more graphically rich menus such as the ones that are used in Microsoft Office Application.
Thank you!
M. Nauman Yousuf
"Mess with the Best, Die like the rest"
|
|
|
|
|
Google around. You should actually do a search here at CP and I think there are a few graphically rich controls...Sorry, too lazy to search for you
"Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon
|
|
|
|
|
if you want to design graphically rich menus like offce 2007, you are better buying a third party control instead of designing it that's just my opinion. Look up www.divelements.co.uk if you want office 2007 style menus.
|
|
|
|
|
Hello,
I have created a setup project and added some dialog boxes to the user interface.
I have added the dialog with 4 text boxes. Then using Orca I have modified this and added a radio button group with 2 radio buttons (A and B).
What I am trying to do is when the user selects A then all text boxes will be enabled. And when the user selects B then only 2 text boxes will be enabled.
This is my first time in using Orca, and anyone explain how to add this condition for selecting the radio buttons.
Many thanks advance,
Steve
|
|
|
|
|
Hello,
I load dynamically an external class library compiled as a .dll file, using this kind of code:
Assembly classLibrary1 = null;
using (FileStream fs = File.Open(path, FileMode.Open))
{
using (MemoryStream ms = new MemoryStream())
{
//... (Read file byte per byte and store it to memorystream)
classLibrary1 = Assembly.Load(ms.ToArray());
}
}
foreach (Type type in classLibrary1.GetExportedTypes())
{
if (type.GetInterface("IPClass") != null)
return Activator.CreateInstance(type) as IPClass;
}
I would like these objects (created from an external assembly) to have access to a single instance of an object I previously created (in the project that loads external classes).
For example:
The static class 'Hello' in my main project has to be accessed by the loaded classes. But only to the 'Hello' instance of the main projet (what I can't apparently do when using class from external assemblies)
Somebody knows how to deal with this issue?
Thank you,
Thomas
|
|
|
|
|
thg971 wrote: The static class 'Hello' in my main project has to be accessed by the loaded classes. But only to the 'Hello' instance of the main projet (what I can't apparently do when using class from external assemblies)
Why doesn't it work? AFAIK, singletons remain single within an AppDomain, so as long as the external assemblies are also loaded in the same AppDomain as your Hello class, they should be accessing the same instance.
If not, then you would have to derive your singleton from MarshalByRef and modify the code in the external assemblies to explicitly accept an instance of your singleton (as a property setter, for example) and then use that instance.
|
|
|
|
|
how can i get index of specified item in multidimension array ?
|
|
|
|
|
By using nested for loops to step through and checking for when the element is equal to your value.
valueSearch = "Value to search for";
for (int x = 0; x < array.GetLength(0); x++)
{
for (int y = 0; y < array.GetLength(1); y++)
{
for (int z = 0; z < array.GetLength(2); z++)
{
if (array[x][y][z] == valueSearch)
{
}
}
}
}
There are 10 types of people in the world, those who understand binary and those who dont.
|
|
|
|
|
I would say that if you need to be searching, arrays aren't the way to go.
But if you must use arrays, the only tip I would add to the above is a break statement jump out of the nested for loops when you actually find the value (no need in running through the rest of the items).
J
Make the logo bigger
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
