|
|
I'm having difficulty identifying the best way to process a dataset I've been provided. It's a fairly large dataset (1.76m records) with two columns (pk1,pk2) in the primary key. e.g., {{1,a},{1,b},{1,c},{1,d},{2,a},{2,b},{3,a},{4,a},{4,b}}
What I need to do is read all the records for pk1 in the dataset {{1,1},{1,2},{1,3},{1,4}}, run some calculations on these, write the results to a different table, then move on to the next pk1. I've tried a couple methods, and run into problems in each. I'm wondering if anyone has any ideas for this.
I'm restricted using the local file system, and cannot install anything new locally: e.g., MS Access, text files (connecting through system.data.oledb), cannot install MySQL or SQLServer.
What I've done thus far, is:
- create a DataTable containing a list of distinct pk1 values [MainList]. {1,2,3,4}
- For each unique value in [MainList],
- Execute a sproc via OleDbDataAdapter.Fill method to populate a DataTable containing all records in the original dataset associated with the current pk1 {{1,a},{1,b},{1,c},{1,d}}
- Process this datatable to create the user's output
- Write the results to a separate set of tables in the same database.
This method takes an unacceptable amount of time, with 98% taken up by the .Fill method.
Next, I tried the same process, but using the OleDbDataReader in lieu of the .Fill(DataTable) process. This improves the speed slightly, but still not where we need it.
Running this file through sequentially, using a text file, and checking the current records key against the prior records key, we processed the entire file in about an hour.
We are currently running VB.NET 2005 with ADO.NET 2.0.
Also, are there any other options on databases available, provided we're not allowed to install something locally, and can only use the local system?
Any input would be appreciated.
|
|
|
|
|
Seems to me that it is a simple set of select statements, unless I am missing something.
A few questions come to mind.
Do you want to do this in the database or the client (VB)?
Do you need to store the results or just output them to the client?
Personally I would use a stored proc, this gives you many tools such as table vars, cursors etc.
1.76m records is not that large.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Thank you, Mycroft,
We would prefer to have all the processing performed in the client, although creating temporary stored procs and deleting them when completed is well within reason, the calculation itself needs to be housed in the client so we want to pass the subsets of data back to the client. The problem (at least as far as I can tell) is getting the data transferred to the client for the calculation.
And yes, the results need to be stored, as the user has a number of options on how s/he can manipulate the results. Although, using an interim step to write it to a different file, then importing it to the database is feasible (since I cannot have an open datareader if I want to write something to the same database).
I hadn't thought about using a cursor for this. Are cursors available using ADO.NET?
|
|
|
|
|
I don't want a display name wrote: although creating temporary stored procs and deleting them when completed is well within reason
Oh no its not reasonable - this implies you do not "own" the database otherwise why delete the proc?
I don't want a display name wrote: We would prefer to have all the processing performed in the client
"prefer" is not a strong enough reason to screw up your application design, there needs to be a good reason to inflict this design on yourself. You need to make a decision, where you want the work to be done. I always prefer the server but that does not preclude using the client. What you should not be doing is using both!
Unless there is a calculation method/function you cannot use in the server then I would pass the parameters to get the resultset to the server proc, process the calculation as close to the datastore as possible (SQL) and return the resultset.
If you must use the client I think you should look into linq. I could do this with datatables and views by iterating through the records but linq sounds like it could do the job with sql like statements.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
You can get the release candidate of SQL Server 2008 from Microsoft Download or through your MSDN subscription.
To download, see SQL Server 2008 RC0 Download[^]. You can also download it through your MSDN subscription[^]. You can also find SQL Server 2008 RC 0 at TechNet Plus[^].
The released version is scheduled for third calendar quarter this year.
This release includes support for upgrading failover clusters from previous versions of SQL Server, as well as several bug fixes and minor improvements that we have made in response to the extensive customer feedback we’ve received from previous CTPs.
More than 250,000 customers and partners have downloaded the previous community technology preview (CTP) releases.
To learn more about SQL Server 2008, see SQL Server 2008, What's New[^]. To get started using the features in SQL Server 2008, see SQL Server 2008 Jump Start[^].
SQL Server 2008 delivers on the data platform vision in four areas: Mission Critical Platform, Dynamic Development, Beyond Relational, and Pervasive Insight.
|
|
|
|
|
Hello.
I have some problems with an application developed in C# and Oracle connection, The Application is a Windows Services that process some actions in an Oracle database in real time.
The problem is that I'm using C# 2003 and Framework 1.1 SP1, and I don´t know why the service lost the connection with the database and the object Oracleconnection.State is "Open".
I think that this is my problem:
http://support.microsoft.com/default.aspx?scid=kb;en-us;830173[^]
But I can´t close and open the connection all the time, I´m only need to reopen the connection when and Exception Happended (But my connection class it´s in other .net solution).
I thought that if I put a thread listening a variable that is changed any time I do an action, if this variable doesn´t change I reopend the connection, but my client doesn´t like this solution.
Anyone can help me ?, if you think on any solution please let me know.
Thank's a lot.
Regards, Jose A. Morey.
|
|
|
|
|
Retain the connection credentials in the web service
retain the sql command object for resubmission
submit command
trap the error when the SQL connection fails
interrogate the error to make sure it is the connection fail error
reestablish the connection
resubmit the SQL action (this being the nasty bit)
Just some ideas!
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Hi,
I have a typical scenario. I want to get
My query is
SELECT StarRate, date FROM BasicSearch WHERE date BETWEEN '2008-06-03' AND '2008-06-05';
My Result is as follows
StarRate date
-------- -----
1 2008-06-03 00:00:00.000
1 2008-06-04 00:00:00.000
1 2008-06-05 00:00:00.000
5 2008-06-04 00:00:00.000
5 2008-06-05 00:00:00.000
But I require the result as below
StarRate date
-------- -----
1 2008-06-03 00:00:00.000
1 2008-06-04 00:00:00.000
1 2008-06-05 00:00:00.000
Because StarRate 1 has all 3 dates
Thanks in advance
|
|
|
|
|
SELECT DISTINCT StarRate, date FROM BasicSearch WHERE date BETWEEN '2008-06-03' AND '2008-06-05';
Bob
Ashfield Consultants Ltd
|
|
|
|
|
Hi
Do you want the output to be only the StarRate witch has all the dates in the date interval?
Try
SELECT StarRate, date FROM BasicSearch WHERE date BETWEEN '2008-06-03' AND '2008-06-05'
AND StarRate in (
SELECT StarRate FROM BasicSearch WHERE date BETWEEN '2008-06-03' AND '2008-06-05'
GROUP BY StarRate
HAVING Count(*) = DATEDIFF("D", '2008-06-03', '2008-06-05') + 1)
This should work if it's uniqueness on StarRate and date
Kjetil
|
|
|
|
|
Nice one. I think I misread the original post, didn't notive it was just the starrate 1's required.
Bob
Ashfield Consultants Ltd
|
|
|
|
|
hi...
SELECT StarRate, date FROM BasicSearch WHERE date BETWEEN '2008-06-03' AND '2008-06-05'
Order By Date
By..
Tc
|
|
|
|
|
I'm reading an Excel file in VB.NET. The columns in the sheets don't have a datatype specified. One of the columns has phone numbers. Some users have entered the numbers with dashes (-), and others without dashes or spaces. When a column has all of the phone numbers with dashes, it shows all the values. The same happens when all the numbers are without dashes. The problem is when some have and some haven't: it reads only the ones that have a dash. 
How can I read all of the cells? Defining the type on the Excel isn't an option, there are hundreds of files with several sheets each.
I'm sorry if this is the wrong forum for this question.
Thank you for your help!
modified on Tuesday, June 10, 2008 9:35 AM
|
|
|
|
|
I finally found the solution. The problem was the way Excel was designed. Let me explain:
When no data type is specified, Excel reads the first 8 rows of that column to try to find out. My file had several lines of heading, leaving just the first row of data to do this. The first value was text, so it assumed the whole column was text, ignoring the numbers. My solution was to tell the user (luckily just one understanding person) to run the program twice, once with a dash and once without. Not the best solution, but it's the best I can do. I hope I can help somebody with this.
|
|
|
|
|
hi every body
Is there any one know that how can i registering a designed ".mdf" file in desktop engine with out using enterprise manager tool?
I mean that i only instaal desktop engine and no thing more, or in other word I wanna use my project with an install shield file to intall it with out any special action from user.
|
|
|
|
|
i have stored procedure with two parameters as follows:
create procedure checkCompatability
(
@compatility varchar output,
@award_num int
)
AS
SET NOCOUNT OFF;
IF EXISTS (SELECT Award_num from table1 where award_num=@award_num)
begin
set @compatability='Y'
print @compatability
end
else
set @compatability='N'
print @compatability
GO
in C# parameters:
SqlCommand checkDone = new SqlCommand();
checkDone.Connection = con1;
checkDone.Connection.Open();
checkDone.CommandType = CommandType.StoredProcedure;
checkDone.CommandText = "checkCompatability";
SqlParameter comp = new SqlParameter("@compatability", SqlDbType.VarChar);
string S = comp.Value.ToString();\\null reference exception occours here
lblresult.Text = S.ToString();
comp.Direction = ParameterDirection.Output;
SqlParameter complete = new SqlParameter("@award_num", SqlDbType.Int);
complete.Direction = ParameterDirection.Input;
int awN = int.Parse(comboAwardNum.Text);
complete.Value =(int) awN;
lblresult.Text = comp.Value.ToString();
checkDone.Parameters.Add(comp);
checkDone.Parameters.Add(complete);
checkDone.ExecuteNonQuery();
checkDone.Connection.Close();
what is the problem with that code?
|
|
|
|
|
You cannot retrieve the value of output parameters until after you have executed the stored procedure and retrieved all the results (if any). (The latter is a SQL Server limitation: when in 'firehose' mode it doesn't return the output parameter values until after the entire result data stream has been sent.)
DoEvents: Generating unexpected recursion since 1991
|
|
|
|
|
do you mean that i should execute the command before initailize the parameters?
|
|
|
|
|
No, set the parameters, execute the proc and THEN get the output value.
Also, stop double posting - this was in the C# forum as well. 
Bob
Ashfield Consultants Ltd
|
|
|
|
|
Hi,
I have just completely installed SQL Server 2005 Developer Edition with all features but there is no icon of "SQL Server Management Studios" in program files. I tried to find in the following directory:
"C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\SqlWb.exe"
but there is no "SqlWb.exe" present there. What should I do.
Thanx
Regards Syed Kamran ul Haq
ALLAH HAFIZ
|
|
|
|
|
Try running SqlRun_Tools.msi file which is in your SQL Server 2005 media in the
Tools\Setup folder.
Bob
Ashfield Consultants Ltd
|
|
|
|
|
Hi all,
Not sure if this is the right board for this question - sorry if not !!!
I have installed Reporting Services and have created a nmuber of reports via the business intelegence IDE. However a customer has asked for something a little different than the usual table format.
They want a report that has a number of text boxes arranged on the page that contain data grabbed from the db. I have managed to get this working but they also want this same format on a new page of the report for each row returned by the SQL statement.
Not sure i have explained this very well but any info you could give would be fantastic as i am really stuck !!!
Thanks,
John
|
|
|
|
|
I think what you want is a header. Go to report on the menu and page header. Put your text boxes in there?
|
|
|
|
|
Hi,
Thanks for the reply! but i am still having problems.
The textboxes are populated from fields in the dataset. when i tried to put them in the header i was told that fields cant be put in the header.
All i need is the same report page replicated for each row in the dataset. but its seeming impossible 
Please help !!!!
Thanks
John
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
