|
I see.
Basically my test data contains a single parent with children, grand children, great grand children (up to the 6th level) totaling 1200 nodes.
2 tables are affected: The main table where the data is contained and another table which just contains the parent/child ids relationships. Only the main table is updated.
The previous code generates the tree, traverses and updates each node using a stored procedure.
I now just moved the tree generation and traversal to C# code and keep a simple stored procedure to update each data.
Then I enclosed the entire thing in a single transaction (using the SqlTransaction class).
Any comments about this?
My concern is that, since it is a single transaction... I may still experience the same problem that we currently have, which is that when the data is updating, other users who are updating data in the same SQL table are stuck until this finishes updating.
Thanks again.
Rafferty
|
|
|
|
|
You may already be doing this or something more sophisticated... Depending on how you are loading the data and if you can check constraints/triggers without database access, you may be able to drop constraints and triggers temporarily. Then afterwards you reapply the constraints and triggers. This may be faster and also let you recover from certain errors as you would log only those rows that you could not load.
|
|
|
|
|
Consider this,
Create an indicator somehow that could be interrogated prior to making a change to the table. If the "bulk update" indicator is true then maybe you could warn the user that the database is currently being updated and they want to try their update at a later time. Or, if you want to get sophiticated, maybe you could create a pending transaction table where you could temporarily store their transaction until the "bulk update" is finished and apply it when done.
Basically, you would be creating a Queue for updates on your table.
Look at sp_GetAppLock as a possible tool for creating your "bulk update" inidcator.
Just a thought.
|
|
|
|
|
All depends how optimistic you want to be.
If you perform the updates using the (RowLock) hint it may restrict the locking. Though a half hour transaction will still end up locking everything in sight until it is done,
The only other thing would be to run your selects using the (NoLock) hint, with the corresponding risk that you will read some updated data that will later get rolled back and so have never existed.
What is more important, stopping the locking, or getting guaranteed data? Your choice.
Having said that Unless I have guaranteed single user access to a database, such as in an overnight job, there is no way I would allow any transaction to lock records for more than a few milliseconds. In my experience, if the Application hangs for half an hour waiting for database access the end users will buy a new system.
|
|
|
|
|
Thanks all for your ideas. Looks like the best way is really to reduce this bulk update into smaller transactions.
Let's see if our client agrees with the change.
Thanks again! You're all very helpful! 
Rafferty
|
|
|
|
|
Hello again,
I converted the large stored procedure into smaller ones, by placing the tree generation and traversal business logic in C# code and just using stored procedures for simple checks and updates. The entire operation, then, is enclosed in a single transaction.
The good news is that the 30-minute operation was optimized to 15 minutes.
The bad news is that the new c# code made the web server too busy that other users couldn't use the application (even though multi-threading is automatically done by the IIS).
What I plan to do is to implement multi-threading manually for this function. Do you think it will solve my problem?
Any other suggestions?
Thanks again.
Rafferty
|
|
|
|
|
In store procedure.i want to display a message.ie
Alter procedure CheckForValidUser
(
@Username varchar(20),
@Password varchar(10)
)
as
select ltrim(rtrim(name_first + ' '+ isnull(name_last,''))) as name,rolename,p.pid, date_exit from hms_person p, HMS_personell pl where pl.username = @Username and password = @Password and pl.pid=p.pid ;
I'm passing two parameters but to display three arguments,i want to get a message to display whether the date_exit is not null is it possible to display as popup
regards
Kankeyan
|
|
|
|
|
AFAIK no.
1. You must be doing all these checks in the business layer of your application.
2. In case something goes wrong in the stored procedure, send the message back to the calling method and it shaould take care of showing it to the user. (Not the exact message but a user freindly one)
50-50-90 rule: Anytime I have a 50-50 chance of getting something right, there's a 90% probability I'll get it wrong...!!
|
|
|
|
|
Danish is correct, you need to trap the error and deal with it in the client.
Remember SQL Server runs on the server, who is going to see a message box ON THE SERVER, you need to do all the work on the client
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
is it possible to display in client side
|
|
|
|
|
I would again say, you should not let that null date to reach your DB. Check it beforehand. For other exception, you can have something like this:
You can use try/catch in your stored procedure. Then, if it fails, you should send the exception back to your data access layer. That or business should inturn change the message to a user friendly one. Also, before doing this make sure you log it in somewhere for future reference.
Once your message has reached UI layer, use MessageBox or Alert as applicable.
In order to understand ho try catch works in SQL, use google and you will find a lot of help.
50-50-90 rule: Anytime I have a 50-50 chance of getting something right, there's a 90% probability I'll get it wrong...!!
|
|
|
|
|
Only through your client application.
Bob
Ashfield Consultants Ltd
Proud to be a 2009 Code Project MVP
|
|
|
|
|
If this is required for debugging purpose, you can use the print statement, or if you want to notify your application users, you need to pass an appropriate value to the client and show the popup in the client application (such as a WinForms Form or an ASP.NET page).
|
|
|
|
|
Hi All,
I Developed an Application in vb.net 2005 using sql server 2005 express. So far I installed in single PCs and is working fine.
But now i want the application to be access by client PCs but update the SQL database on a centralized server PC.
e.g. MASTER_PC will have the SQL database & the main application will run on PC1, PC2 and PC3; so that users can access the database from their respective PCs.
Overall to say, I just need a central location where the SQL database can be stored, and all employees can run and modify the database through their application via internet/LAN(or some other source)
Please suggest me how to go ahead with this:
1) As I have 4 PCs in my office, Do i need to create a LAN ?
2) Do I need to do any major modifications in my application?
Any links/tutorial/references will be highly appriciated.
Thanks
R.S.
|
|
|
|
|
For your connection string, it's easiest to troubleshoot a client if you use DSN connection as opposed to ipaddress. I am not sure if you need to create a LAN since I have always set these up when I already had a LAN.
If you've already networked your PC's, then the main step is modify the system DSN on each pc. This can be done by modifying registry settings or through odbccad32.exe. I am assuming that you will want to call your database by name. With Oracle, you also might create a TNSNames file. This is not necessary for SQL Server. Clients can access the database when the login with VPN or are on your network at the office.
Assuming that your have a Major modifications to your application should not be necessary.
|
|
|
|
|
1. Aren't your 4 PC's on a single network? If they are, all you need is to set up your server to allow remote connections. Then all your PCs should be able to connect to your database.
If they are not on same network, but all are connected to internet, still you can connect to your database. Here[^] is a link to help you.
2. I don't think the code would change.
50-50-90 rule: Anytime I have a 50-50 chance of getting something right, there's a 90% probability I'll get it wrong...!!
|
|
|
|
|
Hi guys, I'm relatively new to databases and have been playing with SQL Server 2008 Express for some time. I want to build a database on stock prices and various other stock-related indicators across time. So basically it's going to have three dimensions: name of the stock, stock attributes, and time. The user will be able to query for certain stock's performance on a particular attribute across a designated time period.
Normally this would be a perfect job for OLAP (multidimensional database). However, due to financial constraint and other reasons  , I am not considering OLAP at the moment. With my limited knowledge on databases, I think I could achieve the above by using:
1) Relational database: if I need to track 1000 stocks, I will construct 1000 tables, with time and stock attributes on each stock table.
2) XML: I am relatively new to XML too, but it seems that with some clever XPath/XQuery coding, dumping all the data into one huge XML database is not a bad idea (or is it?), as long as data can be effectively retrieved.
In terms of speed/performance, maintenance convenience, and server-memory efficiency, which of the above two is a better choice? Or do I really need to migrate to using OLAP?
Sorry if this is a simple question, I am a total newbie regarding databases.
Thanks in advance!!
|
|
|
|
|
I strongly advise against 1,000 tables. This is not a scalable solution and many of your queries will take a very long time as you will need to join many tables.
|
|
|
|
|
I wouldn't store similar information on different stocks in different tables, just the one table with a field added to identify the stock would be fine. It would be able to provide more functionality with less code, as you can now easily search/list over many stocks, and never need to enumerate the tables.
|
|
|
|
|
Thanks for the replies guys. But if I add a field to identify the stocks, it would be one huge table and it won't look as elegant.
By the way, are you suggesting the following structure:
Time StockName Open High Low Close EPS Dividend Return% ... ... ...
1/2/07 Citi 20 23 19 20.5 5m 5 4% ... ... ...
....
....
....
11/25/09 Citi 4 5 3 4.5 2m 2 -10% ... ... ...
1/2/07 BofA 35 37 32 36.4 7m 10 7% ... ... ...
....
....
....
11/25/09 BofA 12 13 11 10 3m 1 -7% ... ... ...
1/2/07 MSFT 45 47 41 42.5 28m 3 7% ... ... ...
....
....
....
11/25/09 MSFT 4 5 3 4.5 2m 2 -10% ... ... ...
If this is true, assuming that the table will contain 1000 stocks with 50 attributes for a 3-year period, this table will need to contain: 1000 x 50 x 260(business days) x 3 = 39,000,000 stock prices. Is this a little bit too large for one table? (If it is, as an alternative perhaps I could break 1000 stocks down into several smaller tables based on the market in which they are traded in? So one table for each market. What do you think?
Thanks!
|
|
|
|
|
James Shao wrote: are you suggesting...
yes I am. I haven't done this with millions of records, my DB apps aren't large, but yes that is what I would do. I avoid tables with identical structure, I only use one Persons table, one Vehicles table, etc.
|
|
|
|
|
Thanks a lot, I'll give this a try and come back if I've encountered errors.
|
|
|
|
|
My databases ARE in the millions of records, Luc is correct have 1 structure with all the information. Do not store any data twice (database design 101) eg is you are have industry or sector for an equity then have an equity table with the static data and a related table with the tick information.
If table size becomes an issue there are many better option than splitting by stock name. Partitioning by time is a better solution (partition by year would do). Doesn't express include analysis services (have not checked)
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Breaking it down by market is not a good idea, but you do need to know what market something traded on. A single stock may trade on multiple markets in the US, even when you ignore after-hours trading and dark pools.
I've personally worked with global financial databases with billions of records. The DBA sets the appropriate indices and partitions and performance was fine.
What you do not want to do is design something fragmented where it is hard to change the structure later without losing your data.
As far as memory usage, you can use SELECT statements and limit the amount of memory that sql server will use. So if you have a 10 GB table, you can tell sql server not to use more than 500 MB of RAM for instance.
|
|
|
|
|
Hi Ted, thank you for the suggestions. I also feel that fragmenting the data is not a good idea. But if I dump them all into one table, that would create big redundancy in the date column (my 1st column), since I'll need to repeat it for every stock I have in my database. Is this okay?
Thanks!
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
