|
What is the nature of this job? Does it create records ? Or read the data? How big is this database ? How are many disks are installed on the server? How are they being used by the database engine? Are the Operating System and database files on different disks?
On the database server from a command prompt type "perfmon" to bring up the Performance Monitoring Utility. You want to add a counter for Logical disk / Avg Disk Queue Length (All disks) Look at the graph while the job is running, if you see high activity on one disk and low activity on another disk, you may want to consider moving some DB files around.
Here are a couple of things to consider:
1) Count the number of rows in the tables involved. Check these values daily, weekly, monthly.
2) Identify the tables that are growing.
3) Are there indexes on these tables? Check the fragmentation of the indexes. Possible create new indexes or revise existing ones.
NOTE: Be careful when adding indexes, it may help your one job, but have a negative impact on the rest of the system.
With the limited information given, I'm giving you some basic areas to investigate. I'll be glad to help you.
|
|
|
|
|
Hi ,
I have A nvarchar value Like This =id1-1,2 ; id2-1,3 ; .......i want return the nvarchar value with replace of id1,id2 values(i have a table id,value)
|
|
|
|
|
REPLACE (Transact-SQL)[^]
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
|
Hi ,
I have A nvarchar value Like This =id1-1,2 ; id2-1,3 ; .......i want return the nvarchar value with replace of id1,id2 values(i have a table id,value)
|
|
|
|
|
Do you ask about using CAST or/and CONVERT?
Or what did you mean?
|
|
|
|
|
I ned to convert below DB2 query to SQL
SUBSTR(xmlserialize(xmlagg(xmltext(CONCAT( ', ',DEL.COVERAGE_CODE))) as VARCHAR(1024)), 3) as COVERAGE_CODE
I am not able to get equivalents of xml related things in SQL. Any help would be appreciated
|
|
|
|
|
|
For that specific query, there are various equivalent methods in SQL:
Concatenating Row Values in Transact-SQL - Simple Talk[^]
STUFF
(
(
SELECT ', ' + DEL2.COVERAGE_CODE
FROM YourTable As DEL2
WHERE DEL2.GroupingField = DEL.GroupingField
FOR XML PATH(''), TYPE
).value('.', 'varchar(max)'),
1, 2, ''
)
Or, if you're using SQL 2017, you can use the new STRING_AGG function:
STRING_AGG (Transact-SQL) | Microsoft Docs[^]
STRING_AGG(DEL.COVERAGE_CODE, ', ')
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Just a random question since I already have a different solution but I had the following situation.
I have a table with a billion rows (actually probably about 1.1 billion.)
I was trying to span the table, read every row and do an analysis.
Certainly couldn't load the entire table. I was using a paged query (limit/count). Each page took about 90 minutes for the query itself. So not really something that was going to allow me to do much analysis.
Any other ideas on spanning it or speeding it up?
At one point I was even considering just dumping it and writing an app to do the analysis outside of the database. That was about the only other solution I had.
The database was MySQL (AWS Aurora actually).
The relevant parts of the table were as follows and the id has a primary key. (I didn't design the table.)
Table T
id char(22)
RefId1 char(22)
RefId2 char(22)
...
|
|
|
|
|
The obvious question is: What kind of analysis?
|
|
|
|
|
Jörgen Andersson wrote: The obvious question is: What kind of analysis?
Since I needed to span the database not sure how that matters unless you would perhaps suggest random sampling.
But in this case I needed to validate that the ids in each row did in fact match my expectations of what they were pointing to.
|
|
|
|
|
jschell wrote: What kind of analysis?
I think that is critical, the design of your analysis is the driving factor and is going to be the one area you can gain the most benefit from optimization.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Ok.
Per the OP table looks like the following
Table T
id char(22)
RefId1 char(22)
RefId2 char(22)
I need to do the following
1. Verify that RefId1 and RefId2 are in a different table either in that order (1,2) or (2,1)
2. Report if neither or only one id is found.
3. Report if more than one match is found
4. If found report if a different column (not documented above) is same as in the second table.
The second table also has a billion rows.
Both tables have indexes on the id, and the two other columns documented above.
|
|
|
|
|
oh that is ugly - the 1,2 - 2,1 is going to cost you.
Can you do it in a couple of steps, inner join on the 2 layouts and eliminate them from the process, possibly even move the non matching records into another table for your reporting analysis (one assumes the majority have valid matching records).
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Mycroft Holmes wrote: oh that is ugly - the 1,2 - 2,1 is going to cost you.
Yep - not my design.
Mycroft Holmes wrote: Can you do it in a couple of steps, inner join on the 2 layouts and eliminate them from the process,
I suspect that join between the two tables will return on the order of a billion rows. Either because there are very few rows that are not matched or there are zero. I am only expecting the first because with legacy data orphans might occur.
Mycroft Holmes wrote: possibly even move the non matching records
I have to find them first. Once I find them I don't consider the analysis to be a problem. Actually for those that do not match I consider it likely they are orphans.
|
|
|
|
|
Split it into groups by taking the first 1 or 2 characters, process each set independently.
Getting desperate here as I'm running out of ideas.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Also an interesting idea.
For base 64 encoding that would give approximately 244000 per query(2 characters0 and, per the other comment, it should be fast enough to use that a page.
The only limitation there would be if the values were not uniformly distributed.
|
|
|
|
|
jschell wrote: if the values were not uniformly distributed
Uhm I would presume the fields are indexed and physical location would be irrelevant.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
It's not uniformly distributed per definition, but the first half of a guid is the time part, and it's stored in reverse order of significance, so for this purpose it could probably be considered as uniformly distributed.
|
|
|
|
|
For clarification on that the value is actually based on the UUID from java. And that UUID would seem to have a uniform distribution because the layout is not strictly ordered by time - the UUID uses the minor time (probably seconds/millis) as the most significant part.
|
|
|
|
|
Another question.
Do you need to keep track on what page you are analysing or is sequential paging good enough?
|
|
|
|
|
Paging was not part of the solution. Paging was necessary only because I could not read the entire collection into memory. However I did need to insure that all of the data was read and that the same row was not read twice.
|
|
|
|
|
Then I believe I have enough info to recommend you to not use pagination the way you do.
SELECT * FROM tbl LIMIT 2000,10; is an extraordinary inefficient query.
It's internally functioning like this pseudocode:
SELECT BOTTOM 10 *
FROM (
SELECT TOP (2000+10) *
FROM T
)In short it selects the 2010 first rows to just throw away the first 2000 rows.
If you can store the max(ID) from the previous page I'd recommend trying this instead
SELECT id
,RefId1
,RefId2
,...
FROM T
WHERE id > @PreviousMaxID
ORDER BY id
LIMIT @PageSizeYou obviously need to have an index on the id column for this to be fast.
|
|
|
|
|
Jörgen Andersson wrote: If you can store the max(ID) from the previous page I'd recommend trying this instead
I like the idea. Unfortunately the id is not sequential (again not my idea.)
So to do the paged query based on that it would require sorting the primary id then paging on that. So perhaps exactly the same or even less efficient.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
