Click here to Skip to main content
15,867,704 members
Articles / Web Development / ASP.NET
Article

Paging of Large Resultsets in ASP.NET

Rate me:
Please Sign up or sign in to vote.
4.92/5 (174 votes)
5 Aug 200411 min read 1M   26.6K   593   138
An article about optimization and performance testing of MS SQL Server 2000 stored procedures used for paging of large resultsets in ASP.NET

Introduction

The paging of a large database resultset in Web applications is a well known problem. In short, you don't want all the results from your query to be displayed on a single Web page, so some sort of paged display is more appropriate. While it was not an easy task in the old ASP, the DataGrid control in the ASP.NET simplifies this to a few lines of code. So, the paging is easy in ASP.NET, but the default behavior of the DataGrid is that all resulting records from your query will be fetched from SQL server to the ASP.NET application. If your query returns a million records this will cause some serious performance issues (if you need convincing, try executing such a query in your web application and see the memory consumption of the aspnet_wp.exe in the task manager). That's why a custom paging solution is required where desired behavior is to fetch only the rows from the current page.

There are numerous articles and posts concerning this problem and several proposed solutions. My goal here is not to present you with an amazing solves-it-all procedure, but to optimize all the existing methods and provide you with a testing application so you can do evaluation on your own. Here is a good starting point article which describes many different approaches and provides some performance test results:

How do I page through a recordset?

I was not satisfied with the most of them. First, half of the methods use old ADO and are clearly written for the "old" ASP. The rest of the methods are SQL server stored procedures. Some of them yield poor response times as you can see from the author’s performance results at the bottom of the page, but several have caught my attention.

Generalization

The three methods I decided to closely look into are the ones the author calls TempTable, DynamicSQL and Rowcount. I'll refer to the second method as the Asc-Desc method in the rest of this text. I don't think DynamicSQL was a good name, because you can apply dynamic SQL logic to the other methods too. The general problem with all these stored procedures is that you have to assess which columns you'll allow sorting for and that won't probably be just the PK column(s). This leads to a new set of problems – for each query you want to display via paging you must have as many different paging queries as you have different sorting columns. This means that you will either have a different stored procedure (regardless of the paging method applied) for each sorting column or you'll try to generalize this to only one stored procedure with the help of dynamic SQL. This has a slight performance impact, but increases maintainability if you need to display many different queries using this approach. Thus, I’ll try to generalize all of the stored procedures in this text with dynamic SQL, but in some cases it will be possible to achieve only a certain level of generalization, so you’ll still have to write separate stored procedures for some complex queries.

The second problem with allowing other sorting columns beside the PK column(s) is that if those columns are not indexed in some way, none of these methods will help. In all of them a paged source must be sorted first and the cost of using ordering by non-indexed column is immense for large tables. The response times are so high that all the procedures are practically unusable in this case (the response varies from couple of seconds to couple of minutes depending on the size of the tables and the starting record being fetched). The indexing of other columns brings more performance issues and may be undesirable, for example it might significantly slow you down in a situation where you have a lot of daily imports.

TempTable

The first one I would comment on is the TempTable method. This is actually a widely proposed solution and I encountered it several times. Here is another article that describes it along with the explanation and a sample how to use custom paging with the DataGrid:

ASP.NET DataGrid Paging Part 2 - Custom Paging

The methods in both articles could be optimized with just the Primary Key data copied to the temp table and then doing the join with the main query. Therefore, the essence of this method would be the following

SQL
CREATE TABLE #Temp (
    ID int IDENTITY PRIMARY KEY,
    PK  /* here goes PK type */
)

INSERT INTO #Temp SELECT PK FROM Table ORDER BY SortColumn

SELECT ... FROM Table JOIN #Temp temp ON Table.PK = temp.PK ORDER BY temp.ID <BR>WHERE ID > @StartRow AND ID < @EndRow
The method can be optimized further by copying the rows to the temp table until the end paging row is reached (SELECT TOP EndRow...), but the point is that in the worst case – for a table with 1 million records you end up with 1 million records in a temp table as well.  Considering all this and having looked upon the results in the article above, I decided to discard this method from my tests.

Asc-Desc

This method uses default ordering in a subquery and then applies the reverse ordering. The principle goes like this

SQL
DECLARE @temp TABLE (
    PK  /* PK Type */ NOT NULL PRIMARY 
)

INSERT INTO @temp 
SELECT TOP @PageSize PK FROM (
    SELECT TOP (@StartRow + @PageSize) 
    PK, 
    SortColumn /*If sorting column is defferent from the PK, SortColumn must <BR>                 be fetched as well, otherwise just the PK is necessary */ 
    ORDER BY SortColumn /* default order – typically ASC */) 
ORDER BY SortColumn /* reversed default order – typically DESC */

SELECT ... FROM Table JOIN @Temp temp ON Table.PK = temp.PK <BR>ORDER BY SortColumn /* default order */
Full Code – Paging_Asc_Desc

RowCount

The base logic of this method relies on the SQL SET ROWCOUNT expression to both skip the unwanted rows and fetch the desired ones:

SQL
DECLARE @Sort /* the type of the sorting column */
SET ROWCOUNT @StartRow
SELECT @Sort = SortColumn FROM Table ORDER BY SortColumn
SET ROWCOUNT @PageSize
SELECT ... FROM Table WHERE SortColumn >= @Sort ORDER BY SortColumn
Full Code – Paging_RowCount

SubQuery

There are 2 more methods I’ve taken into consideration, and they come from different resources. The first one is well known triple query or the SubQuery method. The most thorough approach is the one I’ve found in the following article

Server-Side Paging with SQL Server

Although you'll need to be subscribed, a .zip file with the SubQuery stored procedure variations is available. The Listing_04.SELECT_WITH_PAGINGStoredProcedure.txt file contains the complete generalized dynamic SQL. I used a similar generalization logic with all other stored procedures in this text. Here is the principle followed by the link to the whole procedure (I shortened the original code a bit, because a recordcount portion was unnecessary for my testing purposes).

SQL
SELECT ... FROM Table WHERE PK IN 
    (SELECT TOP @PageSize PK FROM Table WHERE PK NOT IN
        (SELECT TOP @StartRow PK FROM Table ORDER BY SortColumn)
    ORDER BY SortColumn)
ORDER BY SortColumn
Full Code – Paging_SubQuery

Cursor

I’ve found the last method while browsing through the Google groups, you can find the original thread here. This method uses a server-side dynamic cursor. A lot of people tend to avoid cursors, they usually have poor performance because of their non-relational, sequential nature. The thing is that paging IS a sequential task and whatever method you use you have to somehow reach the starting row. In all the previous methods this is done by selecting all rows preceding the starting row plus the desired rows and then discarding all the preceding rows. Dynamic cursor has the FETCH RELATIVE option which does the “magic” jump. The base logic goes like this

SQL
DECLARE @PK /* PK Type */
DECLARE @tblPK TABLE (
    PK /* PK Type */ NOT NULL PRIMARY KEY
)

DECLARE PagingCursor CURSOR DYNAMIC READ_ONLY FOR
SELECT @PK FROM Table ORDER BY SortColumn

OPEN PagingCursor
FETCH RELATIVE @StartRow FROM PagingCursor INTO @PK

WHILE @PageSize > 0 AND @@FETCH_STATUS = 0
BEGIN
    INSERT @tblPK(PK) VALUES(@PK)
    FETCH NEXT FROM PagingCursor INTO @PK
    SET @PageSize = @PageSize - 1
END

CLOSE PagingCursor
DEALLOCATE PagingCursor

SELECT ... FROM Table JOIN @tblPK temp ON Table.PK = temp.PK <BR>ORDER BY SortColumn
Full Code – Paging_Cursor

Generalization of Complex Queries

As pointed out before, all the procedures are generalized with dynamic SQL, thus, in theory, they can work with any kind of complex query. Here is a complex query sample that works with Northwind database.

SQL
SELECT Customers.ContactName AS Customer, 
       Customers.Address + ', ' + Customers.City + ', ' + <BR>                                                Customers.Country AS Address, 
       SUM([Order Details].UnitPrice*[Order Details].Quantity) AS <BR>                                                          [Total money spent] 
FROM Customers
INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID
INNER JOIN [Order Details] ON Orders.OrderID = [Order Details].OrderID
WHERE Customers.Country <> 'USA' AND Customers.Country <> 'Mexico'
GROUP BY Customers.ContactName, Customers.Address, Customers.City, <BR>         Customers.Country 
HAVING (SUM([Order Details].UnitPrice*[Order Details].Quantity))>1000
ORDER BY Customer DESC, Address DESC
The paging stored procedure call that returns the second page looks like this
SQL
EXEC ProcedureName
/* Tables */
'Customers
INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID
INNER JOIN [Order Details] ON Orders.OrderID = [Order Details].OrderID',
/* PK */
'Customers.CustomerID',
/* ORDER BY */
'Customers.ContactName DESC, Customers.Address DESC',
/* PageNumber */
2,
/* Page Size */
10,
/* Fields */
'Customers.ContactName AS Customer,
Customers.Address + '', '' + Customers.City + '', '' + Customers.Country <BR>                                                                  AS Address, 
SUM([Order Details].UnitPrice*[Order Details].Quantity) AS [Total money spent]',
/* Filter */
'Customers.Country <> ''USA'' AND Customers.Country <> ''Mexico''',
/*Group By*/
'Customers.CustomerID, Customers.ContactName, Customers.Address, <BR> Customers.City, Customers.Country 
HAVING (SUM([Order Details].UnitPrice*[Order Details].Quantity))>1000'

Note that in the original query, aliases are used in the ORDER BY clause. You can't do that in paging procedures, because the most time-consuming task in all of them is skipping rows preceding the starting row. This is done in various ways, but the principle is not to fetch all the required fields at first, but only the PK column(s) (in case of RowCount method the sorting column), which speeds up this task. All required fields are fetched only for the rows that belong to the requested page. Therefore, field aliases don't exist until the final query, and sorting columns have to be used earlier (in row skipping queries).

The RowCount procedure has another problem, it is generalized to work with only one column in the ORDER BY clause. The same goes for Asc-Desc and Cursor methods, though they can work with several ordering columns, but require that only one column is included in the PK. I guess this could be solved with more dynamic SQL, but in my opinion it is not worth the fuss. Although these situations are highly possible, they are not that frequent. Even if they are, you can always write a separate paging procedure following the principles above.

Performance Testing

I used these 4 methods in my tests, if you have a better one, I’d be glad to know about it. Nevertheless, I wanted to compare these methods and measure their performance. The first thought was to write an ASP.NET test application with paged DataGrid and then measure page response. Still, this wouldn’t reflect the true response time of the stored procedures, so the console application seemed more appropriate. I also included a web application, not for performance testing, but rather as an example of how DataGrid custom paging works with these stored procedures. They are both incorporated in the PagingTest Solution.

I used the auto generated large table for my tests and inserted around 500 000 records in it. If you don’t have a large table to experiment on, you can download the script for a table design and stored procedure for data generation here. I didn't want an identity column for my PK, I used the uniqueidentifier instead. If you'll use this script, you may consider to add an identity after you generate the table. It will add numbers sorted by PK and you'll have an indication that correct page is fetched when you call a paging procedure with PK sorting.

The idea behind performance testing was to call a specific stored procedure many times through a loop and then measure the average response time. Also, in order to remove caching deviations and to model the real situation more accurately – multiple calls to a stored proc with the same page fetched each time seemed inappropriate. Thus, a random sequence of the same stored procedure with a set of different page numbers was required. Of course, a set of different page numbers assumes fixed number of pages (10 – 20) where each page would be fetched many times, but in a random sequence.

It’s not hard to notice that response times depend on the distance of the fetched page from the beginning of the resultset. The further the starting record is, more records need to be skipped. This is the reason I didn’t include first 20 pages in my random sequence. Instead I used the set of 2N pages. A loop was set to a (number of different pages)*1000. So, every page was fetched around 1000 times (more or less because of a random distribution).

Results

Here are the results I've got - Paging_Results (MS Excell file)
Image 1
Image 2
Image 3
Image 4
Image 5

Conclusion

The methods performed in the following order, starting from the best one - RowCount, Cursor, Asc-Desc and Subquery. The behavior in the lower portion was especially interesting, because in many real situations you'll browse beyond the first five pages rarely, so the Subquery method might satisfy your needs in those cases. It all depends on the size of your resultset and the prediction how frequently will the distant pages be fetched. You might use the combination of methods as well. As for myself, I decided to use the RowCount method wherever possible. It beaves quite nice, even for the first page. The "wherever possible" part stands for some cases where it's hard to generalize this method, then I would use the Cursor (possibly combined with the SubQuery for the first couple of pages).

Update 2004-05-05

The main reason I wrote this article was the feedback from the vast programming community. In a couple of weeks I'll be starting work on a new project. The preliminary analysis showed that there's going to be a couple of very large tables involved. These tables will be used in many complex joined queries and their results will be displayed in the ASP.NET application (with sorting and paging enabled). That's why I invested some time in research and pursue for the best paging method. It wasn't just the performance that interested me, but also the usability and maintainability.

Now the invested time has started to pay off already. You can find a post by C. v. Berkel below (many thanks) in which he found a flaw in the RowCount method. It won't work correctly if the sorting column is not unique. The RowCount method performed the best in my tests, but now I am seriously considering not using it at all. In most cases sorting columns (besides the PK) won't be unique. This leaves me with the Cursor method as the fastest and applicable to most situations. It can be combined with the SubQuery method for the first couple of pages and possibly with the RowCount method for unique sorting columns.

Another thing which may be worth mentioning is that there's a tiny flaw in the Asc-Desc method as well. It always returns the PageSize number of records for the last page and not the actual number (which may be lower than the PageSize). The correct number can be calculated but since I don't intend to use this procedure (because of how it performed), I didn't want to improve it any further.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Software Developer Mono Ltd
Croatia Croatia
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
GeneralRe: Problem with your SubQuery example Pin
HPDiggy6-Jun-06 22:08
HPDiggy6-Jun-06 22:08 
GeneralModification of ASC / DSC Pin
Anonymous27-Sep-05 12:39
Anonymous27-Sep-05 12:39 
GeneralRe: Modification of ASC / DSC Pin
Anonymous3-Oct-05 6:29
Anonymous3-Oct-05 6:29 
GeneralUnions Pin
Wayne Gibson6-Jul-05 10:10
Wayne Gibson6-Jul-05 10:10 
GeneralRe: Unions Pin
DiverKas31-Aug-05 4:07
DiverKas31-Aug-05 4:07 
GeneralRe: Unions Pin
crudmop1-Sep-05 6:59
crudmop1-Sep-05 6:59 
GeneralRe: Unions Pin
DiverKas1-Sep-05 7:18
DiverKas1-Sep-05 7:18 
GeneralResults using SQLServer 2005 Pin
Wayne Gibson4-Jul-05 0:46
Wayne Gibson4-Jul-05 0:46 
Hi,
Just recently found your article, which has helped me a lot. As I'm about to start work on a new project using SQLServer 2005 and wanted to handle the paging myself. So I ran your console app using your test data and stored procedures. These were the results that I obtained:

Stored Procedure - Paging_Asc_Desc, Sorted by - LargeTable.PK
Page AvgTime(ms) Repeated
0001 8.291998 1064
0002 8.498502 1057
0004 8.920390 1071
0008 9.189243 1068
0016 8.790213 1088
0032 8.533528 1082
0064 10.676187 1029
0128 8.900683 1106
0256 8.604966 1080
0512 9.507904 1127
1024 10.932310 1091
2048 10.314348 1035
4096 8.484024 1047
8192 12.947528 1055

Stored Procedure - Paging_Cursor, Sorted by - LargeTable.PK
Page AvgTime(ms) Repeated
0001 8.314010 1119
0002 8.002117 1065
0004 8.108675 1072
0008 8.348525 1046
0016 8.856720 1064
0032 8.296601 1096
0064 8.373274 1135
0128 10.686322 1088
0256 9.724534 1071
0512 8.072444 1052
1024 8.167563 1014
2048 8.192759 1083
4096 8.501365 1059
8192 8.168116 1036

Stored Procedure - Paging_RowCount, Sorted by - LargeTable.PK
Page AvgTime(ms) Repeated
0001 3.153030 1064
0002 3.117965 1031
0004 2.943163 1048
0008 3.452929 1105
0016 2.866718 1048
0032 2.890373 1081
0064 3.255159 1046
0128 3.185968 1053
0256 3.167820 1078
0512 3.162919 1105
1024 5.242944 1062
2048 3.204971 1103
4096 3.127273 1124
8192 3.312748 1052

Stored Procedure - Paging_SubQuery, Sorted by - LargeTable.PK
Page AvgTime(ms) Repeated
0001 2.662898 1053
0002 2.590598 1036
0004 2.663109 1083
0008 3.416987 1143
0016 2.854944 1133
0032 2.631056 1100
0064 2.659625 1043
0128 2.769550 1092
0256 2.871124 1015
0512 2.746807 1050
1024 3.086499 1048
2048 3.160059 1106
4096 3.011994 1044
8192 2.764887 1054

As you can see the SubQuery seems to be performing just as well as the RowCount. So I don't know if there have been any changes to SQLServer from 2000 to 2005 that would affect the results. But it looks like the order using SQLServer 2005 is as follows: SubQuery, RowCount, Paging_Cursor, Paging_Asc_Desc. Has anyone else had a similar experience with 2005? Or found any quicker methods..

Many thanks

Wayne Gibson
GeneralRe: Results using SQLServer 2005 Pin
Wayne Gibson4-Jul-05 3:30
Wayne Gibson4-Jul-05 3:30 
GeneralRe: Results using SQLServer 2005 Pin
Tom Pester5-Jul-05 4:33
Tom Pester5-Jul-05 4:33 
GeneralThank you for posting that tidbit Pin
Tek Boy16-Jul-05 16:18
Tek Boy16-Jul-05 16:18 
QuestionDos sql server support asc/desc method ? Pin
w3Nima2-Jun-05 6:59
w3Nima2-Jun-05 6:59 
AnswerRe: Dos sql server support asc/desc method ? Pin
Anonymous2-Jun-05 14:37
Anonymous2-Jun-05 14:37 
GeneralRe: Dos sql server support asc/desc method ? Pin
w3Nima3-Jun-05 19:59
w3Nima3-Jun-05 19:59 
GeneralRe: Dos sql server support asc/desc method ? Pin
Tom Pester4-Jun-05 1:11
Tom Pester4-Jun-05 1:11 
GeneralRe: Dos sql server support asc/desc method ? Pin
w3Nima4-Jun-05 4:33
w3Nima4-Jun-05 4:33 
Generali can How to get the total records Pin
lfabiano31-May-05 11:26
lfabiano31-May-05 11:26 
GeneralRe: i can How to get the total records Pin
Tom Pester1-Jun-05 1:26
Tom Pester1-Jun-05 1:26 
Generalsp_executesql instead of EXEC Pin
DejaVudew29-Apr-05 7:23
DejaVudew29-Apr-05 7:23 
GeneralAnother way to exec cursors Pin
ArtemL25-Apr-05 0:55
ArtemL25-Apr-05 0:55 
GeneralRe: Another way to exec cursors Pin
Anonymous25-Apr-05 4:04
Anonymous25-Apr-05 4:04 
QuestionHow to make it return an empty table if scroll beyond page? Pin
_joel26-Feb-05 22:07
_joel26-Feb-05 22:07 
GeneralSimple Fix for Row Count Pin
PaigePB8-Jan-05 7:34
PaigePB8-Jan-05 7:34 
GeneralOne that seems to work for me Pin
Riaan Lehmkuhl5-Jan-05 4:06
professionalRiaan Lehmkuhl5-Jan-05 4:06 
Generala small bug in store proceduce (Paging_RowCount.sql) Pin
Doan Hong Ha30-Dec-04 22:26
Doan Hong Ha30-Dec-04 22:26 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.