LINQ Part 3: An Introduction to IQueryable

• Download source - 66 KB
• Download QueryableFun-Exe.zip - 5.4 MB

Introduction

In this series, we've explored IEnumerable and taken a look at the standard methods that extend this interface. Together these form a small, but crucial part of LINQ, known as LINQ to Objects.

Another important aspect of LINQ is its ability to query other data sources (e.g. databases) where the information does not reside in local memory. In fact, it may even exist on an entirely different machine.

To support this very different requirement, LINQ introduces the concepts of query provider (System.Linq.IQueryProvider), expression tree (System.Linq.Expressions.Expression), and queryable sequence (System.Linq.IQueryable). This article will explore these concepts.

Background

With LINQ to Objects, LINQ can simply provide standard methods that operate directly on the sequences that are queried. With other data sources, this would be inefficient.

For example, for a database, we want to make as few round-trips to the server as possible. Also, we want to request as little information as possible. Finally, we want to take advantage of all of the "smarts" of the database product.

To facilitate this, LINQ introduces the following new concepts:

• Query provider (IQueryProvider) - This is specialized software that can interpret a query, so that it efficiently utilizes the underlying resources.
• Expression tree (Expression) - A tree is formed from elements of a query. This tree is later analyzed by the query provider. This should be familiar to anyone who has ever written a parser or compiler.
• Queryable sequence (IQueryable) - This is the approximate equivalent to IEnumerable in LINQ to Objects. It simply pairs a query provider with an expression tree.

As with IEnumerable, LINQ provides a set of standard methods, defined in the System.Linq.Queryable class. These methods all extend IQueryable. They all share identical names and near identical syntax with their counterparts in System.Linq.Enumerable.

While this reduces the learning curve for developers, it is also deceptive. While conceptually similar, these methods accomplish their goal in an entirely different manner: they build the expression tree.

This is the third in a series of articles on LINQ. Links to other articles in this series are as follows:

• LINQ Part 1: A Deep Dive into IEnumerable
• LINQ Part 2: Standard Methods - Tools in the Toolbox
• LINQ Part 3: An Introduction to IQueryable
• LINQ Part 4: A Deep Dive Into a Queryable Extension Method

LINQ Expression Trees

Most of the methods in System.Linq.Enumerable simply create a new instance of IEnumerable that wraps the one on which it operates. Since the underlying sequence generally exists in memory, or is easily acquired, there is no real concern about the mechanism by which it is fetched.

The methods in System.Linq.Queryable operate in a different manner. Each consumes an IQueryable, which is simply a pairing of an expression tree (Expression) with a query provider (IQueryProvider). Each method produces a new IQueryable, where the expression tree has been altered to include additional elements of the query.

Let's consider a simple query:

Source.Students.Where(student => student.LastName == "Marx")

The first IQueryable we need to consider is Source.Students. In this case, the expression tree (IQueryable.Expression) consists entirely of a single node (ConstantExpression). The value of this node (ConstantExpression.Value) is simply the queryable itself.

The next IQueryable we need to consider is the one returned by the Where method. Here things get a lot more complex. In this case, the expression tree (IQueryable.Expression) consists of many nodes. Visually, it appears as follows:

The nodes in the tree are as follows:

1. The call to the Where method, which takes two arguments, each of which appears in the tree.
2. The original IQueryable upon which the Where method operates.
3. A container for the Lambda expression that provides the predicate for the Where method. Note: It seems odd (even to me) that this node is required.
4. The Lambda expression (student => student.LastName == "Marx") that provides the predicate for the Where method.
5. The parameter for the Lambda expression (student).
6. The "equals" comparison that is evaluated within the Lambda expression (student.LastName == "Marx").
7. The left hand operand for the "equals" comparison, which accesses a property/member (LastName).
8. The right hand operand for the "equals" comparison, the constant "Marx".
9. The parameter for the property/member access (student). This is the specific instance where the property value is found.

What Good is an Expression Tree?

So, now we have a nifty expression tree. What use is it?

This is where the query provider (IQueryProvider) comes into the picture. It is responsible for "executing" the query described by the expression tree.

For example, in the case of a database provider, it may do quite a bit. First, it needs to translate the expression tree into a SQL query. So, in our previous example, this may be something like:

SELECT * FROM Students WHERE LastName = 'Marx'

It might then need to execute the query on the database server, fetch the results, instantiate objects for each of the rows, and provide an enumerable of these objects.

A Quick Review

From this article, we've learned the following:

• The extension methods in System.Linq.Queryable operate on instances of IQueryable, by building an expression tree (Expression).
• IQueryable simply pairs the expression tree (Expression) with a query provider (IQueryProvider).
• The query provider (IQueryProvider) is responsible for interpreting the expression tree, executing the query, and fetching the results.

The key word here is "interpreting". Because the query provider (and its underlying resource) is likely more limited than the C# language, there are limitations to what can appear within the query.

In general, the Lambda expressions are limited to fairly vanilla operations: comparisons, string manipulations, and projection (creation of new instances from existing property values). The exact limitations are dependent on the query provider itself.

If you're merely consuming a LINQ queryable, what you've learned so far, is about all you need to understand.

The remainder of this article dives a bit deeper. Feel free to learn more, or skip it, depending on your specific needs.

Exploring More Complex Queries

So far, we've only looked at a very simple query. As the query grows in complexity, the expression tree becomes rather large. To facilitate exploring more complex queries, an application (QueryableFun) is included with this article.

Mostly, I recommend simply playing with the application. It allows the user to run some pre-defined queries. These are selected from the "Queries" drop down list, which appears in the upper right hand corner of the window, in the tool strip.

To execute a query, simply click the "Run" button ().

The application consists of two panels: the expression tree (left) and a tab control (right). Within the tab control, you'll find four tabs. There are as follows:

• Expression - Enter or view C# LINQ expressions here.
• Properties - When you click on a node in the expression tree, this will display the properties for that node.
• Results - A data grid with the results of the query.
• Errors - Any compilation errors that occur while evaluating the expression.

A screen capture of the application appears below. Please understand that this is simply a learning application. It is not intended for any production use. As such its quality is lower than what I would normally produce.

For serious explorations of LINQ, I highly recommend LINQPad. I have zero association with the company. I simply relied on this, quite heavily, when I was teaching myself LINQ. The application comes in both free and paid versions.

IQueryable Members

While this is the most important interface, it is also the simplest. Its primary purpose is to pair an expression tree with a query provider. In fact, it is so simple that a single implementation could satisfy most query providers.

It has only four members, which are as follows:

IQueryProvider Members

While the implementation of this interface is generally quite complex, the interface itself is quite simple. It allows a consumer to create queryables and execute expressions. It has only four members:

QueryableFun Modules

The QueryableFun application is mostly intended to provide a flavor of what expression trees look like. While it is not a very serious effort, it does demonstrate a few clever tricks.

• Roslyn - It provides an example of compiling and executing C# script using Microsoft's Roslyn technology.
• IQueryable - It provides an example of a very simple IQueryable implementation. Instead of the complexity of evaluating and executing a query against an external resource, this implementation simply wraps an underlying collection.
• IQueryProvider - It provides an example of a very simple IQueryProvider implementation. This implementation simply alters the expression tree to replace the original queryable with a queryable for an underlying collection. It then tricks LambdaExpression.Compile() into doing all the heavy lifting.
• Toy data - It provides some "toy" data for LINQ queries, so that no database or more complex resource is required.

The modules in this application are as follows:

In the Transeric.Queryable project, which is also a part of this application, "toy" data is provided in the form of simple POCO classes and collections based upon them.

Further Reading

For further reading, see the following:

IQueryable<T> Interface
https://docs.microsoft.com/en-us/dotnet/api/system.linq.iqueryable-1

IQueryProvider Interface
https://docs.microsoft.com/en-us/dotnet/api/system.linq.iqueryprovider

Expression Class
https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.expression

Queryable Class
https://docs.microsoft.com/en-us/dotnet/api/system.linq.queryable

LINQ to Objects
https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/linq-to-objects

LINQ to SQL
https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/linq/index

LINQPad
https://www.linqpad.net/

History

• 4/20/2018 - The original version was uploaded
• 4/21/2018 - Added links to other articles in this series
• 4/23/2018 - Updated exe and src to handle singleton LINQ methods
• 4/24/2018 - A couple of minor cosmetic and grammar corrections
• 4/25/2018 - Added link to fourth article in series