Update: There's a new article which explains the latest development of this project. Read here.
Introduction
I will show you how I built a start page similar to Google IG in 7 nights using ASP.NET AJAX, .NET 3.0, LINQ, DLinq and XLinq. I have logged my day to day development experience in this article, and documented all the technical challenges, interesting discoveries and important design and architectural decisions. You will find the implementation quite close to the actual Google IG. It has drag and drop enabled widgets, complete personalization of pages, multi-page feature, and so on. It's not just a prototype or a sample project. It's a real living and breathing Open Source start page running at http://dropthings.omaralzabir.com/ which you can use everyday. You are welcome to participate in the development and make widgets for the project.
Updates
- Jan 6, 2007: Scott Guthrie showed me how to improve ASP.NET AJAX client side performance by switching to
debug="false" in web.config. It improves performance significantly. Read here. - Jan 5, 2007: Deployment problem discussed. Read here.
- Jan 4, 2007: Visual Studio 2005 Extensions for .NET Framework 3.0 (Windows Workflow Foundation) required as prerequisite. Read here.
- Jan 4, 2007: Someone asked me if I am picking a fight with Google. I am not. I respect Google very much because they pioneered in this area and I am just a follower. Start Page is a really good project to show all these new technologies.
What is a Web 2.0 AJAX Start Page
A Start Page allows you to build your own homepage by dragging and dropping widgets on the page. You have complete control over what you want to see, where you want to see, and how you want to see. The widgets are independent applications which provide you with a set of features like to-do-list, address book, contact list, RSS feed, etc. Start Pages are also widely known as RSS aggregators, or in general terms, "content aggregators" from a variety of web sources. But you can not only read RSS feeds using your Start Page but also organize your digital life with it. AJAX Start Pages are one step ahead of old school start pages like My Yahoo by giving you state-of-the-art UI with lots of JavaScript effects. They give you a desktop application like look and feel by utilizing AJAX and lots of advanced JavaScript and DHTML techniques.
Some of the popular AJAX Start Pages are Pageflakes, Live, Google IG, Netvibes, Protopage, Webwag etc. Among these, Google IG is the simplest one. The one I have built here is something between real Google IG and Pageflakes in terms of AJAX and client-side richness. Google IG is mostly web 1.0 style postback model, and it's not really that much of AJAX. For example, you see it postback on switching page, adding new modules, changing widget properties, etc. But the one I have built here is a lot more AJAX, providing rich client-side experience close to what you see in Pageflakes.
Features
Build your page by dragging and dropping widgets. You can completely personalize the page by putting what you want and where you want. You can add, remove widgets on your page. You can drag and drop them where you like. Close your browser and come back again, you will see the exact setup as you left it. You can use it without registering as long as you like.
Once you put a lot of content on your page, you will find one page is not enough. You have the option to use multiple pages. You can create as many pages as you like.
Widgets
Widgets provide you with an interesting architecture where you can focus on providing the features relevant to the widget and never worry about authentication, authorization, profile, personalization, storage, framework, etc. All these are something widgets get for granted from their host. Moreover, you can build widgets independent of the host project. You do not need the whole host web application source code in your local development computer in order to build widgets. Just create a regular ASP.NET 2.0 Web site, create a user control, make it do what it's supposed to do in regular postback model without worrying about JavaScript, implement a little interface, and you are done! I have tried my best to create an architecture where you need not worry about AJAX and JavaScript at all. Also, the architecture allows you to use regular ASP.NET 2.0 controls, AJAX Control Toolkit controls, and any extender in ASP.NET AJAX. You also get full server-side programming support and can utilize .NET 2.0 or 3.0. You can use regular ViewState and store temporary states. You can also use ASP.NET Cache in order to cache data for Widgets. It is far better than what you find in current Start Pages where you have to build the whole widget using JavaScript and you need to abide by specific API guidelines and strict "no postback" model. Those who have built widgets for current Start Pages must know what a traumatizing experience widget development really is for them.
Technologies
The client side is built using ASP.NET AJAX RC and the AJAX Control Toolkit. Several custom extenders are used to provide the specialized drag and drop feature.
The middle tier is built using Windows Workflow Foundation, and the data access layer uses DLinq and SQL Server 2005.
Web Layer
There's basically only one page, the Default.aspx. All the client-side features you see are available in Default.aspx and in the form of Widgets. We cannot do postback or too much navigation between pages because that would kill the Web 2.0ness. So, all the features must be provided in one page which never posts back and does not redirect to other pages.
The tabs are just simple <UL> and <LI> inside an UpdatePanel. When you change the page title or add a new page, it does not post back the whole page because only the UpdatePanel that contains the tab refreshes. The other part of the page remains as it is.
public UserPageSetup NewUserVisit( )
{
var properties = new Dictionary<string,object>();
properties.Add("UserName", this._UserName);
var userSetup = new UserPageSetup();
properties.Add("UserPageSetup", userSetup);
WorkflowHelper.ExecuteWorkflow( typeof( NewUserSetupWorkflow ),
properties );
return userSetup;
}
Here, we pass the UserName (which is basically a GUID for a new user), and we get back a UserPageSetup object which contains the user settings and pages and widgets on the first page that is rendered on screen.
Similarly, on the second visit, it just loads the user's setup by executing UserVisitWorkflow.
public UserPageSetup LoadUserSetup( )
{
var properties = new Dictionary<string,object>();
properties.Add("UserName", this._UserName);
var userSetup = new UserPageSetup();
properties.Add("UserPageSetup", userSetup);
WorkflowHelper.ExecuteWorkflow( typeof( UserVisitWorkflow ),
properties );
return userSetup;
}
But how about performance? I did some profiling on the overhead of workflow execution, and it is very fast for synchronous execution. Here's proof from the log you get in the Visual Studio output window:
334ec662-0e45-4f1c-bf2c-cd3a27014691 Activity: Get User Guid 0.078125
b030692b-5181-41f9-a0c3-69ce309d9806 Activity: Get User Pages 0.0625
b030692b-5181-41f9-a0c3-69ce309d9806 Activity: Get User Setting 0.046875
b030692b-5181-41f9-a0c3-69ce309d9806 Activity: Get Widgets in page: 189 0.0625
334ec662-0e45-4f1c-bf2c-cd3a27014691 Total: Existing user visit 0.265625
The first four entries are the time taken by the individual activities during data access. The time entries here are in seconds, and the first four entries represent the duration of the database operations inside activities. The last one is the total time for running a workflow with 5 activities and some extra code. If you sum up all the individual activity execution time for database operations, it is 0.25, which is just 0.015 sec less than the total execution time. This means, executing the workflow itself takes around 0.015 sec, which is almost nothing.
Data Access Using DLinq
DLinq is so much fun. It's so amazingly simple to write the data access layer that generates really optimized SQL. If you have not used DLinq before, brace for impact!
When you use DLinq, you just design the database and then use SqlMetal.exe (comes with the LINQ May CTP) in order to generate a Data Access class which contains all the data access codes and entity classes. Think about the dark age when you had to hand code all entity classes following the database design and hand code data access classes. Whenever your database design changed, you had to modify the entity classes and modify the insert, update, delete, get methods in the data access layer. Of course, you could use third party ORM tools or use some kind of code generators which generate entity classes from database schema and generate data access layer codes. But, do no more, DLinq does it all for you!
The best thing about DLinq is it can generate something called Projection which contains only the necessary fields and not the whole object. There's no ORM tool or Object Oriented Database library which can do this now because it really needs a custom compiler in order to support this. The benefit of projection is pure performance. You do not select fields which you don't need, nor do you construct a jumbo object which has all the fields. DLinq only selects the required fields and creates objects which contains only the selected fields.
Let's see how easy it is to create a new object in the database called "Page":
var db = new DashboardData(ConnectionString);
var newPage = new Page();
newPage.UserId = UserId;
newPage.Title = Title;
newPage.CreatedDate = DateTime.Now;
newPage.LastUpdate = DateTime.Now;
db.Pages.Add(newPage);
db.SubmitChanges();
NewPageId = newPage.ID;
Here, DashboardData is the class which SqlMetal.exe generated.
Say, you want to change a Page's name:
var page = db.Pages.Single( p => p.ID == PageId );
page.Title = PageName;
db.SubmitChanges();
Here, only one row is selected.
You can also select a single value:
var UserGuid = (from u in db.AspnetUsers
where u.LoweredUserName == UserName &&
u.ApplicationId == DatabaseHelper.ApplicationGuid
select u.UserId).Single();
And, here's the Projection I was talking about:
var users = from u in db.AspnetUsers
select { UserId = u.UserId, UserName = u.LoweredUserName };
foreach( var user in users )
{
Debug.WriteLine( user.UserName );
}
If you want to do some paging like select 20 rows from 100 rows:
var users = (from u in db.AspnetUsers
select { UserId = u.UserId, UserName = u.LoweredUserName }).Skip(100).Take(20);
foreach( var user in users )
{
Debug.WriteLine( user.UserName );
}
If you are looking for a transaction, see how simple it is:
using( TransactionScope ts = new TransactionScope() )
{
List<Page> pages = db.Pages.Where( p => p.UserId == oldGuid ).ToList();
foreach( Page page in pages )
page.UserId = newGuid;
UserSetting setting = db.UserSettings.Single( u => u.UserId == oldGuid );
db.UserSettings.Remove(setting);
setting.UserId = newGuid;
db.UserSettings.Add(setting);
db.SubmitChanges();
ts.Complete();
}
Unbelievable? Believe it.
You may have some mixed feelings about DLinq performance. Believe me, it generates exactly the right SQL that I wanted it to do. Use SqlProfiler and see the queries it sends to the database. You might also think all these "var" stuff sounds like late binding in the old COM era. It will not be as fast as strongly typed code or your own hand written super optimal code which does exactly what you want. You will be surprised to know that all this DLinq code actually gets transformed into pure and simple .NET 2.0 IL by the LINQ compiler. There's no magic stuff or no additional libraries in order to run this code in your existing .NET 2.0 project. Unlike many ORM tools, DLinq also does not heavily depend on Reflection.
Day 1: Building the Widget Container Using UpdatePanel
There are two concepts here, one is the Widget Container and the other is the Widget. Widget Container provides the frame which has a header and a body area. The actual widget is loaded in the body area. WidgetContainer is a server control which is dynamically created on the page for each widget instance. The actual Widget is also a server control which is loaded dynamically inside the widget container.
Each Widget contains several UpdatePanels which helps smaller parts of the widgets to get updated without a whole page refresh or a whole Widget refresh. For example, the actual widget which is hosted inside the container is loaded inside an UpdatePanel. So, no matter how many times the actual widget postbacks, the whole widget does not postback or whole the column.
Finding the right combination of UpdatePanel and the distribution of HTML elements inside UpdatePanel was difficult. For example, I first put the whole widget inside one UpdatePanel. It worked nicely, there was only one UpdatePanel per widget so the overhead was small. But the problem was with the extenders which are attached with the HTML elements inside UpdatePanel. When UpdatePanel refreshes, it removes the existing HTML elements and creates new ones. As a result, all the extenders attached to the previous HTML elements get lost unless the extenders are also inside the UpdatePanel. Putting extenders inside UpdatePanel means whenever the UpdatePanel is refreshed, new instances of extenders are created and initialized. This makes the UI experience very slow. You can actually see the slowness visually when you do something on the widget which makes it postback itself.
So, the final idea was to separate the header area and the body area between multiple UpdatePanels. One UpdatePanel hosts the header area and the other UpdatePanel hosts the actual Widget. This way, if you do something on the widget and the widget body refreshes, it does not refresh the header area, and the extenders attached to the header do not get lost. The CustomFloatingBehavior extender is attached with the header. So, the extender itself needs to be inside the UpdatePanel. But, putting the extender inside the UpdatePanel means every time the UpdatePanel refreshes, the extender is created and initialized again. This gives poor performance.
So, the optimal solution so far is have two UpdatePanels per WidgetContainer, one contains the contents of the header, not the whole header itself. So, when the header UpdatePanel refreshes, the DIV which contains the whole header does not get recreated as it is outside the UpdatePanel. This way, we can put the CustomFloatingBehavior extender outside the UpdatePanel too. Thus, the extender can attach with the header container DIV.
The WidgetContainer is quite simple. It has the header area where the title and the expand/collapse/close buttons are, and the body area where the actual Widget is hosted. In the solution, the file "WidgetContainer.ascx" is the WidgetContainer.
<asp:Panel ID="Widget" CssClass="widget" runat="server">
<asp:Panel id="WidgetHeader" CssClass="widget_header" runat="server">
<asp:UpdatePanel ID="WidgetHeaderUpdatePanel" runat="server"
UpdateMode="Conditional">
<ContentTemplate>
<table class="widget_header_table" cellspacing="0"
cellpadding="0">
<tbody>
<tr>
<td class="widget_title"><asp:LinkButton ID="WidgetTitle"
runat="Server" Text="Widget Title" /></td>
<td class="widget_edit"><asp:LinkButton ID="EditWidget"
runat="Server" Text="edit"
OnClick="EditWidget_Click" /></td>
<td class="widget_button"><asp:LinkButton ID="CollapseWidget"
runat="Server" Text="" OnClick="CollapseWidget_Click"
CssClass="widget_min widget_box" />
<asp:LinkButton ID="ExpandWidget" runat="Server" Text=""
CssClass="widget_max widget_box" OnClick="ExpandWidget_Click"/>
</td>
<td class="widget_button"><asp:LinkButton ID="CloseWidget"
runat="Server" Text="" CssClass="widget_close widget_box"
OnClick="CloseWidget_Click" /></td>
</tr>
</tbody>
</table>
</ContentTemplate>
</asp:UpdatePanel>
</asp:Panel>
<asp:UpdatePanel ID="WidgetBodyUpdatePanel" runat="server"
UpdateMode="Conditional" >
<ContentTemplate><asp:Panel ID="WidgetBodyPanel" runat="Server">
</asp:Panel>
</ContentTemplate>
</asp:UpdatePanel>
</asp:Panel>
<cdd:CustomFloatingBehaviorExtender ID="WidgetFloatingBehavior"
DragHandleID="WidgetHeader"
TargetControlID="Widget" runat="server" />
When the page is loaded, for each widget instance, first a widget container is created and then the widget container hosts the actual widget inside it. WidgetContainer works as a gateway between the core framework and the actual Widget, and provides a convenient API for storing state, or changing the state of the widget like expanding/collapsing, etc. WidgetContainer also conveys important messages to the actual widget like when it is collapsed or when it is closed, etc.
protected override void OnInit(EventArgs e)
{
base.OnInit(e);
var widget = LoadControl(this.WidgetInstance.Widget.Url);
widget.ID = "Widget" + this.WidgetInstance.Id.ToString();
WidgetBodyPanel.Controls.Add(widget);
this._WidgetRef = widget as IWidget;
this._WidgetRef.Init(this);
}
Here, the widget container first loads the actual widget from the URL provided in the Widget definition. Then, it puts the widget inside a body panel. It also passes its own reference as IWidgetHost to the actual widget.
WidgetContainer implements the IWidgetHost interface which helps the actual widget to communicate with the framework and the container:
public interface IWidgetHost
{
void SaveState(string state);
string GetState();
void Maximize();
void Minimize();
void Close();
bool IsFirstLoad { get; }
}
The implementations are quite simple. For example, the IWidgetHost.Minimize collapses the widget body area:
void IWidgetHost.Minimize()
{
DatabaseHelper.Update<WidgetInstance>(this.WidgetInstance,
delegate(WidgetInstance i)
{
i.Expanded = false;
});
this.SetExpandCollapseButtons();
this._WidgetRef.Minimized();
WidgetBodyUpdatePanel.Update();
}
First, we update the WidgetInstance row and then we refresh the UI. The actual widget also gets a callback via the IWidget interface.
All the functionality of IWidgetHost was easy to implement except the Close one. When Close is called, we need to remove the widget from the page. This means, the WidgetContainer on the page and the WidgetInstance row in the database need to be removed. Now, this is something the WidgetContainer itself cannot do. It needs to be done by the column container which contains the WidgetContainer. The Default.aspx is the container of all WidgetContainers. So, whenever Close is called, WidgetContainer raises an event to Default.aspx, and Default.aspx does the actual work for removing the widget and refreshing the column.
Day 2: Building a Custom Drag and Drop Extender and Multicolumn Drop Zone
The AJAX Control Toolkit comes with a DragPanel extender which you can use to provide drag and drop support to panels. It also has a ReorderList control which you can use to provide reordering of items in a single list. Our widgets are basically panels with a header which acts as the drag handle and flows vertically in each column. So, it might be possible that we can create a reorder list in each column and use the DragPanel to drag the widgets. But, I could not use ReorderList because:
- The
ReorderList strictly uses an HTML Table to render its items. - The
ReorderList takes the Drag Handle template to create a drag handle for each item. We already have a drag handle created inside a Widget, so we cannot allow the ReorderList to create another drag handle. - I need a client-side callback on drag and drop and reordering of items so that I can make AJAX calls and persist the widget positions.
The next trouble was with the DragPanel extender. The default implement of Drag and Drop in the AJAX Control Toolkit has some problems:
- When you start dragging, the item becomes absolutely positioned, but when you drop it, it does not become static positioned. There's a small hack needed for restoring the original position to "static".
- It does not put the dragging item on top of all items. As a result, when you start dragging, you see the item is being dragged below other items, which makes the drag get stuck sometimes, especially when there's an
IFRAME.
So, I have made a CustomDragDropExtender and a CustomFloatingExtender. CustomDragDropExtender is for the column containers where the widgets are placed. It provides the reordering support. It allows any item right under the container to be ordered, which is marked with a specific class name. Here's how it works:
<asp:Panel ID="LeftPanel" runat="server" class="widget_holder" columnNo="0">
<div id="DropCue1" class="widget_dropcue">
</div>
</asp:Panel>
<cdd:CustomDragDropExtender ID="CustomDragDropExtender1" runat="server"
TargetControlID="LeftPanel" DragItemClass="widget"
DragItemHandleClass="widget_header"
DropCueID="DropCue1" DropCallbackFunction="WidgetDropped" />
LeftPanel becomes a widget container which allows the widgets to be dropped on it and reordered. The DragItemClass attribute on the extender defines the items which can be ordered. This prevents from non-widget HTML Divs from getting ordered. Only the DIVs with the class "widget" are ordered. So, say there are five DIVs with the class named "widget". It will allow reordering of only these five divs:
<div id="LeftPanel" class="widget_holder" >
<div id="WidgetContainer1_Widget" class="widget"> ... </div>
<div id="WidgetContainer2_Widget" class="widget"> ... </div>
<div id="WidgetContainer3_Widget" class="widget"> ... </div>
<div id="WidgetContainer4_Widget" class="widget"> ... </div>
<div id="WidgetContainer5_Widget" class="widget"> ... </div>
<div>This DIV will not move</div>
<div id="DropCue1" class="widget_dropcue"></div>
</div>
It also takes a DropCallbackFunction which it calls when a widget is dropped on the container.
function WidgetDropped( container, item, position )
{
var instanceId = parseInt(item.getAttribute("InstanceId"));
var columnNo = parseInt(container.getAttribute("columnNo"));
var row = position;
WidgetService.MoveWidgetInstance( instanceId, columnNo, row );
}
This allows me to get the widget which was dropped or reordered, the column, and the position. I can then call a Web Sservice and asynchronously inform the server what just happened. The server updates the position of the widgets according to the new placement.
Note: I am not doing a postback, instead calling a Web Service on Drag and Drop. If I do postback, say postback the column UpdatePanel, then the whole column will refresh, which gives a poor drag and drop experience. This is why the drag and drop does not refresh any part of the page and silently calls a Web Service in the background in order to save the position of the dropped widget.
The HTML output contains the column number inside the column DIV as an attribute, and each widget DIV contains the widget instance ID. These two IDs help the server identify what the column is and which widget has been moved.
<div id="LeftPanel" class="widget_holder" columnNo="0">
<div InstanceId="151" id="WidgetContainer151_Widget" class="widget">
The additional attributes are generated from the server side.
Now, making the first extender is really hard. I generally do not openly admit if something was hard for me, so trust me, when I say hard, it is "H A R D". The architecture is just so overwhelming when you start with. But gradually, you will grasp the idea, and you will surely try hard to appreciate the OOP style super slow JavaScript object model that ASP.NET AJAX provides.
Day 3: Building the Data Access Layer and Site Load
It was so easy to build the data access layer using DLinq. First I designed the database:
User contains a collection of pages. Each page contains a collection of WidgetInstances. A WidgetInstance represents one Widget. The Widget table contains the definition of the widget, e.g., the name of the widget and the user control file name which has the code for the widget. WidgetInstance represents an instance of a widget on a column and the row of a page. UserSetting stores some user level settings.
After designing the database, I used SqlMetal.exe and generated the data access class named DashboardData which contains all the entity classes and DLinq implementations for working with the database. DashboardData inherits from the DataContext class which is a base class in the System.Data.Dlinq namespace for all data access classes. It has all the methods for insert, update, delete, select, transaction management, connection management, etc.
I also created a convenient DatabaseHelper class which contains convenient methods for Insert, Update, and Delete. One of the issues with DLinq is that, if your entity travels through multi-tier, then they get detached from the DataContext from where they were initially loaded. So, when you try to update entities again using a different DataContext, you first need to attach the entity instance with the data context, then make the changes and call SubmitChanges. Now the problem is, from the business layer, you do not have access to the DataContext which will be created by the data access layer while updating the entity object. The Business Layer will just send the entity object to the data access component and then the data access layer will do the update by creating a new DataContext. But DLinq requires you to attach the entity object "before" making changes to them. But the regular business layer will make the modifications first and then send to the data access component in order to update the object. So, a traditional attempt like this will fail:
Page p = DashboardData.GetSomePage();
...
...
p.Title = "New Title";
DashboardData.UpdatePage( p );
Somehow you need to do this:
Page p = DashboardData.GetSomePage();
...
...
DashboardData.AttachPage( p );
p.Title = "New Title";
DashboardData.UpdatePage( p );
But this is not possible because this means you cannot make DashboardData stateless. You need to create DataContext inside methods, and somehow you need to store the reference to DataContext between the function calls. This might be OK for a single user scenario, but is not an acceptable solution for multiuser websites.
So, I did this approach:
Page p = DashboardData.GetSomePage();
...
...
DashboardData.Update<Page>( p, delegate( Page p1 )
{
p1.Title = "New Title";
});
Here, the Update<> method first attaches the page object with the DataContext and then calls the delegate passing the reference to the attached object. You can now modify the passed object as if you were modifying the original object inside the delegate. Once the delegate completes, it will be updated using DataContext.SubmitChanges();
The implementation of the Update<> method is this:
public static void Update<T>(T obj, Action<T> update)
{
var db = GetDashboardData();
db.GetTable<T>().Attach(obj);
update(obj);
db.SubmitChanges();
}
Here's an example usage:
WidgetInstance widgetInstance = DatabaseHelper.GetDashboardData().
WidgetInstances.Single( wi => wi.Id == WidgetInstanceId );
DatabaseHelper.Update<WidgetInstance>( widgetInstance,
delegate( WidgetInstance wi )
{
wi.ColumnNo = ColumnNo;
wi.OrderNo = RowNo;
});
The delegate gives us a benefit that you are in the context of the business layer or the caller. So, you can access UI elements or other functions/properties which you need in order to update the entity's properties.
For convenience, I have made Insert<>, Delete<> also. But they are not required as they do not have such an "Attach first, modify later" requirement.
public static void Delete<T>(Action<T> makeTemplate) where T:new()
{
var db = GetDashboardData();
T template = new T();
makeTemplate(template);
db.GetTable<T>().Remove(template);
db.SubmitChanges();
}
Day 4: Building a Flickr Photo and RSS Widget using XLinq
The first widget we will build is a nice Flickr widget.
It downloads Flickr photos as XML feed from the Flickr website and then renders a 3X3 grid with the pictures.
The first step is to download and parse the XML using XLinq. Here's an easy way to prepare an XElement from a URL:
var xroot = XElement.Load(url);
Nown we convert each photo node inside the XML to an object of the PhotoInfo class for convenient processing:
var photos = (from photo in xroot.Element("photos").Elements("photo")
select new PhotoInfo
{
Id = (string)photo.Attribute("id"),
Owner = (string)photo.Attribute("owner"),
Title = (string)photo.Attribute("title"),
Secret = (string)photo.Attribute("secret"),
Server = (string)photo.Attribute("server"),
Farm = (string)photo.Attribute("Farm")
})
But from the screenshot, you see you can navigate between the photos because Flickr actually returns more than 9 photos. So, we need to prepare objects of the PhotoInfo class from only those XML nodes which belong to the current paging index.
Here's how paging is done on the XML:
var photos = (from photo in xroot.Element("photos").Elements("photo")
select new PhotoInfo
{
Id = (string)photo.Attribute("id"),
Owner = (string)photo.Attribute("owner"),
Title = (string)photo.Attribute("title"),
Secret = (string)photo.Attribute("secret"),
Server = (string)photo.Attribute("server"),
Farm = (string)photo.Attribute("Farm")
}).Skip(pageIndex*Columns*Rows).Take(Columns*Rows);
We take only 9 photos from the current pageIndex. The page index is changed when the user clicks the Next or Previous links. The Skip method skips the number of items in the XML, and the Take method takes only the specified number of nodes from the XML.
Once we have the photo objects to render, a 3X3 HTML Table renders the photos:
foreach( var photo in photos )
{
if( col == 0 )
table.Rows.Add( new HtmlTableRow() );
var cell = new HtmlTableCell();
var img = new HtmlImage();
img.Src = photo.PhotoUrl(true);
img.Width = img.Height = 75;
img.Border = 0;
var link = new HtmlGenericControl("a");
link.Attributes["href"] = photo.PhotoPageUrl;
link.Attributes["Target"] = "_blank";
link.Attributes["Title"] = photo.Title;
link.Controls.Add(img);
cell.Controls.Add(link);
table.Rows[row].Cells.Add(cell);
col ++;
if( col == Columns )
{
col = 0; row ++;
}
count ++;
}
The reason why I used HtmlGenericControl instead of HtmlLink is that, HtmlLink does not allow you to add controls inside its Controls collection. This is a limitation of the HtmlLink class.
This was very easy to make using XLinq. Then, I built the RSS Widget which shows RSS Feeds from a feed source. First, I get the URL of the feed from the Widget State and then download the feed XML:
string url = State.Element("url").Value;
int count = State.Element("count") == null ? 3 :
int.Parse( State.Element("count").Value );
var feed = Cache[url] as XElement;
if( feed == null )
{
feed = XElement.Load(url);
Cache.Insert(url, feed, null, DateTime.MaxValue, TimeSpan.FromMinutes(15));
}
Then, I bind the XML to a DataList which shows a list of Hyperlinks:
FeedList.DataSource = (from item in feed.Element("channel").Elements("item")
select new
{
title = item.Element("title").Value,
link = item.Element("link").Value
}).Take(this.Count);
The DataList is very simple:
<asp:DataList ID="FeedList"
runat="Server" EnableViewState="False">
<ItemTemplate>
<asp:HyperLink ID="FeedLink" runat="server" Target="_blank"
CssClass="feed_item_link"
NavigateUrl='<%# Eval("link") %>'>
<%# Eval("title") %>
</asp:HyperLink>
</ItemTemplate>
</asp:DataList>
And that's all!
But there's a bit of tweaking with the state. Each RSS Widget stores the URL in its State. The Widget table has a DefaultState column which contains the predefined URL for the RSS widgets. When an RSS widget is created on the page, the default state is copied to the widget instance's state. XLinq makes it very easy to deal with the simple XML fragments. For example, here's how I read the URL:
public string Url
{
get { return State.Element("url").Value; }
set { State.Element("url").Value = value; }
}
The state XML is like this:
<state>
<count>3</count>
<url>...</url>
</state>
The State property parses the XML and returns it as an XElement which refers to the root node <state>:
private XElement State
{
get
{
if( _State == null ) _State = XElement.Parse(this._Host.GetState());
return _State;
}
}
Day 5: Building Workflows in the Business Layer
Here's a workflow which shows what happens when a user visits the site:
First, we get the UserGuid from the user's name. Then, we use the Guid to load the pages, user settings, and the widgets in the current page. Finally, we prepare a UserPageSetup object which contains all the information required to render the page.
Now, what happens when the user visits the site for the first time? We need to create an anonymous user and create a default page setup for the user and then load the user's page setup again. This is done inside the new user visit workflow, which is like this:
The last activity named "CallWorkflow" calls the User Visit workflow again in order to load the user setup which is just created. So, here we can see some reuse of the workflow.
The activities do very small amount of work. For example, the create new page activity creates a new page and returns the ID:
protected override ActivityExecutionStatus Execute(
ActivityExecutionContext executionContext)
{
DashboardData db = DatabaseHelper.GetDashboardData();
var newPage = new Page();
newPage.UserId = UserId;
newPage.Title = Title;
newPage.CreatedDate = DateTime.Now;
newPage.LastUpdate = DateTime.Now;
db.Pages.Add(newPage);
db.SubmitChanges(ConflictMode.FailOnFirstConflict);
NewPageId = newPage.ID;
return ActivityExecutionStatus.Closed;
}
DashboardFacade, which is the entry point to the business layer, is quite simple. It knows which workflows to invoke on which operations. It just takes the parameters and invokes the right workflow for the operation. For example, it has a NewUserVisit function which does nothing but execute NewUserVisitWorkflow.
public class DashboardFacade
{
private string _UserName;
public DashboardFacade( string userName )
{
this._UserName = userName;
}
public UserPageSetup NewUserVisit( )
{
var properties = new Dictionary<string,object>();
properties.Add("UserName", this._UserName);
var userSetup = new UserPageSetup();
properties.Add("UserPageSetup", userSetup);
WorkflowHelper.ExecuteWorkflow(
typeof( NewUserSetupWorkflow ), properties );
return userSetup;
}
There were three major headaches I had to solve while implementing the business layer using Workflow and DLinq.
- Synchronous execution of workflow in ASP.NET
- Getting objects out of workflow after execution is complete
- Invoke one workflow from another synchronously
Synchronous Execution of Workflow in ASP.NET
Workflow is generally made for asynchronous execution. WorflowRuntime is usually created only once per Application Domain, and the same instance of runtime is used everywhere in the same app domain. In ASP.NET, the only way you can ensure single instance of a WorkflowRuntime and make it available everywhere is by storing it in the HttpApplication. Also, you cannot use the Default scheduler service which executes workflows asynchronously. You need to use ManualWorkflowSchedulerService, which is specially made for synchronous workflow execution.
There's a handy class called WorkflowHelper which does the Workflow creation and execution. Its ExecuteWorkflow function executes a workflow synchronously.
public static void ExecuteWorkflow( Type workflowType,
Dictionary<string,object> properties)
{
WorkflowRuntime workflowRuntime =
HttpContext.Current.Application["WorkflowRuntime"] as
WorkflowRuntime;
ManualWorkflowSchedulerService manualScheduler =
workflowRuntime.GetService
<ManualWorkflowSchedulerService>();
WorkflowInstance instance =
workflowRuntime.CreateWorkflow(workflowType, properties);
instance.Start();
manualScheduler.RunWorkflow(instance.InstanceId);
}
It takes the type of workflow to execute and a dictionary of data to pass to the workflow.
Before running any workflow, first, the WorkflowRuntime needs to be initialized once and only once. This is done in the Global.asax in the Application_Start event.
void Application_Start(object sender, EventArgs e)
{
DashboardBusiness.WorkflowHelper.Init();
}
The WorkflowHelper.Init does the initialization work:
public static WorkflowRuntime Init()
{
var workflowRuntime = new WorkflowRuntime();
var manualService = new ManualWorkflowSchedulerService();
workflowRuntime.AddService(manualService);
var syncCallService = new Activities.CallWorkflowService();
workflowRuntime.AddService(syncCallService);
workflowRuntime.StartRuntime();
HttpContext.Current.Application["WorkflowRuntime"] = workflowRuntime;
return workflowRuntime;
}
Here, you see two services are added to the workflow runtime. One is for synchronous execution, and another is for synchronous execution of one workflow from another.
Invoke One Workflow from Another Synchronously
This was a major headache to solve. The InvokeWorkflow activity which comes with Workflow Foundation executes a workflow asynchronously. So, if you are calling a workflow from ASP.NET which in turn calls another workflow, the second workflow is going to be terminated prematurely instead of executing completely. The reason is, ManualWorkflowSchedulerService will execute the first workflow synchronously and then finish the workflow execution and return. If you use the InvokeWorkflow activity in order to run another workflow from the first workflow, it will start on another thread and it will not get enough time to execute completely before the parent workflow ends.
Here, you see only one activity in the second workflow gets the chance to execute. The remaining two activities do not get called at all.
Luckily, I found an implementation of a synchronous workflow execution at: http://www.masteringbiztalk.com/blogs/jon/PermaLink,guid,7be9fb53-0ddf-4633-b358-01c3e9999088.aspx
It's an activity which takes the workflow as input and executes it synchronously. The implementation of this activity is very complex. Let's skip it.
Getting Objects Out of Workflow After Execution is Complete
This one was the hardest. The usual method for getting data out of a workflow is to use the CallExternalMethod activity. You can pass an interface while calling a workflow, and the activities inside the workflow can call the host back via the interface. The caller can implement the interface and get the data out of the workflow.
It is a requirement that the interface must use intrinsic data types or types which are serializable. Serializable is a requirement because the workflow can go to sleep or get persisted and restored later on. But, DLinq entity classes cannot be made serializable. The classes that SqlMetal generates are first of all not marked as [Serializable]. Even if you add the attribute manually, it won't work. I believe, during compilation, the classes are compiled into some other runtime class which does not get the Serializable attribute. As a result, you cannot pass DLinq entity classes from an activity to a workflow host.
The workaround I found was to pass the object references as properties in the dictionary that we pass to the workflow. As ManualWorkflowSchedulerService runs the workflow synchronously, the object references remain valid during the lifetime or the workflow. There is no cross appdomain call here, so there is no need for serialization. Also, modifying the objects or using them does not cause any performance problem because the objects are allocated in the same process.
Here's an example:
public UserPageSetup NewUserVisit( )
{
var properties = new Dictionary<string,object>();
properties.Add("UserName", this._UserName);
var userSetup = new UserPageSetup();
properties.Add("UserPageSetup", userSetup);
WorkflowHelper.ExecuteWorkflow( typeof( NewUserSetupWorkflow ), properties );
return userSetup;
}
So far so good. But, how do you write DLinq code in a WinFX project? If you create a WinFX project and start writing LINQ code, it won't compile. LINQ requires a special compiler in order to generate C# 2.0 IL out of LINQ code. There's a specialized C# compiler in the "C:\Program Files\Linq Preview\bin" folder which MSBuild uses in order to compile LINQ codes. After a long struggle and comparison between a LINQ project file and a WinFX project file, I found that the WinFX project has a node at the end:
<Import
Project="$(MSBuildExtensionsPath)\Microsoft\Windows Workflow Foundation\
v3.0\Workflow.Targets" />
And the LINQ project has the node:
<Import Project="$(ProgramFiles)\LINQ Preview\Misc\Linq.targets" />
These notes select the right MSBuild script for building the projects. But if you just put the LINQ node in a WinFX project, it does not work. You have to comment out the first node:
After this, it built the code and everything ran successfully.
But workflows with Conditions and Rules did not run. At runtime, the workflows threw a "Workflow Validation Exception". When I use code in the rule, it works. But if I use Declarative Rules in a Condition, then it does not work. Declarative rules are added as an Embedded Resource under the workflow or activity which contains all the rules defined in XML format. It appears that the .rules file does not get properly embedded and the workflow runtime cannot find it while executing a workflow.
Now, this was a dead end for me. If I create a regular WinFX project, then it works fine. But then again, I cannot write LINQ code in a regular WinFX project. So, I have to create a mix of LINQ and WinFX projects and use no Declarative rules. But I so desperately wanted to write rules in workflows and activities. I struggled the whole night on this problem, but found no solution. It was so frustrating. Then in the dawn, when there was absolute silence everywhere and the sun was about to rise, I had a divine revelation to me from heaven:
Thou shalt bring forth the source of misery above thy.
So, I did. I brought the .rules file (source of misery) from under the .cs file to one level upward on the project level. It then looked like this:
For this, I had to open the Project file (.csproj) in Notepad and remove the <DependentUpon> node under the <EmbeddedResource> node:
<ItemGroup>
<EmbeddedResource Include="Activities\CreateDeafultWidgetsOnPageActivity.rules">
</EmbeddedResource>
And it worked! There's absolutely no way in the world I could have known that, right?
Day 6: Page Switch Problem
Widgets need to know whether it's the first time load of the widget or if it is a postback. Normally, when it's a first time load, widgets load all the settings from their persisted state and render the UI for the first time. Upon postback, widgets don't restore settings from the persisted state always; instead, sometimes they update the state or reflect small changes on the UI. So, it is important for users to know when they are being rendered for the first time and when it is a postback.
However, when you have multiple tabs, the definition of first time load and postback changes. When you click on another tab, it's a regular postback for ASP.NET because a LinkButton gets clicked. This makes the Tab UpdatePanel postback asynchronously, and on the server-side, we find out which tab is clicked. Then, we load the widgets on the newly selected tab. But, when the widgets load, they call Page.IsPostBack and they get true. So, widgets assume they are already on the screen and try to do partial rendering or try to access the ViewState. But, this is not true because they did not appear on screen yet and there's no ViewState for the controls on the widget. As a result, the widgets behave abnormally and all ViewState access fails.
So, we need to make sure during tab switch, although it's a regular ASP.NET postback, Widgets must not see it as postback. The idea is to inform widgets whether it is a first time load or not, via the IWidget interface.
On Default.aspx, there's a function SetupWidgets which creates the WidgetContainer and loads the widgets. Here's how it works:
private void SetupWidgets(Func<WidgetInstance, bool> isWidgetFirstLoad)
{
var setup = Context.Items[typeof(UserPageSetup)] as UserPageSetup;
var columnPanels = new Panel[] {
WidgetViewUpdatePanel.FindControl("LeftPanel") as Panel,
WidgetViewUpdatePanel.FindControl("MiddlePanel") as Panel,
WidgetViewUpdatePanel.FindControl("RightPanel") as Panel};
foreach( Panel panel in columnPanels )
{
List<WidgetContainer> widgets =
panel.Controls.OfType<WidgetContainer>().ToList();
foreach( var widget in widgets ) panel.Controls.Remove( widget );
}
Skip the Func<> thing for a while. First, I clear the columns which contain the WidgetContainer so that we can create the widgets again. See the cool LINQ way to find out only the WidgetContainer controls from the Panel's Controls collection.
Now, we create the WidgetContainers for the widgets on the newly selected tab:
foreach( WidgetInstance instance in setup.WidgetInstances )
{
var panel = columnPanels[instance.ColumnNo];
var widget = LoadControl(WIDGET_CONTAINER) as WidgetContainer;
widget.ID = "WidgetContainer" + instance.Id.ToString();
widget.IsFirstLoad = isWidgetFirstLoad(instance);
widget.WidgetInstance = instance;
widget.Deleted +=
new Action<WidgetInstance>(widget_Deleted);
panel.Controls.Add(widget);
}
While creating, we set a public property IsFirstLoad of the WidgetContainer in order to let it know whether it is being loaded for the first time or not. So, during the first time load of Default.aspx or during tab switch, the widgets are setup by calling:
SetupWidgets( p => true );
What you see here is called Predicate. This is a new feature in LINQ. You can make such predicates and avoid creating delegates and the complex coding model for delegates. The predicate returns true for all widget instances, and thus all widget instances see it as first time load.
So, why not just send "true" and declare the function as SetupWidgets(bool). Why go for the black art in Linq?
Here's a scenario which left me no choice but to do this. When a new widget is added on the page, it is a first time loading experience for the newly added widget, but it's a regular postback for existing widgets already on the page. So, if we pass true or false for all widgets, then the newly added widget will see it as a postback just like all other existing widgets on the page and thus fail to load properly. We need to make sure it's a non-postback experience only for the newly added widget, but a postback experience for the existing widget. See how it can be easily done using this Predicate feature:
new DashboardFacade(Profile.UserName).AddWidget( widgetId );
this.SetupWidgets(wi => wi.Id == widgetId);
Here, the predicate only returns true for the new WidgetId, but returns false for the existing WidgetId.
Day 7: Signup
When a user first visits the site, an anonymous user setup is created. Now, when the user decides to signup, we need to copy the page setup and all user related settings to the newly signed up user.
The difficulty was to get the anonymous user's GUID. I tried Membership.GetUser(), passing Profile.UserName which contains the anonymous user name. But, it does not work. It seems Membership.GetUser only returns a user object which exists in the aspnet_membership table. For anonymous users, there's no row in the aspnet_membership table, only in the aspnet_users and aspnet_profile tables. So, although you get the user name from Profile.UserName, you cannot use any of the methods in the Membership class.
The only way to do it is to read the UserId directly from the aspnet_users table. Here's how:
AspnetUser anonUser = db.AspnetUsers.Single( u =>
u.LoweredUserName == this._UserName
&& u.ApplicationId == DatabaseHelper.ApplicationGuid );
Note: You must use LoweredUserName, not the UserName field, and must include ApplicationID in the clause. The Aspnet_users table has an index on ApplicationID and LoweredUserName. So, if you do not include AapplicationID in the criteria and do not use the LoweredUserName field, the index will not hit, and the query will end up in a table scan which is very expensive. Please see my blog post for details on this: Careful-when-querying-on-aspnet.
Once we have the UserId of the anonymous user, we just need to update the UserID column in Page and the UserSetting table to the newly registered user's UserId.
So, first get the new and old UserId:
MembershipUser newUser = Membership.GetUser(email);
AspnetUser anonUser = db.AspnetUsers.Single( u =>
u.LoweredUserName == this._UserName
&& u.ApplicationId == DatabaseHelper.ApplicationGuid );
Guid oldGuid = anonUser.UserId;
Guid newGuid = (Guid)newUser.ProviderUserKey;
Now, update the UserId field of the Pages of the user:
List<Page> pages = db.Pages.Where( p => p.UserId == oldGuid ).ToList();
foreach( Page page in pages )
page.UserId = newGuid;
But here's a catch. You cannot change the field value if it's a primary key, using DLinq. You have to delete the old row using the old primary key and then create a new row using the new primary key:
UserSetting setting = db.UserSettings.Single( u => u.UserId == oldGuid );
db.UserSettings.Remove(setting);
setting.UserId = newGuid;
db.UserSettings.Add(setting);
See DashboardFacade.RegisterAs(string email) for the full code.
Web.config Walkthrough
The web project is a mix of WinFX, LiINQ, and ASP.NET AJAX. So, the web.config needs to be configured in such a way that it allows harmonious co-existence of these volatile technologies. The web.config itself requires a lot of explanation. I will just highlight the areas which are important.
You need to use the LINQ compiler so that the default C# 2.0 compiler does not compile the site. This is done by:
<system.codedom>
<compilers>
<compiler language="c#;cs;csharp" extension=".cs"
type="Microsoft.CSharp.CSharp3CodeProvider,
CSharp3CodeDomProvider"/>
</compilers>
</system.codedom>
Then, you need to put some extra attributes in the <compilation> node:
<compilation debug="true" strict="false" explicit="true">
Now, you need to include the ASP.NET AJAX assemblies and WinFX assemblies:
<compilation debug="true" strict="false" explicit="true">
<assemblies>
<add assembly="System.Web.Extensions, ..."/>
<add assembly="System.Web.Extensions.Design, ..."/>
<add assembly="System.Workflow.Activities, ..."/>
<add assembly="System.Workflow.ComponentModel, ..."/>
<add assembly="System.Workflow.Runtime, ..."/>
You also need to put "CSharp3CodeDomProvider.dll" in the "bin" folder and add references to System.Data.DLinq, System.Data.Extensions, System.Query, and System.Xml.Xlinq. All these are required for LINQ.
I generally remove some unnecessary HttpModule from the default ASP.NET pipeline for faster performance:
<httpModules>
<remove name="Session"/>
<remove name="WindowsAuthentication"/>
<remove name="PassportAuthentication"/>
<remove name="UrlAuthorization"/>
<remove name="FileAuthorization"/>
<add name="ScriptModule" type="System.Web.Handlers.ScriptModule, ..."/>
</httpModules>
How Slow is ASP.NET AJAX
Very slow, especially in Internet Explorer 6. In fact, its slowness is so bad that you can visually see it on the local machine while running on your powerful development computer. Try pressing F5 several times on a page where all the required data for the page are already cached on the server side. You will see the total time it takes to fully load the page is quite long. ASP.NET AJAX provides a very rich object oriented programming model and a strong architecture which comes at a high price on performance. From what I have seen, as soon as you put UpdatePanels on the page and some extenders, the page becomes too slow. If you just stick to the core framework for only the Web Service call, you are fine. But as soon as you start using UpdatePanel and some extenders, it's pretty bad. ASP.NET AJAX performance is good enough for simple pages which have, say, one UpdatePanel and one or two extenders for some funky effects. Maybe, one more data grid on the page or some data entry form. But that's all that gives an acceptable performance. If you want to make a Start Page like a website where one page contains almost 90% of the functionality of the whole website, the page gets heavily loaded with the JavaScript generated by the extenders and UpdatePanels. So, the Start Page is not something that you should make using UpdatePanels and Extenders. You can, of course, use the core framework without doubt for Web Service calls, XML HTTP, login/logout, profile access etc.
Update: Scott Guthrie showed me that changing debug="false" in web.config emits much lighter runtime scripts to the client side and all the validation gets turned off. This results in fast JavaScript execution for the extenders and the UpdatePpanel. You can see the real performance from the hosted site right now. The performance is quite good after this. IE 7, FF, and Opera 9 shows much better performance. But IE 6 is still quite slow, but not as slow as it was before with debug="true" in the web.config.
When you make a Start Page, it is absolutely crucial that you minimize network roundtrips as much as possible. If you study Pageflakes, you will see on first time load, the Wizard is visible right after 100KB of data transfer. Once the wizard is there, the rest of the code and content download in the background. But if you close the browser and visit again, you will see the total data transfer over the network is around 10 KB to 15 KB. Pageflakes also combines multiple smaller scripts and stylesheets into one big file so that the number of connections to the server is reduced and the overall download time is less than the large number of smaller files. You really need to optimize to this level in order to ensure people feel comfortable using the Start Page everyday. Although this is a very unusual requirement, this is something you should try in all AJAX applications because AJAX applications are full of client-side code. Unfortunately, you cannot achieve this using ASP.NET AJAX unless you do serious hacking. You will see that even for a very simple page setup which has only three extenders, the number of file downloadeds is significant:
All the files with ScriptResource.axd are small scripts in the AJAX Control Toolkit and my extenders. The size you see here is after gzip compression, and they are still quite high. For example, the first two are nearly 100 KB. Also, all these are individual requests to the server which could be combined into one JS file and served in one connection. This would result in better compression and much less download time. Generally, each request has 200ms of network roundtrip overhead, which is the time it takes for the request to reach the server and then the first byte of the response to return to the client. So, you are adding 200ms for each connection for nothing. It is quite apparent to ScriptManager which scripts are needed for the page on the server-side because it generates all the script references. So, if it could combine them into one request and serve them gzipped, it could save significant download time. For example, here 12 X 200ms = 2400ms = 2.4 sec is being wasted on the network.
However, one good thing is that, all of these get cached and thus do not download a second time. So, you save significant download time on future visits.
So, the final statement: UpdatePanels and Extenders are not good for websites which push client-side richness to the extreme like AJAX Start Pages, but definitely very handy for not so extreme websites. It's very productive to have designer support in Visual Studio and very good ASP.NET 2.0 integration. It will save you from building an AJAX framework from scratch and all the JavaScript controls and effects. In Pageflakes, we realized there is no point in building a core AJAX framework from scratch, and we decided to use the Atlas runtime for the XmlHttp and Web Service call. Besides the core AJAX stuff, everything else is homemade, including the drag and drop, expand/collapse, fly in/out, etc. These are just too slow using UpdatePanel and Extenders. Both speed and smoothness are very important to Start Pages because they are set as the browser homepage.
Deployment Problem
Due to a problem in ASP.NET AJAX RC version, you can't just copy the website to a production server and run it. You will see none of the scripts are loading because ScriptHandler malfunctions. In order to deploy it, you will have to use the "Publish Website" option to pre-compile the whole site and then deploy the precompiled package.
How to Run the Code
Remember, you cannot just copy the website to a server and run it. It will not run. Something wrong with the ScriptResource handler in the ASP.NET AJAX RC version. You will have to Publish the website and copy the precompiled site to a server.
Next Steps
If you like this project, let's make some cool widgets for it. For example, a To-do-list, Address book, Mail Widget, etc. This can become a really useful Start Page if we can make some useful widgets for it. We can also try making a widget which runs Google IG modules or Pageflakes' flakes on it.
Conclusion
AJAX Start Page is a really complex project where you push DHTML and JavaScript to their limits. As you add more and more features on the client side, the complexity of the web application increases geometrically. Fortunately, ASP.NET AJAX takes away a lot of complexity on the client-side so that you can focus on your core features and leave the framework and AJAX stuff to the runtime. Moreover, DLinq and the cool new features in .NET 3.0 makes it a lot easier to build a powerful data access layer and a business logic layer. Making all these new technologies work with each other was surely a great challenge and a rewarding experience.
Shameless disclaimer: I am the co-founder and CTO of Pageflakes, the coolest Web 2.0 AJAX Start Page. I like building AJAX websites, and I am really, really good at it.
