Blog Archive

Home
jQuery

This post is in our Javascript Primer series, a collections of articles aimed at .Net and back-end developers who are transition to client-side UX development.  In a previous installment, the topic of publish-subscribe design pattern was featured, and this pattern fostered loosely coupled ViewModels.  It is recommended that you read that post as well, as this edition of our primer builds on those concepts.

A basic tenant to good programming is to break down activities to small components.  Smaller functions are easier to maintain.  In many instances the information that the users are working with is best presented in logical groups, dissected so that an improper decision can never be made.  A rich user interface may force you to break a larger object into several ViewModels.  But guess what?  If you need to break information down, will also need to re-assemble this information to transmit back to mother ship to be stored in your database.

Challenges for the ViewModels

There are several challenges ahead for you if need your ViewModels to communicate, and you want to retain the agility provided to you with loose coupling.  If you recall the architecture we established with the publish and subscribe design pattern, all our “pieces” are highly independent from one another.  But we now have a complication.  Independence also means blissful ignorance.  Each ViewModel only knows how to contribute their small portions of information for glorious “mother object”.  The best it can is yield up what it has and somehow, someway, something will piece things together.

Let’s consider a code example:

We have simple project tracker that has a name, a roster of workers and a calendar that displays each workers start date.  Each tab is serviced by a ViewModel.  The “Project Info” tab shares the project name information with “Team”.  The tab “Team” shares the project team data with both the “Project Info” and the “Timeline” tabs.  This is accomplished using the publish and subscribe pattern that was detailed in this post.  You should review that post is your are unfamiliar with postaljs or with the publish and subscribe design pattern.

The challenge here is that we want a “Save” button that will easily gather the data from each tab and save it our project object.   We also want this controlling activity to be flexible and allow each ViewModel to supply the data needed without too much orchestration: in other words, we just ask the ViewModel “What do you have for a project object”, and each ViewModel fulfills its responsibility.

Pipe and Filter Pattern For Chaining Events or Commands

Many of you with a background in server side application development may recognize the pipe and filter pattern.  Simply said, it is a way to chain a set of operations that are to be performed in succession.  This diagram is a depiction of the pipe and filter:

PipesAndFilters

The “Filter” is a function that performs one thing.  Our pipe is the “orchestrator” or directory in that it will forward to each filter as ordered.

For our purposes we will continue to use postaljs as our communication mechanism.  We’ll need a pipeline that will keep track of which filter to execute and forward data to each subsequent filter of the process chain.

var pipeline = {
  index: 0,
  filters: ["edit.getViewOneData","edit.getViewTwoData","edit.finalDestination"]
 };

The property “filters” contains the names of the postaljs topics that will kick off a function.  In this case we will need 3 subscriptions that listen for items listed in the “filters” array.  Each process step, or filter, will be executed in the order that listed in the “filters” array: in this case, “edit.getViewOneData” will be executed first.  The property “index” tracks which filter is being processed.  As each filter is executed, it will increment index, then use index to access the next topic in the chain of events.  Starting off the process is accomplished in this fashion:

// Now start up the pipeline, fire off the first step that will execute the first filter
 postal.publish({
  channel: "pipenfilter",
  topic: pipeline.filters[pipeline.index],
  data: pipeline
 });

//  A sample subscription.  This will be executed first since it is listed first in the pipeline.filters array
postal.subscribe({
  channel: "pipenfilter",
  topic: "edit.getViewOneData",
  callback: function (pipeline, env) {
    fetchViewOneData(pipeline);
}
 });

//  After filter performs, it needs to call the next filter in the chain<
var fetchViewOneData = function(pipeline){
  // ... perform activities with some data

  //  To forward to next step, increment the index
  pipeline.index++;

  //  Now forward to the next filter
  postal.publish({
    channel: "pipenfilter",
    topic: pipeline.filters[pipeline.index],
    data: pipeline
  });
};

For our particular scenario with the project UX above we will have to do a few simple things:

  1. The ViewModels “ProjectInfo”, “Budget” and “Team” will need subscriptions to a “pipenfilter” channel
  2. Each ViewModel will need a corresponding filter function, so the “Team” ViewModel will get a “fetchTeamFilter”
  3. Add an endpoint in our $(document).ready section to act as a controller for the pipeline process.  Since we don’t have a controller this logic can sit here.  Naturally for cleaner production code you’ll want to be a bit cleaner and place this in a controller object.

All of the addition changes are noted with a “New pipe and filter” comment, so you can search through the Javascript to see where the changes have been made.  Most of the updates are just a few lines, and the nice thing is that the code to forward to the next filter is the same each time.  Not much exciting stuff to be seen other than incrementing the index and passing the data to the next topic.

Where No One Has Gone Before (just sayin’)

Now that you can synchronize your data, assemble it back into a proper object, you can begin your next journey.  There’s many long term benefits here that you will begin to realize.  What if you need to add an additional ViewModel, or what if you need to change which ViewModel display the groupings of data?  Without our pipe and filter solution, you would need to change a lot of things.  With what we have described here, the most that have to do is to add an additional filter name to the pipeline list and make sure that your additional ViewModel functioned properly.  The other ViewModels are left untouched.

Another benefit is also be that you can control what functions are fired while allowing the ViewModels to do their thing.  What if you had a ViewModel that published changes to other ViewModels after it finished performing calculations.  By ordering the filters in the pipeline accordingly, you can allow the ViewModels to do that hard work and be assured that all aspects of your data is up to date.  Doing more with less code means you can test and maintain this code with confidence.

This is a continuation of the Javascript Primer series where we focus on topics essential for .Net or back-end developers or who are transitioning to Javascript and client side development.  Many of the topics covered in this series will be related to producing web apps running in the browser.  Some design patterns and practices are also application to NodeJS / server-side Javascript apps as well.

Keeping IT Together

One complaint that many have when embarking on applications of medium complexity is that organizing your code is difficult; indeed, even with frameworks you can still end up with giant god object that contains all data-bindings, algorithms, helpers methods.  If you take the attitude that Javascript is a second class, toy language where you simply store jQuery animation methods and AJAX calls, you quickly end up with files that can be 3-4000 lines long!  Maintaining that is beyond torture.

A common design pattern used to alleviate this issue is the Model View ViewModel (MVVM) pattern.  The goal of MVVM is to divide up the logic of your application into UX, business logic, and data-binding activities into discrete components.  KnockoutJS is a popular framework that supports this approach.  Addy Osmani’s “Design Patterns for Javascript” defines the primary components of MVVM, and it is great primer to read if you are unfamiliar with these terms.  For our purposes:

Model will contain the data elements, such as FirstName, LastName, etc.  Essentially this is a building block for your front-end that will “model” entities such as user, customer, project, etc.

ViewModel will contain the behavior for the UI.  This can be business logic, formatting of data, and bridge the gap between data coming from a server and what is useful for the user to interactive with.  The ViewModel will also use the Models.

View will be the html document.  This will also contain the directives for what to present to the user, and will interact with the ViewModel.

We Still Have A Problem

So even with the MVVM pattern at our disposal we still have to fight bloated ViewModels.  From here out we will use a sample project for our discussion where we want to maintain project information comprised of project name, project description, and project notes, team members and a simple timeline / calendar function.  Sounds straight forward enough.  On our UI we will have a tab for grouping of information, and as information changes on a tab, we want that information available in each subsequent tab.  Again, nothing too earth shattering here.  It should look something like this JSFiddle below.  (NOTE:  For the sake of this exercise all code has been placed in the Javascript section of our JSFiddle.  You should split your code into modules so they can easily managed).

Each tab on this screen has logical groupings of information.  For the sake of argument let’s say that we decide to implement this UI with just one ViewModel for all three tabs.  For databinding we will be using KnockoutJS,  Starting out we have just one ViewModel – ProjectViewModel – that handles all updates; more specifically, we can take advantage of the data-binding facilities here, so when a team member is added, we can detect this change and automatically execute and update to the calendar.

We have three tabs – “Project Info” is pretty simple, and it uses the computed observable teamSize to display the count of the project team. You note that any changes to “Project Name” will be reflected on the “Team” tab.   The tab Team allows you to add a team member.  To support updating the calendar with the team member start date we have create subscription to the projectTeam observable array.  The tab “Timeline” has a calendar that displays the team member start date, and this is updated each time we add a new team member.  Also, any changes to the number of team members will be reflected on the first tab, “Project Info”.

Indeed this works well, since we have the advantage of one ViewModel and can benefit from easily sharing one set of data between all three tabs, But we still have an issue should we start adding more functionality and increase the properties and functions in the ProjectViewModel.  Soon maintaining the ViewModel will become difficult, and testing discrete portions of functionality will rapidly become difficult.  We need to be able to isolate or reduce the impact that the alterations made to the ViewModels have on each other.  This means adhering to the principal of “loose coupling”.

So what does it mean for us to create a ViewModel per tab?  The biggest issue now will be communicating between each ViewModel, and as always, our mantra should be “Keep it simple.  Please.  Pretty please”. So how do we achieve communicating data updates without creating three ViewModels that are still highly dependent on the others?  Our solution is to implement a subscribe and publish message architecture.  While this sounds intimidating, this simply means we will create each ViewModel with a method – a “subscription” – that will wait for a message that has the data we need.  As a corollary, each ViewModel need a way to “publish” any changes to data that occurs.

Super Simple Message with PostalJS

PostalJS is a very simple messaging framework that facilitates sub / pub messaging in a unique way.  In particular, PostalJS allows us to set up a subscription that remains “ignorant” of how the message is generated.  There is no “pre-registration processes” to map out a route between ViewModels; on the contrary, your Javascript object simply declares that will update when a subscription arrives.  This also means that several ViewModels can subscribe to an event and the publisher doesn’t care.

Here is a brief sample taken from github:

var subscription = postal.subscribe({
        channel: "orders",
        topic: "item.add",
        callback: function(data, envelope) {
            // `data` is the data published by the publisher.
            // `envelope` is a wrapper around the data & contains
            // metadata about the message like the channel, topic,
            // timestamp and any other data which might have been
            // added by the sender.
        }
    });

postal.publish({
        channel: "orders",
        topic: "item.add",
        data: {
            sku: "AZDTF4346",
            qty: 21
        }
    });

Looks pretty cool.  And simple.  So for us, our ViewModels will remain ignorant of each other, and only are aware that someone, something has sent data to it.  Another nice by product is that should one of the ViewModels fail, the impact is greatly reduced.  So should the ViewModel with the calendar fail, the other tabs could still collect data and function.  Better yet, adding new features to a ViewModel is going to be easier to achieve since we can test in isolation, and if the new features do not interrupt our sub / pub messaging, we can feel confident that new changes can be rolled into production without bringing down the entire application. Let’s look at the results, then examine the changes to our app’s architecture.

A Brave New World

If you open the Javascript tab of the JSFiddle, you’ll see that we have broken up the ProjectViewModel into 3 ViewModels:  TeamViewModel, TimelineViewModel, and of course the remants of ProjectViewModel.  We’ll take the updates on at time, focusing on the feature in the UI and work back to the code.

Project Name As before, entering the project name in the “Project Info” tab updates the title in the “Team” tab.  Same as before, the update happens when the input box loses focus.  Now let’s focus on the code.  Open up the Javascript in the JSFiddle, and you’ll see some drastic changes.  In the ProjectViewModel we have created a Knockout trigger that publishes a message:

</pre>
<pre data-language="js">_projectName.subscribe(function(newValue){

        //    Now use PostalJS to push message out
        postal.publish({
            channel: "project",
            topic: "edit.projectNameChange",
            data: {
                projectName: _projectName()
            }
        });
    });

So what we are doing here telling Knockout “Anytime there is a change to the observable _projectName, fire off a function that publishes a piece of data with the new value of _projectName”.  That’s it.  ProjectViewModel has fulfilled its responsibility and told anyone who will listen that a change has taken place.  Now find the TeamViewModel object.  Here we have a subscription that waits for a message containing _projectName updates:

</pre>
<pre data-language="js">var projectNameChangeSubscription = postal.subscribe({
        channel: "project",
        topic: "edit.projectNameChange",
        callback: function(data, envelope) {
            _projectName(data.projectName);
        }
    });

We get the data, and merely update an observable on the TeamViewModel.  It should be apparent by now that neither ProjectViewModel nor TeamViewModel “know” about each.  ProjectViewModel says “_projectName update ready” and who can handle it, does so. TeamViewModel just grabs the update, uses it.  That’s it.  The messaging works by creating a “channel” – in this case “project” – further defines the message with a topic – “edit.projectNameChange”.  This means you can a single channel that support multiple topics and narrowly define your event messages.

ProjectTeam Update Play around with the “Team” tab and add some team members with different dates.  The operates just as before.  Under the hood, the code in TeamViewModel that handles the publishing is:

</pre>
<pre data-language="js">_projectTeam.subscribe(function(){
        //    this is how postaljs publishes events
        postal.publish({
            channel: "project",
            topic: "edit.teamUpdate",
            data: {
                projectTeam: ko.toJS(_projectTeam)
            }
        });

    });

As before, we are asking KnockoutJS to watch for changes to the _projectTeam array, and when a change occurs, we publish a different message.  The portion of code – topic: “edit.teamUpdate” distinguishes this event from the previous on.  We simply take our observableArray and convert it a plain Javascript script array and throw out to whoever wants it.  If you look in both ProjectViewModel and TimelineViewModel you’ll see subscriptions that handle this topic.  This brings us to another important aspect of messaging and postaljs’ strength:  should we want to add more ViewModels that need _projectTeam updates, we only need to subscribe to the topic “edit.teamUpdate”.  We could subscribe 50 more times if we wanted.  The publisher / source ViewModel does not need any alteration, and this achieves our goal of loose coupling.

What Are The Other Benefits?


Looking at our ViewModels, you may have noticed that we are exposing less in the return statements, and with each ViewModel handling less functionality they become simpler to read and maintain.  Imagine that the needed to support 20 different data entry input boxes – one monster ViewModel would have a huge return statement.  It makes it hard to keep things straight in one “god” class.

With PostalJS and event messaging we no longer need to expose an entry point in our objects for data to be passed.  The subscriptions allow the ViewModel to handle receiving update messages internally.  A quite common scenario would be responding to an AJAX update.  With PostalJS you can avoid calls to each ViewModel when new data arrives from the server.  Simply publish to the appropriate topic and let the ViewModels respond as they need.

Some gifts just keep on giving, and many times things can just take on a momentum that grow beyond your expectation. Bob Sherwood wrote to Sensei and pointed out that DataTables.net supports multiple column sorting. All you do is hold down the shift key and click on any second or third column and DataTables will add that column to sort criteria. “Well, how come it doesn’t work with the server side solution?” Talk about the sound of one hand clapping. How about that for a flub! Sensei didn’t think of that! Then panic set in – would this introduce new complexity to the DataTablePager solution, making it too difficult to maintain a clean implementation? After some long thought it seemed that a solution could be neatly added. Before reading, you should download the latest code to follow along.

How DataTables.Net Communicates Which Columns Are Involved in a Sort

If you recall, DataTables.Net uses a structure called aoData to communicate to the server what columns are needed, the page size, and whether a column is a data element or a client side custom column. We covered that in the last DataTablePager post. aoData also has a convention for sorting:

bSortColumn_X=ColumnPosition

In our example we are working with the following columns:

,Name,Agent,Center,,CenterId,DealAmount

where column 0 is a custom client side column, column 1 is Name (a mere data column), column 2 is Center (another data column), column 3 is a custom client side column, and the remaining columns are just data columns.

If we are sorting just by Name, then aoData will contain the following:

bSortColumn_0=1

When we wish to sort by Center, then by Name we get the following in aoData”

bSortColumn_0=2

bSortColumn_1=1

In other words, the first column we want to sort by is in position 2 (Center) and the second column(Name) is in position 1. We’ll want to record this some where so that we can pass this to our order routine. aoData passes all column information to us on the server, but we’ll have to parse through the columns and check to see if one or many of the columns is actually involved in a sort request and as we do we’ll need to preserve the order of that column of data in the sort.

SearchAndSortable Class to the Rescue

You’ll recall that we have a class called SearchAndSortable that defines how the column is used by the client. Since we iterate over all the columns in aoData it makes sense that we should take this opportunity to see if any column is involved in a sort and store that information in SearchAndSortable as well. The new code for the class looks like this:


public class SearchAndSortable
{
	public string Name { get; set; }
	public int ColumnIndex { get; set; }
	public bool IsSearchable { get; set; }
	public bool IsSortable { get; set; }
	public PropertyInfo Property{ get; set; }
	public int SortOrder { get; set; }
	public bool IsCurrentlySorted { get; set; }
	public string SortDirection { get; set; }

	public SearchAndSortable(string name, int columnIndex, bool isSearchable,
								bool isSortable)
	{
		this.Name = name;
		this.ColumnIndex = columnIndex;
		this.IsSearchable = isSearchable;
		this.IsSortable = IsSortable;
	}

	public SearchAndSortable() : this(string.Empty, 0, true, true) { }
}


There are 3 new additions:

IsCurrentlySorted - is this column included in the sort request.

SortDirection - “asc” or “desc” for ascending and descending.

SortOrder - the order of the column in the sort request. Is it the first or second column in a multicolumn sort.

As we walk through the column definitions, we’ll look to see if each column is involved in a sort and record what direction – ascending or descending – is required. From our previous post you’ll remember that the method PrepAOData is where we parse our column definitions. Here is the new code:


// Sort columns
this.sortKeyPrefix = aoDataList.Where(x => x.Name.StartsWith(INDIVIDUAL_SORT_KEY_PREFIX))
								.Select(x => x.Value)
								.ToList();

// Column list
var cols = aoDataList.Where(x => x.Name == "sColumns"
							& string.IsNullOrEmpty(x.Value) == false)
						.SingleOrDefault();

if(cols == null)
{
	this.columns = new List();
}
else
{
	this.columns = cols.Value
	.Split(',')
	.ToList();
}

// What column is searchable and / or sortable
// What properties from T is identified by the columns
var properties = typeof(T).GetProperties();
int i = 0;

// Search and store all properties from T
this.columns.ForEach(col =>
{
	if (string.IsNullOrEmpty(col) == false)
	{
		var searchable = new SearchAndSortable(col, i, false, false);
		var searchItem = aoDataList.Where(x => x.Name == BSEARCHABLE + i.ToString())
									.ToList();
		searchable.IsSearchable = (searchItem[0].Value == "False") ? false : true;
		searchable.Property = properties.Where(x => x.Name == col)
										.SingleOrDefault();

		searchAndSortables.Add(searchable);
	}

	i++;
});

// Sort
searchAndSortables.ForEach(sortable => {
							var sort = aoDataList.Where(x => x.Name == BSORTABLE + sortable.ColumnIndex.ToString())
				.ToList();
sortable.IsSortable = (sort[0].Value == "False") ? false : true;
sortable.SortOrder = -1;

// Is this item amongst currently sorted columns?
int order = 0;
this.sortKeyPrefix.ForEach(keyPrefix => {
	if (sortable.ColumnIndex == Convert.ToInt32(keyPrefix))
	{
		sortable.IsCurrentlySorted = true;

		// Is this the primary sort column or secondary?
		sortable.SortOrder = order;

		// Ascending or Descending?
		var ascDesc = aoDataList.Where(x => x.Name == "sSortDir_" + order)
								.SingleOrDefault();
		if(ascDesc != null)
		{
			sortable.SortDirection = ascDesc.Value;
		}
	}

	order++;
	});
});

To sum up, we’ll traverse all of the columns listed in sColumns. For each column we’ll grab the PorpertyInfo from our underlying object of type T. This gives only those properties that will be displayed in the grid on the client. If the column is marked as searchable, we indicate that by setting the IsSearchable property on the SearchAndSortable class. This happens starting at line 28 through 43.

Next we need to determine what we can sort, and will traverse the new list of SearchAndSortables we created. DataTables will tell us what if the column can be sorted by with following convention:

bSortable_ColNumber = True

So if the column Center were to be “sortable” aoData would contain:

bSortable_1 = True

We record the sortable state as shown on line 49 in the code listing.

Now that we know whether we can sort on this column, we have to look through the sort request and see if the column is actually involved in a sort. We do that by looking at what DataTables.Net sent to us from the client. Again the convention is to send bSortColumn_0=1 to indicate that the first column for the sort in the second item listed in sColumns property. aoData will contain many bSortColum’s so we’ll walk through each one and record the order that column should take in the sort. That occurs at line 55 where we match the column index with the bSortColumn_x value.

We’ll also determine what the sort direction – ascending or descending – should be. At line 63 we get the direction of the sort and record this value in the SearchAndSortable.

When the method PrepAOData is completed, we have a complete map of all columns and what columns are being sorted, as well as their respective sort direction. All of this was sent to us from the client and we are storing this configuration for later use.

Performing the Sort

[gigya src="http://listen.grooveshark.com/songWidget.swf" width="204" height="40" flashvars="hostname=cowbell.grooveshark.com&widgetID=23379337&style=water&p=0" allowScriptAccess="always" wmode="window" ](Home stretch so play the song!!)

If you can picture what we have so far we just basically created a collection of column names, their respective PropertyInfo’s and have recorded which of these properties are involved in a sort. At this stage we should be able to query this collection and get back those properties and the order that the sort applies.

You may already be aware that you can have a compound sort statement in LINQ with the following statement:


var sortedCustomers = customer.OrderBy(x => x.LastName)
                              .ThenBy(x => x.FirstName);

The trick is to run through all the properties and create that compound statement. Remember when we recorded the position of the sort as an integer? This makes it easy for us to sort out the messy scenarios where the second column is the first column of a sort. SearchAndSortable.SortOrder takes care of this for us. Just get the data order by SortOrder in descending order and you’re good to go. So that code would look like the following:


var sorted = this.searchAndSortables.Where(x => x.IsCurrentlySorted == true)
.OrderBy(x => x.SortOrder)
.ToList();

sorted.ForEach(sort => {
    records = records.OrderBy(sort.Name, sort.SortDirection,
                         (sort.SortOrder == 0) ? true : false);
});

On line 6 in the code above we are calling our extension method OrderBy in Extensions.cs. We pass the property name, the sort direction, and whether this is the first column of the sort. This last piece is important as it will create either “OrderBy” or the “ThenBy” for us. When it’s the first column, you guessed it we get “OrderBy”. Sensei found this magic on a StackOverflow post by Marc Gravell and others.

Here is the entire method ApplySort from DataTablePager.cs, and note how we still check for the initial display of the data grid and default to the first column that is sortable.


private IQueryable ApplySort(IQueryable records)
{
	var sorted = this.searchAndSortables.Where(x => x.IsCurrentlySorted == true)
										.OrderBy(x => x.SortOrder)
										.ToList();

	// Are we at initialization of grid with no column selected?
	if (sorted.Count == 0)
	{
		string firstSortColumn = this.sortKeyPrefix.First();
		int firstColumn = int.Parse(firstSortColumn);

		string sortDirection = "asc";
		sortDirection = this.aoDataList.Where(x => x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX + "0")
										.Single()
										.Value
										.ToLower();

		if (string.IsNullOrEmpty(sortDirection))
		{
		sortDirection = "asc";
		}

		// Initial display will set order to first column - column 0
		// When column 0 is not sortable, find first column that is
		var sortable = this.searchAndSortables.Where(x => x.ColumnIndex == firstColumn)
											.SingleOrDefault();
		if (sortable == null)
		{
			sortable = this.searchAndSortables.First(x => x.IsSortable);
		}

		return records.OrderBy(sortable.Name, sortDirection, true);
	}
	else
	{
		// Traverse all columns selected for sort
		sorted.ForEach(sort => {
		records = records.OrderBy(sort.Name, sort.SortDirection,
		(sort.SortOrder == 0) ? true : false);
		});

		return records;
	}
}

It’s All in the Setup

Test it out. Hold down the shift key and select a second column and WHAMO – multiple column sorts! Hold down the shift key and click the same column twice and KAH-BLAMO multiple column sort with descending order on the second column!!!

The really cool thing is that our process on the server is being directed by DataTables.net on the client. And even awseomer is that you have zero configuration on the server. Most awesome-est is that this will work with all of your domain objects, because we have used generics we can apply this to any class in our domain. So what are you doing to do with all that time you just got back?

ImageOn the quest to provide a rich user interface experience on his current project, Sensei has been experimenting with KnockoutJS by Steve Sanderson.  If you haven’t reviewed it’s capabilities yet  it would be well worth your while.  Not only has Steve put together a great series of tutorials, but he has been dog fooding it with Knockout.  The entire documentation and tutorial set is completed used Knockout.  Another fine source is Knockmeout.net by Ryan Niemeyer.  Ryan is extremely active on StackOverflow answering questions regarding Knockout, and also has a fine blog that offers very important insight on developing with this framework.

KnockoutJS is a great way to re-organize your client side code.  The goal of  this post is not to teach you KnocoutJS; rather, Sensei wants to point out other benefits – and a few pitfalls – to adopting its use.  In years past, it’s been difficult to avoid writing spaghetti code in Javascript.  Knockout forces you to adopt a new pattern of thought for organizing your UI implementation.  The result is a more maintainable code base.  In the past you may have written code similar to what Sensei use to write.  Take for example assigning a click event to a button or href in order to remove a record from a table:

&lt;table&gt;
  &lt;thead&gt;&lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;a onclick=&quot;deleteRecord(1); return false;&quot; href=&quot;#&quot;&gt;Customer One&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;1313 Galaxy Way&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;a onclick=&quot;deleteRecord(2); return false;&quot; href=&quot;#&quot;&gt;Customer Two&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;27 Mockingbird Lane&lt;/td&gt;
    &lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;

&lt;script type=&quot;text/javascript&quot;&gt;
function deleteRecord(id){
  //  Do some delete activities ...
}
&lt;/script&gt;

You might even went as far as to assign the onclick event like so:

$(document).ready(function(){
  $(&quot;tr a&quot;).on('click', function(){
    //  find the customer id and call the delete record
  });
});

The proposition offered by Knockout is much different.  Many others much more conversant in design patterns and development than Sensei can offer better technical reasons why you sound use Knockout.  Sensei likes the fact that it makes thinking about your code much simpler.  As in:

 
<td><a data-bind="click: deleteRecord($data)" href="#">Customer One</a></td>

Yep, you have code mixed in with your mark up, but so what.  You can hunt down what’s going on, switch to your external js file to review what deleteRecord is supposed to do.  It’s as simple as that.  Speaking of js files, Knockout forces you to have a more disciplined approach organizing your javascript.  Here is what the supporting javascript could look like:

var CustomerRecord = function(id, name){
  //  The items you want to appear in UI are wrapped with ko.observable
  this.id = ko.observable(id);
  this.name = ko.observable(name);
}

var ViewModel = function(){
var self = this;
  //  For our demo let's create two customer records.  Normally you'll get Json from the server
  self.customers = ko.observableArray([
    new CustomerRecord(1, &quot;Vandelay Industries&quot;),
    new CustomerRecord(2, &quot;Wiley Acme Associates&quot;)
  ]);

  self.deleteRecord = function(data){
    //  Simply remove the item that matches data from the array
    self.customers.remove(data);
  }
}

var vm = new ViewModel();
ko.applyBindings(vm);

That’s it.  Include this file with your markup and that’s all you have to do.  The html will change too.   Knockout will allow you to produce our table by employing the following syntax:

<tbody data-bind=”foreach: customers”>
<tr>
<td><a href=”#” data-bind=”click:  deleteRecord($data)”><span data-bind=”text: id”></span></a></td>
<td><span data-bind=’text: name”></span></td>
<tr>
</tbody>

These Aren’t the Voids You’re Looking For

So we’re all touchy feely because we have organization to our Javascript and that’s a good thing.  Here’s some distressing news – while Knockout is a great framework, getting the hang of it can be really hard.  Part of the reason is Javascript itself.  Because it’s a scripting language, you end up with strange scenarios where you have a property that appear to have the same name but different values.  You see, one of the first rules of using Knockout is that observables ARE METHODS.  You have to access them with (), as in customer.name(), and not customer.name.  In other words, in order for you to assign values to an observable you must:


customer.name(&quot;Vandelay Industries&quot;);

//  Don't do this - you create another property!!

customer.name = &quot;Vandelay Industries&quot;;

What? Actually, as you probably have surmised, you get .name() and .name, and this causes great confusion when you are debugging your application in Firebug.  Imagine you can see that customer.name has a value when you hit a breakpoint, but its not what you’re looking for.  Sensei developed a tactic to help verify that he’s not insane, and it works simply.  When in doubt, go the console in Firebug and access your observable via the ViewModel; so in our case you could issue:

vm.customer.name();

When name() doesn’t match your expectation you’ve most likely added a property with a typo.  Check with

vm.customer.name;

It sounds silly, but you can easily spend a half hour insisting that you’re doing the right thing, but you really confusing a property with a method.  Furthermore, observable arrays can also be a source of frustration:

// This is not the length of the observable array. It will always be zero!!!
vm.customers.length == 0;

// You get the length with this syntax
vm.customers().length;

Knock ‘em inta tamarra, Rocky

Had Sensei known the two tips before starting he would have save a lot of time.  There are many others, and they are best described by Ryan Niemeyer in his post 10 things to know about Knockout from day one.  Read this post slowly.  It will save you a lot of headache.  You may familiar with jQuery and Javascript, but Knockout introduces subtle differences that will catch you off guard.  That’s not a bad thing, it’s just different than what you may be used to.  Ryan also makes great use of JS Fiddle and answers most of his StackOverflow questions by using examples.  Those examples are in many cases easier to learn from than the tutorial since the scope is narrower than the instruction that Steve Sanderson gives.  It really allows you play along as you learn.

ImageOn the quest to provide a rich user interface experience on his current project, Sensei has been experimenting with KnockoutJS by Steve Sanderson.  If you haven’t reviewed it’s capabilities yet  it would be well worth your while.  Not only has Steve put together a great series of tutorials, but he has been dog fooding it with Knockout.  The entire documentation and tutorial set is completed used Knockout.  Another fine source is Knockmeout.net by Ryan Niemeyer.  Ryan is extremely active on StackOverflow answering questions regarding Knockout, and also has a fine blog that offers very important insight on developing with this framework.

KnockoutJS is a great way to re-organize your client side code.  The goal of  this post is not to teach you KnocoutJS; rather, Sensei wants to point out other benefits – and a few pitfalls – to adopting its use.  In years past, it’s been difficult to avoid writing spaghetti code in Javascript.  Knockout forces you to adopt a new pattern of thought for organizing your UI implementation.  The result is a more maintainable code base.  In the past you may have written code similar to what Sensei use to write.  Take for example assigning a click event to a button or href in order to remove a record from a table:

&lt;table&gt;
  &lt;thead&gt;&lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;a onclick=&quot;deleteRecord(1); return false;&quot; href=&quot;#&quot;&gt;Customer One&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;1313 Galaxy Way&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;&lt;a onclick=&quot;deleteRecord(2); return false;&quot; href=&quot;#&quot;&gt;Customer Two&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;27 Mockingbird Lane&lt;/td&gt;
    &lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;

&lt;script type=&quot;text/javascript&quot;&gt;
function deleteRecord(id){
  //  Do some delete activities ...
}
&lt;/script&gt;

You might even went as far as to assign the onclick event like so:

$(document).ready(function(){
  $(&quot;tr a&quot;).on('click', function(){
    //  find the customer id and call the delete record
  });
});

The proposition offered by Knockout is much different.  Many others much more conversant in design patterns and development than Sensei can offer better technical reasons why you sound use Knockout.  Sensei likes the fact that it makes thinking about your code much simpler.  As in:

 
<td><a data-bind="click: deleteRecord($data)" href="#">Customer One</a></td>

Yep, you have code mixed in with your mark up, but so what.  You can hunt down what’s going on, switch to your external js file to review what deleteRecord is supposed to do.  It’s as simple as that.  Speaking of js files, Knockout forces you to have a more disciplined approach organizing your javascript.  Here is what the supporting javascript could look like:

var CustomerRecord = function(id, name){
  //  The items you want to appear in UI are wrapped with ko.observable
  this.id = ko.observable(id);
  this.name = ko.observable(name);
}

var ViewModel = function(){
var self = this;
  //  For our demo let's create two customer records.  Normally you'll get Json from the server
  self.customers = ko.observableArray([
    new CustomerRecord(1, &quot;Vandelay Industries&quot;),
    new CustomerRecord(2, &quot;Wiley Acme Associates&quot;)
  ]);

  self.deleteRecord = function(data){
    //  Simply remove the item that matches data from the array
    self.customers.remove(data);
  }
}

var vm = new ViewModel();
ko.applyBindings(vm);

That’s it.  Include this file with your markup and that’s all you have to do.  The html will change too.   Knockout will allow you to produce our table by employing the following syntax:

<tbody data-bind=”foreach: customers”>
<tr>
<td><a href=”#” data-bind=”click:  deleteRecord($data)”><span data-bind=”text: id”></span></a></td>
<td><span data-bind=’text: name”></span></td>
<tr>
</tbody>

These Aren’t the Voids You’re Looking For

So we’re all touchy feely because we have organization to our Javascript and that’s a good thing.  Here’s some distressing news – while Knockout is a great framework, getting the hang of it can be really hard.  Part of the reason is Javascript itself.  Because it’s a scripting language, you end up with strange scenarios where you have a property that appear to have the same name but different values.  You see, one of the first rules of using Knockout is that observables ARE METHODS.  You have to access them with (), as in customer.name(), and not customer.name.  In other words, in order for you to assign values to an observable you must:


customer.name(&quot;Vandelay Industries&quot;);

//  Don't do this - you create another property!!

customer.name = &quot;Vandelay Industries&quot;;

What? Actually, as you probably have surmised, you get .name() and .name, and this causes great confusion when you are debugging your application in Firebug.  Imagine you can see that customer.name has a value when you hit a breakpoint, but its not what you’re looking for.  Sensei developed a tactic to help verify that he’s not insane, and it works simply.  When in doubt, go the console in Firebug and access your observable via the ViewModel; so in our case you could issue:

vm.customer.name();

When name() doesn’t match your expectation you’ve most likely added a property with a typo.  Check with

vm.customer.name;

It sounds silly, but you can easily spend a half hour insisting that you’re doing the right thing, but you really confusing a property with a method.  Furthermore, observable arrays can also be a source of frustration:

// This is not the length of the observable array. It will always be zero!!!
vm.customers.length == 0;

// You get the length with this syntax
vm.customers().length;

Knock ‘em inta tamarra, Rocky

Had Sensei known the two tips before starting he would have save a lot of time.  There are many others, and they are best described by Ryan Niemeyer in his post 10 things to know about Knockout from day one.  Read this post slowly.  It will save you a lot of headache.  You may familiar with jQuery and Javascript, but Knockout introduces subtle differences that will catch you off guard.  That’s not a bad thing, it’s just different than what you may be used to.  Ryan also makes great use of JS Fiddle and answers most of his StackOverflow questions by using examples.  Those examples are in many cases easier to learn from than the tutorial since the scope is narrower than the instruction that Steve Sanderson gives.  It really allows you play along as you learn.

Some gifts just keep on giving, and many times things can just take on a momentum that grow beyond your expectation.  Bob Sherwood wrote to Sensei and pointed out that DataTables.net supports multiple column sorting.  All you do is hold down the shift key and click on any second or third column and DataTables will add that column to sort criteria.  ”Well, how come it doesn’t work with the server side solution?”  Talk about the sound of one hand clapping.  How about that for a flub!  Sensei didn’t think of that!  Then panic set in – would this introduce new complexity to the DataTablePager solution, making it too difficult to maintain a clean implementation?  After some long thought it seemed that a solution could be neatly added.  Before reading, you should download the latest code to follow along.

How DataTables.Net Communicates Which Columns Are Involved in a Sort

If you recall, DataTables.Net uses a structure called aoData to communicate to the server what columns are needed, the page size, and whether a column is a data element or a client side custom column.  We covered that in the last DataTablePager post.  aoData also has a convention for sorting:

bSortColumn_X=ColumnPosition

In our example we are working with the following columns:

,Name,Agent,Center,,CenterId,DealAmount

where column 0 is a custom client side column, column 1 is Name (a mere data column), column 2 is Center (another data column), column 3 is a custom client side column, and the remaining columns are just data columns.

If we are sorting just by Name, then aoData will contain the following:

bSortColumn_0=1

When we wish to sort by Center, then by Name we get the following in aoData”

bSortColumn_0=2

bSortColumn_1=1

In other words, the first column we want to sort by is in position 2 (Center) and the second column(Name) is in position 1.  We’ll want to record this some where so that we can pass this to our order routine.  aoData passes all column information to us on the server, but we’ll have to parse through the columns and check to see if one or many of the columns is actually involved in a sort request and as we do we’ll need to preserve the order of that column of data in the sort.

SearchAndSortable Class to the Rescue

You’ll recall that we have a class called SearchAndSortable that defines how the column is used by the client.  Since we iterate over all the columns in aoData it makes sense that we should take this opportunity to see if any column is involved in a sort and store that information in SearchAndSortable as well.  The new code for the class looks like this:

public class SearchAndSortable
    {
        public string Name { get; set; }
        public int ColumnIndex { get; set; }
        public bool IsSearchable { get; set; }
        public bool IsSortable { get; set; }
        public PropertyInfo Property{ get; set; }
        public int SortOrder { get; set; }
        public bool IsCurrentlySorted { get; set; }
        public string SortDirection { get; set; }

        public SearchAndSortable(string name, int columnIndex, bool isSearchable,
                                bool isSortable)
        {
            this.Name = name;
            this.ColumnIndex = columnIndex;
            this.IsSearchable = isSearchable;
            this.IsSortable = IsSortable;
        }

        public SearchAndSortable() : this(string.Empty, 0, true, true) { }
    }

There are 3 new additions:

IsCurrentlySorted - is this column included in the sort request.

SortDirection - “asc” or “desc” for ascending and descending.

SortOrder - the order of the column in the sort request.  Is it the first or second column in a multicolumn sort.

As we walk through the column definitions, we’ll look to see if each column is involved in a sort and record what direction – ascending or descending – is required. From our previous post you’ll remember that the method PrepAOData is where we parse our column definitions. Here is the new code:

//  Sort columns
this.sortKeyPrefix = aoDataList.Where(x =&gt; x.Name.StartsWith(INDIVIDUAL_SORT_KEY_PREFIX))
                                            .Select(x =&gt; x.Value)
                                            .ToList();

//  Column list
var cols = aoDataList.Where(x =&gt; x.Name == &quot;sColumns&quot;
                                            &amp; string.IsNullOrEmpty(x.Value) == false)
                                     .SingleOrDefault();

if(cols == null)
{
  this.columns = new List();
}
else
{
  this.columns = cols.Value
                       .Split(',')
                       .ToList();
}

//  What column is searchable and / or sortable
//  What properties from T is identified by the columns
var properties = typeof(T).GetProperties();
int i = 0;

//  Search and store all properties from T
this.columns.ForEach(col =&gt;
{
  if (string.IsNullOrEmpty(col) == false)
  {
    var searchable = new SearchAndSortable(col, i, false, false);
    var searchItem = aoDataList.Where(x =&gt; x.Name == BSEARCHABLE + i.ToString())
                                     .ToList();
    searchable.IsSearchable = (searchItem[0].Value == &quot;False&quot;) ? false : true;
    searchable.Property = properties.Where(x =&gt; x.Name == col)
                                                    .SingleOrDefault();

    searchAndSortables.Add(searchable);
  }

  i++;
});

//  Sort
searchAndSortables.ForEach(sortable =&gt; {
  var sort = aoDataList.Where(x =&gt; x.Name == BSORTABLE + sortable.ColumnIndex.ToString())
                                            .ToList();
  sortable.IsSortable = (sort[0].Value == &quot;False&quot;) ? false : true;
                sortable.SortOrder = -1;

  //  Is this item amongst currently sorted columns?
  int order = 0;
  this.sortKeyPrefix.ForEach(keyPrefix =&gt; {
    if (sortable.ColumnIndex == Convert.ToInt32(keyPrefix))
    {
      sortable.IsCurrentlySorted = true;

      //  Is this the primary sort column or secondary?
      sortable.SortOrder = order;

     //  Ascending or Descending?
     var ascDesc = aoDataList.Where(x =&gt; x.Name == &quot;sSortDir_&quot; + order)
                                                    .SingleOrDefault();
     if(ascDesc != null)
     {
       sortable.SortDirection = ascDesc.Value;
     }
   }

   order++;
 });
});

To sum up, we’ll traverse all of the columns listed in sColumns. For each column we’ll grab the PorpertyInfo from our underlying object of type T. This gives only those properties that will be displayed in the grid on the client. If the column is marked as searchable, we indicate that by setting the IsSearchable property on the SearchAndSortable class.  This happens starting at line 28 through 43.

Next we need to determine what we can sort, and will traverse the new list of SearchAndSortables we created. DataTables will tell us what if the column can be sorted by with following convention:

bSortable_ColNumber = True

So if the column Center were to be “sortable” aoData would contain:

bSortable_1 = True

We record the sortable state as shown on line 49 in the code listing.

Now that we know whether we can sort on this column, we have to look through the sort request and see if the column is actually involved in a sort.  We do that by looking at what DataTables.Net sent to us from the client.  Again the convention is to send bSortColumn_0=1 to indicate that the first column for the sort in the second item listed in sColumns property.  aoData will contain many bSortColum’s so we’ll walk through each one and record the order that column should take in the sort.  That occurs at line 55 where we match the column index with the bSortColumn_x value.

We’ll also determine what the sort direction – ascending or descending – should be.  At line 63 we get the direction of the sort and record this value in the SearchAndSortable.

When the method PrepAOData is completed, we have a complete map of all columns and what columns are being sorted, as well as their respective sort direction.  All of this was sent to us from the client and we are storing this configuration for later use.

Performing the Sort

[gigya src="http://listen.grooveshark.com/songWidget.swf" width="204" height="40" flashvars="hostname=cowbell.grooveshark.com&widgetID=23379337&style=water&p=0" allowScriptAccess="always" wmode="window" ](Home stretch so play the song!!)

If you can picture what we have so far we just basically created a collection of column names, their respective PropertyInfo’s and have recorded which of these properties are involved in a sort.  At this stage we should be able to query this collection and get back those properties and the order that the sort applies.

You may already be aware that you can have a compound sort statement in LINQ with the following statement:

var sortedCustomers = customer.OrderBy(x =&gt; x.LastName)
                                           .ThenBy(x =&gt; x.FirstName);

The trick is to run through all the properties and create that compound statement. Remember when we recorded the position of the sort as an integer? This makes it easy for us to sort out the messy scenarios where the second column is the first column of a sort. SearchAndSortable.SortOrder takes care of this for us. Just get the data order by SortOrder in descending order and you’re good to go. So that code would look like the following:

var sorted = this.searchAndSortables.Where(x =&gt; x.IsCurrentlySorted == true)
                                     .OrderBy(x =&gt; x.SortOrder)
                                     .ToList();

sorted.ForEach(sort =&gt; {
             records = records.OrderBy(sort.Name, sort.SortDirection,
             (sort.SortOrder == 0) ? true : false);
});

On line 6 in the code above we are calling our extension method OrderBy in Extensions.cs. We pass the property name, the sort direction, and whether this is the first column of the sort. This last piece is important as it will create either “OrderBy” or the “ThenBy” for us. When it’s the first column, you guessed it we get “OrderBy”. Sensei found this magic on a StackOverflow post by Marc Gravell and others.

Here is the entire method ApplySort from DataTablePager.cs, and note how we still check for the initial display of the data grid and default to the first column that is sortable.

private IQueryable ApplySort(IQueryable records)
{
  var sorted = this.searchAndSortables.Where(x =&gt; x.IsCurrentlySorted == true)
                                                .OrderBy(x =&gt; x.SortOrder)
                                                .ToList();

  //  Are we at initialization of grid with no column selected?
  if (sorted.Count == 0)
  {
    string firstSortColumn = this.sortKeyPrefix.First();
    int firstColumn = int.Parse(firstSortColumn);

    string sortDirection = &quot;asc&quot;;
    sortDirection = this.aoDataList.Where(x =&gt; x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX +                                                                    &quot;0&quot;)
                                                    .Single()
                                                    .Value
                                                    .ToLower();

    if (string.IsNullOrEmpty(sortDirection))
    {
      sortDirection = &quot;asc&quot;;
    }

    //  Initial display will set order to first column - column 0
    //  When column 0 is not sortable, find first column that is
    var sortable = this.searchAndSortables.Where(x =&gt; x.ColumnIndex == firstColumn)
                                                        .SingleOrDefault();
    if (sortable == null)
    {
      sortable = this.searchAndSortables.First(x =&gt; x.IsSortable);
    }

    return records.OrderBy(sortable.Name, sortDirection, true);
  }
  else
  {
      //  Traverse all columns selected for sort
      sorted.ForEach(sort =&gt; {
                             records = records.OrderBy(sort.Name, sort.SortDirection,
                            (sort.SortOrder == 0) ? true : false);
      });

    return records;
  }
}

It’s All in the Setup

Test it out. Hold down the shift key and select a second column and WHAMO – multiple column sorts! Hold down the shift key and click the same column twice and KAH-BLAMO multiple column sort with descending order on the second column!!!

The really cool thing is that our process on the server is being directed by DataTables.net on the client.  And even awseomer is that you have zero configuration on the server.  Most awesome-est is that this will work with all of your domain objects, because we have used generics we can apply this to any class in our domain.  So what are you doing to do with all that time you just got back?

Source code has been yet again updated!! Read about the changes in DataTablePager Now Has Multi-Column Sort Capability For DataTables.Net If you are new to DataTables.Net and Sensei’s paging solution and want to detailed study of how it works, work through this post first, then get the latest edition.  Note, code links in this post are to the first version.

The last episode of server-side paging with DataTablerPager for DataTables.Net we reviewed the basics of a server-side solution that paged records and returned results in the multiples as specified by DataTables.Net.  You will want to have read that post before preceding here.  The older version of the source is included in that post as well as this will help get you acclimated.  The following capabilities were reviewed:

  • The solution used generics and could work with any collection of IQueryable.  In short any of your classes from you domain solution  could be used.
  • Filtering capability across all properties was provided.  This included partial word matching, regardless of case.
  • Ordering of result set was in response to the column clicked on the client’s DataTables grid.

DataTablePager Enhancements

This past month Sensei has added new capabilities to the DataTablePager class that makes it an even better fit for use with DataTables.Net.  The new features are:

  • Dynamically select the columns from the properties of your class based on the column definitions supplied by DataTables.Net!!!
  • Exclude columns from sort or search based on configuration by DataTables.Net
  • Mix columns from your class properties with client-side only column definitions; e.g. create a column with <a href>’s that do not interfere with filtering, sorting, or other processing.

Before we jump into the nitty-gritty details let’s review how DataTables.Net allows you to control a column’s interaction with a data grid.  Grab the new source code to best follow along.

DataTables.Net Column Definition

You would think that there would be quite a few steps to keep your server-side data paging solution in concert with a client side implementation, and that would mean customization for each page.   DataTables.Net provides you with fine control over what your columns will do once displayed in a data grid.  Great, but does that mean a lot of configuration on the server side of the equation?  As well soon see, no, it doesn’t.  What is done on the client for configuration will be that you need to do.

The structure aoColumnDefs is the convention we use for column configuration.  From the DataTables.Net site:

aoColumnDefs: This array allows you to target a specific column, multiple columns, or all columns, using the aTargets property of each object in the array (please note that aoColumnDefs was introduced in DataTables 1.7). This allows great flexibility when creating tables, as the aoColumnDefs arrays can be of any length, targeting the columns you specifically want. The aTargets property is an array to target one of many columns and each element in it can be:

  • a string – class name will be matched on the TH for the column
  • 0 or a positive integer – column index counting from the left
  • a negative integer – column index counting from the right
  • the string “_all” – all columns (i.e. assign a default)

So in order for you to include columns in a sort you configure in this manner:

/* Using aoColumnDefs */
$(document).ready(function() {
	$('#example').dataTable( {
		&quot;aoColumnDefs&quot;: [
			{ &quot;bSortable&quot;: false, &quot;aTargets&quot;: [ 0 ] }
		] } );
} );

} );

In other words we are defining that the first column – column 0 – will not be included in the sorting operations.  When you review the columns options you’ll see you have options for applying css classes to multiple columns, can include a column in filtering, can supply custom rendering of a column, and much more.

In the example that we’ll use for the rest of the post we are going to provide the following capability for a data grid:

  1. The first column – column 0 – will be an action column with a hyperlink, and we will want to exclude it form sort and filtering functions.
  2. Only display a subset of the properties from a class.  Each of these columns should be sortable and filterable.
  3. Maintain the ability to chunk the result set in the multiples specified by DataTables.Net; that is, multiples of 10, 50, and 100.

Here is the configuration from the aspx page SpecifyColumns.aspx:

&quot;aoColumnDefs&quot; : [
   {&quot;fnRender&quot; : function(oObj){
      return &quot;&lt;a href=&quot;&amp;quot;center.aspx?centerid=&amp;quot;&quot;&gt;Edit&lt;/a&gt;&quot;;
   },
     &quot;bSortable&quot; : false,
     &quot;aTargets&quot; : [0]},
   {&quot;sName&quot; : &quot;Name&quot;,
     &quot;bSearchable&quot; : true,
     &quot;aTargets&quot;: [1]},
   {&quot;sName&quot; : &quot;Agent&quot;,
    &quot;bSearchable&quot; : true,
    &quot;bSortable&quot; : true,
    &quot;aTargets&quot; : [2]
   },
   {&quot;sName&quot; : &quot;Center&quot;, &quot;aTargets&quot;: [3]},
   {&quot;fnRender&quot; : function(oObj){
            return &quot;2nd Action List&quot;;
         },
     &quot;bSortable&quot; : false,
     &quot;aTargets&quot; : [4]},
   {&quot;sName&quot; : &quot;CenterId&quot;, &quot;bVisible&quot; : false, &quot;aTargets&quot; : [5]},
   {&quot;sName&quot; : &quot;DealAmount&quot;, &quot;aTargets&quot; : [6]}
]
  1. Column 0 is our custom column – do not sort or search on this content.  Look at oObj.aData[4] – this is a column that we’ll return but not display.  It’s referred so by the position in the data array that DataTables.Net expects back from the server.
  2. Columns 1 – 3 are data and can be sorted.  Note the use of “sName”.  This will be included in a named column list that corresponds to the source property from our class.  This will be very important later on for us, as it allows us to query our data and return it in any order to DataTables.Net.  DataTables will figure out what to do with it before it renders.
  3. Threw in another custom column.  Again, no sort or search, but we’ll see how this affects the server side implementation later on.  Hint – there’s no sName used here.
  4. Another data column.

To recap, we want to be able to define what data we need to display and how we want to interact with that data by only instructing DataTables.Net what to do.  We’re going to be lazy, and not do anything else – the class DataTablePager will respond to the instructions that DataTables.Net supplies, and that’s it.  We’ll review how to do this next.  Sensei thinks you’ll really dig it.

DataTablePager Class Handles your Client Side Requests

If you recall, DataTables.Net communicates to the server via the structure aoData.  Here is the summary of the parameters.  One additional parameter that we’ll need to parse is the sColumns parameter, and it will contain the names and order of the columns that DataTables.Net is rendering.  For our example, we’ll get the following list of columns if we were to debug on the server:

,Name,Agent,Center,,CenterId,DealAmount

These are all the columns we named with sName, plus a place holder for those custom columns that not found in our class.  This has several implications.  For one, it will mean that we will no longer be able to simply use reflection to get at our properties, filter them and send them back down to the client.  The client is now expecting an array where each row will have 7 things, 5 of which are named and two place holders for items that the client wants to reserve for itself.  Hence the convention of passing an empty item in the delimited string as shown above.

It will also mean that we’ll have to separate the columns that we can filter or sort.  Again this is the reason for leaving the custom column names blank.  In other words, we’ll have to keep track of the items that we can search and sort.  We’ll do this with a class called SearchAndSortable:

public class SearchAndSortable
    {
        public string Name { get; set; }
        public int ColumnIndex { get; set; }
        public bool IsSearchable { get; set; }
        public bool IsSortable { get; set; }
        public PropertyInfo Property{ get; set; }

        public SearchAndSortable(string name, int columnIndex, bool isSearchable, bool isSortable)
        {
            this.Name = name;
            this.ColumnIndex = columnIndex;
            this.IsSearchable = isSearchable;
            this.IsSortable = IsSortable;
        }

        public SearchAndSortable() : this(string.Empty, 0, true, true) { }
    }

This will summarize what we’re doing with our properties.   The property ColumnIndex will record the position in sColumn where our column occurs.  Since we’ll need access to the actual properties themselves we’ll store these in the SearchAndSortable as well so that we can reduce the number of calls that use reflection. DataTablePager uses a List of SortAndSearchables to track what’s going on.  We fill this list in the method PrepAOData()

//  What column is searchable and / or sortable
            //  What properties from T is identified by the columns
            var properties = typeof(T).GetProperties();
            int i = 0;

            //  Search and store all properties from T
            this.columns.ForEach(col =&gt;
            {
                if (string.IsNullOrEmpty(col) == false)
                {
                    var searchable = new SearchAndSortable(col, i, false, false);
                    var searchItem = aoDataList.Where(x =&gt; x.Name == BSEARCHABLE + i.ToString())
                                     .ToList();
                    searchable.IsSearchable = (searchItem[0].Value == &quot;False&quot;) ? false : true;
                    searchable.Property = properties.Where(x =&gt; x.Name == col)
                                                    .SingleOrDefault();

                    searchAndSortables.Add(searchable);
                }

                i++;
            });

            //  Sort
            searchAndSortables.ForEach(sortable =&gt; {
                var sort = aoDataList.Where(x =&gt; x.Name == BSORTABLE + sortable.ColumnIndex.ToString())
                                            .ToList();
                sortable.IsSortable = (sort[0].Value == &quot;False&quot;) ? false : true;
            });

We’ll get the properties from our class. Next we’ll traverse the columns and match the property names with the names of the columns. When there is a match, we need to query aoData and get the column search and sort definitions based on the ordinal position of the column in the sColumns variable. DataTables.Net convention for communicating this is the form of:

bSortable_ + column index => “bSortable_1″ or “bSearchable_2″

We take care of that with this line of code:

var searchItem = aoDataList.Where(x =&gt; x.Name == BSEARCHABLE +
                                     i.ToString())
                                     .ToList();
searchable.IsSearchable = (searchItem[0].Value == &quot;False&quot;) ? false : true;

Now we go through the list of properties again but this time determine if we should sort any of the columns. That happens in the section //Sort. In the end we have a list of properties that corresponds with the columns DataTables.Net has requested, and we have defined if the property can be search (filtered) or sorted.

For filtering DataTablePager recall that we use the method GenericSearchFilter().  The only alteration here is that we only will add the properties to our query that are defined as searcable:

//  Create a list of searchable properties
            var filterProperties = this.searchAndSortables.Where(x =&gt;
                                        x.IsSearchable)
                                          .Select(x =&gt; x.Property)
                                          .ToList();

The rest of the method is unaltered from the prior version. Pretty cool!! Again, we’ll only get the properties that we declared as legal for filtering. We’ve also eliminated any chance of mixing a custom column in with our properties because we did not supply an sName in our configuration.

The method ApplySort() required one change. On the initial load of DataTable.Net, the client will pass up the request to sort on column 0 even though you may have excluded it. When that is the case, we’ll just look for the first column that is sortable and order by that column.

//  Initial display will set order to first column - column 0
//  When column 0 is not sortable, find first column that is
var sortable = this.searchAndSortables.Where(x =&gt; x.ColumnIndex ==
                                         firstColumn)
                              .SingleOrDefault();
if(sortable == null)
{
   sortable = this.searchAndSortables.First(x =&gt; x.IsSortable);
}

return records.OrderBy(sortable.Name, sortDirection, true);

After we have filtered and sorted the data set we can finally select the only those properties that we want to send to the client.  Recall that we have parsed a variable sColumns that tells what columns are expected.  We’ll pass these names onto extension method PropertiesToList().  This method will only serialize the property if the column is include, and since we have already paired down our data set as a result of our query and paging, there is very little performance impact.  Here is the new PropertiesToList method:

public static ListPropertiesToList(this T obj, List propertyNames)
{
   var propertyList = new List();
   var properties = typeof(T).GetProperties();
   var props = new List();

   //  Find all &quot;&quot; in propertyNames and insert empty value into list at
   //  corresponding position
   var blankIndexes = new List();
   int i = 0;

   //  Select and order filterProperties.  Record index position where there is
   //  no property
   propertyNames.ForEach(name =&gt;
   {
      var property = properties.Where(prop =&gt; prop.Name == name.Trim())
         .SingleOrDefault();

      if(property == null)
      {
         blankIndexes.Add(new NameValuePair(name, i));
      }
      else
      {
         props.Add(properties.Where(prop =&gt; prop.Name == name.Trim())
                                    .SingleOrDefault());
      }
      i++;
   });

   propertyList = props.Select(prop =&gt; (prop.GetValue(obj, new object[0]) ?? string.Empty).ToString())
                                        .ToList();

   //  Add &quot;&quot; to List as client expects blank value in array
   blankIndexes.ForEach(index =&gt;; {
      propertyList.Insert(index.Value, string.Empty);
   });

   return propertyList;
}

You might ask why not just pass in the list of SearchAndSortTable and avoid using reflection again. You could, but remember at this point we have reduced the number of items to the page size of 10, 50 or 100 rows, so your reflection calls will not have that great an impact. Also you should consider whether you want to simply have a function that will select only those properties that you need. Using SearchAndSortable would narrow the scope of utility, as you can use this method in other areas other than prepping data for DataTables.Net.

Now It’s Your Turn

That’s it.  Play with the page named SpecifyColumns.aspx.  You should be able to add and remove columns in the DataTable.Net configuration and they will just work.  This will mean, however, that you’ll have to always define your columns in your aspx page.  But since we worked really hard the first time around, DataTablePager will still be able to create paged data sets for any class in your domain.

Source code is here.  Enjoy.

Source code has been updated!! Read about the changes in Dynamically Select Columns with Server-Side Paging and Datatables.Net If you are new to DataTables.Net and Sensei’s paging solution and want to detailed study of how it works, work through this post first, then get the latest edition.  Note, code links in this post are to the first version.

A central theme for 2010 has been fluency, or the continual practice of certain methods to such a degree that your performance improves and you produce increasingly polished, effective solutions.  For software development this has meant tools to save time and increase quality.  It also means keeping an eye toward making the users of your solutions more efficient as well.  In the spirit of “fluent solutions”, Sensei will end the year with a post that examines how to create a data paging solution for the jQuery data grid plug-in DataTables.Net.

DataTables can turn a HTML table into a fully functional data grid like the one offered by Telerik.  This plug-in offers client side sorting, filtering/ search,  as well as support for server-side processing processing of data.  It is an extremely feature rich tool created by Allan Jardine, and is itself worthy of a series of posts.  For this post on data paging Sensei recommends that you read through these examples to get an idea of what the data paging service needs to achieve.

Let’s get started with the goals we need to achieve when providing server-side data paging support:

  • Send data to client in the multiples or “chunks” that the client requests, and respond when the size of the sets requested is changed by the user.
  • Re-order the data set if the user clicks on a column heading.  Honor the data set size when returning the data.
  • Filter across all columns of data based on user input.  Implement this as partial matches, and again, honor the data set size.

Remember this is about flexibility, so we have the additional goals of:

  • Create a solution that can be reused.
  • Provide a mechanism to accommodate any type of .Net class using generics.

Essentially we want to be able to write code like so:

var tenants = tenantRepository.GetAll();
var dataTablePager = new DataTablePager();
var returnDataSet = dataTablePager.Filter(requestParms, tenants);

Before we proceed, Sensei wants to acknowledge those really smart people whose ideas contributed to this solution:

Zack Owens – jQuery DataTables Plugin Meets C#

Jeff Morris - Using Datatables.net JQuery Plug-in with WCF Services

Dave Ward – ASMX ScriptService mistake – Invalid JSON primitive

You may want to download the source before reading the rest of this post.

Communicating with DataTables

DataTables uses the following parameters when processing server-side data:

Sent to the server:

Type Name Info
int iDisplayStart Display start point
int iDisplayLength Number of records to display
int iColumns Number of columns being displayed (useful for getting individual column search info)
string sSearch Global search field
boolean bEscapeRegex Global search is regex or not
boolean bSortable_(int) Indicator for if a column is flagged as sortable or not on the client-side
boolean bSearchable_(int) Indicator for if a column is flagged as searchable or not on the client-side
string sSearch_(int) Individual column filter
boolean bEscapeRegex_(int) Individual column filter is regex or not
int iSortingCols Number of columns to sort on
int iSortCol_(int) Column being sorted on (you will need to decode this number for your database)
string sSortDir_(int) Direction to be sorted – “desc” or “asc”. Note that the prefix for this variable is wrong in 1.5.x where iSortDir_(int) was used)
string sEcho Information for DataTables to use for rendering

Reply from the server

In reply to each request for information that DataTables makes to the server, it expects to get a well formed JSON object with the following parameters.

Type Name Info
int iTotalRecords Total records, before filtering (i.e. the total number of records in the database)
int iTotalDisplayRecords Total records, after filtering (i.e. the total number of records after filtering has been applied – not just the number of records being returned in this result set)
string sEcho An unaltered copy of sEcho sent from the client side. This parameter will change with each draw (it is basically a draw count) – so it is important that this is implemented. Note that it strongly recommended for security reasons that you ‘cast’ this parameter to an integer in order to prevent Cross Site Scripting (XSS) attacks.
string sColumns Optional – this is a string of column names, comma separated (used in combination with sName) which will allow DataTables to reorder data on the client-side if required for display
array array mixed aaData The data in a 2D array

The data sent back is in the following form depicted below. Note that aaData is merely an array of strings – there is no column information. This will present a challenge in that you will not be able to simply serialize a collection and pass back the results.

{
    &quot;sEcho&quot;: 3,
    &quot;iTotalRecords&quot;: 57,
    &quot;iTotalDisplayRecords&quot;: 57,
    &quot;aaData&quot;: [
        [
            &quot;Gecko&quot;,
            &quot;Firefox 1.0&quot;,
            &quot;Win 98+ / OSX.2+&quot;,
            &quot;1.7&quot;,
            &quot;A&quot;
        ],
        [
            &quot;Gecko&quot;,
            &quot;Firefox 1.5&quot;,
            &quot;Win 98+ / OSX.2+&quot;,
            &quot;1.8&quot;,
            &quot;A&quot;
        ],
        ...
    ]
}

As you may be aware, if you wish to use ASP.Net web services to serialize JSON you must POST to the service and instruct it to interpret your parameters as JSON. DataTables will POST variables as value pairs and this won’t work for us when POSTing to a web service. We’ll have to translate the variables to a usable format. Luckily DataTables allows us to intervene with the following code, where we create a JSON string by serializing a structure called aoData:

&quot;fnServerData&quot;: function ( sSource, aoData, fnCallback ) {

		        	var jsonAOData = JSON.stringify(aoData);

			        $.ajax( {
                                        contentType: &quot;application/json; charset=utf-8&quot;,
				        type: &quot;POST&quot;,
				        url: sSource,
				        data: &quot;{jsonAOData : '&quot; + jsonAOData + &quot;'}&quot;,
				        success: function(msg){
				            fnCallback(JSON.parse(msg.d));
				        },
				        error: function(XMLHttpRequest, textStatus, errorThrown) {
                            alert(XMLHttpRequest.status);
                            alert(XMLHttpRequest.responseText);

                        }
			        });

Our web service can now de-serialize aoData and parse the appropriate parameters. This gives us important items such as how many records to display, what columns to sort on, and what search terms should be applied in a filter.

DataTablePager Class

DataTablePager.cs is the work horse of our solution.  It will sort, filter and order our data, and as an extra, serialize the results in format required by aaData.  Here’s the constructor:

public DataTablePager(string jsonAOData, IQueryable queryable)
        {
            this.queryable = queryable;
            this.type = typeof(T);
            this.properties = this.type.GetProperties(BindingFlags.Public | BindingFlags.Instance);
            this.aoDataList = new List&gt;();
            this.sortKeyPrefix = new List();

            PrepAOData(jsonAOData);
        }

The parameter jsonAOData is the JSON string that contains the variables iDisplayStart, iDisplayLength, etc.  These will be parsed by the method PrepAOData.  The parameter queryable is the collection of records that will be filtered and parsed into JSON format required by DataTables.

The method Filter() coordinates all of the work.  It’s pretty simple what we want to do:  filter our data based on each column containing the search term, sort the result, then pull out the number of records we need to include in the page, and finally convert the collection into the format DataTables understands.

public FormattedList Filter()
        {
            var formattedList = new FormattedList();

            //  What are the columns in the data set
            formattedList.Import(this.properties.Select(x =&gt; x.Name + &quot;,&quot;)
                                                 .ToArray());

            //  Return same sEcho that was posted.  Prevents XSS attacks.
            formattedList.sEcho = this.echo;

            //  Return count of all records
            formattedList.iTotalRecords = this.queryable.Count();

            //  Filtered Data
            var records = this.queryable.Where(GenericSearchFilter());
            records = ApplySort(records);

            //  What is filtered data set count now.  This is NOT the
            //  count of what is returned to client
            formattedList.iTotalDisplayRecords = (records.FirstOrDefault() == null) ? 0 : records.Count();

            //  Take a page
            var pagedRecords = records.Skip(this.displayStart)
                     .Take(this.displayLength);

            //  Convert to List of List
            var aaData = new List&gt;();
            var thisRec = new List();

            pagedRecords.ToList()
                    .ForEach(rec =&gt; aaData.Add(rec.PropertiesToList()));

            formattedList.aaData = aaData;

            return formattedList;
        }

That said, there is some trickery that goes on in order to make this happen because we are creating a solution to is going to work with any IQueryable to we supply. This means that the filtering and the sorting will need to be dynamic.

To make the filtering dynamic we will build expression trees that will convert each property to a string, convert the string to lower case, then execute a Contains method against the value of that property.  The method GenericSearchFilter() called on line 16 accomplishes this with the following lines of code:

//  Except from GenericSearchFilter
MethodInfo convertToString = typeof(Convert).GetMethod(&quot;ToString&quot;, Type.EmptyTypes);

 var propertyQuery = (from property in this.properties
        let toStringMethod = Expression.Call(                                          Expression.Call(Expression.Property(paramExpression, property), convertToString, null),                                                            typeof(string).GetMethod(&quot;ToLower&quot;, new Type[0]))
         select Expression.Call(toStringMethod, typeof(string).GetMethod(&quot;Contains&quot;), searchExpression)).ToArray();

We get an array of Expressions that when executed will tell us if the value matches our search term. What we want is to include the item if ANY of the properties is a match, so this means we have to use and OR for all of the properties. That can be accomplished with:

for (int j = 0; j &lt; propertyQuery.Length; j++)
{
  //  Nothing to &quot;or&quot; to yet
  if (j == 0)
  {
    compoundOrExpression = propertyQuery[0];
  }

  compoundOrExpression = Expression.Or(compoundOrExpression,
                                              propertyQuery[j]);
}

So with what is listed above we would be able to match all properties with against a single search term. Pretty cool. But DataTables raises the bar even higher. If you were to go to the samples page and filter using multiple partial words you would find that you could perform some very effective searches with phrases like “new chic”. This would select all records that had properties containing “new” OR “chic”. Imagine the scenario where your user wants to finds all cities “New York” or “Chicago”. We’ve all been there where we have a grid and can only search for one term, or worse, where we have to add a row to a search filter grid and constantly push a “query” button to perform our searches. DataTables does all of the with one search box – just type and the filtering begins.

GenericSearchFilter() handles that scenario. First the search term is parsed into individual terms if there is a ” ”  supplied in the string. This means we will have to perform the propertyQuery for each term that we have. To return all of the records that correspond to each term we still need to perform the OR in groups, but then we need to AND these predicates together so we can get all of the groups per individual term. Here’s the source edited slightly for readability:

//  Split search expression to handle multiple words
var searchTerms = this.genericSearch.Split(' ');

for (int i = 0; i &lt; searchTerms.Length; i++) {    var searchExpression = Expression.Constant( searchTerms[i].ToLower());                   //  For each property, create a contains expression   //  column =&gt; column.ToLower().Contains(searchTerm)
  //  Edited for clarity - create the array propertyQuery logic is here ...
  var propertyQuery = ...

  //  Inner loop for grouping all OR's for this search term
  for (int j = 0; j &lt; propertyQuery.Length; j++)   {     //  Nothing to &quot;or&quot; to yet     if (j == 0)     {       compoundOrExpression = propertyQuery[0];     }     compoundOrExpression = Expression.Or(compoundOrExpression, propertyQuery[j]);   }   //  First time around there is no And, only first set of or's   if (i == 0)   {     compoundAndExpression = compoundOrExpression;   }   else   {     compoundAndExpression = Expression.And(compoundAndExpression, compoundOrExpression);   } } 

So GenericSearchFilter will build a humongous expression tree for all the properties in your class. To make this usable for the Where we convert it using Expression.Lambda and our Where clause just goes about its merry way. Because we have used generics, you can supply any class from your assemblies. One caveat, and Sensei is trying to find a resolution. If you have a string property to that is set to null, the expression tree fails. You’ll note that in the classes supplied in the sample, the properties that are of type string in the Tenant class are defaulted to empty in the constructor.  A small price to pay for some great functionality. To sort our data we use the method ApplySort():

 private IQueryable ApplySort(IQueryable records)         {             string firstSortColumn = this.sortKeyPrefix.First();             int firstColumn = int.Parse(firstSortColumn);             string sortDirection = &quot;asc&quot;;             sortDirection = this.aoDataList.Where(x =&gt; x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX +
                                                                      &quot;0&quot;)
                                                .Single()
                                                .Value
                                                .ToLower();

            if (string.IsNullOrEmpty(sortDirection))
            {
                sortDirection = &quot;asc&quot;;
            }

            return records.OrderBy(this.properties[firstColumn].Name, sortDirection, true);
        }

An extension method OrderBy will accept the name of column, the sort direction as parameters. The parameter initial will indicate if we are sorting mulitple times, so we can accomplish multi-property sort with syntax like

var sortedRecords = records.OrderBy(&quot;State&quot;, &quot;desc&quot;, true)
                                      .OrderBy(&quot;City&quot;, &quot;asc&quot;, false);

public static IOrderedQueryable OrderBy(this IQueryable source, string property, string sortDirection, bool initial)
        {
            string[] props = property.Split('.');
            Type type = typeof(T);
            ParameterExpression arg = Expression.Parameter(type, &quot;x&quot;);
            Expression expr = arg;
            foreach (string prop in props)
            {
                // use reflection (not ComponentModel) to mirror LINQ
                PropertyInfo pi = type.GetProperty(prop);
                expr = Expression.Property(expr, pi);
                type = pi.PropertyType;
            }
            Type delegateType = typeof(Func&lt;,&gt;).MakeGenericType(typeof(T), type);
            LambdaExpression lambda = Expression.Lambda(delegateType, expr, arg);

            string methodName = string.Empty;

            //  Asc or Desc
            if (sortDirection.ToLower() == &quot;asc&quot;)
            {
                //  First clause?
                if (initial &amp;&amp; source is IOrderedQueryable)
                {
                    methodName = &quot;OrderBy&quot;;
                }
                else
                {
                    methodName = &quot;ThenBy&quot;;
                }
            }
            else
            {
                if (initial &amp;&amp; source is IOrderedQueryable)
                {
                    methodName = &quot;OrderByDescending&quot;;
                }
                else
                {
                    methodName = &quot;ThenByDescending&quot;;
                }
            }

            object result = typeof(Queryable).GetMethods().Single(
                    method =&gt; method.Name == methodName
                            &amp;&amp; method.IsGenericMethodDefinition
                            &amp;&amp; method.GetGenericArguments().Length == 2
                            &amp;&amp; method.GetParameters().Length == 2)
                    .MakeGenericMethod(typeof(T), type)
                    .Invoke(null, new object[] { source, lambda });
            return (IOrderedQueryable)result;
        }

All good things …
It’s been a long ride, this post. A lot of code discussed, a lot of ground covered. The solution is here.  As always, play around and see how this can help you. If anything breaks, tell Sensei. If you have improvements, tell Sensei. DataTables is a great tool for your arsenal, hopefully the DataTablePager can help you integrate paging with large datasets as part of your solution offering.

Right now Sensei wants to sign off by toasting to you for wading through all of this, and for having the desire to build up your skills.  Obtaining fluency in what you do is a hard road to travel, but it’s worth it because you get things done quicker and better with each session.

Be sure to read about the latest version in “Dynamically Select Columns with Server-Side Paging and Datatables.Net

ActiveEngine Software by ActiveEngine, LLC.