Blog Archive

Home
Problem Solving

ski_and_snowboard_snowsport1Enterprise level software can provide marvelous benefits with process automation; yet, how many tales can you recount of the company that has implemented Enterprise level procurement / ERP / BPM, etc and it was a death march. Pure and simple it was a nightmare, accounting couldn’t close the books for over 5 weeks, and this continued for almost a year. True story.

This post is not to be seen as refutation of automation, business process management, statistics, and economics. People, organizations, and companies need direction, require a director to set the tone, the pace and cadence for reaching goals. But there is a danger that we treat processes like science with the belief that when business behaves in a more rational fashion, things will operate like physics or mathematics. While it is true that applying scientific approach and principals to business can help, at its essence, business is not science. Here’s another analogy – while you should apply science to the field of medicine, in the end, the body and mind heal in manners that are not scientific. Ask any doctor about misdiagnosis and miracles. If he or she hasn’t been humbled by their lack of understanding of healing, change doctors!

We have to careful with model-abstraction worship. Let me tell you, the model is not the process – it’s a representation. Look at the two intrepid skiing figures above. Do you think they are thinking of the mechanics or the physics of what they are experiencing? Torque on the knees? Now put yourself in their instructors or trainers shoes and ask “how would I really go about communicating what needs to be learned to survive this type of run?”

General_Consultation_Workflow_diagram_v68An organization is comprised of the interactions of people, and these series of steps, while seeming out of control, create the cadence of your business. At the level where the work is completed, if you were to ask someone to describe what they do, it will start with a list items, not a diagram like the one above. At the level where the work is done,the people pushing the buttons, experience a different reality than what management experiences. If your goal is to assist those people, then your first step is not to alienate them by diagramming their busy day with pseudo scientific terms and weird charts. Remember, the director who can’t get her actors to relate to the story makes an awful film. You have to relate to others on a fundamental level, and for the most part in business, people don’t relate to this diagram. A picture is worth a thousand words, but for those whose inbox is flooded with requests, a diagram is not a picture, it a a thousand confusing questions. “Why is the box blue? Can I have a diamond instead?”

“Ok”, you ask, “what do we do instead?” Well, as luck would have, there is a good way to implement process, get approvals, and have consistency without over orchestrating business process and creating confusion. The approach is to view the decisions that people make as gateways between important steps. This is best illustrated with an example, so let’s model a bonus approval process for a sales organization.

We have a sales team who is closing deals weekly, and for each deal a manager and vice president will need to weigh in on the merits of the award. Fairly standard in most businesses, and in many cases this is managed with email. The bonus manager is the fulcrum of this process, as it he who currently reminds the sales agents that they have not supplied the correct supporting materials, as well as gently nudging the vice president of sales that she needs to get into the system and complete approvals so that the request can be released to payroll before the Wednesday paycheck run. Your role, dear reader, is to get the steps of this process that are inside the bonus managers head and create something that they will engage in. To create a fan of the new process, it’s better that the process come from the bonus manager – they have to do it, so if they create it, they own it!

Remember the diatribe against diagrams? You can’t ask the bonus manager to diagram their process, as they’re too busy. But have them list the steps – yes a simple list. It may be a very limited set of items. You get this:

  • Fill Out Bonus Request
  • Bonus Manager Reviews
  • Vice President Reviews
  • Payroll Issues payment

You can prod the bonus manager further, and get them to categorize the end states of this process. We’ll add Bonus Denied and Bonus Paid, as this will represent all the stops in the process. Now the list looks like this:

  • Fill Out Bonus Request
  • Bonus Manager Reviews
  • Vice President Reviews
  • Payroll Issues payment
  • Bonus Denied
  • Bonus Paid

The next thing to do is ask “What decisions get you from step to step? For example, what do you do when work is incomplete from the Sales Agent?” The answer – “Today? I email them and request more info”. Lets model this action as a phrase starting with a verb:

Request More Info

Now you start a helpful chart that looks like this:

Step Decision Next Step
Bonus Manager Review Request More Info Fill Out Bonus Request

 

Simple. It’s in English, and there is direct understanding of what happens. What’s more, these are the phrases, thoughts, and language used by the key person of the process. Their “vocab” is the context of how they think, and it’s not a stretch for understand what is occurring, no threat from the oppressive science of process modeling. You could even do this at lunch! Here is what the end product will look like:

Step Decision Next Step
Fill Out Bonus Request Complete Bonus Manager Review
Bonus Manager Review Request More Info Fill Out Bonus Request
Bonus Manager Review Approve VP Review
VP Review Deny Bonus Denied
VP Review Approve Payroll Issues Payment
Payroll Issues Payment Complete Bonus Paid



There’s more good news. ActiveEngine Software has a product that will take this list and create a workflow. Here is a demo video. Thanks for reading.

faradaypennytwinsOf late, Sensei needs to keep a clear head.  That has meant learning to segment ideas and really, really, really focus on streamlined features.  This is hard.  Not because Sensei has a plethora of great ideas.  That would be a nice problem to have.  Many times in software development you end up this guy:

This is the state where you have things you want to accomplish, yet even when you pair things down to the “essential”, other essential, critical factors must be taken into consideration before you create a mess.  This is life calling, and that string which suspends that giant sword that you noticed hovering over your head is about to snap.  There is a good chance that you need more discipline, better execution tactics, far better honed chops, you name the metaphor.  Sensei has been at this game for over 22 years, and still the speed that thought takes to become reality is way too slow.

With great sarcasm you can remind your self that some of the best work lays ahead, but the reality is that you still need to fight to be fluent, you have to claw your way to a Zen state of mind / no-mind.  So chose, the art of bushido or the art of BS.  Or maybe work smarter and enjoy life.

Before Sensei leaves you, ponder this:  does “being done” mean that you’ve dropped off a product but have to get on the phone in order to make changes, and maybe now that you are struggling why couldn’t you figure out to take time when it was more critical to be fluent with your productivity?

 

This is the fourth in a series of posts for ApprovaFlow, an alternative to Windows Workflow written in C# and JSON.Net.  Source code for this post is here.

[gigya src="http://listen.grooveshark.com/songWidget.swf" width="250" height="40" flashvars="hostname=cowbell.grooveshark.com&songID=1211234&style=metal&p=0" wmode="window"]

Last Time on ApprovaFlow

In the previous post we discussed how the Pipe and Filter pattern facilitated a robust mechanism for executing tasks prior and after a transition is completed by the workflow state machine.  This accomplished our third goal and to date we have completed:

Model a workflow in a clear format that is readable by both developer and business user. One set of verbiage for all parties.  Discussed in Simple Workflows With ApprovaFlow and Stateless.

•  Allow the state of a workflow to be peristed as an integer, string, etc.  Quickly fetch state of a workflow.  Discussed in Simple Workflows With ApprovaFlow and Stateless.

•  Create pre and post processing methods that can enforce enforce rules or carry out actions when completing a workflow task.  Discussed in ApprovaFlow:  Using the Pipe and Filter Pattern to Build a Workflow Processor

These goals remain:

• Introduce new functionality while isolating the impact of the new changes. New components should not break old ones

• Communicate to the client with a standard set of objects. In other words, your solution domain will not change how the user interface will gather data from the user.

• Use one. aspx page to processes user input for any type of workflow.

• Provide ability to roll your own customizations to the front end or backend of your application.

It’s the Small Changes After You Go Live That Upset You

The goal we’ll focus on next is Introduce new functionality while isolating the impact of the new changes. New components should not break old ones, as it’s the small upsetters that lurk around the corner that your users will think up that will keep you in the constant redeployment cycle. If we implement a plug-in system, then we can prevent the new features from breaking the current production system. Implementing these changes in isolation will lead to faster testing, validation and happier users.

We lucked out as our implementation of the Pipe And Filter pattern forced us to create objects with finite functionality.  If you recall each step in our workflow chain was implemented as a filter derived from FilterBase and this lends itself nicely to creating plug-ins.  The Pipe and Filter pattern forces us to have a filter for each unique action we wish to carry out.  To save data we have a SaveData filter, to validate that a user can supply a Trigger we have the ValidateUserTrigger, and so on.

“Great, Sensei, but aren’t we still constrained by the fact that we have to recompile and deploy any time we add new filters?  And, if I have to do that, why bother with the pattern in the first place?”

Well, we can easily reduce the need for re-deploying the application through the use of a plugin system where we read assemblies from a share and interrogate them by searching for a particular object type on application start up.  Each new feature will be a new filter.  This means you will be working with a small project that references ApprovaFlow to create new filters without disturbing the existing architecture.   We’ll also create a manifest of approved plug-ins so that we can control what is used and institute a little security since we wouldn’t want any plugin to be introduced surreptitiously.

Plug-in Implementation

The class FilterRegistry will perform the process of reading a share, fetching the object with type FilterBase, and register these components just like we do with our system components.  There are a few additions since the last version, as we now need to read and store the manifest for later comparison with the plug-ins.  The new method ReadManifest takes care of this new task:

<pre><code>private void ReadManifest()
{
string manifestSource = ConfigurationManager.AppSettings["ManifestSource"].ToString();

Enforce.That(string.IsNullOrEmpty(manifestSource) == false,
“FilterRegistry.ReadManifest – ManifestSource can not be null”);

var fileInfo = new FileInfo(manifestSource);

if (fileInfo.Exists == false)
{
throw new ApplicationException(“RequestPromotion.Configure – File not found”);
}

StreamReader sr = fileInfo.OpenText();
string json = sr.ReadToEnd();
sr.Close();

this.approvedFilters = JsonConvert.DeserializeObject>>(json);
}

</code></pre>

The manifest is merely a serialized list of FilterDefinitions. This is de-serialized into a list of approved filters.With the approved list the method LoadPlugin performs the action of reading the share and matching the FullName of the object type between the manifest entries and the methods in the assembly file:

<pre><code>

public void LoadPlugIn(string source)
{
Enforce.That(string.IsNullOrEmpty(source) == false,
“PlugInLoader.Load – source can not be null”);

AppDomain appDomain = AppDomain.CurrentDomain;
var assembly = Assembly.LoadFrom(source);

var types = assembly.GetTypes().ToList();

types.ForEach(type =>
{
var registerFilterDef = new FilterDefinition();

// Is type from assembly registered?
registerFilterDef = this.approvedFilters.Where(app => app.TypeFullName == type.FullName)
.SingleOrDefault();

if (registerFilterDef != null)
{
object obj = Activator.CreateInstance(type);
var filterDef = new FilterDefinition();
filterDef.Name = obj.ToString();
filterDef.FilterCategory = registerFilterDef.FilterCategory;
filterDef.FilterType = type;
filterDef.TypeFullName = type.FullName;
filterDef.Filter = AddCreateFilter(filterDef);

this.systemFilters.Add(filterDef);
}
});
}

</code></pre>

That’s it. We can now control what assemblies are included in our plug-in system.  Later we’ll create a tool that will help us create the manifest so we do not have to managed it by hand.

What We Can Do with this New Functionality

Let’s turn to our sample workflow to see what possibilities we can develop.  The test CanPromoteRedShirtOffLandingParty from the class WorkflowScenarios displays the capability of our workflow.  First lets review our workflow scenario.  We have created a workflow for the Starship Enterprise to allow members of a landing party to request to be left out of the mission.  Basically there is only one way to get out of landing party duty and that is if Kirk says it’s okay.  Here are the workflow’s State, Trigger and Target State combinations:

State Trigger Target State
RequestPromotionForm Complete FirstOfficerReview
FirstOfficerReview RequestInfo RequestPromotionForm
FirstOfficerReview Deny PromotionDenied
FirstOfficerReview Approve CaptainApproval
CaptainApproval OfficerJustify FirstOfficerReview
CaptainApproval Deny PromotionDenied
CaptainApproval Approve PromotedOffLandingParty

Recalling the plots from Star Trek, there were times that the medical officer could declare the commanding officer unfit for duty. Since the Enterprise was originally equipped with our workflow, we want to make just a small addition – not a modification – and give McCoy the ability to allow a red shirt to opt out of the landing party duty.

Here’s where our plugin system comes in handy.  Instead of adding more states and or branches to our workflow we’ll check for certain conditions when Kirk makes his decisions, and execute actions.  In order to help out McCoy the following filter is created in a separate project:

<pre><code>

public class CaptainUnfitForCommandFilter : FilterBase
{
protected override Step Process(Step input)
{
if(input.CanProcess & input.State == “CaptainApproval”)
{
bool kirkInfected = (bool)input.Parameters["KirkInfected"];

if(kirkInfected & input.Answer == “Deny”)
{
input.Parameters.Add(“MedicalOverride”, true);
input.Parameters.Add(“StarfleetEmail”, true);
input.ErrorList.Add(“Medical Override of Command”);
input.CanProcess = false;
}
}

return input;
}
}

</code></pre>

This plug-in is simple: check that the state is CaptainApproval and when the answer was “Deny” and Kirk has been infected, set the MedicalOverride flag and send Starfleet an email.

The class WorkflowScenarioTest.cs has the method CanAllowMcCoyToIssueUnfitForDuty() that demonstrates how the workflow will execute. We simply add the name of the plug-in to our list of post transition filters:

<pre><code>
string postFilterNames = “MorePlugins.TransporterRepairFilter;Plugins.CaptainUnfitForCommandFilter;SaveDataFilter;”;
</code></pre>

This portion of code uses the plug-in:

<pre><code>

// Captain Kirt denies request, but McCoy issues unfit for command
parameters.Add(“KirkInfected”, true);

step.Answer = “Deny”;
step.AnsweredBy = “Kirk”;
step.Participants = “Kirk”;
step.State = newState;

processor = new WorkflowProcessor(step, filterRegistry, workflow);
newState = processor.ConfigurePipeline(preFilterNames, postFilterNames)
.ConfigureStateMachine()
.ProcessAnswer()
.GetCurrentState();

// Medical override issued and email to Starfleet generated
bool medicalOverride = (bool)parameters["MedicalOverride"];
bool emailSent = (bool)parameters["StarfleetEmail"];

Assert.IsTrue(medicalOverride);
Assert.IsTrue(emailSent);
</code></pre>

Now you don’t have to hesitate with paranoia each time you need introduce a variation into your workflows. No more small upsetters lurking around the corner. Plus you can deliver these changes faster to your biggest fan, your customer. Source code is here.   Run through the tests and experiment for your self.

Source code has been yet again updated!! Read about the changes in DataTablePager Now Has Multi-Column Sort Capability For DataTables.Net If you are new to DataTables.Net and Sensei’s paging solution and want to detailed study of how it works, work through this post first, then get the latest edition.  Note, code links in this post are to the first version.

The last episode of server-side paging with DataTablerPager for DataTables.Net we reviewed the basics of a server-side solution that paged records and returned results in the multiples as specified by DataTables.Net.  You will want to have read that post before preceding here.  The older version of the source is included in that post as well as this will help get you acclimated.  The following capabilities were reviewed:

  • The solution used generics and could work with any collection of IQueryable.  In short any of your classes from you domain solution  could be used.
  • Filtering capability across all properties was provided.  This included partial word matching, regardless of case.
  • Ordering of result set was in response to the column clicked on the client’s DataTables grid.

DataTablePager Enhancements

This past month Sensei has added new capabilities to the DataTablePager class that makes it an even better fit for use with DataTables.Net.  The new features are:

  • Dynamically select the columns from the properties of your class based on the column definitions supplied by DataTables.Net!!!
  • Exclude columns from sort or search based on configuration by DataTables.Net
  • Mix columns from your class properties with client-side only column definitions; e.g. create a column with <a href>’s that do not interfere with filtering, sorting, or other processing.

Before we jump into the nitty-gritty details let’s review how DataTables.Net allows you to control a column’s interaction with a data grid.  Grab the new source code to best follow along.

DataTables.Net Column Definition

You would think that there would be quite a few steps to keep your server-side data paging solution in concert with a client side implementation, and that would mean customization for each page.   DataTables.Net provides you with fine control over what your columns will do once displayed in a data grid.  Great, but does that mean a lot of configuration on the server side of the equation?  As well soon see, no, it doesn’t.  What is done on the client for configuration will be that you need to do.

The structure aoColumnDefs is the convention we use for column configuration.  From the DataTables.Net site:

aoColumnDefs: This array allows you to target a specific column, multiple columns, or all columns, using the aTargets property of each object in the array (please note that aoColumnDefs was introduced in DataTables 1.7). This allows great flexibility when creating tables, as the aoColumnDefs arrays can be of any length, targeting the columns you specifically want. The aTargets property is an array to target one of many columns and each element in it can be:

  • a string – class name will be matched on the TH for the column
  • 0 or a positive integer – column index counting from the left
  • a negative integer – column index counting from the right
  • the string “_all” – all columns (i.e. assign a default)

So in order for you to include columns in a sort you configure in this manner:

/* Using aoColumnDefs */
$(document).ready(function() {
	$('#example').dataTable( {
		&quot;aoColumnDefs&quot;: [
			{ &quot;bSortable&quot;: false, &quot;aTargets&quot;: [ 0 ] }
		] } );
} );

} );

In other words we are defining that the first column – column 0 – will not be included in the sorting operations.  When you review the columns options you’ll see you have options for applying css classes to multiple columns, can include a column in filtering, can supply custom rendering of a column, and much more.

In the example that we’ll use for the rest of the post we are going to provide the following capability for a data grid:

  1. The first column – column 0 – will be an action column with a hyperlink, and we will want to exclude it form sort and filtering functions.
  2. Only display a subset of the properties from a class.  Each of these columns should be sortable and filterable.
  3. Maintain the ability to chunk the result set in the multiples specified by DataTables.Net; that is, multiples of 10, 50, and 100.

Here is the configuration from the aspx page SpecifyColumns.aspx:

&quot;aoColumnDefs&quot; : [
   {&quot;fnRender&quot; : function(oObj){
      return &quot;&lt;a href=&quot;&amp;quot;center.aspx?centerid=&amp;quot;&quot;&gt;Edit&lt;/a&gt;&quot;;
   },
     &quot;bSortable&quot; : false,
     &quot;aTargets&quot; : [0]},
   {&quot;sName&quot; : &quot;Name&quot;,
     &quot;bSearchable&quot; : true,
     &quot;aTargets&quot;: [1]},
   {&quot;sName&quot; : &quot;Agent&quot;,
    &quot;bSearchable&quot; : true,
    &quot;bSortable&quot; : true,
    &quot;aTargets&quot; : [2]
   },
   {&quot;sName&quot; : &quot;Center&quot;, &quot;aTargets&quot;: [3]},
   {&quot;fnRender&quot; : function(oObj){
            return &quot;2nd Action List&quot;;
         },
     &quot;bSortable&quot; : false,
     &quot;aTargets&quot; : [4]},
   {&quot;sName&quot; : &quot;CenterId&quot;, &quot;bVisible&quot; : false, &quot;aTargets&quot; : [5]},
   {&quot;sName&quot; : &quot;DealAmount&quot;, &quot;aTargets&quot; : [6]}
]
  1. Column 0 is our custom column – do not sort or search on this content.  Look at oObj.aData[4] – this is a column that we’ll return but not display.  It’s referred so by the position in the data array that DataTables.Net expects back from the server.
  2. Columns 1 – 3 are data and can be sorted.  Note the use of “sName”.  This will be included in a named column list that corresponds to the source property from our class.  This will be very important later on for us, as it allows us to query our data and return it in any order to DataTables.Net.  DataTables will figure out what to do with it before it renders.
  3. Threw in another custom column.  Again, no sort or search, but we’ll see how this affects the server side implementation later on.  Hint – there’s no sName used here.
  4. Another data column.

To recap, we want to be able to define what data we need to display and how we want to interact with that data by only instructing DataTables.Net what to do.  We’re going to be lazy, and not do anything else – the class DataTablePager will respond to the instructions that DataTables.Net supplies, and that’s it.  We’ll review how to do this next.  Sensei thinks you’ll really dig it.

DataTablePager Class Handles your Client Side Requests

If you recall, DataTables.Net communicates to the server via the structure aoData.  Here is the summary of the parameters.  One additional parameter that we’ll need to parse is the sColumns parameter, and it will contain the names and order of the columns that DataTables.Net is rendering.  For our example, we’ll get the following list of columns if we were to debug on the server:

,Name,Agent,Center,,CenterId,DealAmount

These are all the columns we named with sName, plus a place holder for those custom columns that not found in our class.  This has several implications.  For one, it will mean that we will no longer be able to simply use reflection to get at our properties, filter them and send them back down to the client.  The client is now expecting an array where each row will have 7 things, 5 of which are named and two place holders for items that the client wants to reserve for itself.  Hence the convention of passing an empty item in the delimited string as shown above.

It will also mean that we’ll have to separate the columns that we can filter or sort.  Again this is the reason for leaving the custom column names blank.  In other words, we’ll have to keep track of the items that we can search and sort.  We’ll do this with a class called SearchAndSortable:

public class SearchAndSortable
    {
        public string Name { get; set; }
        public int ColumnIndex { get; set; }
        public bool IsSearchable { get; set; }
        public bool IsSortable { get; set; }
        public PropertyInfo Property{ get; set; }

        public SearchAndSortable(string name, int columnIndex, bool isSearchable, bool isSortable)
        {
            this.Name = name;
            this.ColumnIndex = columnIndex;
            this.IsSearchable = isSearchable;
            this.IsSortable = IsSortable;
        }

        public SearchAndSortable() : this(string.Empty, 0, true, true) { }
    }

This will summarize what we’re doing with our properties.   The property ColumnIndex will record the position in sColumn where our column occurs.  Since we’ll need access to the actual properties themselves we’ll store these in the SearchAndSortable as well so that we can reduce the number of calls that use reflection. DataTablePager uses a List of SortAndSearchables to track what’s going on.  We fill this list in the method PrepAOData()

//  What column is searchable and / or sortable
            //  What properties from T is identified by the columns
            var properties = typeof(T).GetProperties();
            int i = 0;

            //  Search and store all properties from T
            this.columns.ForEach(col =&gt;
            {
                if (string.IsNullOrEmpty(col) == false)
                {
                    var searchable = new SearchAndSortable(col, i, false, false);
                    var searchItem = aoDataList.Where(x =&gt; x.Name == BSEARCHABLE + i.ToString())
                                     .ToList();
                    searchable.IsSearchable = (searchItem[0].Value == &quot;False&quot;) ? false : true;
                    searchable.Property = properties.Where(x =&gt; x.Name == col)
                                                    .SingleOrDefault();

                    searchAndSortables.Add(searchable);
                }

                i++;
            });

            //  Sort
            searchAndSortables.ForEach(sortable =&gt; {
                var sort = aoDataList.Where(x =&gt; x.Name == BSORTABLE + sortable.ColumnIndex.ToString())
                                            .ToList();
                sortable.IsSortable = (sort[0].Value == &quot;False&quot;) ? false : true;
            });

We’ll get the properties from our class. Next we’ll traverse the columns and match the property names with the names of the columns. When there is a match, we need to query aoData and get the column search and sort definitions based on the ordinal position of the column in the sColumns variable. DataTables.Net convention for communicating this is the form of:

bSortable_ + column index => “bSortable_1″ or “bSearchable_2″

We take care of that with this line of code:

var searchItem = aoDataList.Where(x =&gt; x.Name == BSEARCHABLE +
                                     i.ToString())
                                     .ToList();
searchable.IsSearchable = (searchItem[0].Value == &quot;False&quot;) ? false : true;

Now we go through the list of properties again but this time determine if we should sort any of the columns. That happens in the section //Sort. In the end we have a list of properties that corresponds with the columns DataTables.Net has requested, and we have defined if the property can be search (filtered) or sorted.

For filtering DataTablePager recall that we use the method GenericSearchFilter().  The only alteration here is that we only will add the properties to our query that are defined as searcable:

//  Create a list of searchable properties
            var filterProperties = this.searchAndSortables.Where(x =&gt;
                                        x.IsSearchable)
                                          .Select(x =&gt; x.Property)
                                          .ToList();

The rest of the method is unaltered from the prior version. Pretty cool!! Again, we’ll only get the properties that we declared as legal for filtering. We’ve also eliminated any chance of mixing a custom column in with our properties because we did not supply an sName in our configuration.

The method ApplySort() required one change. On the initial load of DataTable.Net, the client will pass up the request to sort on column 0 even though you may have excluded it. When that is the case, we’ll just look for the first column that is sortable and order by that column.

//  Initial display will set order to first column - column 0
//  When column 0 is not sortable, find first column that is
var sortable = this.searchAndSortables.Where(x =&gt; x.ColumnIndex ==
                                         firstColumn)
                              .SingleOrDefault();
if(sortable == null)
{
   sortable = this.searchAndSortables.First(x =&gt; x.IsSortable);
}

return records.OrderBy(sortable.Name, sortDirection, true);

After we have filtered and sorted the data set we can finally select the only those properties that we want to send to the client.  Recall that we have parsed a variable sColumns that tells what columns are expected.  We’ll pass these names onto extension method PropertiesToList().  This method will only serialize the property if the column is include, and since we have already paired down our data set as a result of our query and paging, there is very little performance impact.  Here is the new PropertiesToList method:

public static ListPropertiesToList(this T obj, List propertyNames)
{
   var propertyList = new List();
   var properties = typeof(T).GetProperties();
   var props = new List();

   //  Find all &quot;&quot; in propertyNames and insert empty value into list at
   //  corresponding position
   var blankIndexes = new List();
   int i = 0;

   //  Select and order filterProperties.  Record index position where there is
   //  no property
   propertyNames.ForEach(name =&gt;
   {
      var property = properties.Where(prop =&gt; prop.Name == name.Trim())
         .SingleOrDefault();

      if(property == null)
      {
         blankIndexes.Add(new NameValuePair(name, i));
      }
      else
      {
         props.Add(properties.Where(prop =&gt; prop.Name == name.Trim())
                                    .SingleOrDefault());
      }
      i++;
   });

   propertyList = props.Select(prop =&gt; (prop.GetValue(obj, new object[0]) ?? string.Empty).ToString())
                                        .ToList();

   //  Add &quot;&quot; to List as client expects blank value in array
   blankIndexes.ForEach(index =&gt;; {
      propertyList.Insert(index.Value, string.Empty);
   });

   return propertyList;
}

You might ask why not just pass in the list of SearchAndSortTable and avoid using reflection again. You could, but remember at this point we have reduced the number of items to the page size of 10, 50 or 100 rows, so your reflection calls will not have that great an impact. Also you should consider whether you want to simply have a function that will select only those properties that you need. Using SearchAndSortable would narrow the scope of utility, as you can use this method in other areas other than prepping data for DataTables.Net.

Now It’s Your Turn

That’s it.  Play with the page named SpecifyColumns.aspx.  You should be able to add and remove columns in the DataTable.Net configuration and they will just work.  This will mean, however, that you’ll have to always define your columns in your aspx page.  But since we worked really hard the first time around, DataTablePager will still be able to create paged data sets for any class in your domain.

Source code is here.  Enjoy.

Source code has been updated!! Read about the changes in Dynamically Select Columns with Server-Side Paging and Datatables.Net If you are new to DataTables.Net and Sensei’s paging solution and want to detailed study of how it works, work through this post first, then get the latest edition.  Note, code links in this post are to the first version.

A central theme for 2010 has been fluency, or the continual practice of certain methods to such a degree that your performance improves and you produce increasingly polished, effective solutions.  For software development this has meant tools to save time and increase quality.  It also means keeping an eye toward making the users of your solutions more efficient as well.  In the spirit of “fluent solutions”, Sensei will end the year with a post that examines how to create a data paging solution for the jQuery data grid plug-in DataTables.Net.

DataTables can turn a HTML table into a fully functional data grid like the one offered by Telerik.  This plug-in offers client side sorting, filtering/ search,  as well as support for server-side processing processing of data.  It is an extremely feature rich tool created by Allan Jardine, and is itself worthy of a series of posts.  For this post on data paging Sensei recommends that you read through these examples to get an idea of what the data paging service needs to achieve.

Let’s get started with the goals we need to achieve when providing server-side data paging support:

  • Send data to client in the multiples or “chunks” that the client requests, and respond when the size of the sets requested is changed by the user.
  • Re-order the data set if the user clicks on a column heading.  Honor the data set size when returning the data.
  • Filter across all columns of data based on user input.  Implement this as partial matches, and again, honor the data set size.

Remember this is about flexibility, so we have the additional goals of:

  • Create a solution that can be reused.
  • Provide a mechanism to accommodate any type of .Net class using generics.

Essentially we want to be able to write code like so:

var tenants = tenantRepository.GetAll();
var dataTablePager = new DataTablePager();
var returnDataSet = dataTablePager.Filter(requestParms, tenants);

Before we proceed, Sensei wants to acknowledge those really smart people whose ideas contributed to this solution:

Zack Owens – jQuery DataTables Plugin Meets C#

Jeff Morris - Using Datatables.net JQuery Plug-in with WCF Services

Dave Ward – ASMX ScriptService mistake – Invalid JSON primitive

You may want to download the source before reading the rest of this post.

Communicating with DataTables

DataTables uses the following parameters when processing server-side data:

Sent to the server:

Type Name Info
int iDisplayStart Display start point
int iDisplayLength Number of records to display
int iColumns Number of columns being displayed (useful for getting individual column search info)
string sSearch Global search field
boolean bEscapeRegex Global search is regex or not
boolean bSortable_(int) Indicator for if a column is flagged as sortable or not on the client-side
boolean bSearchable_(int) Indicator for if a column is flagged as searchable or not on the client-side
string sSearch_(int) Individual column filter
boolean bEscapeRegex_(int) Individual column filter is regex or not
int iSortingCols Number of columns to sort on
int iSortCol_(int) Column being sorted on (you will need to decode this number for your database)
string sSortDir_(int) Direction to be sorted – “desc” or “asc”. Note that the prefix for this variable is wrong in 1.5.x where iSortDir_(int) was used)
string sEcho Information for DataTables to use for rendering

Reply from the server

In reply to each request for information that DataTables makes to the server, it expects to get a well formed JSON object with the following parameters.

Type Name Info
int iTotalRecords Total records, before filtering (i.e. the total number of records in the database)
int iTotalDisplayRecords Total records, after filtering (i.e. the total number of records after filtering has been applied – not just the number of records being returned in this result set)
string sEcho An unaltered copy of sEcho sent from the client side. This parameter will change with each draw (it is basically a draw count) – so it is important that this is implemented. Note that it strongly recommended for security reasons that you ‘cast’ this parameter to an integer in order to prevent Cross Site Scripting (XSS) attacks.
string sColumns Optional – this is a string of column names, comma separated (used in combination with sName) which will allow DataTables to reorder data on the client-side if required for display
array array mixed aaData The data in a 2D array

The data sent back is in the following form depicted below. Note that aaData is merely an array of strings – there is no column information. This will present a challenge in that you will not be able to simply serialize a collection and pass back the results.

{
    &quot;sEcho&quot;: 3,
    &quot;iTotalRecords&quot;: 57,
    &quot;iTotalDisplayRecords&quot;: 57,
    &quot;aaData&quot;: [
        [
            &quot;Gecko&quot;,
            &quot;Firefox 1.0&quot;,
            &quot;Win 98+ / OSX.2+&quot;,
            &quot;1.7&quot;,
            &quot;A&quot;
        ],
        [
            &quot;Gecko&quot;,
            &quot;Firefox 1.5&quot;,
            &quot;Win 98+ / OSX.2+&quot;,
            &quot;1.8&quot;,
            &quot;A&quot;
        ],
        ...
    ]
}

As you may be aware, if you wish to use ASP.Net web services to serialize JSON you must POST to the service and instruct it to interpret your parameters as JSON. DataTables will POST variables as value pairs and this won’t work for us when POSTing to a web service. We’ll have to translate the variables to a usable format. Luckily DataTables allows us to intervene with the following code, where we create a JSON string by serializing a structure called aoData:

&quot;fnServerData&quot;: function ( sSource, aoData, fnCallback ) {

		        	var jsonAOData = JSON.stringify(aoData);

			        $.ajax( {
                                        contentType: &quot;application/json; charset=utf-8&quot;,
				        type: &quot;POST&quot;,
				        url: sSource,
				        data: &quot;{jsonAOData : '&quot; + jsonAOData + &quot;'}&quot;,
				        success: function(msg){
				            fnCallback(JSON.parse(msg.d));
				        },
				        error: function(XMLHttpRequest, textStatus, errorThrown) {
                            alert(XMLHttpRequest.status);
                            alert(XMLHttpRequest.responseText);

                        }
			        });

Our web service can now de-serialize aoData and parse the appropriate parameters. This gives us important items such as how many records to display, what columns to sort on, and what search terms should be applied in a filter.

DataTablePager Class

DataTablePager.cs is the work horse of our solution.  It will sort, filter and order our data, and as an extra, serialize the results in format required by aaData.  Here’s the constructor:

public DataTablePager(string jsonAOData, IQueryable queryable)
        {
            this.queryable = queryable;
            this.type = typeof(T);
            this.properties = this.type.GetProperties(BindingFlags.Public | BindingFlags.Instance);
            this.aoDataList = new List&gt;();
            this.sortKeyPrefix = new List();

            PrepAOData(jsonAOData);
        }

The parameter jsonAOData is the JSON string that contains the variables iDisplayStart, iDisplayLength, etc.  These will be parsed by the method PrepAOData.  The parameter queryable is the collection of records that will be filtered and parsed into JSON format required by DataTables.

The method Filter() coordinates all of the work.  It’s pretty simple what we want to do:  filter our data based on each column containing the search term, sort the result, then pull out the number of records we need to include in the page, and finally convert the collection into the format DataTables understands.

public FormattedList Filter()
        {
            var formattedList = new FormattedList();

            //  What are the columns in the data set
            formattedList.Import(this.properties.Select(x =&gt; x.Name + &quot;,&quot;)
                                                 .ToArray());

            //  Return same sEcho that was posted.  Prevents XSS attacks.
            formattedList.sEcho = this.echo;

            //  Return count of all records
            formattedList.iTotalRecords = this.queryable.Count();

            //  Filtered Data
            var records = this.queryable.Where(GenericSearchFilter());
            records = ApplySort(records);

            //  What is filtered data set count now.  This is NOT the
            //  count of what is returned to client
            formattedList.iTotalDisplayRecords = (records.FirstOrDefault() == null) ? 0 : records.Count();

            //  Take a page
            var pagedRecords = records.Skip(this.displayStart)
                     .Take(this.displayLength);

            //  Convert to List of List
            var aaData = new List&gt;();
            var thisRec = new List();

            pagedRecords.ToList()
                    .ForEach(rec =&gt; aaData.Add(rec.PropertiesToList()));

            formattedList.aaData = aaData;

            return formattedList;
        }

That said, there is some trickery that goes on in order to make this happen because we are creating a solution to is going to work with any IQueryable to we supply. This means that the filtering and the sorting will need to be dynamic.

To make the filtering dynamic we will build expression trees that will convert each property to a string, convert the string to lower case, then execute a Contains method against the value of that property.  The method GenericSearchFilter() called on line 16 accomplishes this with the following lines of code:

//  Except from GenericSearchFilter
MethodInfo convertToString = typeof(Convert).GetMethod(&quot;ToString&quot;, Type.EmptyTypes);

 var propertyQuery = (from property in this.properties
        let toStringMethod = Expression.Call(                                          Expression.Call(Expression.Property(paramExpression, property), convertToString, null),                                                            typeof(string).GetMethod(&quot;ToLower&quot;, new Type[0]))
         select Expression.Call(toStringMethod, typeof(string).GetMethod(&quot;Contains&quot;), searchExpression)).ToArray();

We get an array of Expressions that when executed will tell us if the value matches our search term. What we want is to include the item if ANY of the properties is a match, so this means we have to use and OR for all of the properties. That can be accomplished with:

for (int j = 0; j &lt; propertyQuery.Length; j++)
{
  //  Nothing to &quot;or&quot; to yet
  if (j == 0)
  {
    compoundOrExpression = propertyQuery[0];
  }

  compoundOrExpression = Expression.Or(compoundOrExpression,
                                              propertyQuery[j]);
}

So with what is listed above we would be able to match all properties with against a single search term. Pretty cool. But DataTables raises the bar even higher. If you were to go to the samples page and filter using multiple partial words you would find that you could perform some very effective searches with phrases like “new chic”. This would select all records that had properties containing “new” OR “chic”. Imagine the scenario where your user wants to finds all cities “New York” or “Chicago”. We’ve all been there where we have a grid and can only search for one term, or worse, where we have to add a row to a search filter grid and constantly push a “query” button to perform our searches. DataTables does all of the with one search box – just type and the filtering begins.

GenericSearchFilter() handles that scenario. First the search term is parsed into individual terms if there is a ” ”  supplied in the string. This means we will have to perform the propertyQuery for each term that we have. To return all of the records that correspond to each term we still need to perform the OR in groups, but then we need to AND these predicates together so we can get all of the groups per individual term. Here’s the source edited slightly for readability:

//  Split search expression to handle multiple words
var searchTerms = this.genericSearch.Split(' ');

for (int i = 0; i &lt; searchTerms.Length; i++) {    var searchExpression = Expression.Constant( searchTerms[i].ToLower());                   //  For each property, create a contains expression   //  column =&gt; column.ToLower().Contains(searchTerm)
  //  Edited for clarity - create the array propertyQuery logic is here ...
  var propertyQuery = ...

  //  Inner loop for grouping all OR's for this search term
  for (int j = 0; j &lt; propertyQuery.Length; j++)   {     //  Nothing to &quot;or&quot; to yet     if (j == 0)     {       compoundOrExpression = propertyQuery[0];     }     compoundOrExpression = Expression.Or(compoundOrExpression, propertyQuery[j]);   }   //  First time around there is no And, only first set of or's   if (i == 0)   {     compoundAndExpression = compoundOrExpression;   }   else   {     compoundAndExpression = Expression.And(compoundAndExpression, compoundOrExpression);   } } 

So GenericSearchFilter will build a humongous expression tree for all the properties in your class. To make this usable for the Where we convert it using Expression.Lambda and our Where clause just goes about its merry way. Because we have used generics, you can supply any class from your assemblies. One caveat, and Sensei is trying to find a resolution. If you have a string property to that is set to null, the expression tree fails. You’ll note that in the classes supplied in the sample, the properties that are of type string in the Tenant class are defaulted to empty in the constructor.  A small price to pay for some great functionality. To sort our data we use the method ApplySort():

 private IQueryable ApplySort(IQueryable records)         {             string firstSortColumn = this.sortKeyPrefix.First();             int firstColumn = int.Parse(firstSortColumn);             string sortDirection = &quot;asc&quot;;             sortDirection = this.aoDataList.Where(x =&gt; x.Name == INDIVIDUAL_SORT_DIRECTION_KEY_PREFIX +
                                                                      &quot;0&quot;)
                                                .Single()
                                                .Value
                                                .ToLower();

            if (string.IsNullOrEmpty(sortDirection))
            {
                sortDirection = &quot;asc&quot;;
            }

            return records.OrderBy(this.properties[firstColumn].Name, sortDirection, true);
        }

An extension method OrderBy will accept the name of column, the sort direction as parameters. The parameter initial will indicate if we are sorting mulitple times, so we can accomplish multi-property sort with syntax like

var sortedRecords = records.OrderBy(&quot;State&quot;, &quot;desc&quot;, true)
                                      .OrderBy(&quot;City&quot;, &quot;asc&quot;, false);

public static IOrderedQueryable OrderBy(this IQueryable source, string property, string sortDirection, bool initial)
        {
            string[] props = property.Split('.');
            Type type = typeof(T);
            ParameterExpression arg = Expression.Parameter(type, &quot;x&quot;);
            Expression expr = arg;
            foreach (string prop in props)
            {
                // use reflection (not ComponentModel) to mirror LINQ
                PropertyInfo pi = type.GetProperty(prop);
                expr = Expression.Property(expr, pi);
                type = pi.PropertyType;
            }
            Type delegateType = typeof(Func&lt;,&gt;).MakeGenericType(typeof(T), type);
            LambdaExpression lambda = Expression.Lambda(delegateType, expr, arg);

            string methodName = string.Empty;

            //  Asc or Desc
            if (sortDirection.ToLower() == &quot;asc&quot;)
            {
                //  First clause?
                if (initial &amp;&amp; source is IOrderedQueryable)
                {
                    methodName = &quot;OrderBy&quot;;
                }
                else
                {
                    methodName = &quot;ThenBy&quot;;
                }
            }
            else
            {
                if (initial &amp;&amp; source is IOrderedQueryable)
                {
                    methodName = &quot;OrderByDescending&quot;;
                }
                else
                {
                    methodName = &quot;ThenByDescending&quot;;
                }
            }

            object result = typeof(Queryable).GetMethods().Single(
                    method =&gt; method.Name == methodName
                            &amp;&amp; method.IsGenericMethodDefinition
                            &amp;&amp; method.GetGenericArguments().Length == 2
                            &amp;&amp; method.GetParameters().Length == 2)
                    .MakeGenericMethod(typeof(T), type)
                    .Invoke(null, new object[] { source, lambda });
            return (IOrderedQueryable)result;
        }

All good things …
It’s been a long ride, this post. A lot of code discussed, a lot of ground covered. The solution is here.  As always, play around and see how this can help you. If anything breaks, tell Sensei. If you have improvements, tell Sensei. DataTables is a great tool for your arsenal, hopefully the DataTablePager can help you integrate paging with large datasets as part of your solution offering.

Right now Sensei wants to sign off by toasting to you for wading through all of this, and for having the desire to build up your skills.  Obtaining fluency in what you do is a hard road to travel, but it’s worth it because you get things done quicker and better with each session.

Be sure to read about the latest version in “Dynamically Select Columns with Server-Side Paging and Datatables.Net

ActiveEngine Software by ActiveEngine, LLC.