Since CSVs are generally flat we are explicitly in what we fetch (see $fields variable) this causes Raven to pull only from Lucene index (a thus a lot faster) rather than using the index as doc id lookup. To get full compatibility with Excel you have to be a little careful with encoding, otherwise you have to fallback to the import wizard, with ascii it is double click open, fortunately in this instance we live within the ascii charset. 

At GistHub: http://gist.github.com/548970

$baseUrl = "http://localhost:8080"

$outputFile = "salesexport.csv"

$fields = "Manufacturer,ProductName,RetailPrice,ProductCode"

$fieldsArray = $fields.Split(',')

$startAt = 0;

$pagingSize = 100

# Build your Index in RavenDB - best to do your hard work there.

# script  supports flat indexes so we use fields in query to ensure it pulls

# from Lucene rather than a Lucene to DocID lookup - and again much faster

[System.Reflection.Assembly]::Load("System.Web.Extensions, Version=3.5.0.0," `

+ "Culture=neutral, PublicKeyToken=31bf3856ad364e35")

$serializer = New-Object System.Web.Script.Serialization.JavaScriptSerializer

$serializer.MaxJsonLength = [System.Int32]::MaxValue # probably too laxed

$webClient = New-Object System.Net.WebClient

#clear the file and write the columns

#using ascii since Excel presumes CSV and has problem open .csv directly when BOM 

#present. if you need unicode use utf8, save file as .txt and use wizard

[String]::Join(",",($fieldsArray | %{ ('"' + $_ + '"')})) | Out-File -Encoding ascii $outputFile

do {

$indexUrl = "$baseUrl/indexes/Shop/Brochure?start=$startAt&pageSize=$pagingSize&" `

    + [string]::Join("&",($fieldsArray | % { "fetch=$_" } ))

$response = $webClient.DownloadString($indexUrl)

$jsonDic = $serializer.DeserializeObject($response)

$results = $jsonDic["Results"]

$script:totalResults =$results.Length

foreach($result in $results)

{

# ravendb includes the __document_id in the ouput as the last key so 

# we select only fields we specify by using a select -first

[string]::Join(",", (% { $result.Keys } | select -first $fieldsArray.Length |`

% { ('"' + $result[$_].Replace(",","\,") + '"') } )) `

| Out-File -Encoding ascii $outputFile -Append

# if you want custom mapping just use $result[column-name] to build your own

}

$startAt = $startAt + $pagingSize

} while ( $script:totalResults -ne 0 )

Again we use the .Net serializer rather than Newtonsoft.Json due to strange errors that I dont have time to troubleshoot, this exports more than 20,000 records in less than a minute (on a modest desktop). Enjoy

 
(simple google docs drawing)

Initially I played with Rhino ETL but even that was overkill (and a little slow – but a beautiful framework) so inspired from recently looking at psake I decided to see how far I could get with Powershell sanely. The great thing about using a admin-friendly scripting language like powershell if that is easy to modify if your schema changes and it’s extremely easy to deploy, although it might not be the best code. Due to Powershell’s full .Net support and RavenDB’s restful interface it was pretty easy to get this up and running – and a good opportunity to further enhance my Powershell skills.

Here’s the result – a rough and simple ETL powershell script for moving data from a SQL Server table to RavenDB, it tries a PATCH first incase the doc already exists, failing that it will do a PUT, since given other properties are set from different sources an overwrite of the document is not going to work for us.

# requires curl on PATH - if you installed mysysgit with it added to all PATHS then

# you should be okay, otherwise download from http://curl.haxx.se 

# presume integrated security edit connection otherwise

$sqlserver = ".\SQLEXPRESS" 

$db = "DATAVENDOR_CAR"

$baseUrl = "http://localhost:8080"

# first value selected must be the doc id - ideally create dedicated view instead

$sqlQuery =  "SELECT CarId, Manufacturer, Model," `

  + "Title, BodyStyle, Doors, CO2, Trim, DriveTrain, Transmission," `

  + "FuelType, EngineSize, ManufacturerRetailPrice, SourceUpdated FROM CarView"

$connection_string = ("Data Source=$sqlserver; Initial Catalog=$db;Integrated Security=SSPI")

$conn = New-Object System.Data.SqlClient.SqlConnection($connection_string)

# newtonsoft.json created weird errors 

[System.Reflection.Assembly]::Load("System.Web.Extensions, Version=3.5.0.0, Culture=neutral,

 PublicKeyToken=31bf3856ad364e35")

$serializer = New-Object System.Web.Script.Serialization.JavaScriptSerializer

$conn.Open()

$cmd = $conn.CreateCommand()

$cmd.CommandText = $sqlQuery

$reader = $cmd.ExecuteReader()

$derivativeIds = New-Object 'System.Collections.Generic.List[string]'

while ($reader.Read()) {

    $docid = $reader.GetValue(0)

    $url = ([System.Uri]("$baseUrl/docs/" + $docid)).AbsoluteUri.ToString();

    # build a list of SET operations - either update or create property

    # only affects properties existly set in the operations, does not overwrite 

    # whole document. 

    $listOfSetOperations = New-Object 'System.Collections.Generic.List

[System.Collections.Generic.Dictionary[string,object]]'

    for ($i = 1; $i -lt $reader.FieldCount; $i = $i +1)    { 

        $dic = New-Object 'System.Collections.Generic.Dictionary[string,object]'

        $dic.Add("Type","set")

        $dic.Add("Name",$reader.GetName($i))

        $dic.Add("Value",$reader.GetValue($i))

        $listOfSetOperations.Add($dic)

     }

    $jsonPayLoad = $serializer.Serialize($listOfSetOperations).Replace("`"","'")

    $response = curl -w '%{HTTP_CODE}' -X PATCH $url "-d $jsonPayLoad"

    # doc does not exist so lets insert it. I go this route rather than a single update-or-insert PUT 

    # because if the doc already exists on the ravendb it will have additional properties associated with

    # that have been added/modified since (i.e. new properties dont exist in this sql source db)

    if($response -eq '404')  {

        $dictionary = New-Object 'System.Collections.Generic.Dictionary[string,object]'

        for ($i = 0; $i -lt $reader.FieldCount; $i = $i +1)    {

            $dictionary.Add($reader.GetName($i),$reader.GetValue($i))

         }

        $dictionarySerialized = $serializer.Serialize($dictionary).Replace("`"","'") 

        curl -X PUT $url "-d $dictionarySerialized"

    }

}

Example also available on gisthub here, adapt it per your needs, http://gist.github.com/543499.

I had strange errors relating to Value property with Newtonsoft.Json, so I fell back to .Net Javascript serializer. We could additionally clean this up by using WebClient instead of relying on curl. And of course this is void of any real error handling or performance analytics and as junior to Powershell any best-practices, feel free to extend it, adapt it, it isn’t meant to be beautiful code so don’t punish me too hard.

Later in the week I plan to batch the requests into groups of 50 (or a configurable amount), per the documentation at http://ravendb.net/documentation/docs-http-batching-2 this should significantly improve throughput, I’ll update the script on gisthub.

I used PowerGUI script editor as the main IDE (although it does crash a lot – have process explorer / task manager handy), but for a free tool it’s a good start. For debugging Raven communication add –v parameter to the curl command, or use Fiddler, or plugin WebClient with some verbosity. Also note, by default RavenDb requires authentication for anything other than GET requests so you’ll need to either change the behaviour via the config file or ensure your http requests are authenticated. 

My spiking experience this afternoon.

When doing CQRS with a message bus behind the controller, you’ll be tempted to think the only solution is do fire and forget-for-now (remaining completely pure), but often the command system will be processing commands in a fraction of a second, if you hang about a few hundred milliseconds you will probably be able to know if the command succeed or fails, without this you either assume success (fake the result based on the command) or can at the best presume in progress (eg. Amazon – Processing your order – We will email soon).

Traditionally if you are waiting for an expensive  operation in the web layer could be dangerous, you could easily exhaust the thread pool if the operation is long running (even a few hundred milliseconds could be exhausted with a few hundred or so concurrent connections).

However with a true asynchronous programming model you need not keep the thread when the operation is not CPU bound.  In the case of file IO and socket IO, the responsibility is moved to the device drivers (a operating system I/O completion port), when the operation is completed, an event loop will reawaken a thread to continue processing.

In a nutshell if your ASP.Net request is waiting half a second for a response from the command layer it need not be taking up a thread (presuming the right APM practices are followed), when the non CPU bound operations completes an internal event loop will grab a thread (which may or may not be the same that setup the request) and continue processing. This is very similar to asynchronous programming model that has made NodeJS and Twisted on Ruby popular in recent months but the same technology has existed in .Net since 2.0 and some parts since 1.1 (albeit not cleanly in the web layer).

In NServiceBus the event loop is the NServiceBus worker threads that are created when the bus is started, when a reply arrives on the bus that correlates with message sent it will execute the callback provided (if one exists), in our case this signals to AsyncManager to grab a thread, restore the context (request etc..) and continue processing – calling the XYZCompleted event (with the args the callbacks gives to the AsyncManager).

My example is using the Async controller functionality in ASP.Net MVC 2.0.  I am using NServiceBus to Send* a message to a specific endpoint and wait for the reply.  Without Async controller functionality we would have to rely on a more inefficient methods such ManualResetEvent which will keep one of your precious ASP.Net web-worker threads tied to the one request, even if you push it back to a ThreadPool it is still taking a thread until you embrace a true APM (asynchronous programming model).

* (note: we dont do pub/sub from the web layer, we push the command in to the system where, when processed, may or may not generate a serious of events that then can be subsequently publish to interested subscribers)

using System;

using System.Linq;

using System.Web.Mvc;

using NBusEcoClient.Messages;

using NBusEcoClient.Web.Filters;

using NServiceBus;

namespace NBusEcoClient.Web.Controllers

{

    public class BookPurchaseController : AsyncController

    {

        private readonly IBus bus;

        public BookPurchaseController(IBus bus)

        {

            this.bus = bus;

        }

        [AsyncTimeout(250)]

        [SwallowNsbTimeoutAsCompletionResult]

        public void OrderAsync(OrderCommand orderCommand)

        {

            AsyncManager.OutstandingOperations.Increment();

            if (orderCommand.IsValid())

            {

                bus.Send(orderCommand).Register(ar =>

                                        {

                                            AsyncManager.Parameters["completionResult"] =

                                                ar.AsyncState as CompletionResult;

                                            AsyncManager.OutstandingOperations.Decrement();

                                        }, orderCommand);

                return;

            }

            AsyncManager.Parameters["completionResult"] = 

                new CompletionResult {ErrorCode = (int) CommandResult.Failure};

            AsyncManager.OutstandingOperations.Decrement();

        }

        public ActionResult OrderCompleted(CompletionResult completionResult)

        {

            if (completionResult.ErrorCode == (int) (CommandResult.NoError))

            {

                return OrderSuccessful();

            }

            if (completionResult.ErrorCode == (int) (CommandResult.StillProcessing))

            {

                // Not received a reply for > 250ms, perhaps cmd servers are  busy.

                // We'll return in progress - command received and valid

                return OrderInProgress();

            }

            if (completionResult.ErrorCode == (int) (CommandResult.Failure))

            {

                // Command has failed – is it a domain failure or not. // missing: affect on tranx?

                OrderNotAcceptedMessage domainError = completionResult.Messages

                    .Where(m => m.GetType().Equals(typeof (OrderNotAcceptedMessage)))

                    .Cast<OrderNotAcceptedMessage>().FirstOrDefault();

                return OrderFailure(domainError != null ? 

                    domainError.ErrorMessage : "Generic error");

            }

            throw new ArgumentException("Unexpected completion result: " + 

                completionResult.ErrorCode);

        }

        protected ActionResult OrderFailure(string message)

        {

            ViewData["Error"] = message;

            return View("OrderFailure");

        }

        protected ActionResult OrderSuccessful()

        {

            return View("OrderProcessed");

        }

        protected ActionResult OrderInProgress()

        {

            return View("OrderInProgress");

        }

    }

}

Because our controller inherits from AsyncController rather than Controller we split our action over the non CPU bound operation (i.e. waiting for a correlated reply) , in this case Order is called just like a synchronized action, the AsyncController takes care of choosing the proper method dispatcher based on XAsync  and XCompleted where X is the action  name.

Because we actually expect the timeout to be non-fatal occurrence (infact it will probably be regular) we need to coerce the TimeoutException and handle it as a non-exception. One could use an ActionFilter like this to do this:

 public class SwallowNsbTimeoutAsCompletionResult : ActionFilterAttribute

    {

        public override void OnActionExecuted(ActionExecutedContext filterContext)

        {

            bool hasTimeoutException = (filterContext.Exception as TimeoutException) != null;

            if( hasTimeoutException)

            {

                var actionDescriptor =

                    filterContext.ActionDescriptor as ReflectedAsyncActionDescriptor;

                filterContext.ExceptionHandled = true;

                filterContext.Result = actionDescriptor.CompletedMethodInfo

                            .Invoke(filterContext.Controller

                                    , new object[]

                                            {

                                    new CompletionResult

                                        {

                                            ErrorCode = (int) (CommandResult.StillProcessing)

                                        }

                                }) as ActionResult;

            }

        }

    }

If you dont extend further then in simple examples you can simplify and use this instead:

[HandleError(ExceptionType = typeof(TimeoutException), View = "InProgress")]

A more tricky step that remains is to clear out the correlated map on the IBus, this will leak and we are no longer interested in correlating the response to the callback* after the timeout as a view would have been returned to the client and connection lost,  (ajax scenarios may differ will polling (and more so with websockets).

* note: you may still be interested in the response but just not the callback handling it

I haven’t got a great way to do this yet since the field is private but here’s a snippet to get you going (maybe schedule a cleanup every minute). If only NServiceBus was on GitHub, I’ve uninstalled SVN now

 var messageIdToAsyncHandlerPrivateProperty = bus.GetType()

                            .GetField("messageIdToAsyncResultLookup", BindingFlags.Instance | BindingFlags.NonPublic);

                    var mappingDictionary = messageIdToAsyncHandlerPrivateProperty.GetValue(bus)

                        as IDictionary<string,NServiceBus.Unicast.BusAsyncResult>;

                    lock(mappingDictionary) {

                        var toRemove = mappingDictionary.Where(busResult => true);  

                        // change predicate to something like to determine non interested one

                        // busResult => ((IMessageWithExpiringCallBackInterest)(busResult.State)).Sent > 1sec

                        foreach(var removeSubject from toRemove)

                            mappingDictionary.Remove(removeSubject);

                    }

I have asked UdiDahan about making this protected so we can extend UnicastBus to handle such scenarios.

If you are interested in the reply after the response has gone then add a normal Handler and make sure the message handler is associated when the bus is created, both the callback and handler will fire (the callback first if exists), you may need to set a flag (isHandled) in the header of current message context if both are going to do similar behaviours. However don’t use a handler on the web layer to for example send “order failed” email, this should be published from the domain layer and something subscribes to it and publishes, maybe in same process or maybe different process/node communicated pub/sub with NServiceBus.

On service side you return a result and optionally a message, NServiceBus sends these as one CompletionResult (or at least aggregates them into one). For example:

// success

            bus.Return((int)CommandResult.NoError);

            // command failure

            bus.Reply(new OrderNotAcceptedMessage() { ErrorMessage = "Credit not available" } );

            bus.Return((int)CommandResult.Failure);

Disclaimers:

• This is not production code, it was just a dirty spike, Im not an expert, I may be wrong, I wouldnt use this code in production!
• MVC: You obviously don’t do string magic like that, you should probably embrace Thunderdome principle and you could probably move a lot of NSB/Async logic to something like MyOpinionatedAsyncController.
• Commands. No distinguish between domain error and failures, infact the wording is probably wrong here, and I need to investigate further about scenarios where transaction is rollbacked.
• This is just an eye opener to some techniques, I hope you learnt something

References:

• Asynchronous programming chapter in C# via CLR 2nd edition
• Memory leak for uncalled callbacks in NSB
• NServiceBus documentation
• Using an Asynchronous Controller in ASP.NET MVC
• Async source sample from MSDN

Twitter folks that discuss CQRS, NserviceBus, regularly etc.. mattcodes (me), UdiDahan (author of NSB), @andreasohlund, @Daneel3001, @neilrobbins, @pvjds

Take the following C’# snippet I’ve cooked up: (1)

using System;

using System.Collections.Generic;

namespace ConsoleApplication1

{

    internal class Program

    {

        private static void Main(string[] args)

        {

            var myIntegerPrinters = new List<Action>();

            for (int i = 0; i < 5; i++)

            {

                int currentIndex = i;

                myIntegerPrinters.Add(() => Console.WriteLine(currentIndex));

            }

            foreach (Action intPrinter in myIntegerPrinters)

            {

                intPrinter.Invoke();

            }

            Console.Read();

        }

    }

}

This code (1) prints out 0, 1, 2, 3, 4 as expected, since we create a closure around currentIndex which is set on each iteration of the loop to the current value of i (copied by value). 

Now try this in JavaScript (Firebug console window will suffice): (2)

            var myIntegerPrinters = [];

            for (var i = 0; i < 5; i++)

            {

                var currentIndex = i;

                myIntegerPrinters.push(function () { console.log(currentIndex); } );

            }

            for each (var intPrinter in myIntegerPrinters)

            {

                intPrinter();

            }

You might expect this to produce the same result as the C# code as we too create a closure around currentIndex, however this snippet prints out 4,4,4,4,4. Yikes.

This had me puzzled for a minute or two and then it dinged.

JavaScript does not have block scope, only function scope, and this is critical to remember not just for where you define your variables but also in the case of closures considerations.

The code below (3) is by meaning the same as (2), for illustration we move the currentIndex out of the for loop block, essentially due to absence of block scope this how JavaScript interprets the code from (2), as you can see each closure will reference the same currentIndex as the others, which is incremented up until 4.

            var currentIndex = 0;

            var myIntegerPrinters = [];

            for (var i = 0; i < 5; i++)

            {

                currentIndex++;

                myIntegerPrinters.push(function () { console.log(currentIndex); } );

            }

            for each (var intPrinter in myIntegerPrinters)

            {

                intPrinter();

            }

Of course, this prints out 4,4,4,4,4. In listing (2) despite it appearing as new declaration, JavaScript just silently ignores the repeated var and refers to the current currentIndex in the scope (the function scope). C#, the strict language that it is, due to its block scope, would produce an error that it cannot be declared due to different meaning in the scope etc.. so you have a safeguard from the compiler there which JavaScript does not have.

Inner function scope to the rescue, as Robert Nyman illustrates (in between all the crazy celebrity lust – what is all that about?), you can overcome this like this:

var myIntegerPrinters = [];

    for (var i = 0; i < 5; i++)

    {

       myIntegerPrinters.push((function (currentIndex) { 

            return new function() { console.log(currentIndex); };

         }(i)));

    }

    for each (intPrinter in myIntegerPrinters)

    {

        intPrinter();

    }

This of course, due to function scope, prints out 0,1,2,3,4. And voila we have achieve the same as the C# code in listing (1) albeit with a little more work. 

Just a heads up for C# devs embracing JavaScript and its coolness, be careful of the absence of block scope, JavaScript won’t tell you or warn you and it could be that bug that you spend half a day on See ØreDev and Robert Nyman talk for more JavaScript goodness.

NodeJS (http://www.nodejs.org) is an event driven non-blocking (all operations are non-blocking) JavaScript hosting engine, very lightweight and very fast – due to its reliance on the performant Google V8 engine, it’s also very easy using NodeJS to expose JSON data services in just a dozen lines of code.  The non-blocking architecture of NodeJS might seem unconventional at first compared to C#/Java, especially the heavy use of callbacks everywhere, but essentially it means that you no longer have threads waiting for IO completion operations (this handed off the OS and underlying interrupts). NodeJS is single threaded – but when the thread is running it is entirely non-blocking thus will complete as fast as your processor can handle, this single threading nature doesn’t limit capacity but the exact opposite, thread context switching is expensive.

In my example, I collect the arrival details of airplanes destined for the beautiful Koh Samui every five minutes and place them into a global cache, the request logic then just reads the cache and spits out a JSON representation, from the perspective of the client the operation is as fast as one can achieve.

Given the time I spent in the early days of my career writing scrapping software (in C#) for a popular money comparison site, I thought it would be nice to revisit the area in a more dynamic event fashion, thus the collection operation in my case is web scraping, but this could easily be any other procedure such as database* or file reads, again asynchronous in nature out the box. *support varies – PostgreSQL seems to be the choice at the moemnt.

If you’re familiar with JavaScript in the browser then the remainder of code should be identical to what you’d use elsewhere, i.e.  basic language constructs, regular expressions, nature of callbacks and closures etc..

For the page retrieval logic, due to the way flightstats.co.uk splits the data by 3hr period on their site, we have to fire off eight page requests to get an overview of the full 24hrs, this is all done asynchronously and afforded by the event driven nature of NodeJS, no need to explicitly setup worker threads or a thread pool, just pure simplisticity.

Creating a server endpoint to publish the data takes just 5 lines. A JSON stingify utility is already included with NodeJS as an added treat (very useful for debugging too – output the whole object).

Note: To build NodeJS you’ll need to download the latest package from http://nodejs.org/ and follow the installation instructions. On Ubuntu Linux you’ll also need to install cmake and buildessentials from apt-get before you install NodeJS so that you have the cxx compiler.

Note: This code is exampleware to demonstrate the power of NodeJS, there is no error handling, the regular expressions are fragile, and flightstats.co.uk already expose developer friendly API via web services that should be used instead of regular expression.

var sys = require('sys')

var http = require('http');

(function(options) {

  var arrivingFlights = new Array();

  var util = { trim: function( text ) {

        return (text || "").replace( rtrim, "" );

    }};

  var rtrim = /^(\s|\u00A0)+|(\s|\u00A0)+$/g

  var baseUrl = "/go/weblet?guid=c228b59beca1b817:44617f82:117d601a4c6:221f&weblet=status&action=AirportFlightStatus&airportCode=USM&airportQueryType=1";

  var host = "www.flightstats.com";

  var refer = "http://www.samuiairportonline.com/flight-status";

  var userAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6";

  var retrieve = function(url,complete) 

  {

      var flightStat = http.createClient(80, host);

      var request = flightStat.request("GET", url,

                      {"Host": host

                    , "Referer": refer

                    , "User-Agent": userAgent });

      request.addListener('response', function(response)

      {      

      response.setBodyEncoding("utf8"); 

      var body = "";

      response.addListener('data', function(chunk)  {   body += chunk;  });

      response.addListener('end', function()  {   complete(body);  });

      });

      request.close();

  }

  var updateFlightTimes = function() 

  {

    var completedCounter = 0; // reset 

    var lastArrivingFlights = new Array();

    // copy over the global collection only after all 8 result sets have returned

    var completed = function() {   if(completedCounter++ && completedCounter == 8 )    arrivingFlights = lastArrivingFlights;      };

    var processTable = function(page)

    {

      var tableContent = page.match(/class="tableListingTable"[^>]+>([^]*)<\/table>/gmi)[0];

      var rowRegExp =  /<tr>\s*?<td[^>]*?>\s*?<a[^>]*?>([^<]*?)<\/a>\s*?(?:<span[^>]*>.*?<\/span>)?\s*?<\/td>\s*?<td[^>]*?>([^<]*?)<\/td>\s*?<td[^>]*?>\s*?\(\s*?<a[^>]*>.*?<\/a>\s*?\)([^<]*?)<\/td>\s*?<td[^>]*?>([^<]*?)</mgi;

      while(rowMatch = rowRegExp.exec(tableContent)) 

      {

    lastArrivingFlights.push({ code: util.trim(rowMatch[1]), 

                  carrier: util.trim(rowMatch[2]), 

                  origin: util.trim(rowMatch[3]),     

                    arriveTime: util.trim(rowMatch[4]) });    

      }

      completed();     

    };

    // full 24hr period - have to pull 8 pages back

    for(var i = 1; i <= 8; i++)

    {

      retrieve(baseUrl + "&airportQueryTimePeriod=" + i,processTable);

    }

  }

  updateFlightTimes();

  setInterval(updateFlightTimes,options.updateInterval); 

  http.createServer(function (req, res) {

      res.writeHead(200, {'Content-Type': 'application/json'});

      res.write(JSON.stringify(arrivingFlights));

      res.close();

  }).listen(8000);

})({ updateInterval: 300000 });

Code is also on fixee.org at http://fixee.org/paste/88y4i22/

I really like Closures, however I hate debugging when it involves closures, for the simple fact that my favourite JavaScript tool Firebug with Firefox has severe problems evaluating (during a break) on variables that form part of the closure.  An unhelpful “Reference Error: x is not defined” is all we get, where x is the variable name.  Hovering over the variable or using the command window does not work either.

An experiment on previous versions of FF/FB and in Ubuntu linux was even more disappointing with “break on all errors” not working correctly either, so from what I understand this is a non-platform specific issue (please correct me otherwise).

I use this simple code to illustrate (jQuery dependency) the issue with debugging closures:

$(function()

  {

     var myName = "matt";

      var otherNames = ["ted","john","matt","fred"];

     $.each(otherNames, function()

           {

                   var result;

                   if(this == myName)

                   {

                      result = this + " is SAME as "  + myName;

                   } else {

                       result = this + " is not same as " +  myName;

                   }

                    $('<p>' + result + '</p>').appendTo('#matchResults');

           });

  });

​

Note: For the jQuery savy amongst us I concede that this is overly verbose.

For the paste bin complete with fancy HTML click here.

Here’s the result in Firefox 3.6:

As you can see the runtime (Firefox JavaScript engine – TraceMonkey) can correctly evaluate myName (otherwise we’d never branch into the if  block) but unfortunately either due to a Firebug bug or the optimization that the TraceMonkey engine makes we can not see the value in the Watch window. Some have suggested that due to optimizations there is not enough profile data to determine myName from the debugging environment.

Google Chrome and its V8 engine does much better in this respect, not only can it evaluate myName, but it also gives me a breakdown of what variables are local and which are part of the closure. Sweet. Unfortunately in this instance, it wouldn’t break on the breakpoint I set, but at least it stopped on the next available point (outside the if/else block), presumably this is because Chrome’s V8 engine was able to optimize the branch (maybe due to the data being available at load time). Chrome also does syntax highlighting on JavaScript – just not on inline scripts.

Next up, Internet Explorer 8, I didn’t expect much from the IE team (after a let down with IE6 and IE7), but their new Developer Tools proved me wrong, loaded very fast, evaluated myName correctly and even nice color syntax highlighting, you can tell they have been influenced by Firebug design but why not? It works very nice.

I include Opera 10.5 for good measure however despite their trumpeting about DragonFly I found it very poor like a pre v1 beta of Firebug. One can not add anything to the watch window, can not hover on source code to expose values, I even had to switch to the command window to evaluate myName (at least it did better than Firebug in this case though).

I know the Apple fanboys would hit me with their new iPads if I didn’t include Safari 4, I wish I had suitable VM to download this bloat to (30.2mb just for a browser), the scarifies I make in the name of blogging.  But hold up, this looks familiar, the same Web Inspector as Chrome – I always thought WebKit was just the renderer. But something interesting has happened unlike Chrome, this breaks in the correct place, and reveals the closure variable, this is exactly what I wanted. This may be a postitive side-effect that Safari’s JavaScript engine is less optimizing than Chrome’s V8 but for debugging I can spare a few extra nanoseconds. I eat my words, Safari for JavaScript debugging is pretty good. Like Chrome, Safari also does syntax highlighting on JavaScript – just not on inline scripts.

In conclusion, there are more tools for debugging JavaScript against the DOM than just Firebug, whilst Firebug is still my favourite at least I’ve discovered some alternatives for when Firebug just can’t deliver. Surprisingly Safari is now my first backup choice.

I’d be interested in hearing workarounds (other than promoting the variable to global) and possible configuration options to disable JavaScript optimizations when debugging – please leave in the comments.

I’ve just finished working on a service which exports information from a legacy database once per day, in a nutshell retrieve, map, publish.

In a four-step process it works as follow:

• Query the data layer for records modified after last run date (~24hrs)
• Load the records into a domain model (responsibility of NHibernate)
• Convert the collection to our flat shared schema DTOs (AutoMapper)
• Publish the messages to the bus (NServiceBus / MSMQ)

Whilst performance wasn’t a noted priority it still had to complete the daily job within a few hours and with the resources a basic server afforded. In the initial stages of implementation, I had my integration tests running against a SQLite database and performance wise everything was progressing smoothly. However when we pointed it against a production database (SQL Server) with a six figure number of aggregate instead of less than 100 for test data we soon hit problems, dreaded out of memory exceptions. 

It didn’t take long to pinpoint the issue was related the data access strategy, bringing all the entities out of the database at once made the Session balloon at shocking rate, in a hindsight a bad approach to begin with. This was a showstopper, and with it approaching 5pm, and happy hour on the beach finishing at 7pm, I could already see this turning out to be disappointing.

I contemplated a few ideas, changing some settings with regards to mutability (we are doing read-only), switching to an IStatelessSession, and even polluting the consumer to use some paging mechanism…. I played around for while and then I thought about a yield inside the service layer, essentially implementing my own iterator to keep the unit of work small and thus underlying NHibernate session minimal.

In my service layer and wherever possible I favour IEnumerable over IList, for the definition I had:

IEnumerable<Derivative> GetDerivativesWithPriceChanges(DateTime laterThan)

And for the code (refactor due):

  public IEnumerable<Derivative> GetDerivativesWithPriceChanges(DateTime laterThan)
        {
            Expression<Func<Derivative, bool>> predicate = d => d.PricingUpdated > laterThan;

            int bufferSize = 10; // make configurable, inject
            int marker = 0;
            int lastAmountRetrieved = 0;

            do
            {

                using (UOW.Start())
                {
                    var selection
                        = new Func<IQueryable<Derivative>, IQueryable<Derivative>>
                            (d => d.Skip(marker).Take(bufferSize));

                    var derivatives = derivativeRepository.GetByQuery(predicate, selection).ToList();

                    lastAmountRetrieved = derivatives.Count;

                    foreach (var derivative in derivatives)
                    {
                        yield return derivative;
                    }
                }

                marker = marker + bufferSize;

            } while (lastAmountRetrieved == bufferSize);

            yield break;

        }

)

Inside GetDerivativesWithPriceChanges, I’m paging through the database opening and closing the Unit of work after a specified number of entities are returned. Note: UOW system provides the underlying ISession to the repositories, although my UOW differs slights I recommend analysing Ayende’s Rhino Commons for a good example.

Best of all the consumer of the service is none the wiser, the code for paging was already on the base repository (as simple as Skip(x).Take(y) with NHibernate.Linq), and as an added benefit we have quicker feedback to our logging system and on our integration tests.  It’s certainly not the purist implementation, but since I was working with read-only data it turned out perfect for this scenario and I was enjoying a Heineken by the beach by 6.30pm. I imagine I will eventually refactor the some of the logic for readonly paging/uow streaming to its own class, but for today, check-in, and close solution.

A few lessons learnt:

1. Memory is not never infinite.
2. Long running operations are bad for integration tests, these tests are often already a magnitude slower than unit tests so keep them trim.
3. Always consider the data and quantity of which you’ll be working with in production. If unsure, ask.
4. The problem is rarely solely NHibernate but usually the high-level approach taken to tackle the problem, don’t damn the NHibernate Session.

In the past I’ve hated leaking ICriteria/DetachedCriteria out of the repository, in addition with a lot of the business logic being pushed down into infrastructure code it always felt a little dirty.  Whatever approach I’ve taken there has always been some aspect that didn’t feel right, I’ve always been concerned that I’ve have NH—>DB rather than it just being an implementation of the infrastructure that’s wired in at the end, thankfully with NHibernate and Linq there is some light at the end of the tunnel. I’ve haven’t finalised my approach yet, I’ve still got some internal debating to do about location of the query and services and possibly the integrating the specification pattern (now it’s easy to consume), but I think I’m going to take a slight reversal on my previous post.

This is what I’m liking.

public virtual IEnumerable<T> GetByQuery
            (Expression<Func<T, bool>> predicate,
             Func<IQueryable<T>, IQueryable<T>> selectionPredicate)
        {
            using (ITransaction tx = Session.BeginTransaction())
            {
                try
                {
                    IQueryable<T> query = Session.Linq<T>().Where(predicate);
                   return selectionPredicate(query);
                }
                catch (HibernateException hibernateException)
                {
                    tx.Rollback();
                    throw new RepositoryException(hibernateException);
                }
            }
        }

In my service layer or outer repository etc..:

            Expression<Func<Hotel, bool>> predicate = hotel=>
                  hotel.Stars > 3 && hotel.Country == Country.UK

           Func<IEnumerable<Hotel>,IEnumerable<Hotel>> selectionPredicate = col =>
                          col.Skip(10).Take(10); // pagingSetup.GetQueryForPage(requestedPage);

            var hotels = hotelRepository.GetByQuery(predicate, selectionPredicate);
            return hotels;
   

I fired up Ayende’s cool NH Profiler and the above does get converted to a single neat parameterized SQL statement with correct selection of the paging. This also significantly aids in testability as injecting a in-memory repository in place of NH into the service layer is now super easy just execute the expressions as is on the collection (I won’t even post the code for that)…  In future I might not even be working with NHibernate – so this approach gives me great flexibility with no query logic trapped in HQL or the NH criteria api – who knows – it could be couchdb etc..

Creation

NHibernate requires each mapped type to have a parameterless constructor – However, this parameterless constructor does not need to pollute your domain model as it can be hidden from external code by marking it with protected or protected internal whilst your parameterized constructors can be public , meaning you can still enforce integrity at the constructor level, thus never allowing for creation of invalid entities. If for some reason you never want to create a Dog without a date of birth you could enforce this whilst still using NHibernate by doing something as simple as this:

public class Dog
{
    protected internal  Dog() { }

    public Dog(DateTime dateOfBirth)
    {
        this.dateOfBirth = dateOfBirth;
    }

    private DateTime dateOfBirth;
    public DateTime DateOfBirth
    {
        get
        {
            return dateOfBirth;
        }
        protected set
        {
            dateOfBirth = value;
        }
    }
}

(You could optionally use an automatic property if using C# 3.0)

Where you don’t require a separate factory I’ve personally taken to creation static methods to give some more semantic meaning to the creation. Here’s a trivial example, but this style really comes into its own when you have multiple constructor parameter sets that have the same signature, not only do the static method names gives meaning, but you can’t vary a constructor unless you vary its parameter set too.

        public static Dog CreateFromEstimatedAge(int estimatedAge)
        {
            var dog = new Dog {isEstimated = false,
                               dateOfBirth = DateTime.Now.AddYears(-estimatedAge)};
            return dog;
        }

2) Encapsulation

Classes that contain nothing but public virtual type name { get; set; } are potentially a big code smell.  To preserve encapsulation one needs to carefully consider the visibility of each member and any constraints that should be imposed on the setters.  For example, if you had a database key that was a simple auto-increment identity that required no visibility then you have no good reason to define a public setter for this property. On brownfield projects you should also consider if you need to map everything that’s in the database, if you’ve already gone past that point, use either the inbuilt tools or Resharper to find usage of properties (or rather where no usage exists).

2b) Don’t expose IList

These are big smells as far as I’m considered, when you do something like person.Children.Add(new Child("Fabio")) you’re violating the "tell, don’t ask principle", you should be delegating the work of adding a child to the person, however subtly in this case you’re asking for the collection of children and doing the work yourself.  You should expose methods "AddChild()" – which throws it own meaningful exception when constraints are violated, and return an IEnumerable of Children via the .Children property (getter only). I disagree with throwing a not implemented via your own collection or not supported via a read only list since where a class implements a particular interface it should be substitutable without worry, I shouldn’t have to know that underneath it is a read only collection or a list stole of its owed behaviour.

Most consumers of an IList I’ve seen in projects use only the iterator and the Count property – which is available via Linq extensions method of Count() – perhaps not as efficient but I’m willing to make this tiny sacrifice. Linq extensions method also provide numerous other operations over IEnumerables including sorting, selection, projection etc.. Expose the lowest common interface of IEnumerable and you’ll be able to change the underlying concrete implementation as you please.  Avoid creating your own collection type to handle constraints as this adds extra work and in my experience quite often the constraints that govern the collection are owned and unique to the particular type of parent.

4) Validation

You should be throwing a suitable exception if immediately possible when a call to a setter is attempting to put the object into an invalid state. So person.Age = -5 should throw ArgumentException. If a string property can not be null or empty, throw an ArgumentNullException when null and an ArgumentException when empty, follow the BCL conventions.  Where more complex validation is carried out and it is the responsibility of the object to perform this validation then move away from properties, where properties run for more than a few lines or make calls to foreign objects (exception of lazy loading) then its time to consider moving them to methods.

I’ve seen a couple of people object to immediate validation, preferring to allow consumers to push the object into a known invalid state and then call IsValid or Validate or something similar (and even some delegating this to Interceptors etc..), but you can still achieve summary/delayed validation whilst not weakening your domain integrity which I’ll demonstrate in another blog post in a few days.

5) Repository

There’s been a lot of talk on generic repositories recently, I’m definitely a fan of hiding the generic repository in an inner repository, exposing find all by criteria style methods on the domain repository gives away too much of its own responsibility as well probably leaking some ORM specific dependencies out of your repository (think DetachedCriteria), and even NHibernate.ObjectNotFoundException etc.. This style of repository I think works quite well:

using System;

namespace Cruisers.Ecommerce.Domain.Repository
{
  public interface ICruiseRepository
  {
      IEnumerable<ICruise> GetCruisesOnDate(DateTime date);
      IEnumerable<ICruise> GetCruisesThatAreDiscounted();
      ICruise GetCruiseById(int id);
      ICruise GetCruiseByBrokerCodeName(string codeName);
  }
}

namespace Cruises.Ecommerce.DataAccess
{
    public class NHCruiseRepository : ICruiseRepository
    {
        private readonly IDaoRepository<Cruise> daoRepository;

        public NHCruiseRepository(IDaoRepository<Cruise> daoRepository)
        {
            this.daoRepository = daoRepository;
        }

        public IEnumerable<ICruise> GetCruisesOnDate(DateTime date)
        {
            throw new System.NotImplementedException();
        }

        public IEnumerable<ICruise> GetCruisesThatAreDiscounted()
        {
            throw new System.NotImplementedException();
        }

        public ICruise GetCruiseById(int id)
        {
            try
            {
               return daoRepository.Get(id);
            } catch(NHibernate.ObjectNotFoundException nhException)
            {
               // throw new ArgumentOutOfRangeException("id","cruise not found",nhException);
                throw new CruiseNotFoundException("cruise with id " + id + " not found.",nhException);
            }
        }

        public ICruise GetCruiseByCodeName(string codeName)
        {
            throw new System.NotImplementedException();
        }
    }

}

namespace Cruisers.Common
{
    public interface IDaoRepository<T, K>
    {
        T Get(K id);
        IList<T> FindAll(DetachedCritiera critiera);
    }
}

But eventually you may reach a point when you have to consider sorting, paging, etc.. in these cases one can create a Query object to nicely encapsulate sorting and paging options and provide methods to accept such e.g.

IEnumerable<ICruise> GetCruisesOnDate(OnDateQuery onDateQuery)

As a further improvement in 2.1 we’ll be able to elegantly use the specification pattern (you can today but it is some work) and have the NH repository convert the predicates to NH queries, I intend to spike an example from the trunk soon. NH 2.1 will give us the ability to use Linq natively, but I recommend you avoid exposing this as you’ll have query logic (admittedly NH clean) thrown throughout your application, instead we harness the forthcoming Linq power to use clean specifications instead.

Here’s a few snippets that I’ve used in the past few days. They are a bit verbose at this stage, but it’s all part of the learning, hopefully one day I might be answering your Powershell questions over on StackOverflow.com.

1) Run this in a solution directory and it will produce a summary of classes that contain Tests, it will sum the number of tests and group the tests by Directory. Admittedly this took me 15 minutes to write, but probably would take less than a minute if I had to restart

PS C:\dev\pxan> gci -rec -include "*.cs"  | where { ($numberTests = $(gc $_.FullName | where { $_.ToString() -match "\[Te
st\]" }).Count );($numberTests -gt 0) } | where { $numberTests -gt 0 } | select @{n="Name";e= { $_.Name }}, @{n="Directo
ry"; e={$_.Directory }},@{n="NumberOfTests";e={ $numberTests }} | group -property Directory | foreach { Write-Host $_.Na
me; (foreach { $_.Group } | Select Name, NumberOfTests ) }

C:\dev\pxan\Shark.Client\Tests

Name                                                                                                      NumberOfTests
----                                                                                                      -------------
FormBucketTests.cs                                                                                                   14
FormDataExtractorTests.cs                                                                                            17
ProcSessionTests.cs                                                                                                  13

2) Show me the top five CPU consuming processes,  hmm.. This Windows Live Mesh consumes a lot more power than FolderShare.

PS C:\Documents and Settings\Matt Freeman> gps | sort CPU -desc | select -first 5

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
-------  ------    -----      ----- -----   ------     -- -----------
    695      24    62800      81644   226   206.94    504 Moe
    526      19    51548      38548   346    54.52   2784 WindowsLiveWriter
    459       9     3412       1656    45    26.41    692 lsass
   1116       0        0        240     2     9.52      4 System
    173       6    26436      29200    78     9.44   1864 nod32krn

3) Occasionally I find myself needing to search through a directory of files for a particular string in a file (eg. a database credentials in some PHP app etc..). You could substitute the extra $true to filter further down by modified date, or file size, or extension etc..

PS C:\dev\phpcms> gci -rec | where { $fileName = $_.Name; $true}  |  gc -ErrorAction SilentlyContinue | where { $_.Contains
("password") }   | foreach { Write-Host $fileName " ::::: "  $_  }
setup.php  :::::    $password = "abc";
adminsetup.php :::::    $password = "abc";

I’ve found that the standard powershell application fine for performing most shell tasks although this week I’m going to wrap some of snippets I used regularly into scripts in which I’d probably use an IDE like PowerShell Analyzer for such.

What’s also pretty cool is that one can use powershell from build scripts so I forsee this replacing some of less obviously syntax I’ve had to use to achieve tasks in MSBuild and NAnt directly.