Creating a simple NodeJS app with Mongo

Okay, I woke up this morning (6am) with a need to create a simple reporting dashboard to display the coverage results from OpenCover when it dog-foods its own tests. Now that OpenCover has no reported bugs, I decided to use my spare time to investigate other technologies for a while.

What I needed was simple 'online' storage to capture results from the build system and the ability to extract that data into charts. Normally I'd probably knock up a simple rails app because it is easy to do, however I decided, probably due to the heat, to use the following:

• node.js - a technology I haven't used but have meant to for a while; I also like the JavaScript syntax better than ruby (it's a personal thing)
• mongodb - a database I am familiar with
• google charts - free; as in beer.
• heroku - free; well my intended usage will be.

A quick time-boxed search of the web about how to use node with mongodb and create a RESTful API and I settled on the following packages:

• mongoose - for interacting with the mongo database
• restify - for creating a simple rest server
• nodemon - monitors changes in your app and restarts; sounds useful

I'll assume other packages will be added to the mix as challenges present themselves. It's now 7am and time for breakfast and then the fun starts... 

And a few hours later we have a simple storage system hosted on heroku all we need now is the charts

The repository can be found on github. 

I am sure it will evolve over time but it was very simple to get to this stage by leveraging the work of all those who have gone before.

MongoDB, Mongoid, MapReduce and Embedded Documents.

I am using Mongoid to store some data as documents in a MongoDB database and then run some MapReduce queries against the data. Now I have no trouble with mapping data from normal documents and an embedded document but I could not extract data from an embedded collection of documents i.e.

class Foo
  include Mongoid::Document

  #fields
  field :custom_id, :type => String

  #relations
  embeds_many :bars

end

class Bar
  include Mongoid::Document

  #fields
  field :custom_field, :type => String

  #relations
  embedded_in :Foo

end

First it looks like that we need to run the map part of the MapReduce against the parent document and not the child i.e. Foo.map_reduce(...) will work find documents but Bar.map_reduce(...) does not, however that is not surprising as it is also not possible to count all Bar documents by doing Bar.all.count in the rails console.

Now a MapReduce query in MongoDB is done as a pair of JavaScript scripts, the first does the map by emitting a mini-document of data and the second that aggregates the data in some manner. So thinking I had a collection (array) my first attempt to map data from the embedded document was this:

MAP:

function() {
  if (this.bars == null) return;
  for (var bar in this.bars){
    emit(bar.custom_field, { count: 1 });
  }
}

REDUCE:

function(key, values) {
  var total = 0;
  for ( var i=0; i< values.length; i++ ) {
    total += values[i].count;
  }
  return { count: total };
}

This produced an unusual result such that there was only a single aggregated document with a null key and the count was the total number of child documents (summed across all the parents).

Now I could have just broken the child document out and not embedded it but I didn't want to break the model over something so trivial that must, in my eyes, be possible.

After much googling and reading of forum posts, I couldn't find any samples. I eventually observed of some 'unusual' syntax on an unrelated topic which led me to rewrite the map script into this:

function() {
  if (this.bars== null) return;
  for (var bar in this.bars){
    emit(this.bars[bar].custom_field, { count: 1 });
  }
}

Which produced the expected results. Okay this was probably obvious to anyone who knows MongoDB+MapReduce well but it took me a while to find out and it still isn't that intuitive, though I think I now know why it is this way, so I thought I'd write it up as a bit of a reference.

The "Pigs" and "Chickens" fable

I think anyone who is anyone who has heard of Agile and Scrum have heard of the Pigs and Chickens story and how it describes those who are committed to the delivery of the project, "Pigs", and those who are just involved, "Chickens." If not click on the image and learn more about it.

However I was just recently re-reading "Death March" by Edward Yourdon (1st Edition) and I came across this response to the parable, in the context of commitment whilst on a death march.

“I’m not sure you will find any old pigs in development perhaps more chickens. I think that kind of commitment continues until (inevitably?) you get into the first death march project – then there is a rude awakening. Either the pig realises what’s happening, this is the slaughterhouse! RUN!! Or the pig is making bacon…” - Paul Mason (Death March).

I just found it quite amusing and thought I should share...

Mutation Testing; a use for re-JIT?

Where to start...
Mutation testing is described as modifying a program in small amounts and then executing the original 'passing' tests that exercise that code and then watching them fail. It is a way of making sure your tests are actually testing what you believe they are testing.

Setting the stage...
So how can we do this with .NET? Well first we need to know what tests execute what code and we can use OpenCover for that when it is using it's tracking by test feature. With that feature we can see which tests execute which sequence points and also see what branches were exercised, it is this later information we can take advantage of when creating a mutation testing utility.

New toys to play with...
Now this mutation tester is going to be working at the IL level and as such we could use the JIT (Just-in-time) compilation feature that is used with OpenCover (and PartCover). However that would mean a complicated instrumentation that we would then have to control which path we would want to exercise, or we could have simpler instrumentation but that would require the process under test (e.g. nunit, mstest, ...) to be stop and started each time to allow new code to be exercised. With .NET 4.5 (in preview at the time of writing) there is a re-JIT compilation feature that we could use instead and this would allow us to use simple instrumentation without needing to stop and start the process under test. There are a number of limitations of re-JIT but after reviewing them (several times) I don't think any are actually show stoppers.

However to make the Re-JIT useful we need a way of executing a test or tests repeatedly without having to restart the application under test and this isn't possible with nunit and mstest. However it should be possible to use the test runners from AutoTest.Net if we host them directly or in a separate process that can be communicated with.

A plan...
So the flow will be something like this (I wonder how will this will stand up to the test of time) I haven't looked at the latest profiler API in-depth but documentation on MSDN) and David Broman's Blog seem to indicate this should be possible.

• Run OpenCover to produce an XML file with a list of what tests exercised what branches
• For each branch point =>
• if first branch of method then store the original IL (as we will need repeated access to this IL)
• (re)instrument the method that contains that branch point and using the original IL of that method invert the logic of only that point
• execute each test that exercises that branch point => record pass, fail
• if last branch of method then revert method to original IL

All it needs is a name...
All of this will be hosted on GitHub under OpenMutate. Let the games begin....

Unusual coverage in VB.NET

Recently a user posted on StackOverflow on why he was seeing unusual coverage results in VB.NET with MSTEST and Visual Studio. The the question already had answers that helped the questioner but I decided to delve a little deeper and find out why the solution proposed worked.

The issue was that in his code sample the End Try was not being shown as covered even though he had exercised the Try and the Catch parts of his code.

First I broke his sample down into something simpler and I have highlighted the offending line.

 07  Function Method() As String  
 08    Try  
 09      Return ""  
 10    Catch ex As Exception  
 11      Return ""  
 12    End Try
 13  End Function  

In debug we can extract the following sequence points (I am, obviously, using OpenCover for this.)

<SequencePoints>  
  <SequencePoint offset="0" ordinal="0" uspid="261" vc="0" ec="32" el="7" sc="5" sl="7"/>  
  <SequencePoint offset="1" ordinal="1" uspid="262" vc="0" ec="12" el="8" sc="9" sl="8"/>  
  <SequencePoint offset="2" ordinal="2" uspid="263" vc="0" ec="22" el="9" sc="13" sl="9"/>  
  <SequencePoint offset="19" ordinal="3" uspid="264" vc="0" ec="30" el="10" sc="9" sl="10"/>  
  <SequencePoint offset="20" ordinal="4" uspid="265" vc="0" ec="22" el="11" sc="13" sl="11"/>  
  <SequencePoint offset="40" ordinal="5" uspid="266" vc="0" ec="16" el="12" sc="9" sl="12"/>  
  <SequencePoint offset="41" ordinal="6" uspid="267" vc="0" ec="17" el="13" sc="5" sl="13"/>  
</SequencePoints>  

(where sl = start line, el = end line, sc = start column, ec = end column and offset = IL offset in decimal)

However these only make sense when you look at the IL...

.method public static
    string Method () cil managed
{
    // Method begins at RVA 0x272c
    // Code size 43 (0x2b)
    .maxstack 2
    .locals init (
        [0] string Method,
        [1] class [mscorlib]System.Exception ex
    )

    IL_0000: nop
    IL_0001: nop
    .try
    {
        IL_0002: ldstr ""
        IL_0007: stloc.0
        IL_0008: leave.s IL_0029

        IL_000a: leave.s IL_0028
    } // end .try
    catch [mscorlib]System.Exception
    {
        IL_000c: dup
        IL_000d: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::SetProjectError(class [mscorlib]System.Exception)
        IL_0012: stloc.1
        IL_0013: nop
        IL_0014: ldstr ""
        IL_0019: stloc.0
        IL_001a: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
        IL_001f: leave.s IL_0029

        IL_0021: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
        IL_0026: leave.s IL_0028
    } // end handler

    IL_0028: nop

    IL_0029: ldloc.0
    IL_002a: ret
} // end of method Module1::Method

Now as you can see the End Try line that is causing concern would only be marked as hit (assuming they are using similar instrumentation to OpenCover) if the code reached IL instruction at offset 40 (IL_0028) however when one looks at the IL produced it is not possible to see how you would ever reach that instruction due to the odd IL produced (leave.s is a small jump like instruction that is used to exit try/catch/finally blocks) and if you follow the code you see that you will always reach a leave.s that jumps to IL_0029 first.

In release the IL changes to something more like what I was expecting beforehand and it has no unusual extra IL...

.method public static
    string Method () cil managed
{
    // Method begins at RVA 0x2274
    // Code size 30 (0x1e)
    .maxstack 2
    .locals init (
        [0] string Method,
        [1] class [mscorlib]System.Exception ex
    )

    .try
    {
        IL_0000: ldstr ""
        IL_0005: stloc.0
        IL_0006: leave.s IL_001c
    } // end .try
    catch [mscorlib]System.Exception
    {
        IL_0008: dup
        IL_0009: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::SetProjectError(class [mscorlib]System.Exception)
        IL_000e: stloc.1
        IL_000f: ldstr ""
        IL_0014: stloc.0
        IL_0015: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
        IL_001a: leave.s IL_001c
    } // end handler

    IL_001c: ldloc.0
    IL_001d: ret
} // end of method Module1::Method

but so do the sequence points...

<SequencePoints>
  <SequencePoint offset="0" ordinal="0" uspid="33" vc="0" ec="22" el="9" sc="13" sl="9"/>
  <SequencePoint offset="15" ordinal="1" uspid="34" vc="0" ec="22" el="11" sc="13" sl="11"/>
  <SequencePoint offset="28" ordinal="2" uspid="35" vc="0" ec="17" el="13" sc="5" sl="13"/>
</SequencePoints>

So now one will never see your try/catch lines marked covered, so this is not helpful.

So lets try changing your code as suggested and go back to debug (because that is where you will be running coverage from usually.)

15   Function Method2() As String
16        Dim x As String
17        Try
18            x = ""
19        Catch ex As Exception
20            x = ""
21        End Try
22        Return x
23    End Function

Again we look at the sequence points...

<SequencePoints>
  <SequencePoint offset="0" ordinal="0" uspid="268" vc="0" ec="33" el="15" sc="5" sl="15"/>
  <SequencePoint offset="1" ordinal="1" uspid="269" vc="0" ec="12" el="17" sc="9" sl="17"/>
  <SequencePoint offset="2" ordinal="2" uspid="270" vc="0" ec="19" el="18" sc="13" sl="18"/>
  <SequencePoint offset="17" ordinal="3" uspid="271" vc="0" ec="30" el="19" sc="9" sl="19"/>
  <SequencePoint offset="18" ordinal="4" uspid="272" vc="0" ec="19" el="20" sc="13" sl="20"/>
  <SequencePoint offset="31" ordinal="5" uspid="273" vc="0" ec="16" el="21" sc="9" sl="21"/>
  <SequencePoint offset="32" ordinal="6" uspid="274" vc="0" ec="17" el="22" sc="9" sl="22"/>
  <SequencePoint offset="36" ordinal="7" uspid="275" vc="0" ec="17" el="23" sc="5" sl="23"/>
</SequencePoints>

and the IL...

.method public static
    string Method2 () cil managed
{
    // Method begins at RVA 0x282c
    // Code size 38 (0x26)
    .maxstack 2
    .locals init (
        [0] string Method2,
        [1] string x,
        [2] class [mscorlib]System.Exception ex
    )

    IL_0000: nop
    IL_0001: nop
    .try
    {
        IL_0002: ldstr ""
        IL_0007: stloc.1
        IL_0008: leave.s IL_001f
    } // end .try
    catch [mscorlib]System.Exception
    {
        IL_000a: dup
        IL_000b: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::SetProjectError(class [mscorlib]System.Exception)
        IL_0010: stloc.2
        IL_0011: nop
        IL_0012: ldstr ""
        IL_0017: stloc.1
        IL_0018: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
        IL_001d: leave.s IL_001f
    } // end handler

    IL_001f: nop
    IL_0020: ldloc.1
    IL_0021: stloc.0
    IL_0022: br.s IL_0024

    IL_0024: ldloc.0
    IL_0025: ret
} // end of method Module1::Method2

So for the End Try to be covered we need line 21 to be hit and that is offset 31 (IL_001F) and as it can be seen both leave.s instructions jump to that point so now that line will be marked as covered.

Adding OpenCover to TeamCity

Adding OpenCover to the latest version of TeamCity (6.5) couldn't be easier however if you need help follow these simple steps.

1) Download and install OpenCover
2) Download and install ReportGenerator (actually unzip)
3) Register the OpenCover profiler DLLs using the regsvr32 utility

regsvr32 /s x86\OpenCover.Profiler.dll
regsvr32 /s x64\OpenCover.Profiler.dll


4) Using TeamCity add a new Build Step to your configuration
5) Choose Command Line as the runner type then choose Custom Script for the Run option.
6) Now all is needed is to set up the command to run the profiler against your tests e.g. for OpenCover the working directory is set to main\bin\debug and so we have

"%env.ProgramFiles(x86)%\opencover\opencover.console.exe" "-target:..\..\..\tools\NUnit-2.5.10.11092\bin\net-2.0\nunit-console-x86.exe" -targetargs:"OpenCover.Test.dll /noshadow" -filter:"+[Open*]* -[OpenCover.T*]*" "-output:..\..\..\opencovertests.xml"
"%env.ProgramFiles(x86)%\ReportGenerator\bin\ReportGenerator.exe" ..\..\..\opencovertests.xml ..\..\..\coverage

7) Finally setup the artifacts so that you can view the results in TeamCity e.g.

%teamcity.build.workingDir%\opencovertests.xml
%teamcity.build.workingDir%\coverage\**\*.*


And there you have it, OpenCover running under TeamCity and visual reports provided by ReportGenerator. I am sure you will find ways to improve upon this for your own builds.

The problem with sequence coverage. (part 2)

Previously I mentioned why just relying on sequence coverage is not a good idea as it is possible to have 100% sequence coverage but not 100% code coverage. However I only described a scenario that used a branch that had 2 paths i.e. the most common form of the conditional branches, but there is one other member of the conditional branch family that exists in IL and that is the switch instruction; this instruction can have many paths. This time I am using the code from the Newtonsoft.Json library because a) it has tests and b) it is very well covered at 83% sequence coverage, but only 72% (by my calculations) branch coverage. The subject of this investigation is BsonReader::ReadType(BsonType) this method has a very large switch statement, that actually is defined as a switch statement in IL, with a default and several fall-throughs; a fall-through is where two or more case statements call the same code. The method itself has 98% sequence coverage and 82% branch coverage; the only code that is uncovered is the handler for the default: path.

which is not unexpected as it is a handler for an Enum which should not be set to any value that is not part of the allowed values. Looking at the branch coverage report we have the following results (the switch instruction we are interested in is at IL offset 8.

Now the first path (0) is unvisited, but we knew that, so the next unvisited branch is #14 and the next is #17; luckily for us the enum in question that is used by the switch instruction is well defined.

And as such we can thus deduce that the method is never called during testing with the values Symbol and TimeStamp but the code that they would call is covered; in fact we can see from the code that both these enum values are part of the switch/case statement and are part of fall-throughs. So again we see how branch coverage helps identify 'potential' issues and test candidates.