Application Tracing

So OpenCover is as feature complete as I care to take it at the moment, I may do this one feature involving Windows Store applications should I have a need for it, and I decided to not continue with OpenMutate as I can't really find a need for it other than an exploratory investigation into reJIT.

I do have one more itch to scratch when it comes to profilers and that is application tracing and this may allow me to play with other technologies which I'll list later. This itch started a few months back, perhaps 6+ months ago, when I was trying to integrate some commercial tracing applications to an application I was working on and they both died horribly and I started to look for alternatives and found nothing available. Now I could have started the project then but I decided to pester both vendors until they fixed the problem which they eventually did (within a week or two of each other or so it seemed to me) and I integrated one of the solutions and moved on... but the itch never went away.

So what am I thinking... well a profiler (obviously) with 32/64 support (again given) and making obscene abuse of the COR_PRF_MONITOR_ENTERLEAVE functionality. The problem here is I don't really know what people will want to track (hey that is why there are companies that do this thing with BAs and such like to decide on this) so in the first instance I'll go at tracing everything (which will probably be very slow) and go from there.

This leads to the next problem, data, lots of it, lots and lots of it, and that data is going to need a home but a home I can then use to create reports at some point. For this I am thinking asynchronous, initially a queue and potentially an event source like data store like EventStore or NEventStore. The advantage of an event source would allow the ability to regenerate the views once we know what they are, perhaps something along the lines of Splunk or SplunkStorm would come into play.

So a name... always the hardest part but thankfully we have the internet and online dictionaries so I've gone with OpenDiscover.

Mutation Testing; a use for re-JIT?

Where to start...
Mutation testing is described as modifying a program in small amounts and then executing the original 'passing' tests that exercise that code and then watching them fail. It is a way of making sure your tests are actually testing what you believe they are testing.

Setting the stage...
So how can we do this with .NET? Well first we need to know what tests execute what code and we can use OpenCover for that when it is using it's tracking by test feature. With that feature we can see which tests execute which sequence points and also see what branches were exercised, it is this later information we can take advantage of when creating a mutation testing utility.

New toys to play with...
Now this mutation tester is going to be working at the IL level and as such we could use the JIT (Just-in-time) compilation feature that is used with OpenCover (and PartCover). However that would mean a complicated instrumentation that we would then have to control which path we would want to exercise, or we could have simpler instrumentation but that would require the process under test (e.g. nunit, mstest, ...) to be stop and started each time to allow new code to be exercised. With .NET 4.5 (in preview at the time of writing) there is a re-JIT compilation feature that we could use instead and this would allow us to use simple instrumentation without needing to stop and start the process under test. There are a number of limitations of re-JIT but after reviewing them (several times) I don't think any are actually show stoppers.

However to make the Re-JIT useful we need a way of executing a test or tests repeatedly without having to restart the application under test and this isn't possible with nunit and mstest. However it should be possible to use the test runners from AutoTest.Net if we host them directly or in a separate process that can be communicated with.

A plan...
So the flow will be something like this (I wonder how will this will stand up to the test of time) I haven't looked at the latest profiler API in-depth but documentation on MSDN) and David Broman's Blog seem to indicate this should be possible.

• Run OpenCover to produce an XML file with a list of what tests exercised what branches
• For each branch point =>
• if first branch of method then store the original IL (as we will need repeated access to this IL)
• (re)instrument the method that contains that branch point and using the original IL of that method invert the logic of only that point
• execute each test that exercises that branch point => record pass, fail
• if last branch of method then revert method to original IL

All it needs is a name...
All of this will be hosted on GitHub under OpenMutate. Let the games begin....

Questions about open source and liability in the workplace

Last weekend I attended DDDSydney and one of the most interesting sessions was a panel session about Microsoft and Opensource (Open Source & Microsoft Ecosystem); though as these things go, it went quickly off(ish) topic as expected by the panelists whom I'll refer to as the crazy drupal girl and the 3 stooges (honestly no offence folks, it was highly entertaining).

However it got me thinking about the number of projects where I come have across an unusual bit of open source software that has some use (but has not found a niche or has since been surpassed) and I find that this was introduced by a developer as it was their pet open source project. Now the first question is "what is the liability under this scenario?"

Did the developer ask first as they should before using any open source software on a project? If so then the company accepted the situation but what happens if they did not (or what not made aware) are they still liable or is the developer liable? I assume it would be the company as they should be having some sort of oversight but for small overworked teams where process may not be as strong this may get overlooked.

The other issue is what happens if you introduce your pet open source software project and then you leave, who supports it? How do you separate the open source project needs and the day-job, when they are so intermingled? Does the remaining team support it, do they have the skills? What happens if the parting was acrimonious in nature then they, the team, raised a legitimate issue would you fix it, or leave them to stew?

I don't have answers to the above, I did title this "Questions about...", that can be applied universally the answer to most I suppose is "it depends". Each situation will be different I suspect but I think these type of questions should be asked by any company hoping to use open source software and developers wishing to introduce it, whether that are contributors or not.

Personally I have decided to NOT introduce the open source software I develop into my workplace, yes they could use it and find it useful but they can also afford commercial alternatives. If someone else suggested it, I'd have to make sure there was an agreement should an issue arise that affects them, that if they want it fixed quick then I may have to use 'work' time i.e. no guarantees that it would be done that evening or even that week; after all it is supposed to be fun and not stressful.

How do we get Users out of [open source] Welfare?

Okay an odd title but something I've been thinking about for some time and I suppose is the source of much frustration I have been having whilst maintaining PartCover; I am hoping to reverse the situation with OpenCover.

Categorizing open source users

First I'd like to explain that I like to roughly categorize people involved in open source like thus:

Contributors - these are the guys and gals at the pit-face, developing software, writing documentation and generally striving to make an open source product better.

Investors - these individuals use open source software and help to make the product better via feedback and raising, and following up, issues (probably as it is in their interest to do so).

Benefactors - usually companies that give tools to open source developers or sponsor a project in other ways i.e. free licenses or free hosting e.g. NDepend, JetBrains and GitHub.

Angels - these people provide invaluable advice in just managing a open source project and may not be actively involved in the development itself but just keep you sane.

Community - Our main user base, users of open source but don't actively contribute back and hence why sometimes I refer to them as Welfare. Maybe in the case of the this group it is just a failure to engage, the product just works and they have no need to be involved outside of viewing forums and stackoverflow. But I feel that without the involvement of this group a lot of open source software, no matter how good, can fall by the wayside.

But how do we get them involved? Well first we have to find them, in my case with PartCover as the project had been abandoned the users stopped raising issues on the SourceForge forums and tended to ask questions on other outlets such as StackOverflow, SharpDevelop or Gallio forums and mailing lists.

Finding the users

I scoured the internet and compiled a list of popular places that PartCover was mentioned or supported. I was surprised to find that PartCover was used or supported by SharpDevelop, TeamCity and TypeMock amongst others (and yet again I am surprised it was abandoned and not adopted by anyone sooner).

StackOverflow seems to be the main place where people ask questions and to keep track of questions I have subscribed to an RSS feed for the partcover tag; and as soon as an opencover tag becomes available, or I get enough rep to create it, I'll subscribe to that.

Twitter is also quite a common medium nowadays so I have also set up the following search filter "opencover OR partcover -rt -via" to see if anyone mentions either of the projects.

Engaging the users

Now I have found the users, or the majority of, I started notifying these lists, forums and projects that PartCover was alive again (and I have started to do the same to inform them about OpenCover). Hopefully bringing them back or at least notifying them that if they have really big issues there is somewhere to go.

Involving the Community users

This is the big ask and I don't have an answer. If the product works then they don't need to talk to the forums or declare their appreciation of a job well done. I think sites like ohloh are trying to address the balance. Some OS projects have a donate button, but I am not sure we are doing open source for money, though some projects do eventually go commercial, anyone else can pick up the original code and develop it. Maybe the users don't know how to be involved, in the case of my OS projects they are quite specialised and the learning curve may be too much for some. But I don't think you have to just be involved in projects you use a lot.

Possible ways to get involved

If you are good at graphics why not offer to knock up some graphics for use on web-sites and in the application. [I am quite lucky that Danial Palme added support for PartCover and OpenCover to his Report Generator tool and has done a much better job than I would ever do.]

If you are good at installers, or even if you want to learn more about them, offer to manage them on behalf of the project.

If there is a project you like, support them on the forums like StackOverflow and help other users.

Perhaps update the wikis and forums, sometimes the users know how a product works or can be used better then the developers.

If your company uses a lot of open source, why not buy some licenses for useful software tools and donate them, geeks love shiny new toys; quite a few vendors such as NDepend will donate licenses to open source projects.

If you have an issue, try to help the developers as much as possible to resolve it by supplying as much information as you can and repeatable samples, remember the developers are international and doing this in their own time (as you probably know trying to repeat a scenario from scant information is very frustrating) and maintain contact whilst it is being resolved and let them know when it is.

Okay that's me done on the subject for now, suggestions anyone?

OpenCover First Beta Release

Okay, the first post on a blog I created many, many months ago and still not got round to starting. Why the delay? Well, just been busy and not a lot to say; actually some would say I have too much to say it's just not publishable.

But now I am happy to announce that the first release of OpenCover is now available on GitHub https://github.com/sawilde/opencover/downloads.

"So what?" I hear you say, "we have NCover, dotCover and PartCover [and probably many others with the word cover in the name,] do we need another code coverage tool?" Well, I think the answer is "Yes!" but before I say why a brief history.

About a year ago I adopted PartCover when I found it lost and abandoned and only supporting .NET2; also PartCover has a large user base, SharpDevelop, Gallio, to name but two, and I felt it was a shame to just let it fall by the wayside. I had also done some work on CoverageEye (another OpenSource tool that was originally hosted on GotDotNet and has since vanished) whilst working for a client in the UK, so I felt I had a fighting chance to do the upgrade to .NET4; I don't know if my changes ever got uploaded to GotDotNet as I was not in charge of that.

The adoption was far from easy for a number of reasons, one of which was I was surprised just how little C++ I could actually remember and it's changed a bit since I last used it in anger. Also due to lack of communication with the original developers meant that I was on my own in working out a) how it worked and b) just what the issues are (a lot of the reported issues had long since been abandoned by the reporters).

At the beginning of the adoption I cloned the SourceForge repository to GitHub, git being the in-thing at the time, and after I was eventually admitted access to SourceForge I attempted to maintain both repositories. Due to the lack of permissions on SourceForge, no matter how many times I asked, I eventually abandoned SourceForge and kept all development to GitHub; I also updated the SourceForge repository with a lot of ReadMe posts to point to GitHub.

So upgrading PartCover progressed and thankfully bloggers such as David Broman had already covered the subject matter about upgrading .NET2 profilers to .NET4 and things to look out for. That, it would turn out, was the easy bit.

PartCover had 3 main issues (other than lack of .NET4 support)

1) Memory usage

2) 64 bit support

3) If the target crashed then you got no results.

I'll tackle each of these in turn:

1) Memory - PartCover builds a model of each assembly/method/instrumented point in memory; though I managed to cut down memory usage by moving some of the data gathering to the profiler host it wasn't enough - PartCover also added 10 IL instructions (23 bytes) for each sequence point identified + 4 bytes allocated memory for the counter.

2) 64 bit support - PartCover used a complex COM + Named Pipe RPC, which thankfully just worked but I couldn't work out how to upgrade it to 64 bit (a few other helpers have offered and then gone incommunicado, I can only assume the pain was too much).

3) Crashing == no results - this was due to the profiler being shutdown unexpectedly and the runtime not calling the ::Shutdown method and as such all that data not being streamed to the host process; thankfully people were quite happy to fix crashing code so not a major issue but still an annoyance.

All of this would take major rework of substantial portions of the code and the thought was unbearable. I took a few stabs at bits and pieces but got nowhere.

Thankfully I had received some good advice and though I tried to apply it to PartCover I realised the only way was to start again, taking what I had learned from the guys who wrote PartCover and some ideas I had come across from looking at other opensource tools such as CoverageEye and Mono.Cecil.

OpenCover was born.

This time I created a simple COM object supporting the interfaces and then made sure I could compile it in both 32 and 64 bit from day one.

I then decided to make the profiler as simple as possible, so it is maintainable and move as much of the model handling to the profiler host, thank heavens for Mono.Cecil. The only complex thing was deconstructing the IL and reassembling it after it had been instrumented. OpenCover only inserts 3 IL instruction (9/13 bytes depending on 32/64 bit) per instrumented point; it forces a call into the profiler assembly itself and this C++ code then records the 'hit'.

Finally I decided I had to get the data out of the profiler and into the host as soon as possible. I toyed with WCF and WWSAPI but this also meant I had no XP support, but at least I could test other ideas. However if my target/profiler crashed I would loose the last packet of data; not drastic but not ideal. Eventually I bit the bullet and switched to using shared memory.

The switch to shared memory has brought a number of benefits one of which is the ability to handle a number of processes under the same profiling session, both 64 and 32 bit and to aggregate the results as they all use the same shared memory. I have yet to work out how to set this up via configuration files but anyone wishing to experiment can do so via modifying the call to ProfilerManager::RunProcess in the OpenCover.Host::Program.

So this is where we are now, OpenCover has been released (beta obviously) and as of time of writing some people have actually downloaded it. I am now braced for the issues to come flooding/trickling in.

Feel free to download and comment, raise issues on GitHub, get involved; Daniel Palme, he of Report Generator fame, is hopefully going to upgrade his tool to include OpenCover.