Moles coverage support

I've run into a problem that basically results in zero coverage from ncover.

moles.runner (part of the Moles and Pex framework) is required to run tests that have Moles in them. Basically, it instruments the assembly so that types can be detoured.

However, when running under the moles runner, only the moles framework itself is code covered, even with the coverall flag set. The assembly's that moles is running all show with 0 coverage.

Here's what my msbuild task looks like:

<Exec WorkingDirectory ="$(MSBuildProjectDirectory)" 
      Command="&quot;$(NCover)&quot; //coverAll //xml &quot;Output\%(TestFiles.FileName).Coverage.xml&quot; &quot;$(molesRunner)&quot; &quot;%(TestFiles.FullPath)&quot; /runner:&quot;$(Nunit)&quot; /args:/xml:&quot;Output\%(TestFiles.FileName).xml&quot;"/>

I'd appreciate any ideas or suggestions.

Robert

RE: Moles coverage support

From what I can tell, Pex and Moles use the .NET profiler API (as does NCover), and at present only one profiler can run at a time, so I'm not sure that you'll be able to run NCover with your test suite.

RE: Moles coverage support

Ah, that's too bad. Makes our code coverage look worse than it is. :(

We're putting all of our moled tests into a category so that we can exclude them and run the rest of the tests, so we at least get some coverage reports with NCover.

Thanks for the response.

Robert

RE: Moles coverage support

I had found some workaround to enable Code Coverage for moled assemblies at this article -

http://ctp.social.msdn.microsoft.com/Forums/en/pex/thread/9b2f683c-d73e-49f4-b35a-c424dc509791

a) I had to set an environment variable for moles to do CLR profiler chaining. This was specific to NCover 3.0. I found via running regsvr32 NCover.lib.x86.dll then looking in the registry for the guid it registers, usually under \HKLM\Software\\Classes\Typelib\ set CLRMONITOR_EXTERNAL_PROFILERS={9721F7EB-5F92-447C-9F75-79278052B7BA} b) I had to tell NCover to attach to child processes. I used the //pn argument NCover.console.exe //pn moles.runner.x86.exe

I tried to follow this workaround but it didn't seem to make difference.

Can you please let me know if this workaround really works?

Also do I need to specify argument //pn as moles.runner.x86.exe if I use //coverall for NCover console

RE: Moles coverage support

I was able to collect coverage by chaining NCover profiler and Moles as that post suggests. You can download my sample project and check it out. This is the simplest of examples so this unsupported chaining may introduce other problems as use becomes more complex.

This project is built and tested as 64-bit only. I have used NUnit, not MSTest. After building the project you will find a command file CoverWithMoles.cmd in the output directory. This file contains the environment setting for the profiler chaining and sample command.

Environment setting: set CLRMONITOR_EXTERNAL_PROFILERS={9721F7EB-5F92-447c-9F75-79278052B7BA}

Sample Command: ncover.console //pn moles.runner.exe //eas "moles.runner;mscorlib.Moles" "c:\Program Files\Microsoft Moles\bin\moles.runner.exe" TestingMolesNunit.dll /runner:"C:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0\nunit-console.exe"

Download the example: http://downloads.ncover.com/CoverWithMoles.zip

NCover: v3.4.18

RE: Moles coverage support

Is there a way to achieve the same through the NCover Explorer?

I also tried to run the batch file and I get the following output and I still get 0% code coverage.

C:\LinkController\Agt.Fault.Test\bin\Debug>set CLRMONITOR_EXTERNAL_PROFILERS={9 21F7EB-5F92-447c-9F75-79278052B7BA}

C:\LinkController\Agt.Fault.Test\bin\Debug>ncover.console //pn moles.runner.exe //eas "moles.runner;mscorlib.Moles" "c:\Program Files\Microsoft Moles\bin\moles runner.exe" Agt.Fault.Test.dll /runner:"C:\Program Files\Microsoft Visual Studi 10.0\Common7\IDE\MSTest.exe" NCover Complete Trial v3.4.18.6937 x86 (Administrator) Copyright (c) 2007 - 2010 Gnoso Inc. License Key: M4AFCQR9TV4IV28B Registered to Customer #: 71240 for 1 Machine 21 days left in the trial period.

Command: "c:\Program Files\Microsoft Moles\bin\moles.runner.exe" Command Args: "Agt.Fault.Test.dll" "/runner:C:\Program Files\Microsoft Visual S udio 10.0\Common7\IDE\MSTest.exe" Working Directory: Excluded Assemblies: moles.runner;mscorlib.Moles

Program Output Process 'moles.runner' [PID 6024] has begun profiling. Microsoft Moles Runner v0.94.51023.0 -- http://research.microsoft.com/moles -- NET v4.0.30319 Copyright (c) Microsoft Corporation 2007-2010. All rights reserved.

instrumenting...Process 'moles.runner' [PID 6024] has finished profiling.

End Program Output

Execution Time: 3.8438 s Symbol Coverage: 0.00% Branch Coverage: 0.00%

Coverage File: C:\LinkController\Agt.Fault.Test\bin\Debug\coverage.nccov

moles.runner.exe is returning exit code #-667

C:\LinkController\Agt.Fault.Test\bin\Debug>

RE: Moles coverage support

Hi sub_zero,

You should be able to run all of the required test parameters using NCover Explorer, except for setting the CLRMONITOR, of course.

Speaking of which, is there a copy/paste error in your post, or did you set CLRMONITOR_EXTERNAL_PROFILERS={9 21F7EB-5F92-447c-9F75-79278052B7BA}?

I ask because that GUID is missing a '7' in the initial string, as compared to the correct one below:

CLRMONITOR_EXTERNAL_PROFILERS={9721F7EB-5F92-447C-9F75-79278052B7BA}

Also, the dev who created the test Moles projects used NUnit only, not MSTest, so unfortunately we can't be sure that set-up will work.

I hope this helps!

NCover Support

RE: Moles coverage support

The GUID was a copy and past error when I execute there is no missing 7.

Can you guys try with MSTest because it doesn't seem to work with that.

Thanks

RE: Moles coverage support

Thanks for clarifying.

We're always trying to expand the flexibility of NCover, but there has been limited interest in Moles so far.

The dev team is dedicated to the next NCover release right now, but I'll contact you directly if there's any new information about Moles and MSTest.

Best regards,

NCover Support

RE: Moles coverage support

Dear NCover support,

We face the same issue. The workarounds did not work in my case, apparently the behavior is different under X64 machines. Or the numbers obtained by NCover are not valuable.

Is there any chance to see this issue solved if we open an official support ticket for this ?

From the testing I did it seams to be a problem around environment variables that conflict beteen the NCover and Moles startups.

Kind Regards

RE: Moles coverage support

Hi degreefs,

Thanks to you and sub_zero for your interest in using NCover with Moles.

I discussed this issue with the dev team; the developer's intention was to provide a trial-of-concept and an example project that might help others in the community better understand chaining NCover with other profilers.

However, because chaining profilers is complex and precarious, the NCover Moles project remains an unsupported experiment.

The dev team will continue to consider the impact of Moles as they work on the next NCover release, but there are currently no plans to further test or develop the existing Moles sample project.

Best Regards,

NCover Support

RE: Moles coverage support

I was able to get round the issue of the molesrunner returning exit code 667 by removing the {} from the GUID when setting the environment variable. However, I still get no coverage. I think I have an issue with the way the command is constructed in my msbuild task though, as the output shows:

NCover.Console.exe //pn moles.runner.exe //x MolesCoverage.xml //eas "moles.runner;mscorlib.Moles" moles.runner.exe Example.Moles.Test.dll /runner:"C:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0"\nunit-console.exe