Visual Studio 2015 – Smart Unit Tests


I’ve been reviewing the new Visual Studio 2015.  I currently use both 2013 Professional (at work) and 2012 professional (at home).  So I’m going to blog about a feature I stumbled across while I was examining the unit testing enhancements of VS 2015.

About Visual Studio 2015

The version of VS 2015 that I’m looking at is the Preview version.  Unfortunately, there is only one version of the preview and that is the Ultimate version.  So I’m not sure which features will make it into the Professional version, which will most likely be the version that I’ll purchase in the future.  The Professional Version of VS 2013 without MSDN is currently (as of this writing) selling for $499.  The Ultimate version sells for $13,299.  Yikes!  Premium sells for $6,119.  Also Yikes!  I can’t imagine a company trying to staff a dozen or more developers with a per-seat cost that high.  The cost of the PC becomes an accounting round-off.

OK, I’m going to stop complaining about the price. The feature that is only available in the Ultimate and Premium that I would like to see in the Professional version is the code coverage unit testing feature.  This seems like a basic function of code testing and there are a lot of third party add-on packages that are a cheaper than $6k.  Here’s the feature comparison chart for VS 2013: Feature Comparison

Code Coverage

This is a slight diversion from my original blog topic, but I’m going to show how the code coverage works and why I like it so much.  First, I created an object called MyMathClass, and I created two methods: Divide and Multiply, which perform the functions that you would expect.  Then I created unit tests for the Divide method but not the multiply method.  Under the “Test” menu is a menu item “Analyze Code Coverage” and “All Tests”, which runs the unit tests and shows a “Code Coverage Results” window at the bottom of the screen:

Next, I navigated to the object by double-clicking on the Multiple in the code coverage results screen and the coverage is shown in colored highlights:

As you can see, the code that is not covered by any unit tests shows as red.  All code covered by unit tests is a pale blue/green color.  This is a very trivial example, but I loaded a large program with lots of unit tests into VS 2015 and ran this feature.  I could see instantly which methods and code inside methods was not covered.
Smart Unit Tests
So I was reading through the VS 2015 features website: VS 2015 Features and I stumbled across a feature called “Smart Unit Tests”.  This feature will generate unit tests to cover all cases for each method in an object.  Basically, you right-click inside the object (but outside your methods) and select “Smart Unit Tests”.  This will create unit tests grouped by the method name.  Be aware that there is a drop-down in the smart unit tests window to select which group of tests to view:

If you select a method from this drop-down, then select all the tests in the window and hit the save button (right-click and save, or click on the floppy-disk icon in the smart unit tests window), a unit test project with all the unit test code will be stored:
The code for each unit test is stored in the individual .g.cs sub-files, which are auto-generated.  You’ll need to select each method in the smart unit tests windows drop-down, select all tests and hit the save button for each method in order to add all methods to this test project (yeah, it was painful).  As you can see from my screenshot above, I have added both the unit tests for the divide and the multiply methods.
I was actually surprised at the unit tests that were created for my methods.  For the divide operation, which I purposely made an integer divide, it created a unit test to include an int.MinValue as the numerator and a -1 for the denominator, which caused the method to overflow.  Even though the unit tests are computer generated (and include a lot of messy auto-generated code), I can get an idea of what should be tested.  My big question now, is what kind of unit tests would be generated for complex real-world methods?
I loaded the battleship game AI demo from one of my previous blog posts, changed the object and methods to public (just to allow unit testing) and ran the smart unit test.  I noticed that there are a lot of complex problems that the “smart” unit tester cannot figure out.  First, there was a problem with the Sunk() method tests.  This method tests the playing board to determine if a ship has been sunk.  The issue with the unit tests is that the board was not initialized.  Technically, this is an actual coding issue, since I built this program as a throw-away  console application.  So the map data should have been contained in an object that would be instantiated with some data.  The unit tests would still need to have knowledge of what the object represents and unless ships are mapped to positions in the map board, then a true or false cannot be correctly tested.  In addition to that, the unit tests ran with input values 0, 1, 2 and 6.  There is no length of ship 0 and it missed ship lengths for 4 and 5.  This occurred because the unit test generator had no knowledge of what the Sunk() method is supposed to do.
Another issue I ran into was called “path bounds exceeded” and a unit test was not created for the input used.  The case ran for too many iterations and the smart unit test program thought there was a possible endless loop.  There is a message that indicates that I can select “Fix” from the menu (right-click on the test) and it will change the setting to use 20,000 iterations.  That did not solve the problem and technically, this method does not need a unit test since it doesn’t return a value and it performs no function except to test the maximum searches needed to sink random ship locations.
My conclusion on this feature (and this is just my initial quick-pass) is that it would be handy as an additional check on the unit tests that the developer creates to see if there are possible fence-post values that were missed for inputs, or input combinations that could cause a failed unit test.  Another possible use is to generate a set of unit tests that can be copied into the actual unit test project and cleaned up (i.e. more human readable).  Then the unit tests can be analyzed to make sure they make sense.  Otherwise, it appears to me that the unit tests that are generated are making assumptions about the methods under test that are not correct.  Which is no better than having no unit tests at all.

For more information on this feature you can click here.


Leave a Reply