|
This is a fairly long post, so for the lazy I'll offer a summary up front:
TLDR: There's a lot of potential here; but some major limitations in what they try to test, how they present results, and the need to manually setup any non dependency injected state severely limit the easy button promises MSDN has made. Hopefully a future version will be better.
(If it wasn't for the negative results, and inability to take screenshots of work owned code; this probably would be a better fit as an article than as a long post here.)
This is a semi-followup to my travails with the elephants in corporate IT[^] and "hey that looks cool" replies to insider posts I can' be bothered to find now.
Because I ended up with a laptop only running Win7 (not 8.1 or 10 like I wanted); the only item on my list that I was able to give a whirl to was "Smart Unit Tests"[^]
The basic idea behind them is that by combining various code analysis and generation tools MS Research has been fiddling with over the last few years they're able to look at a codebase, analyze all the paths through it, and generate a set of unit tests to execute every possible path. For developers who have to work with large legacy code bases that weren't written with a unit test suite and whose maintenance budgets are too small to retrofit heavy coverage now, (*waves hand mournfully*), the potential gain here would be that an auto-generated test suite would flag any time changes made in one place propagated elsewhere in the code/altered edge cases/etc; allowing regression bugs to be spotted and fixed early instead of piling up until a pound the keyboard test was ran (or worse when the application went to production).
Unfortunately, my initial impressions are that while they've got a lot of useful bits and pieces; poor integration/developer experience and apparent functional limitations mean that the product is still in the watch and hope for future versions to make it usable stage.
Usability problems I've encountered:
0) The coarsest granularity that it appears to be possible to request test generation at is the class level. Meaning that if your application has several hundred classes you need open each individual source file, right click on the class name (in my case after scrolling past a boilerplate copyright comment header), and select the generate option from the context menu, wait several seconds to several minutes while the tools process that class....
1) After you've generated test cases for a class, there's no way to save the entire set in a single shot. Instead you have to view them by method, do a select all, and then save the selection.
2) While the results UI shows how many "blocks" of code it was able to generate coverage for in an X out of Y form I couldn't find a way to map the covered/not covered blocks back to the code so I could see what it was missing.
3) Issues 0) and 1) make creating a test suite for a non-trivial legacy app a major pain even if it is able to do everything MSDN marketing promises it can. (Without doing so I can't alleviate 2) by using an external code coverage tool.) However...
Functional limitations I've encountered:
A) My codebase uses factories to create objects indirectly; and the tools weren't smart enough to find the places where I coded Foo MyFoo = BarLibraryFactoryCreateFoo(/*...*/); and reported that it couldn't guess how to create Foo objects. In throwing up its hands, Smart Unit Tests, also failed to guess that it could use the internal static Foo Create(/*...*/) methods called by the factory methods or the non public constructors that the Create(/*...*/) methods called. It was at least easy to click fix and edit the auto-generated test factory manually to do object creation.
B) It appears to ignore anything that touches a winform UI control entirely. Since large portions of my legacy code are contained within event handlers, or methods called from the event handlers that read/write values from the UI controls directly; this means that a very large portion of my code remains untestable.
C) It's unable to generate any useful test results in methods where file IO is involved. It's only able to drill down as far as generating file not found errors. Guess what the other large chunk of legacy business logic I was hoping to use an easy button to bring under test is tangled with.
While it would be nice to refactor the code base to separate the business logic out from the UI and IO layers, unless the money fairy drops a check with enough money to keep me paid for a year, it's not going to happen. Even if it did; the act of decoupling the logic from everything else would almost require that I write a unit test suite as I went.
D) It's unable to figure out that it needs/how to effectively create any dependencies that are accessed via global/static class/etc means instead of being provided via dependency injection. (Yes, this is crappy design. Again, see: legacy code; and money fairy, non-existence of.) This meant that when I pointed it at a calculation library I was only able to get minimal coverage before it hit the limits of what it was able to do.
E) This is more of a meta issue; but the tests it generates that trigger an exception all fail. In a lot of cases, this probably is technically correct as it shows the code puking on unexpected nulls (vs an intentional null should throw decision); but it means that even if the other 4 functional problems are all overcome creating a regression detecting test suite would require a lot of manual editing to make the exception thrown tests pass when the exception was thrown.
Things I will(might) try when(if) I find more ambition to mess with this:
0) Poke at menus and throw questions as google to find out if there are options I didn't see to make usability problems 0) - 2) or functional limitation E) go away.
1) Modify the auto-generated test startup code to initialize enough state to work around D) and possibly C) so that the tools have the opportunity to generate a deeper set of tests. (Hopefully via shared testsuite wide code, not on a per class constructor basis.) D)'d be the easier starting place here, since it's the one component of the app I currently do have a good unit test suite for; but extending coverage to the file IO heavy code in C) would be more likely to get my PM onboard with spending a bit of money to go from my screwing with it on my own time to getting paid to push beyond an initial can it do anything useful prototype.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason?
Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful?
--Zachris Topelius
Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies.
-- Sarah Hoyt
|
|
|
|
|
Thanks for sharing...interesting read...Happy New Year 
|
|
|
|
|
It seems like nothing has changed in almost 30 years. In 1986 I worked on a study effort for the U.S. Air Force entitled "Automated Integration Verification and Validation Test Case Generation for Ada". The purpose was to automatically generate test case data sets for an arbitrary body of Ada source code.
The conclusion of the study was basically it couldn't be done economically.
Software Zen: delete this;
|
|
|
|
|
Honestly, I suspect that if you had written a testing friendly code base, but then stopped writing tests to go with it for some reason (under supervised noob, rushed maintenance dev, etc), I think it's probably good enough to be useful in backfilling gaps. It could be useful in greenfield dev as a test suite validator too; the horror of legacy code is still beyond it however. If they port it to F# (currently it's C# only, doesn't even speak VB.net yet); the statelessness of the functional koolaid surrounding that language might make it an easier target.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason?
Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful?
--Zachris Topelius
Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies.
-- Sarah Hoyt
|
|
|
|
|
That would make for a great blog post or an article with code examples of your findings. 
It'd be sad to lose it in the noise of the lounge - put it up on a blog at least!
Marc
|
|
|
|
|
That would require torturing myself by voluntarily entering the hell known as web dev to plonk one onto my webhost; and to commit to a continual stream of torture in the future installing updates and repairing damage every time the platform I picked got pwnd.
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, waging all things in the balance of reason?
Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful?
--Zachris Topelius
Training a telescope on one’s own belly button will only reveal lint. You like that? You go right on staring at it. I prefer looking at galaxies.
-- Sarah Hoyt
|
|
|
|
|
Yeah, I can sympathize with that.
Still, it was an excellent post worthy of some degree of perpetuity.
Marc
|
|
|
|
|
For all CPians: May the new year bring a bright rainbow with a huge pot of gold where it touches down!
You just cannot fix stupid.
|
|
|
|
|
Cornelius Henning wrote: May the new year bring a bright rainbow with a huge pot of gold where it touches down!
I've noticed that the pot of gold is usually on the end of the arch.
But, happy new year to you too, and may you catch a Leprechaun who will take you to the right end of the rainbow!
Marc
|
|
|
|
|
Bring happiness & peace for us...
|
|
|
|
|
I used MaCafee internet security in 2014. now I bought Kaspersky. in the installation, I just find out it is very difficult to remove MacaFee. So next year I will stay with Kaspersky. I have tried all popular internet security software and I may stay with Kaspersky.
just my personal feeling, not promotion for anyone.
diligent hands rule....
|
|
|
|
|
Southmountain wrote: I just find out it is very difficult to remove MacaFee
Yes it is, and even using the removal tool, there is garbage staying behind. I had it once. My luck, it was about time to upgrade more things so I directly format and reinstalled clean enviroment.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
There's a good tutorial from the man himself here[^]
|
|
|
|
|
that video is really funny 
diligent hands rule....
|
|
|
|
|
McAfee is a virus.
Marc
|
|
|
|
|
well, there is a saying about fighting fire with fire isn't there 
|
|
|
|
|
There is no such thing as Internet Security.
|
|
|
|
|
You do know that the worst place to mention which Internet security products you use is the Internet, don't you?
I wanna be a eunuchs developer! Pass me a bread knife!
|
|
|
|
|
Kaspersky is Russian. Just saying. 
There are only 10 types of people in the world, those who understand binary and those who don't.
|
|
|
|
|
Programmer: I stress out every time I deploy to production
Psychiatrist: Go enterprise. U rarely commit & even then u can blame someone else
Sharing my 33 cents worth on SSRS, .NET, Sharepoint and more...
My Blog: http://dotnetsme.com
My book on client-side reporting services with visual studio: http://www.apress.com/book/view/9781590598542
|
|
|
|
|
Then he will have to learn the API for that, my choice, go "to another psychiatrist".
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
Oh boy... double trouble
Sharing my 33 cents worth on SSRS, .NET, Sharepoint and more...
My Blog: http://dotnetsme.com
My book on client-side reporting services with visual studio: http://www.apress.com/book/view/9781590598542
|
|
|
|
|
Deleted by Marc.
modified 1-Jan-15 19:09pm.
|
|
|
|
|
On OriginalGriff's behalf, "DO NOT SHOUT!, Capital letters are considered as shouts." So please write them in a formal way which won't be rude for any of us. If you want us to pay some attention to your line, write them in italic.
Coming to your blog, you've done nothing until now, so how can you expect us to do anything for you? First of all, you need to create it, run it then tell us! If you're going to involve us as volunteers in the beginning, or are going to ask us to, then we're going to ignore this request. Don't just say, "I've created this", show us, that you've created it.
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
We have a place available to share code, get feedback and improve. It's called Code Project.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
