|
SuiteSpot
MXUnit2.0, code named SuiteSpot Ideas for MXUnit 2.0. It looks like we might do some minor releases between now and then, but based on our talks, it looks like we want to accomplish a major release. Here's a start:
Everyone is welcome to comment or edit! TestSuite ImprovementsThe mxunit TestSuite class may change dramatically and it will be a good challenge to make reverse compatible with 1.0. In addition, the TestResult class will likely be different, too. The TestSuite should follow a composite structure (suites of suites of suites) and maybe the annotations provide instructions to the framework on how to "build" the TestSuite and not necessarily how to run it. But, it's unclear at this point what would be the best approach. Ideas anyone? The TestNG xml-based testsuite is interesting and I wonder if we could that as the primary testsuite configuration? So, instead of extending TestSuite, we could write and xml configuration file to tell mxunit how to run tests. It would be nice if both the Ant task and a runner could be used for the same file. Also, not sure how this plays into the plugin, but maybe a testsuite could be run by selecting the xml config, or if the xml config file is present it takes priority? For the moment, I can see a rich Composite of objects, each representing an abstract Test. The run() method can then recursively run all tests in the suite and add the results to a similar Composite TestResult class. Both the TestSuite and TestResult can be modeled as a tree and each part in the TestSuite or TestResult composites would be node in the tree. With this approach, the Eclipse plugin would likely change. It possibly could look a lot like JUnit. ToDo: Look @TestNGs plugin and test results! Ant Task 2.0In order to be more flexible, the Ant task could implement a more abstract interface such as: <mxunittask server="localhost" port="8500"
defaultrunner="/mxunit/runner/HttpAntRunner.cfc"
verbose="true">
<test name="mytest">
<param>
<name>type</name>
<value>directory</value>
</param>
<param>
<name>recurse</name>
<value>true</value>
</param>
<param>
<name>dir</name>
<value>/webapps/myapp/tests/*</value>
</param>
</test>
</mxunittask>
One thing to be wary of, is that with the possible changes to TestSuite and TestResult, and with possibly considering a TestNG-like approach, xml-based test suite, there "may" be some conflict ... maybe not, but just something to be aware of. Better MessagesCreating custom messages, adding them as an argument to an assertion, is a typical practice in xUnit; e.g., assertEquals(expected,actual, "my message"). But maybe it's possible to automate this, to some degree, so that when a failure does, if fact, occur, it will be easy for the developer to know what failed. This would save the time required to write the custom message and allow for faster test writing. Data Driven TestsTestNG does a great job at allowing data to be provided to tests without having to change the test. This would be a powerful and useful feature to add to MXUnit. E.g., <cffunction name="testSomething" mxunit:dataprovider="query|struct|array|xml|method"> <cfset assertEquals(.25, delta(q.val1,q.val2)) /> </cffunction> <cffunction name="delta" access="private"> <cfargument name="arg1" /> <cfargument name="arg2" /> <!--- Delta computation here ... ---> <cfreturn deltaVal /> </cffunction> In the above example, MXUnit would execute the private method, delta, for each row provided by the query object. In TestNG, there are two annotations: @Test(dataprovider={method-name}) and @DataProvider, which identifies the method that returns the data; TestNG requires Object for the data provider return type. So, using that scenario, an MXUnit example might look like: <cffunction name="testSomething" mxunit:dataprovider="com.foo.myprovider.method()">??? Or <cffunction name="testSomething" mxunit:dataprovider="myprovider"> <cfset assertEquals(.25, delta(q.val1,q.val2)) /> </cffunction> <cffunction name="myProvider" access="private"> <cfquery name="q" datasource="ds"> select * from Foo </cfquery> <cfreturn q /> </cffunction> <cffunction name="delta" access="private"> <cfargument name="arg1" /> <cfargument name="arg2" /> <!--- Delta computation here ... ---> <cfreturn deltaVal /> </cffunction> So, the first example just specifies the data whereas the second example seems clearer and specifies how or where to get the data. I think in CF we could be loose about the dataprovider output format as long as the fields can be found within data. Expected ExceptionInstead of writing the somewhat cumbersome and counter-intuitive: try{
myMethodThatShouldThrowAndException();
fail("yep this failed ...");
}
catch(any e){
//no worries
}We could instead write: <cffunction name="testExpectedFailure" mxunit:expectFailure="database,application,mycustom.Exception">
<cfset myMethodThatShouldThrowAndException() />
</cffunction>Cedric and Hani must have thought this was pretty important, because they made it the very first test design pattern in the TestNG book! TestNG would choose to specify the above using the @Test annotation like so: <cffunction name="testExpectedFailure" mxunit="@Test(expectedExceptions='database','application','mycustom.Exception')"> It's unclear at the moment why they parameterized @Test for exception handling. I'm clearly missing something ... Built in dependency injection for mockingWe already have injectMethod(...), which is really nice. Is there any need to do this on a larger scale or to encompass behaviour verifcation? Maybe an inject annotation; e.g., mxunit:inject="..."? ultra-lightweightMXUnit should be smaller. It was suggested on the list that there might be a core set of classes and then the other stuff. I do think we can make it smaller. Maybe use JQuery instead of ExtJS? Regardless, I think we can make it more light-weight. |
I believe we should ignore the @Test annotation because in mxunit we've adopted the convention of "if it's public, it's a test". So I like your ExpectFailure?() option better.
one thing though: i know it's just an example, but your version uses parens and such, and i don't think that's going to be helpful for CF. they do it in java annotations because that's how java annotations work. but cf annotations are just strings, no behavior. so i'm thinking we'd want to make life easy on ourselves and just do something like <cffunction name="testExpectedFailure" mxunit:ExpectedFailure?="database,application,mycustom.Exception"
I agree - executing any public method by default is the way mxunit currently behaves and we should keep that. If users want to intentionally omit something, they should be able to say that: mxunit:skip="true|false" or ...?
I agree,too that using the same syntax as Java may be burdensome. It surely is extra typing, which, for some reason, I'm becoming more sensitive too lately. So, just by virtue of having mxunit:foo, would then indicate that "foo" is the target annotation and the framework, hopefully, would delegate the processing of that to that specific annotation; essentially decoupling it from the framework.
One thing we probably want to make sure of is that we will be able to process more than one annotation per test; e,g, mxunit:expectFailure="foo,bar" and mxunit:group="mygroup" ...