This post is part of the SATF work.

The more simulation acceptance tests there are in a project, the more value they can bring. Therefore, it makes sense to talk about a test suite – an organized bunch of tests. The section shows how a multicomponent test is glued into a whole, states the options for the test code markup and explains how a test suite is organized.

As discussed in the “Test structure” section, each simulation acceptance test fixture consists of three files describing physics environment, software configuration and the test scenarios. The name of the test fixture is what glues together all the files – and determines the fixture identity. It is introduced by the test scenarios code as the single necessary parameter of the SimTestFixture attribute – the attribute designating the test fixture class. Other fixture files must have the same names, differing only in extensions (environ.xml and manifest.xml). The service configuration file names could be arbitrary because they are stated explicitly in the correspondent manifest file.

All the test suite fixtures are compiled into a single .Net assembly, whose name is passed to the test runner service as a parameter, along with the path to the directory (under MRDS root) where xml fixture files reside. The assembly itself is placed into the standard MRDS binary folder with all of its dependencies. On its start, the test runner loads the assembly and proceeds to tests collection. Among all assembly classes, it searches for those with the SimTestFixture attribute. For each of them, it could find corresponding environment and manifest files by the fixture name. Two more parameters of the attribute worth mentioning here are Wip and Ignore. The latter, evidently, makes the test runner ignore the fixture altogether, allowing to temporarily suspend some problematic tests. The Wip parameter, standing for Work In Progress, is adopted from the Cucumber tool. It allows to suspend execution of all but Wip tests, thereby speeding up the pure development phase considerably.

The test code markup is a somewhat isolated but very important issue. Apart from marking the test classes and methods, its main task is not to be on the way of a test developer. The conventional .Net unit tests markup way, the attributes, is easily adopted in the part of the test fixture and test classes designation. Necessary information fits easily into the attribute parameters. The situation with the test methods is not so straightforward, because, first, all of them, in contrast to conventional tests, have nontrivial signatures. And second, all of them are optional. Two approaches to the test methods designation are provided: via attributes and via interfaces. The attribute approach, being in line with class level attributes, provides the special attribute for each method. One drawback is that the method’s signature can only be verified at run time. The other is that a developer must either remember signatures by heart, or copy-and-paste them all the time. The second approach amounts to adding the special one-method interfaces (IPrepare, IStart, etc.) to the interface list of a test class. Then a standard automated tool generates stubs for the interface methods, thus preventing errors in the test methods signatures. The interface approach disadvantage is heterogeneity: the class level part of the simulation test markup is performed via attributes, while the method level one – via interfaces.

[SimTestFixture("fixture_name", PhysicsTimeStep = 0.001f)]
public class TestFixture{
    [SetUp]
    public void SetUp(SimulationTesterService testerService){
    }
    [SimTest(2.5f)]
    public class TestWithAttributes{
        [Fixture]
        public TestFixture Fixture { get; set; }
        [Prepare]
        public void PrepareEntities(IEnumerable entities){
        }
        [Start]
        public IEnumerator Start(){
            yield break;
        }
        [Test]
        public IEnumerator Test(Action @return, IEnumerable
                           entities, double elapsedTime){
            @return(true);
            yield break; 
        }
    }
}

[SimTestFixture("fixture_name", PhysicsTimeStep = 0.001f)]
public class TestFixture{
    [SetUp]
    public void SetUp(SimulationTesterService testerService){
    }
    [SimTest(2.5f)]
    public class TestWithIfaces : 
                 IFixture, IPrepare, IStart, ITest{
        public TestFixture Fixture { get; set; }
            
        public void Prepare(IEnumerable entities){
        }

        public IEnumerator Start(){
            yield break;
        }

        public IEnumerator Test(Action @return, IEnumerable 
                           entities, double elapsedTime){
            @return(true);
            yield break;
        }
    }
}

Attribute and interface test markup

This basic set of tools and conventions has proved to be quite sufficient for all practical purposes of the mobile robot control system development. As a matter of fact, it evolved to meet, if only roughly, all the needs of this project.