This post is part of the SATF work.
The valuable artefact of auto acceptance testing is a test suite. This section reviews the test suite of the system which was developed in parallel and served as a main testbed for the simulation acceptance testing framework itself. It also shows some curious tests of a side project with an entertaining cartoon as a digestive. Continue reading
This post is part of the SATF work.
The picture would not be complete without describing the simulation auto acceptance tests problems revealed over the mobile robot control system development. The first and the most problem is that Microsoft abandoned MRDS support. Since this deprives me of the hope to see any of the graver MRDS flaws ever fixed, I could wrap up the section right here. But then I have to name these flaws, at least. And to look impartial, I will balance them with some homebred flaws. Continue reading
This post is part of the SATF work.
The good news is that all the words in the STF posts before were not in vain. It turned out that simulation acceptance tests make, at least, some sense, if not money. This section presents the various, sometimes unexpected, aspects of that sense. Continue reading
This post is part of the SATF work.
Proper time handling within MRDS turned out to be an absolutely non-trivial task. But its nice part is an opportunity to demonstrate and exploit some of the advanced MRDS features. Namely, the service “polymorphism”, touched in the timer generic service contract; the Observer pattern DSS implementation employed in the timer subscriptions and notifications; and the pattern of interaction between a simulation entity and a service. Continue reading
This post is part of the SATF work.
The more simulation acceptance tests there are in a project, the more value they can bring. Therefore, it makes sense to talk about a test suite – an organized bunch of tests. The section shows how a multicomponent test is glued into a whole, states the options for the test code markup and explains how a test suite is organized. Continue reading
This post is part of the SATF work.
The procedure stated in the “Test flow” section defines the entities involved in simulation auto acceptance testing. But the MRDS mechanics and general unit testing patterns are what determines the actual structure of a test. An example of that structure was presented early on in the “Test example” section, although now, MRDS have been reviewed, it starts making more sense. Since the current section is actually an explanation of what was presented there, I do not reference the section explicitly below, but imply it as a primary example. A simulation auto acceptance test consists of three parts: physics environment description, software description and test scenario. Continue reading
This post is part of the SATF work.
Ideas from the “Solution” and “Probabilistic testing” sections applied to the reality and devices of the MRDS framework gave birth to the simulation acceptance test execution procedure depicted on the picture below. Every step of it will be illustrated by the Monte Carlo localization test of the “Test example” section. Continue reading
This post is part of the SATF work.
This is a good time to review Microsoft Robotics Developer Studio architecture, since all the following will use the terms and notions of that system. Quite naturally, it strongly influenced simulation testing framework algorithms design and data structures. MRDS is a .NET-based framework which consists of three main components: DSS, CCR and simulation engine, plus tooling. Continue reading
This post is part of the SATF work.
The section elaborates on one particular set of simulation auto acceptance tests – the Monte Carlo localization service tests. These tests are used as an example in the most of the following material, so they are worth of individual scrutiny. By the same token, the MC localization case is complex enough to illustrate the most important aspects of the testing framework including probabilistic testing. Although its implementation part looks Greek at the moment (further sections will clarify it), it is also presented, along with the general description of the tests and problem, to provide the taste of what the whole idea is worth in reality. Continue reading
This post is part of the SATF work.
One particular testing issue requires some special measures, but could also be addressed within the approach proposed in the “Solution” section. The issue is testing of the nondeterministic aspects of robotics systems, which are inherent and abundant within the field. If the outcome of a given test is nondeterministic i.e., every single run of the test, the initial configuration being the same, could end up with either a failure or a success, then neither of them would imply the definitive answer to the real question behind the test – an attempt to assess some nondeterministic process. There are two distinctive flavours of this problem. Continue reading