Model-based software development (MBD) with tools like Simulink and TargetLink in the automotive industry has grown over the past years and is a well-established development methodology today. Despite all the advantages, it can’t be denied that MBD introduces one additional step in the development process. In addition to the production C code there is also the model level and test cases can be executed on both. But should we focus on model or code testing in order to get an efficient and ISO 26262 compliant process. Let’s have a look at our choices:
Test only the code
Since many development teams have written their code manually in the past, they are very used to test on the code level. They might still use the same test solution as before because they are familiar with it. As the code is what finally goes into production, this approach is compliant to the ISO 26262 and therefore a valid approach for safety critical projects.
However, the code is still treated like the handwritten code from the past. The model is treated like some kind of PowerPoint and does not play a role during testing. This means first of all that we are missing out on one of the greatest benefits of MBD, as testing and debugging on the model is more transparent and convenient than on the code level. And even worse, if an issue is detected during testing of the code, the developer has to find the place in the model that generates the issue and fix it there. Once this is done the code needs to be generated again and only then can be tested to check if the problem is fixed (or not), which leads to time consuming iterations.
Test only the model
The second option in the model or code testing comparison, is to test the model only. We take all the advantages that come with model-based development into account and do not care about the generated code. Testing and debugging are intuitive and transparent and it looks like a consistent approach.
The downside of this approach is that it is unclear how the code will behave. There are usually at least minor differences between model and code because you may use floating-point on the model and a fixed-point representation on the code. Even if both use identical data types the behavior can be different and code generators integrated in Matlab/Simulink (e.g. TargetLink or EmbeddedCoder) are not certified to always guarantee the same behavior on model and code level. This means we’d have no evidence about the correctness of the code which goes into production
This approach would also not be compliant with the ISO 26262, which asks for testing being done as close as possible to the final product.
Testing the model and the code
Testing on model and code level combines the advantages of both test approaches mentioned before and leads to an intuitive model-based testing workflow that also considers the code and therefore the final product. It is an ISO 26262 compliant approach and covers all aspects of testing. First, you get fast response if you test on the model and you keep also the focus on the code. The functional testing is done on the model. A back-to-back comparison ensures a correct translation into code.
This is also explicitly what the ISO 26262 standard recommends. Chapter 6, Section 9.4.6 NOTE 4 states, that software unit testing can be carried out on the model level, if followed by a back-to-back comparison between model and code. So it’s not a question of model or code testing but to look at both.
However, since we take two implementation levels into account now, the efficiency of this approach depends on the handling of the model and the code as well as the handling of the test data. This can become quite complex and lead to a not well integrated toolchain of different applications and scripts. It would be in particular very inefficient to have one test tool for the model and another one for the code. First of all, it would mean that you have to create a test environment for both of them. If an issue is fixed you need to go back and test again with both tools and beside the additional testing effort you have to take care to keep the test data consistent between the two tools. That creates a lot of effort and is easily prone to errors.
To face these challenges, an efficient and well-integrated tool support is needed that takes the model and the code into account, integrates different test methods for functional testing and back-to-back comparisons, provides automation capabilities to reduce manual effort and should be qualified for the ISO 26262 standard.
Conclusion
To summarize the model or code testing comparison, we can clearly see, that the model-based development is well-established today and will spread and improve even more.
To enjoy the benefits of the model-based development approach also in the testing process and still remain ISO 26262 compliant, the test workflow needs to take the model and the code into account. To face the additional effort of testing model and code, the test tool must be thoroughly designed to support model-based testing as well as code testing.
