by Norbert Ruck, Ralf Lulei, Thorsten Geiselhart and Gerd Rockweiler
Testing is generally understood as concerned with finding defects in systems under development. The objective is to gain confidence in knowing whether a technical development functions as expected and whether it behaves in a consistent manner. In this sense the test is not meant to be a guarantee for absolute correctness. Instead testing makes it possible to determine the circumstances and the likelihood that deviations from expected outcomes will occur. In any case testing should reduce the number of defects in the system be able to demonstrate as far as possible that the system functionality is implemented.
In many sectors failures can be handled with low priority because the end user doesn’t notice them or because the system usability does not suffer. When people can be endangered by such failures, safety takes on a different significance. This is why testing is given a high level of significance in the automotive sector. This is accommodated by international standards and to no small degree by internal company processes. These regulate in detail the path from customer requirements to test cases so that a complete and documented process can be established upon which a car’s control unit can be developed which poses absolutely no risk to its occupants.
The so-called V-model has proven its worth here by providing a sequence with increasing levels of detail for the development of software projects, which we also find used in the automotive sector (see diagram).
An automobile manufacturer fundamentally lays down the requirements on a software system in a customer requirements specification document (in general, for an automobile’s
control unit). The functional specification then sets out how these requirements will be implemented. This is the basis for the creation of test cases which are linked to the functional specification and thereby enable evidence to be provided that test cases exist for all requirements. Reviews verify the correctness and completeness of each step.
Within the life-cycle of a product a series of documents are to be produced, examined and reviewed so they can ultimately be corrected or extended. In many cases a standard MS-Office solution is employed which is in any case already in company use. This probably means that the customer requirements and functional specification are created with a word processor whereas the test case specifications and test protocols are mostly stored in a spreadsheet. Even though this practice benefits from low procurement and learning costs, the disadvantage is that the compliance with required standards and process requirements are made more difficult.
This is particularly noticeable when reacting to requirements changes, to ever faster software release cycles and when showing that all requirements are covered by test cases. For these purposes the use of a special software solution is recommended which replicates the development process. Marquardt GmbH, a company which amongst other things produces software for the control units used by several large automobile makers, employs the IRqA (Integral Requisite Analyzer Requirement tool) from Visure Resolutions. The objective here is to achieve the customer’s requirements as quickly as possible with maximum quality while at the same time demonstrating full implementation and ensuring traceability across all documents. And this has to be within the entire product life-cycle. We will now show the implementation of this process at Marquardt GmbH.
Requirements management tool
Within the tool itself there are separate areas called “Engineering Processes“, which include the management of actual requirements and the management of test scenarios. Various roles which are implemented in the tool permit individual user rights to be allocated for these areas. In this way the software developer receives read-only rights for the „Test Scenario Process“ and at the same time has write privileges for the „Requirements Management Process“. For the software tester the opposite applies. The separate areas within the requirements management tool make such a role-specific separation possible and appropriate, all required data can be viewed but only role-specific information can be modified.
A further advantage which results from the use of a common database is traceability. This means the linking from customer requirements (in the requirements document) to the functional specification (in the functional requirements document) through to validation that a complete coverage of all requirements in the system tests can be shown. This is presented in a so-called traceability matrix, which shows which requirements are tested by which test cases and where possible holes in the test coverage are present. A further major advantage arises when a change is made on the requirements side, irrespective of whether the requirements document or the functional specification is changed. These changes show up as “suspected links” in the traceability matrix. These show at a glance which requirements have changed and for which test cases modifications will be needed. In this manner the searching and location through various versions of the documentation history becomes redundant. Everything is shown in the central requirement management tool.
The data base within IrqA permits all users to access current data at the same time. A checkout mechanism ensures that only a single specific user can change a particular data record at a given time. In this case other users can instantly read the data without requiring that the author first check the data record in. In this way any dependent data records which may exist can be immediately modified. In addition, further users have the possibility to work with other information. This allows work to take place on various data records in parallel and leads to significant synergy effects.
The history function allows earlier versions of data records to be compared with the current
version. At the end of each development cycle a so-called baseline can be set for all data records or for partial areas. This represents a “freeze” of the captured data, which later allows access to the exact data record that was current, reviewed and released in this software release.
Engineering process – Test scenario
The administration of test requirements at Marquardt GmbH takes place in a five stage hierarchical structure. This is constructed as follows: test root – test scenario – test object – test group – test case.
Test root is the basis for all data records within the IRqA engineering process. All test scenarios are summarized in the test root. According to this nomenclature a test scenario contains all test objects. In this sense the system test, for example, is represented as a test scenario. According to the scale of the project the tests of individual system components are also test scenarios. For complex projects with many components there are several test scenarios and for simple projects with just one component a single test scenario is sufficient.
Each test scenario is divided into several test objects. The test objects are divided up into the functional aspects of the item under test (e.g. CAN, LIN, operational voltage) and the test objects divided up into test groups within each of these functional aspects. This results in easier administration of the test cases. Test objects within the LIN test scenarios can, for example, be Wake Up, LIN-failure or communications error.
The test cases themselves are at the lowest level in the hierarchy. This consists of a unique code, the link to the appropriate requirement and a name from which the test content can be deduced. A list of codes and names can be made available as a high level specification at the request of the customer. The actual test case content and procedure is represented by descriptions of preconditions, actions, results and remarks. Furthermore, each test case is classified according to risk, and with the help of an additional attribute the distinction between positive and negative test case can be made. Further attributes are available for managing the test cases.
The block structure is administered hierarchically within the engineering process. In this manner the test root, the test scenarios and the test objects each receive their own block, whose attributes are inherited from the test root. This guarantees a consistent management of test cases across all blocks. the attributes are standardized within Marquardt GmbH, although project-specific extensions are possible.
Usability of IRqA
Compared to simple text processing, the data base supported requirements management offers major advantages for Marquardt GmbH. As described in the chapters above, the various aspects (e.g. traceability, user rights etc. are well covered and performed by this tool. The usage of a program like this takes some getting used to at first and sometimes appears a little cumbersome. The initial set-up and also large changes can utilize an import/ export interface which offers the possibility to save time by exchanging data with standard Office solutions. In the Requirements Engineering Process the interfaces are to Microsoft Word und Excel as well as the convenient XRI (XML for Requirements Interchange). In the Test Scenario Engineering Process the selection of interfaces is limited to Microsoft Word und Excel.
The administration and viewing of various requirements in IRqA opens up a level of visibility which would never have been possible amongst the multitude of requirements in various documents, standards and norms. The displayed data records can be filtered according to attribute, text or such like. Via the previously mentioned block structure the data records can be displayed for just one or several blocks. By the setting of levels it is possible to show hierarchical views, assorted list views or reduced views dependent on attributes. Project-specific or user-specific views can be stored in the data base.
The technical content (test cases) of a requirement consists of a unique code, a descriptive name, the detailed test case description and its classification. This information uniquely describes a test requirement and using the link allows the coverage (traceability) to the customer requirements document and functional specification document to be shown. This is important for ensuring test coverage and for providing information, also for the OEM. Showing coverage isn’t enough though for planning the scope, depth and time aspects of test execution. Apart from the test requirement’s technical details the description therefore also includes information which better describe its planning and maturity aspects. This information is therefore also considered just as much a component of the test case’s description as the technical inputs and needs to be stored with all the other information in the requirements tool.
For test process use Marquardt GmbH has divided the additional information into two categories. On one hand the planning information and on the other the life-cycle attributes.
These attributes are there purely to support the planning process and do not describe the functions or their content. In this way the release in which a test requirement will be checked is captured. Normally the number of test requirements grows in the course of project in proportion to the number of requirements. It isn’t therefore possible to describe each and every test requirement right from the start of a release. A further attribute is assigned so that the release after which the test requirement was introduced or the test execution in which the test requirement was performed can be documented. This procedure makes it possible to individually group the test suites (i.e. the test cases to be executed) for each test execution, which also permits retrospective traceability and, if where doubt exists, a repetition. The attribute can be an indicator of increasing test scope and also a marker for test cases which are no longer used due to changed requirements.
„test phase“ describes the type of test case according to the ISO/IEC 15504 standard. This standard requires various test cases for a new release. In order to avoid administering multiple test projects in the requirements management tool the test requirements in a project are allocated to a particular test. A test requirement can relate to a software test, a system integration test or a system test.
In the software test process time delays and unplanned bug-fixes occur quite frequently. This is taken into account by the test team when creating the test case by assigning a regression level for each test requirement. This additional information is not used during a regular test execution but can be used by the tester to prioritize the test should the entire time period planned for a test no longer be available.
When a test execution is being prepared for the next test it is important to know how many and which particular test requirements will be checked on which test harness so that these can be allocated to the various test systems. Do we need to check them on a fully automatic test system or do we need to perform some or all of them manually on a test bench? This information is of great significance for the detailed scheduling of the test execution because a manual test normally needs appreciably more time than an automatic test.
In addition this information provides a status regarding the reliability of the test results. The execution of an automatic test script is exactly reproducible compared with a manual test, which can suffer from variability of timing and sequence which under some circumstances can influence the results.
Up to now the elements of a test requirement described have related to functional information and planning attributes. For creating a test requirement, the development status also provides an indication of the test case’s quality. On the basis of this maturity information it is possible to determine whether a test requirement is already available for a test execution or whether it still needs to be completed or changed. In order to accept a test requirement which has been created by the test designer for test execution it must first be submitted to a review. This entails explaining to the author of the functional specification and to software quality assurance why the test requirement was created that way and why particular attributes were allocated. Only then is it possible for the
test case to be used for the test execution.
Values of development status are:
– In Work: The test requirement is currently under construction and its contents and additional information are not yet completely defined.
– In Review: The test requirement is ready for review. For release the author of the functional specification and software quality assurance must give their approval.
– Agreed: The test requirement has reached the highest maturity level and may now be integrated into the test execution.
This attribute is additional for test requirements which are to be executed in fully automatic test suites. The attribute captures additional information about the status of the automation. In a similar manner to the software development of control equipment, there are various states which must be transitioned and documented:
– is the test requirement already implemented in the test automation language?
– has the test input file been reviewed and released?
– has the test input file been put into operation?
The entered attributes make it easier for the person with test responsibility to plan the test execution and to properly justify the quality of the testing requirements to a third party. As a result of the data administration in the requirements management tool every project member can easily see the current progress, the test coverage, the status of the automation or the maturity of the test requirements. It is not necessary to check back on whether the documents in the version management tool are the right ones; we can rely on the fact that the current data can always be accessed in the requirements-management tool.
All information needed for the test execution and the creation of metrics can be exported from the requirements management tool as a protocol. This contains both technical and planning-specific attributes. The previously described filters or block filters are used to enable the export to be performed. During the test execution the protocol is supplemented by the automatic test system or the tester with information relevant to the test execution, in particular the results of the executed tests. The information collected, test results and further attributes are then used for various metrics. These are based primarily on the entered test requirement attributes but also use the linking information to the requirements in the functional specification and, for example, the date of the last modification. All metrics are calculated and displayed without further effort so that they can immediately be made visible to the management, the project leader or the OEM.
For the software validation various process work packages have been defined which were developed according to ISO/IEC15504. These work packages will be continually extended and adapted in order to meet the future demands of the automobile makers and to be able to offer an even higher level of quality. To achieve this the interfaces between the three pillars of software validation will be adapted (see diagram) so that the Marquardt process always meets the current requirements.
For test case development a syntax has been defined so that a common test case title and description can be achieved. Using this standardized description it should be possible in the future to use a test case generator to create the test scripts for the various fully automatic test systems. In addition, the test specification will be exported from the requirements management tool into the test case generator and translated into a test script which suites the test system (see diagram).
The common syntax will additionally permit an easier introduction into other projects for every member of staff. All test case information is held in the data base supported requirements management tool.
In the future it is planned to use test case libraries which the test case designer can use when creating the test specification. Test cases which come into question for a library are those which relate to test objects which can be standardised across several projects, such as those relating to the bus systems used, diagnostic functions, or the behaviour of the control units in various operational states. An important and desirable aspect for testing would be the possibility to access the requirements management tool directly from the test application on the test bench in order that test suites can be executed. The data for the relevant test cases in the test run should be buffered in advance of the test or read on real time. After the test run the results would then be automatically stored in the requirements management tool. This would permit, amongst others, the automatic versioning of all test-relevant data in a tool.
These ideas should ensure that Marquardt GmbH can continue in the future to react quickly to the changing market and its increasing requirements in order to ensure the quality which the engineers themselves, the OEM and, last but not least, the automobile passengers deserve.
The authors are responsible for the system test in the software validation at Marquardt GmbH.