Eugene Hutorny

This post describes how SISAM can be used to build a simple environment for automated testing of embedded applications.

Overview

For developing applications with a long life cycle it is very important to ensure that a new release contributes no regression comparing to the previous release. This may become a challenge when an application provides number of capabilities, operates in number of modes or implements number of states. A common approach for addressing this challenge is to establish an automated test environment.

Figure 1 shows a typical automated test environment. It consists of

Batch Controller – a (software) component for running tests in a batch and aggregating overall batch result Test Tool – a component that executes single tests and produce test outputs Comparator – a component that compares test outputs against masters and reports discrepancies Test Cases Repository – a storage of test batches, test cases, test input data and masters Test Log Repository – a storage of test output data and test discrepancies

Figure 1

Typical sequence of events for automated testing:

• Actor uses Batch Controller to start a specific test batch
• Batch Controller reads test cases that constitutes the batch
• Batch Controller calls Test Tool for executing a test case from the batch
• Test Tool reads input test data for the specified test case, applies stimuli to the Testable Application, captures, preprocesses and records application responses
• Batch Controller, upon completion of a test case (all test cases), calls Comparator to compare a test output to a corresponding test master.
• Comparator compares test output and master, writes discrepancies (if any) and signals success/failure
• Batch Controller, upon completion of all test cases, aggregates test results and signals a batch result (success/failure) to the Actor.

A simple test environment

A simple environment for automated testing can be implemented as the following

Test Tool

To use this simplest automated test environment for testing embedded applications, the Test Tool should be a command-line utility, capable to apply stimuli to the testable application and capture the application response to a file.

SISAM & SITEST

SISAM is PIC-based application capable to accept commands via RS232, actuate pins according to the command, acquire data from one or two channels and send acquired data via RS232. SITEST is a command line interface to SISAM. It accepts a sequence of characters from the command line, sends these commands to SISAM and writes data to standard output. These features make SISAM a good candidate for simple testing environment. Figure 2 shows architecture diagram for a test tool build on SISAM and SITEST.

Figure 2

Filters

Due to nature of embedded applications acquired data may vary from run to run even for valid runs. This variability is very application specific; it may make impossible output-to-master comparison, and thus may easily bury the idea of automated testing. To address this issue, acquired data should be filtered and coarsened to the level at which all valid runs would produce identical outputs while invalid runs would still have some difference in the output data. Such a filter should be a command line utility that reads from standard input and writes to standard output. Of course, writing a filter requires additional efforts, however, from authors’ experience, implementing a simple filter does not require any significant efforts.

Function Generators

SISAM is only capable to provide static stimuli for a period of data acquisition. Therefore, if the test conditions require some alternating stimuli applied in parallel with data acquisition, a "function generator" can be used in conjunction with SISAM. A "function generator" is an embedded application that accepts a static input (function ID) and generates a variable function on output.