fsm testing

8/3/2019 FSM Testing

1/52

Model Based testing:

FSM-based Testing

Instructor: Rachida DssouliEmail: [email protected]

Office: EV 007.648

URL: http://www.ciise.concordia.ca/~dssouli

October, 2007


2/52


3/52

FSM

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

S1 is an initial state

Is a transition

it has a starting state

S1,

and an ending state S2

Its label is t1

The input is 1 and an

output 1

/ separates the input

from the output

T1: 1/1


4/52

: Ds --> S: Ds --> Y

Mealy Machine

stateset

initialstate

M = < S, S1, X, Y, Ds, , >

inputset

outputset

spec.domain

transferfunction

outputfunction

Ds S x X

partially defined (specified), deterministic, initialized

S = {S1, S2, S3, S4}

X = {1, 2}

Y = {1, 2}Ds = S x X - {}

S1 S2

S4 S3

t1: 1/1

t2: 2/2

t4: 2/2t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/?

?

An FSM ExampleAn FSM Example


5/52

1) Output fault: point a in FSM fault model.

2) Transfer fault: point b in FSM fault model.

3) Transfer fault with additional states: point c in FSM fault

model.4) Additional or missing transitions: point d in FSM fault model.

5) additional or missing states

Fault Model for Finite State Machine (FSM)Fault Model for Finite State Machine (FSM)


6/52

Specification

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

S1 S2

S4 S3

t1: 1/2

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

Output Fault on transition t1

Implementation under test

IUT


7/52

S1 S2

S4 S3

t1: 1/1

t2: 2/2 t4: 2/2t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3

:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

Transfer fault on t2The ending state is now S3

Specification IUT


8/52

S1 S2

S4 S3

t1: 1/1

t2: 2/2

t4: 2/2t3:1/1

t6: 2/2

t7

: 1/2

t8: 2/2

t5: 1/?

?

S1 S2

S4 S3

t1: 1/1

t2: 2/2 t4: 2/2t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

Transfer fault on t5 with

Additional state

Specification IUT


9/52

Example of implementation with additional

state

Example of implementation with additional

state

S1

S2

S0

b/f

a/e

a/f

c/f

b/fc/ec/e

b/e

a/f

I0

b/f

a/e

a/f

c/e

b/e

I1

I2

b/fc/e

a/f

c/

c/f

a/f

I 1

I 2

I o

b/f

a/e

a/f

b/f

c/ec/e

b/e

I 3

a/e

b/f

c/e

Specification Impl. 1

Impl. 2


10/52

Example of a test suiteExample of a test suite

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

TS = { r.1.1.2.1, r.2.2.1.2.2}

A test suite is a set of inputsequences starting from the

initial state of the machine

r.1.1.2.1

r.2.2.1.2.2

Test Case MS

1.1.2.2

2.2.1.2.2

MIMI

1.1.2.2

2.2.1.2.2

1.1.2.2

2.2.2.2.2

Conforming Non-conforming

Pass TS Fail to pass TS


11/52

Possible changes made by a developerPossible changes made by a developer

Type 1: change the tail state of a transition

Type 2: change the output of a transition

Type 3: add a transition; and Type 4: add an extra state.

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/?

?

No limitation on the numberof such changes allows for

an infinite set of possible

implementations !!!


12/52

Fault model for FSM specificationsFault model for FSM specifications

For the given transition: change the output (output fault) change the next state (transfer fault)if a new state can be added, then

assume an upper bound on thenumber of states in implementations.

For the example above, there are (SxO)SxI = 4x74x5=2820 mutantswith up to 4 states. Among them, 36 mutants represent single(output or transfer) faults, as only 9 transitions are specified.

An example of a very specific fault domain: Only the transitions

related to data transfer may be faulty. These are 4 transitions.This results in only 284 mutants (faulty implementations in

mplf).s3 s4

DT1/IDATind,AK1

DT0/IDATind,AK0

DT0/AK0 DT1/AK1

mutations

s1

IDISreq/DR

CR/ICONinds3s2 s4

ICONresp/CCDT1/IDATind,AK1

DT0/IDATind,AK0

DT0/AK0 DT1/AK1

IDISreq/DRIDISreq/DR


13/52

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/1

t7: 1/2

t8: 2/2

t5: 1/2

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2


14/52

Test Derivation Methods


15/52

Transition tour [Nait 81]

For a given FSM S, a transition tour is a sequence whichtakes the FSM S from the initial state, traverses everytransition at least once, and returns to the initial state .

Detects all output errors, There is no guarantee that all transfer errors can bedetected.

Fault detection power


16/52

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

a/x

b/x

b/y

a/x

a/y

b/x

1

2

3

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

The specification S

A transition tour is :a.a.a.b.b.b

The implementation I1 contains anoutput error. Our transition tour will

detect it.

The implementation I2 contains a transfererror. Our transition tour will not detect it.

SI1

I2

Transition Tour ExampleTransition Tour Example


17/52

An input sequence is a distinguishing sequence (DS )

for an FSM S, if the output produced by the FSM S isdifferent when the input sequence is applied to eachdifferent state. A DS is used as a state identificationsequence.

Detects all output errors, Detects all transfer errors, ADS may not be found for a given FSM.

DS-method [Gonenc 70]


18/52

DS method Example

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

The specification S

S

A distinguishing sequence is :b.b

If we apply it from : state 1 we obtain y.y state 2 we obtain y.x state 3 we obtain x.y

a/x

b/x

b/ya/x

a/y

b/y

1

2

3

I2

A test case which allow thedetection of the transfer error is :

a.b.b.b

If we apply it from the initial state of : the specification we obtain x.x.y.y the implementation we obtain x.x.x.x

Impl.


19/52

DS method

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

Phase 1: Identification of all states/ State cover

From state 1, we can reach state 2 with b/y

and state 3 with a/x

We assume that the reset exist,

Q = { , a, b}

DS = b.b

Test suite = {r.b.b, r.a.b.b, r.b.b.b}

Phase 2, to cover all transitions for output faults

and transfer faults

P = { , a, b, a.b, a.a, b.b, b.a}

Test suite:{r.b.b, r.a.b.b, r.b.b.b, r.a.b.b.b, r.a.a.b.b,

r.b.b.b.b, r.b.a.b.b}


20/52

General methodology for state identification based methods

A) Test generation based on Specification

A-1) Find the Q set or the State cover: minimal inputs that

reach a state from the initial one

A-2) Find the P set or Transition cover: that will cover all remaining transitions

Generate Test Suites using Q and P sets

B) Fault detection

B-1) Apply the generated test suites to the specification to obtain Expected Outputs

B-2) Apply the generated test suites to the implementation to obtain Observed Outputs

Compare the expected and observed outputs (test results)

If they are different then the verdict is fail otherwise it is a pass for the applied test suites.


21/52

The test cases are :

state 1:

state 3 :

state 2 :

Test case structure:

preamble.tested transition.state identification

a.b.bb.b.ba.a.b.b

a.b.b.bb.a.b.bb.b.b.b

DS method Example


22/52

The UIO-method can be applied if for each state of thespecification, there is an input sequence such that theoutput produced by the machine, when it is initially in thegiven state, is different than that of all other states.

The UIOv-method is a variant of the UIO-method. it checkthe uniqueness of the applied identification sequences onthe implementation, meaning that each identification

sequence must be applied on each state of theimplementation and the outputs are compared with thoseexpected from the specification.

UIO-Method [Sabnani 88]

and UIOv-Method [Vuong 89]


23/52

a/x

b/x

b/ya/x

a/y

b/y

1

2

3

The specification S

SUIO sequences are :

state 1 : a.b state 2 : a.a state 3 : a

We assume the existence of areset transition with no output(r/-) leading to the initial statefor every state of S

A transition cover set is :P={e, a, a.b, a.a, b, b.a, b.b}

The test sequences generatedby the UIO-method are :

r.a.b, r.a.a, r.a.b.a.b, r.a.a.a.a,r.b.a.a, r.b.a.a.b, r.b.b.a

UIO ExampleUIO Example


24/52

The W-method involves two sets of input sequences : W-set is a characteristic set of the minimal FSM, andconsists of input sequences that can distinguish

between the behaviors of every pair of states P-set is a set of input sequences such that for eachtransition from state A to state B on input x, there areinput sequences p and p.x in P such that p takes theFSM from the initial state into state A.

Method W[Chow 78]


25/52

a/e

a/f

b/f

b/e

c/eb/f

c/e

c/f

a/f

1

3

2

The specification S

We assume the existence of areset transition with no output(r/-) leading to the initial statefor every state of S

A characterization set is W={a, b}W1 state 1 : a/e,W2 state 2 : a/f, b/f W3 state 3 : b/e

W = Union of all Wi

A transition cover set for the specificationS is :

P={e, a, b, c, b.a, b.b, b.c, c.a, c.b, c.c}

The W-method generates the

following test sequences: (P.W) =r.a, r.b, r.a.a, r.a.b, r.b.a, r.b.b, r.c.a,r.c.b, r.b.a.a, r.b.a.b, r.b.b.a, r.b.b.b,r.b.c.a, r.b.c.b, r.c.a.a, r.c.a.b, r.c.b.a,r.c.b.b, r.c.c.a, r.c.c.b

W method Example


26/52

This method is a generalization of the UIOv method which isalways applicable. It is as the same time an optimization ofthe W-method. The main advantage of the Wp-method, overthe W-method, is to reduce the length of the test suite.

Instead of using the set W to check each reached state si,only a subset of W is used in certain cases. This subset Widepends on the reached state si, and is called an identificationset for the state si.

Wp method [Fujiwara 90]


27/52

The specification S

A characterization set is W={a, b} for W method for state 1 : a/e for state 2 : a/f, b/f for state 3 : b/e

We assume the existence of a

reset transition with no output(r/-) leading to the initial statefor every state of S

The identification sets are : W1={a}, distinguishes the state 1 fromall other states W2={a, b}, distinguishes the state 2

from all other states W3={b}, distinguishes the state 3from all other states

a/e

a/f

b/f

b/e

c/eb/f

c/e

c/f

a/f

1

3

2

Example of Wp method (1/3 )Example of Wp method (1/3 )

effb

ffea321state

Derivation of W


28/52

A state cover set for the specification S is : Q={, b, c}

A transition cover set for the specification S is :P={, a, b, b.c, b.a, b.b, c, c.a, c.c, c.b}

P-Q={a, b.c, b.a, b.b, c.a, c.c, c.b}

Based on these sets, the Wp-method yields the following testsequences :

Phase 1: Q.Wi = {r.a1, r.b.a2, r.b.b2, r.c.b3}The ending state Wi is given in subscript

Phase 2 : (P-Q).Wi ={r.a.a2, r.a.b2, r.b.c.a2, r.b.c.b2, r.b.a.a1,r.b.b.b3, r.c.a.b3, r.c.c.a2, r.c.c.b2, r.c.b.a1}

Example of Wp method (2/3)Example of Wp method (2/3)

W 1 : { a/e } , W 2 : { a/f, b/f } , W 3 : { b/e }


29/52

a/e

a/f

b/f

b/e

c/eb/f

c/e

c/f

a/f

1 2

3

A faulty implementationI

I contains a transfer error 2-

a/f->1 (fat arrow) instead of2-a/f->2 as defined in thespecification S

The application of the test sequencesobtained in Phase 2 leads to thefollowing sequences of outputs :

e.f, e.f, f.f.f, f.f.f, f.f.e, f.f.e, e.f.f, e.e.f,e.e.f, e.e.e

The output printed in bigger size isdifferent from the one expectedaccording to the specification. Therefore,the transfer error in the implementation

is detected by this test sequence.

Example of Wp method (3/3)Example of Wp method (3/3)


30/52

Test derivation based on FSM (Resum)Test derivation based on FSM (Resum)

Transition tour

guaranteed coverage only for output faults

Methods using state identification

with coverage guarantee for output and transfer faults. Three cases:

number of states same for implementation I and specification S

number of states for I possibly larger than for S, but bounded

coverage only for a selected set of transitions (fault function)

Methods without coverage guarantee

Hand made test suite without test derivation procedure

Single long test sequence vs. set of shorter test cases (e.g. test case forspecific transition, test purpose)

Usually, each test case requires reset to initial state; correct resetassumption


31/52

Transition Tour example

Transition tourTT: t1, t4, t3, t9, t2, t3, t6, t7, t8

TT (input/expected output): a/1.b/2.a/1.a/2.b/2.a/1.b/2.a/2.b/2

S1 S2

S4 S3

t1: a/1

t2: b/2t4: b/2

t3:a/1

t6: b/2

t7: a/2

t8: b/2t9

: a/2

Test hypothesis: Initially connected machine


32/52

All state identification Methods

Distinguishing Sequence, UIO, W

Test hypothesis

H1) Strongly connected machine

H2) Contain no equivalent states

H3) deterministic

H4) Completely specified machine

H5) the failure which increases the number of states doesnt occur

The method is applied in two phases from the initial state

phase 1) -sequence to check that each state defined by the specification also

exist in the implementation.

phase 2) -sequence to check all the individual transitions in the specification for

correct output and transfer in the implementation.


33/52


34/52

W method

Assume that the reset exist and it brings the machine from any state to the initial state.

a) Find characterization set W and generate the set of test cases for the specification S

using the W method.

b) Does S have a DS sequence? If not explain why?

a/0

b/0

b/1a/1

a/0

S0

S2

S1

b/0


35/52


36/52

S1

S0

S2

a/0

b/1

a/0b/0

a/1

b/0

S0State

Input

Output

a

0

S1 S2

a a

1 0

S0 S1 S2

b b b

1 0 0

S0 S1 S2

a.b a.b a.b

0.1 1.0 0.0

Derive a DS of length up to 2 for S

a.b is a DS for S

Specification S

Comment: a as input at each state will loop on the state, sequence of a.a. cannot be a DS, the output will

be 0.0.. or 1.1

Transition tour:

Input

Output

a.b.a.b.a.b

0.1.0.0.1.0

Examples Suite

Q set: permits to reach each state


37/52

Q set: permits to reach each state

from the initial state

Q = { , b,b.b}

The first b to reach the state S2

b.b to reach the state S1.

P set is transition cover, permits to execute

each transition at least one starting from the

initial stateS1

S0

S2

a/0

b/1

a/0b/0

a/1

b/0

S0

a

bb

a

b

b

b

b

b

b

b

a

How to derive P set: find allPath starting from the size1 and up and each transition

should be traversed at least once

P = {, a, b, b.a, b.b, b.b.a, b.b.b}

more than one p set may exist, this depends on the alternative

paths that the automata may have.


38/52

The goal of the Phase 1 is identification of

the states in the implementationDS = a.b, Q = { , b,b.b}, P = {, a, b, b.a, b.b, b.b.a, b.b.b}

Phase 1

Q.DS = {r.a.b, r.b.a.b, r.b.b.a.b} Expected output of phase 1is:

{-.0.1, -.1.0.0, -.1.0.1.0}

Phase 2 ( DS in bold)

P.DS= {r.a.b, r.a.a.b,r.b.a.b, r.b.a.a.b, r.b.b.a.b, r.b.b.a.a.b, r.b.b.b.a.b}

{-,0.1, -.0.0.1, -.1.0.0.0, -.1.0.1.0, -.1.0.1.1.0, -.1.0.0.0.1}

S1

S0

S2

a/0

b/1

a/0b/0

a/1

b/0

Note that, the test suites for phase 1 and 2 should be

Derived from the specification and applied to the

implementation to check it for output and

transfer faults.


39/52

S1

S0

S2

a/0

b/1

a/0b/0

a/1

b/0

S1

S0

S2

a/0

b/1

a/0

b/0

a/1

b/0

Specification SImplementation I

Apply the transition tour to the implementation I and comment

Transition tour applied to S

Input

Output of S

Output of I

a.b.a.b.a.b

0.1.0.0.1.0

0.1.0.0.1.0

The implementation I has a transfer fault,

the fault is not detected byTransition tour.

Transition tour detects all output faults but

Doesnt guarantee the detection of transfer faults


40/52


41/52

S2

S0

S1

a/0

b/0

a/0b/0

c/0

a/1

C/0

b/0

c/1

Specification S

State

Input

Output

S0 S1 S2 S0 S1 S2 S0 S1 S2 S0 S1 S2

a a a b b b c c c a.c a.c a.c

0 0 1 0 0 0 1 0 0 0.1 0.0 1.1

Derive a UIO sequence for S

UIO state S0 = c/1

UIO state S2 = a/1

UIO state S1 = a/0.c/0

Transition tour for S

a.b.a.b.c.a.c.b.c

0.0.0.0.0.1.1.0.0


42/52


43/52

completeness: completely specified or partially specified

connectedness: strongly connected or initialy connected

reducibility: reduced or non-reduced

determinism: deterministic or non-deterministic

13

Assumptions about specificationsAssumptions about specifications


44/52

Assumptions about implementationsAssumptions about implementations

t7: 1/2

S1 S2

S4 S3

t1: 1/1

t2: 2/2t4: 2/2t3:1/1

t6: 2/2

t8: 2/2

r/-

r/-

r/-

r/-

Deterministic

Completely defined

react to any input

Limited extra states

Reliable reset

not necessary

15

R l it t ti ti


45/52

Regularity, a testing assumptionRegularity, a testing assumption

This type of assumption allows to limit testing to a finite set of behaviors in thecase of systems that exhibit an infinite behaviors. Examples are

programs (or specifications) with loops and integer input and outputparameters

finite state machines reactive systems, en general

Principle: assume that the implementation has a regular behavior, whichmeans that the number of control states of the implementation is limited.

If the number of states is not bigger than the corresponding numberof states of the specification, then all loops (of the specification) haveto be tested only once.

This is the idea behind the FSM fault model where the number of

implementation states is limited to n, or to some number m > n. This is also the idea behind certain approaches for testing

program loops and for testing in respect to specifications in theform of abstract data types.

Independency, a testing assumption

Independency, a testing assumption


46/52

Independency, a testing assumptionp y g p

Principle:

The different submodules of the system under test are

independent, and faults in one module do not affect the possibilityof detecting the faults in the other modules.

This is a controversial assumption:

In most complex systems, modules or components are dependent.The reasons are:

they share resources (e.g. memory)

they have explicit interactions

Example:

several connections supported by a protocol entity

test only one connection in detail (it is independent of the

others) the others need not be tested, since they are all equal

(uniformity assumption, see below)


47/52

Independency (suite)Independency (suite)

The independency relation is a reasonable assumption in certain cases.

Example:

Equipment to test

Entity N Entity NEntity N

Entity N+1

SAPSAP SAP

Uniformity, a testing assumption

Uniformity, a testing assumption


48/52

y, g p

Uniformity assumption / Congruence

Origin: Partition Testing [Weyuker 91]

Principle There exist similar behaviors. If they are grouped under an

equivalence relation, then it is sufficient to test one behavior ofeach equivalence class for conformance testing.

Special cases:

Principle of partition testing: Apply test for at least onerepresentative for each partition of the input domain (software

testing, EFSM testing)

Equivalent actions for EFSM Equivalent states for FSM

F i i t t d t i i


49/52

Fairness in respect to non-determinismFairness in respect to non-determinism

Many systems have a non-deterministic nature. In particular, theparallelism of distributed systems introduces many possible interleaving

of individual actions within the different system components.

The assumption is that all the execution paths effectively realizedduring testing cover all paths that are pertinent for detecting thepossible implementation faults.

a/1

a/2

a/4

s1

s2

s3s4

non-determinism

Partially defined FSMs

Partially defined FSMs


50/52

Partially defined FSM sPartially defined FSM s

Non-specified transitions need not be tested. However different

interpretations of undefinedness have an impact on testing:

completeness assumption

non-specified transition is implicitly defined, e.g. stay in same

state (as in SDL), or go to an error state

methods for completely defined FSMs may be applied, however,test will rely on implied transitions

dont care

no specific behavior is specified non-specified transitions must be avoided by test cases

robustness tests may be applied to check the reaction of theimplementation for non-specified situations

forbidden not possible to invoke non-specified transitions


51/52

Fault Coverage Evaluation

M th d f F lt C E l ti

Methods for Fault Coverage Evaluation


52/52

Methods for Fault Coverage EvaluationMethods for Fault Coverage Evaluation

The definition of fault coverage always depends on fault model!

Exhaustive mutation analysis

Monte-Carlo simulation method

Deciding completeness

minimize an FSM which is given in the form of the TS, if its minimalform is equivalent to the given FSM then TS is complete (the max #states is assumed), otherwise it is not complete [see Yao]

Structural Analysis

it evaluates the fault coverage of a given test suite by directly

analyzing the test suite against the given FSM. Count the number ofstates distinguished and transitions checked by the test suite. Anumeric measure easy to evaluate (linear complexity) [see Yao]

Different possible measures

compare number of implementations (common approach)

compare the log of number of implementations (corresponds tocounting transitions covered) [called order coverage by Yao]

fsm testing

Documents