COEN 286: Software Quality Assurance and Testing -- fall, 2001

Test Design

Prof. John Noll

Santa Clara University

1. Terminology

Test Case

Specification of

1. environment
2. inputs
3. expected results

Test or Test Procedure

Test case(s) plus instructions for running them:

1. set-up
2. execution
3. evaluation
4. tear-down

Implementation of one or more test cases.

Test Suite

Collection of tests. We make a distinction between tests and test suites because of shadowing and granularity.

2. Design Issues

Shadowing

Test A shadows B if

• B depends on results of A and A fails.
• B follows A and A corrupts the environment somehow.

Granularity

This is a measure of how many test cases a test contains.

• Valid input tests should be large to reduce overhead: we expect them to succeed.
• Invalid input tests should have one case, as we expect the system under test to report an error at least, perhaps halt or fail gracefully.

Strategy

Requirements-based

Black box testing, cases derived from requirements specification.

Function-based

Black box testing, cases derived from functional specification.

Internals-based

White box testing, cases derived from design or code.

Coverage

A measure of test suite completeness, in terms of

• requirements tested;
• functions tested;
• code statements, branches, conditions, paths executed.

3. Test Case Development Methods

• Happy Path

  Test cases and input values chosen from those known or expected to work.

• Equivalence Partitioning

  Divide the input into equivalence classes.

  1. Identify equivalence classes for each input type.
     • Range - one valid, two invalid classes.

       Year between 1902 and 2038

     • Number of inputs (two to four args) - one valid, two invalid classes.

       Recurring type entries shall have exactly three fields.

     • Set (enumeration) - one valid class (the set), one invalid class (the set complement).

       Weekly, Monthly, Quarterly type.

     • Exists/constraint (must be) - one valid class (input satisfying constraint), one invalid class (input not satisfying constraint).

       Start date must precede End date.

     • Sub-classes - an equivalence class may be divided into sub-classes if inputs are handled differently depending on value.

  2. Write test cases to cover each valid equivalence. Combine these into as few tests as possible.
  3. Write test cases to cover each invalid equivalence class. Specify a separate test for each of these.

• Boundary Value Analysis

  Derive test cases from boundaries of equivalence partitions.

  1. Add equivalence partitions for output to input classes.
  2. For each input and output class:
     1. write valid test cases for representative value and high and low values.
     2. write invalid test cases for values just beyond boundary.

• Error Guessing

  Test designer uses skill and experience to devise test cases to uncover errors.

  • null input.
  • long input.
  • random input.
  • almost correct input.
    • spaces in strings.
    • quoted strings.
    • all CAPS.
  • negative numbers.

4. Testing Activities

4.1 Low Level

Testing individual components, in isolation and combination.

• Unit testing - To discover discrepancies between interface specification and actual behavior. 'make test'
• Integration testing - To discover errors in interfaces between components.
  • bottom-up
  • top-down Produces a demonstrable program early on.

4.2 High Level

• Function testing - To detect discrepancies between functional spec. and actual behavior.
• System testing - Attempt to demonstrate system does not meet original requirements. Cannot be methodical, as requirements are not specific enough.
  • Volume
  • Load/stress
  • Security
  • Usability
  • Peformance
  • Resource usage
  • Configuration
  • Compatibility
  • Installability
  • Recovery
  • Serviceability
  • Reliability
• Usability testing
• Acceptance testing - To compare product to current needs of (real) end users.
  • Alpha - End users inside development organization.
  • Beta - Specially selected external users.
• stress
• performance

4.3 Retesting

• Progression - Testing to validate functionality the first time.
• Regression - Testing to validate functionality is still present.