COEN 286: Risk Analysis exercise

Software Testing and Quality Assurance -- winter, 2008

Prof. John Noll

Santa Clara University

$Id: risk-analysis.xpmlbody,v 1.3 2008/02/01 08:19:44 jnoll Exp $
IMPORTANT: please analyze the Active Processes and Process Detail pages only ; you don't have to analyze the other pages. Due $risk_a_due, at the beginning of class. BE SURE TO SPELL CHECK AND PROOFREAD YOUR WORK!

1. Overview

Risk analysis can help prioritize verification and validation activities by ranking potential problems according to the probability and consequence of their occurring. This exercise requires you to do a qualitative risk analysis on the PEOS web interface, as described in the next section.

2. Tasks

Perform the following tasks as specified.

  1. Analyze risks
  2. Repeat the following tasks until you are satisified with the result:

    1. Analyze probability
    2. For each function (primary and secondary) in the PEOS web interface, assess the probability that the function will fail to behave as required, based on your assessment of the technical difficulty of implementing the function.

      Use the function inventory as the source for functions (rather than your own inventory).

      IMPORTANT: please analyze the Active Processes and Process Detail pages only ; you don't have to analyze the other pages.

      Use the qualitative scale described in the risk analysis procedure. The probability is an assessment of the likelihood that the product implementation will fail to meet the requirement as specified, not that the user will make a mistake. In other words, we are interested in the probability that the product will fail to detect invalid input, or fail to correctly handle valid input, or otherwise behave incorrectly. We are not interested in the probability that a user will use a function incorrectly, or provide invalid input, or otherwise misuse the product.

    3. Explain probablity
    4. Explain the reason for your assessment.

    5. Analyze impact
    6. Assess the impact of a function's failure, based on your understanding of the product's purpose and your assessment of how a failure of that kind might affect the end user or customer, and/or the developer's bottom line. You should consult the project authority if you can't determine the impact with confidence. Again, use the qualitative scale described in the risk analysis procedure.

      One way to do this is to identify the function(s) most and least likely to fail, then compare other functions to these. Determine the most severe failure in terms of impact, and assign this an impact of Definitely High. Then compare other functions to this ``benchmark.''

      The customer's requirements list can give another clue, as these are ordered by importance: items at the bottom of the list are less important, and therefore would have lower impact if they fail.

    7. Explain impact
    8. Write a brief justification for the value you chose.

    9. Document result
    10. For each risk analyzed, write a separate entry, including the following bulleted or numbered sub-sections:
      1. Function description.
      2. Probability assignment and justification.
      3. Impact assignment and justification.
      4. Exposure calculation (value to be added later).
      Example Format
      1. ...
      2. ...
      3. Function: File->Load
        • Probability: definitely unlikely. Reason: this is a technically straightforward feature involving simply opening a file and reading its contents. Most of the hard work is done by the operating system.
        • Impact: definitely high. Reason: if a file cannot be loaded the tool is useless.
        • Exposure: 1 * 9 = 9
      4. ...
      5. ...
      6. ...

      Note: You may find that many functions have the same justification for their probability or impact. If this is the case, write an introductory paragraph explaining the broader failure or impact categories into which groups of functions fall; then refer to these categories in your analysis.

  3. Determine exposure
  4. Repeat the following tasks until you are satisified with the result:

    1. Calculate exposure
    2. For each function analyzed previously, calculate the exposure associated with the function, based on your quantitative assessment:
            Exposure = Risk * Impact
          
      Where the values for Risk and Impact are obtained as follows:
      Assessment Risk Value Impact Value
      Definitely High 8 9
      Somewhat High 6 7
      Somewhat Low 3 4
      Definitely Low 1 1
      (These values are specified in the risk analysis procedure.)

    3. Document exposure
    4. Add the exposure calculation to the results.

      Note: it's important to do the exposure calculation after you have done the qualitative assessment, so that the values for each level do not influence your assessement.

  5. Sort results
  6. Sort the analyzed functions by exposure, from highest to lowest.

  7. Create report
  8. Submit a formatted document, in 12pt type, stapled in the upper left hand corner with a single sturdy staple. Include a cover page with the document title (name of this exercise), your name and student ID, the name of this course, and the date.

    The entries should be ordered by exposure, from highest to lowest.

    Be sure to read and follow the Quality Standards described on the Course Information page.

    If for some reason you cannot attend class on the due date, notify me via email ahead of time, and submit your results via email as a PDF attachment (do NOT send Word doc files or other formats). When printed, your attachment must conform to the requirements listed above.

    The final document is due February 7, at the beginning of class.

3. Assessment

This exercise will be assessed on the following criteria:
  1. Completeness: are all functions covered?
  2. Correctness: is the analysis plausible? Are the failures associated with the product, not its users? Are the calculations correct?
  3. Presentation: is the writing clear? Is the formatting professional?