COEN 286: Software Quality Assurance and Testing -- fall, 2001

Risk Analysis Exercise

Prof. John Noll

Santa Clara University

1. Overview

Risk analysis can help prioritize verification and validation activities by ranking potential problems according to the probability and consequence of their occurring.

We define these concepts as follows:

Risk

The probability of an adverse event occurring.

Example: the system crashes, causing an airplane crash.

Impact

The consequence (usually expressed as cost) of an adverse event occurring.

Example: the cost of the lost airplane, plus the compensation to the families of each of the passengers, plus lost future customers.

Exposure

A measure of the ``importance'' of the risk, expressed as the risk impact multiplied by the probability:

Exposure = Risk * Impact

By calculating the exposure associated with each risk, we gain a number by which risks can be ranked, and thus our activities prioritized.

The problem with this approach is that it is frequently difficult to accurately estimate risk probability and impact. Insurance companies maintain voluminous records of historical data to aid in making these estimates. We, however, don't have such resources, and fortunately don't need them: since we are mainly interested in ranking risks to prioritize our activities, we only need to establish the relative impact among risks.

This exercise requires you to do a qualitative risk analysis on your chosen product, as described in the next section.

2. Instructions

Considering again the Budget Program from last year, do the following:

1. Identify five product requirements. State each requirement as concisely and unambiguously as possible.
2. For each requirement, identify a risk associated with that requirement.
3. Assign a probability to the risk, using the following subjective scale:
   • Definitely likely.
   • Somewhat likely.
   • Somewhat unlikely.
   • Definitely unlikely.
   Briefly explain why you characterized the probability as you did.
4. Assign an impact to the the risk, using the following subjective scale:
   • Definitely high impact.
   • Somewhat high impact.
   • Somewhat low impact.
   • Definitely low impact.
   Briefly explain why you characterized the impact as you did.
5. Calculate the risk exposure for each risk, by assigning the following arbitrary values to your subjective evaluations:
   • Definitely likely: .9
   • Somewhat likely: .6
   • Somewhat unlikely: .4
   • Definitely unlikely: .1
   • Definitely high impact: 9
   • Somewhat high impact: 6
   • Somewhat low impact: 4
   • Definitely low impact: 1
6. Order the risks from highest to lowest exposure.
7. Answer the following questions:
   1. Does the ordering you achieve agree with your intuition about the relative importance of the risks you identified? Why or why not?
   2. Which would you be inclined to trust more? Why?

This technique exploits a simple two step qualitative analysis to produce a quantifiable risk ordering. For each risk, you must establish

1. Is it likely or not?
2. How likely or not likely is it?

1. Is the impact high or low?
2. How high or low is it?

3. Deliverables

Submit a formatted document in 12pt type, stapled in the upper left hand corner with a single sturdy staple. Include a cover page with the document title (name of this exercise), your name and student ID, the name of this course, and the date.

For each risk identified, write a separate section, including the following bulleted or numbered sub-sections:

1. Requirement description.
2. Risk description.
3. Probability assignment and justification.
4. Impact assignment and justification.
5. Exposure calculation.

Write an additional section containing your analysis.

The final document is due at the beginning of class on October 15.

In addition to the requirements enumerated above, be sure to read and follow the Quality Standards described on the Course Information page.

If for some reason you cannot attend class on the due date, notify me via email ahead of time, and submit your results via email as in-line plain ASCII text in Unix format, or a PDF attachment (do NOT send Word doc files or other formats). When printed, your attachment must conform to the requirements listed above.

3.1 Example Format

1. The event manager shall process incoming events in the order in which they are received.
   • Risk: events are processed out of order.
   • Probability: definitely unlikely. Reason: events are retrieved one at a time from the operating system and handled immediately; there is no opportunity for them to be re-ordered.
   • Impact: somewhat low. Reason: an out of order event would abort the current workflow, but the replay facility can be used to restart the workflow and advance it to the place where it was at the time of abort. So the user would only lose some time, but no actual work.
   • Exposure: .1 * 4 = .4
2. ...
3. ...
4. ...
5. ...
6. Analysis
   1. Does the ordering you achieve agree with your intuition about the relative importance of the risks you identified? Why or why not?

      No. I would have thought out of order events would have been a high priority risk, because it seems like a serious event.

   2. Which would you be inclined to trust more? Why?

      I would trust the analysis technique described above, because I invented it.