What happens when you are tasked with testing a non-trivial product with which you have no familiarity? Bach describes a ``General Functionality and Stability Test Procedure'' that one could apply to this situation. Originally designed for certifying products for Windows 2000, it allows the tester to start from no product knowledge, and leads to an assessment of product fitness.
Your task is to perform the ``General Functionality and Stability Test Procedure'' (GFSTP) on the PEOS web interface. (Note: your username is the first six letters of your last name; your password is the last four digits of your student ID.)
First, read Bach's paper describing GFSTP. The GFSTP approach has five phases:
In this phase, your main objective is to become familiar with the product. Start by reading a PEOS paper, which explains the product's desired features at a high level. Then, use the product. (Your user name is the first six letters of your last name (all lowercase) as it appears on the official class roster. Your password is the last four digits of your student id.) From these documents, you should be able to derive the product's purpose. You may consult the product authority for clarification.
Note: the product's purpose is not what it does, but what problem it solves for, or what benefit it provides to, its users.
The main source for product functions is the product itself. Develop an outline describing the product's functions in a hierarchical structure. Where possible, use the function name, title, or prompt as it appears on the product's UI; if the function doesn't have one, invent a descriptive name and put it in square brackets ("[]"). Also describe the function's type (button, link, menu, dialog, etc.).
Bach says the outline should contain only primary functions. However, for this exercise you should include all functions. If you do this, it will be easier for the product authority to verify both the completeness of your inventory, and your judgment of what is primary or secondary. Also, by having the entire function inventory specified as an outline, it will be easier to identify groups of functions that might constitute areas of instability as described in the next step.
How deep should your inventory go? Should you include controls or components that are incorporated from third party libraries? Should you inventory functions that don't appear to affect the product directly, such as file browsers? In general, if you are in doubt, include the function: it's better to be complete than leave out some important feature. For any given feature, ask yourself what your ``normal user'' would think. If this person would not distinguish a function as separate from the product, you should include it in your inventory. Note: don't inventory any feature that is part of the browser or operating system that would change if you used a different browser or platform. In other words, don't inventory features that are not invoked from the pages loaded by the product.
Remember that your role is to add value by assessing whether the product is ready for shipment, and thus is ready and able to generate revenue for the organization that hired you. As such, your job is to represent the users and customers of the product, not its developers or managers. Don't make simplifying assumptions for the sake of expediency, lest you miss some important feature that causes the product to flop.
It's possible that this attitude will create a function inventory that is too large to test in the available time. If so, you now have a strong case to argue for more testing time; if no more time is available, you can appeal to the product authority for help in determining which functions to test in the available time.
Follow Bach's guidelines (pp. 17 & 18) for identifying areas of potential instability. A process of ad hoc or exploratory testing will also help you uncover the soft spots in the implementation.
You will almost certainly revise and extend this section after performing the actual tests.
According to Bach, this phase has five steps:
In this phase, you should write down tests that produce failures, and the results of those tests (so the product authority can verify that you did in fact find a failure).
Document outright failures and ``problems'' (tests that produce ``quirky, annoying, erroneous, or otherwise concerning behavior''). Include as part of your documentation:
The last step is to design a consistency verification test that tests ``the most important primary functions ... to assure that the product behaves consistently ...'' The intent is to give a person installing the product a way to verify that the installation worked correctly. The consistency verification test should cover the ``most important'' primary functions, and should be detailed enough so that a person who is not familiar with the product can perform the steps correctly.
Submit a printed (hard copy) report including the following:
Here is a
partial example showing the approach applied to the
Linux convdate command.
Note: use the example as intended, as a guide to formatting, not test design; it is not complete.
In addition to the requirements enumerated above, be sure to read and follow the Quality Standards described on the Course Information page.
If for some reason you cannot attend class on the due date, notify me via email ahead of time to arrange for electronic submission.
The final document is due January 31, at the beginning of class.
Note: I suggested during the break last Thursday to use a file in /tmp. If you do this, however, it must be in linux.students.engr.scu.edu/tmp.
Another way to do this is to create a world readable file in your home directory, then make your home directory traversable:
% cd ~ % echo "A test resource" > my_resource % chmod a+r my_resource % chmod a+x .
Generated Sat Mar 15 11:22:20 2008