COEN 286: Test Implementation Exercise

The Canoo WebTest tool is a framework for creating automated web site tests. The tests are specified as XML documents having constructs corresponding to each of the major test concepts outlined above:

In this example, the test suite (within the target tags) contains one test (bracketed by testSpec tags), which in turn implements one test case. The test case input is specified by the invoke tag, which says that the example.html page should be retrieved. The expected output is encoded in a verifytext tag as the string ``This is the Web Test page.''

For this exercise, we are going to exploit the idea that a test suite can contain other test suites: you will implement one test procedure per file, thus each test suite specified by a target tag contains one test. Your top-level test suite then becomes the collection of these test files.

You should do this exercise in three major phases, as described below. Do not be tempted to write all of your tests, then run and debug them all at once. The XML specification syntax is straightforward and fairly easy to use, but like all languages for computer consumption it has a strict syntax that must be followed exactly; furthermore, XML is burdened with an excess of punctuation, so it's easy to make mistakes.

The right approach is to start early and work incrementally, verifying your work at each increment by actually running the tests. Unless you are extremely facile with ANT, it's unlikely you will be able to complete this exercise if you wait until the night before it's due.

2. Tasks

Set Up Environment

Peform the following tasks:

Login to the Design Center

Produces: home_directory

The Design Center provides remote access to the Unix and Linux workstations via Telnet and SSH. The Design Center (DC) provides to host names that will resolve dynamically to one of the Unix or Linux workstations: sol.dc.engr.scu.edu for Unix (Solaris) and linux.dc.engr.scu.edu for Linux.

To access the DC via telnet, open your Telnet client and connect to either sol.dc.engr.scu.edu or linux.dc.engr.scu.edu. You will be prompted to enter your login name and password:

Connected to nova30.dc.engr.scu.edu.
Escape character is '^]'.


SunOS 5.9


                        #######          #####
  ####    ####   #      #               #     #
 #       #    #  #      #               #     #
  ####   #    #  #      ######           ######
      #  #    #  #            #   ###         #
 #    #  #    #  #      #     #   ###   #     #
  ####    ####   ######  #####    ###    #####

                                 #####    ###
 #    #   ####   #    #    ##   #     #  #   #
 ##   #  #    #  #    #   #  #        # #     #
 # #  #  #    #  #    #  #    #  #####  #     #
 #  # #  #    #  #    #  ######       # #     #
 #   ##  #    #   #  #   #    # #     #  #   #
 #    #   ####     ##    #    #  #####    ###

________________________________________________________________________________
 
                Santa Clara University Engineering Design Center
________________________________________________________________________________
login: foobar
Password: 
-bash-2.05b$

The ``-bash-2.05b$ '' string is your ``shell'' prompt (the shell is the Unix command interpreter, to which you issue commands as described in the sequel).

By default, the login program will start your shell in your ``home directory,'' which is like your desktop folder on a personal computer. You can find your home directory name using the `pwd' command:

	    -bash-2.05b$ pwd
	    /home/foobar

Create Working Directory

Requires: home_directory
Produces: working_directory

Now, create a directory in your Design Center account with name equal to your first name in lower case. For example, if your name is ``Able Student'' then your directory should be called able; if your name is ``Paya Tension'' then your directory should be called paya. (This will be important later as it ensures that your tar file will have the correct structure and will be easily identifiable as yours.)

Verify Canoo Installation

Requires: working_directory
Produces: canoo_smoke_test

This task will ensure that you can run the Canoo WebTest tool. First, import the Java environment and Ant build tool:

	      -bash-2.05b$ setup jdk
	      -bash-2.05b$ setup ant

This will set your shell CLASSPATH and PATH environment variables so you can run java. Then, copy the example test into a file called test-canoo.xml into your /home/astudent. Then, execute the following command to run the test:

	      -bash-2.05b$ ant -buildfile test-canoo.xml

You should see some output, but no failures.

Create Taskdef File

You may have noticed that the example test contains two blocks - taskdef and config - that specify certain global information about the test environment.

The taskdef block must appear in every Canoo test file. As such, it represents duplicate data that could potentially get out of sync. To prevent this, put the taskdef block in a separate ant project called ``taskdef'' within a file called taskdef.xml:

	      <?xml version="1.0" ?>
	      <project name="taskdef">
	      <property name="webtest.home" value="/home/jnoll/src/canoo-webtest"/>

	      <taskdef file="${webtest.home}/webtestTaskdefs.properties">
	      <classpath>
	      <fileset dir="${webtest.home}" includes="**/lib/*.jar"/>
	      <fileset dir="${webtest.home}" includes="**/lib/base/*.jar"/>
	      <fileset dir="${webtest.home}" includes="**/lib/build/*.jar"/>
	      <fileset dir="${webtest.home}" includes="**/lib/optional/*.jar"/>
	      </classpath>
	      </taskdef>

	      <property name="username" value="yourusername"/>
	      <property name="password" value="yourpassword"/>

	      </project>

Then, in each of your test files, you can include the taskdef block by importing it into each of your test files:

	      <import file="taskdef.xml"/>

Note: property definitions have to be placed outside of the taskdef block.

For more examples, see the Canoo examples page.

Set Up Authentication

Requires: canoo_smoke_test
Produces: test_skeleton

In order for the Canoo WebTest tool to be able to access authenticated pages from the PEOS web interface, you must specify a user name and password in each invoke block:

	    <invoke stepid='Get Worklist Page'
	    url='/home/jnoll/protected/index.html'
	    username='user'
	    password='password'/>

This is inconvenient, as it must be duplicated everywhere a username and password are required. To get around this, define properties to contain your username and password:

	    <property name="username" value="user"/>
	    <property name="password" value="password"/>

Put these in your taskdef.xml file so they will be available to all of your tests; set the username and password properties to have the correct values for your login.

You can then refer to them as follows:

	    <invoke stepid='Get Worklist Page'
	    url='/home/jnoll/protected/index.html'
	    username='${user}'
	    password='${password}'/>

Save this skeleton into a file (skeleton.xml) in your working directory (/home/astudent).

Note: the skeleton has only the most rudimentary verification steps. You should be more thorough in verifying the results of any test to be certain the actual results match the expected results.

Also note that the skeleton contains authentication info for the `guest' account; you will have to change the username, password, and process_filename to correct values for your account. To find the correct value for process_filename, login to PEOS with your user name and password, then navigate to the bind resources page. Open the source code viewer (click View->Page Source in Firefox), and search for process_filename. Replace the value in skeleton.xml with the value you find in the page source; this is your process state file.

Create Config File

In the skeleton, You will notice a block similar to the following:

	    <config
	    host="linux.students.engr.scu.edu"
	    port="80"
	    protocol="http"
	    basepath="~jntestus/jitt"
	    summary="true"
	    saveresponse="true"
	    resultpath="/tmp"
	    resultfile="guest">
	    <header name="User-Agent" value="Mozilla/4.0"/>
	    <header name="Accept-Language" value="en"/>
	    </config>

This contains global information that tells canoo where to find the web site, where to save result pages, what browser to emulate, etc. These data are used in every testspec block, and thus represent another example of information that can be copied and therefore get out of sync. To solve this problem, put the config block in a file called config.xml, then replace any config blocks in your testspecs with an ``entity'' reference, as follows (the `&' and `;' are important):

	    <testSpec name="Baseline">
	    &config;
	    <steps>
	    ...
	    <steps>
	    <testSpec>

This example config.xml will work in the Design Center.

Note: the skeleton has a header section at the top similar to the following:

	  <?xml version="1.0"?>
	  <!DOCTYPE project [
	  <!ENTITY config     SYSTEM "config.xml">
	  ]>

This section declares the ``config'' entity so you can refer to it in your testspecs; as such, every one of your test files needs this header section.

Create Buildfile

Requires: working_directory
Produces: buildfile

The Canoo WebTest tool is based on Apache ant, a general tool for building projects from sources. As such, you can take advantage of ant's facilities to automate running of your tests, using a top-level buildfile.

A buildfile is an XML file containing tags that specify operations to perform on files or other objects that are part of a project. For this exercise, the buildfile will just invoke ant recursively on each of your test files, as shown in the following example:

	    <project name='PeosTest' basedir='.'>
  <ant antfile='active-processes-test.xml'/>
  <ant antfile='create-process-test.xml'/>
  <ant antfile='delete-process-test.xml'/>
  <ant antfile='next-previous-test.xml'/>
  <ant antfile='process-link-test.xml'/>
  <ant antfile='process-outline-test.xml'/>
  <ant antfile='resource-binding-test.xml'/>
  <ant antfile='start-finish-test.xml'/>
</project>

Initially, create a buildfile (buildfile.xml) to run the example test:

	    <project name='WebTest' basedir='.'>
	    <ant antfile='example-test.xml'/>
	    </project>

As you create tests, add additional ant tags to invoke each of your tests, replacing example-test.xml with the name of the test file.

Create Tests

Peform the following tasks:

Foreach target page

Visit Home Page

All of your tests should begin by retrieving the PEOS web interface home page. Use invoke to do this. From the home page, you can use clicklink to follow links from the home page to specific feature pages for further testing.

For this assignment, test the following links and pages:

Create Process
Active Process (link to instance list)
Active Processes/process name (link to process view)
Process Outline/action name (link to action detail)
Action Detail/Next & Previous (links)
Action Detail/Start & Finish (buttons)
Resource Binding

Note: for this test, you have to replace the value of process_filename in the url with the name of the process data file associated with your login. To find this, login to PEOS with your user name and password, then navigate to the bind resources page. Open the source code viewer (click View->Page Source in Firefox), and search for process_filename. Replace the value in skeleton with the value you find in the page source; this is your process state file.

Active Processes/Delete (button)

Put the tests for each of the pages and their children into separate files (eight total).

Navigate to Target Page

Follow Link or Submit Form

Follow Target Link

The clicklink tag is used to follow a link. This can be used, for example, to visit a certain page by following a link from the home page.

Submit Form

Fill In Form

To visit a result page by submitting a form, use the setinputfield, setcheckbox, and setselectfield tags to fill in the form:

The setinputfield tag specifies the content to be filled into a textbox.
The setcheckbox tag selects a checkbox while the setradiobutton selects one of a set of (possibly mutually exclusive) radio buttons. Which one to use depends on how the input item is specified in the form: input type ``checkbox'' requires setcheckbox, while input type ``radiobutton'' requires setradiobutton.
The setselectfield tag allows you to select an entry in a dropdown menu.

Submit Form

Once the form fields have been set, use the clickbutton tag to submit the form. Note: clickbutton does not work with buttons that invoke JavaScript functions.

Foreach feature in result page

Test Features

verify title

Each page has a page title that usually appears in the titlebar at the top of a web browser. This is separate from the page content that is displayed in the main browser window, and is specified using the HTML title tag.

The verifytitle tag can be used to verify the page title.

verify static text

Some features are static text: labels for input fields, legends, etc. These can often easily be verified using verifytext.

verify markup

Some features are banners or titles that appear within heading tags. The verifyelement and verifyelementtext tags can be used to verify that these features are present and have the correct content.

verify dynamic content

Certain result content will be derived from the input data entered into a form. Verify these results using either verifytext or verifyxpath.

verify table cells

Dynamic result content that is formatted in a table can be verified using verifyxpath, by explicitly specifying the table cell to be examined, as a row and column:

		    <verifyxpath stepid="verify cell of column 1 in row 3 of the second table."
		    text="cell contents"
		    path="/html/body/table[2]/tbody/tr[3]/td[1]">
		    </verifyxpath>

You have to include the ``tbody'' element in the path, because Canoo will insert this even if it's not there in the original page.

Note: the Canoo documentation mentions a table block that can be added to verifytext for specifying table elements to match against the specified text. Unfortunately, this is difficult to apply to pages that don't have 'name' or 'id' attributes attached to the tables.

Verify Tests

Requires: tests

Be sure all tests run, IN THE DESIGN CENTER ENVIRONMENT and either pass or expose a legitimate failure:

	  -bash-2.05b$ ant -f buildfile

Note: ant is rather verbose by default. See the ``hints'' below for a way to make it quieter.

Create README file

Requires: working_directory
Produces: readme
In your test directory, create a file called README (note spelling: no extension, all upper case) containing the following:

Your name and student id.
A line identifying each of your tests, with a short description of the feature or operation it tests.
A paragraph explaining how to run your tests.

Create Deliverables

Peform the following tasks:

Clean Working Directory

Requires: working_directory
Produces: working_directory
First, clean out your implementation directory so that it contains only your tests, your README file, and buildfile if present (no editor backup files, CVS directories, etc.).

Remove Example Test

Requires: buildfile
Produces: buildfile

Remove

	    <ant antfile='example-test.xml'/>

from your buildfile (buildfile.xml), and remove example-test.xml and skeleton.xml from your working_directory (/home/astudent).

Create TAR File

Requires: working_directory
Produces: tar_file
Then, do the following (assuming your directory is called your-first-name:

	    -bash-2.05b$ cd ..
	    -bash-2.05b$ tar cvf your-first-name.tar your-first-name

Submit TAR File

Requires: tar_file
Produces: tar_file.submitted

Submit, via email before , a message with subject line ``286 test implementation'' and your tar file as a base-64 encoded attachment of type ``application/x-tar''. Be sure the subject line is exactly as specified as I use procmail to capture these; if your message doesn't have the right subject, it will be lost.

Also be sure that the Canoo WebTest tool runs each test correctly before submitting; multiple submissions are not allowed.

COEN 286: Test Implementation Exercise

Software Testing and Quality Assurance -- winter, 2008

Prof. John Noll

Santa Clara University

1. Overview

2. Tasks

Set Up Environment

Login to the Design Center

Create Working Directory

Verify Canoo Installation

Create Taskdef File

Set Up Authentication

Create Config File

Create Buildfile

Create Tests

Foreach target page

Visit Home Page

Navigate to Target Page

Follow Link or Submit Form

Follow Target Link

Submit Form

Fill In Form

Submit Form

Foreach feature in result page

Test Features

verify title

verify static text

verify markup

verify dynamic content

verify table cells

Verify Tests

Create README file

Create Deliverables

Clean Working Directory

Remove Example Test

Create TAR File

Submit TAR File

3. Assessment

4. Hints