Test Plan
From TeamF::CGT
Revision: 4056
Abstract
This document is the Test Plan of the Corporate Governance Toolkit (CGT), which includes the Brix Toolkit and Brix UserApplication. It outlines the testing strategies, tools and procedures, which will be adopted by the development and testing teams in order ensure that the system delivered to the client meets the team's quality goals.
Introduction
Purpose
This document describes the Testing strategies that will be used by the Team to test the System. It specifies each type of Testing that will be conducted according to the Testing strategies.
Overall objectives of this document:
- Define the Testing methodology
- Define the Testing schedule
- Define the Testing strategies
- Define the procedures for conducting each test specified in Testing strategies
- Define the procedure for creating test cases
- Define Team member’s roles and responsibilities required for Testing
Scope
The Test Plan (TP) will only include the following information:
- Testing strategies that will be adopted by the Team to test the System
- Testing tools to be used in Testing
- Testing needs and responsibilities
- Test activity suspension and resumption criteria
- Testing deliverables that will be produced
Testing Methodology
Bottom up testing procedure will be adopted for the CGT project. There are five phases for the entire testing procedure: unit testing, integration testing, system integration testing, system testing and deployment testing.
An agile testing method will apply to each testing phase. That is implementation and testing in pair. The general procedure is listed below:
- One or more implementers will be paired up with one or more testers.
- Implementers release their code to test, when they feel it is ready. Relevant testers then write test cases to test the code throughly.
- When bugs are found during the testing stage, testers publish them on the bug tracking tools and notify corresponding implementers immediately. Please see Bug Tracking and Reporting for more detail.
- When bugs are fixed, implementers notify relevant testers, then they retest the code.
- When testing arrives at other stages, such as intergration testing, system testing and depolyment testing, several people in the testing team will form a new group and be assigned new tasks.
Testing Schedule
A combined Design/Implementation/Testing schedule plan will be produced from the design team. Please refer to the combined schedule plan for more detail. (More detail needs to be added here, after the completion of the schedule plan from design team).
Testing Tools
The following tools shall be used for testing in this project:
- NUnit v2.2.8 (Regression testing framework)
- NUnitAsp v1.5.1 (ASP.NET pages testing)
- Subversion v1.3 (Version control over source code)
- Internet Explorer v6.0+ (Web browser used in running the web-based simulations)
- gVim v6.3 (Text editor)
- Microsoft SQL Server Express (Server used to maintain database for testing)
Bug Tracking Tool
The following tool shall be used for bug tracking in this project:
- Mantis v1.0.0a3 (Bug tracking tool)
Bug Tracking and Reporting
The Mantis bug reporting tool will be used to maintain the bugs found in the CGT.
Bug Reporting Procedure
Bugs need to be reported when they are found outside of scheduled testing. Bugs found during scheduled testing will be dealt accordingly with the type of the test (see each testing section). When a team member discovers a bug, the following procedure should be followed:
- he should create a new bug report in Mantis within 24 hours of discovering the bug;
- he should email the Implementation Manager with details of the new bug within 24 hours of discovering the bug;
- the Implementation Manager must assign the bug to a team member to fix within 48 hours of receiving the email;
Bug Tracking Procedure
Once the bug has been reported and assigned to a team member to fix, the following process shall be followed:
- the team member shall attempt to fix the bug;
- the Testing Manager shall assign a person to write a test case and test that bug;
The tester should make sure the following procedure is complied with:
- the test case should be added to any relevant regression test suite;
- the person testing the fixed code should update the bug report in Mantis if it has been fixed correctly according to the new test case;
- the Implementation Manager should be notified by the person who tested the fixed code stating that the bug has been fixed.
If the team member cannot fix the bug, he should notify the Implementation Manager, who shall then either:
- Reassign the bug to another team member; or
- If he determines that the bug cannot or should not be fixed, update the bug report accordingly and organise a meeting with the Implementation Team to decide what should be done.
Unit Testing
Definition
Unit testing is the testing of an individual class of software. The test for each unit must correctly function before declaring the outcomes of it.
Objectives of Unit Testing
- Identify unit level defects by causing failures to occur.
- Identify defects that are not easily identified by other forms of testing.
- Help the developer to determine the unit:
- Is complete.
- Fulfills its requirements.
- Works as designed.
- Is ready for integration testing.
Preconditions of Unit Testing
- A unit has been chosen to test.
- The requirements of unit has been determined.
- The methods of the unit to be tested has been chosen.
- The unit's assertions, which are class invariant, method preconditions and postconditions have been specified.
Completion criteria of Unit Testing
- Object-oriented classes
- Blackbox testing
- Methods of the classes except getter or setter has been tested. This includes testing return value, the outbound exceptions, the state of the object, the message to be sent, and the inbound exceptions.
- Every constructor has been tested.
- Blackbox testing
- Webpage
- Tag is syntactically correct.
- Every link functions correctly.
- Forms and buttons function correctly.
Responsibilities
- The class developer tests his/her own class before releasing it to the test team.
- Members of testing team perform testing on the class.
- Testing team publishes the results of test on bug tracking tools.
- Notify implementation team immediately, if bugs are found.
Tasks
- Test design
- Design assertions or exceptions.
- May design by using input domain.
- May design by using boundary value testing.
- May design by using Error guessing.
- Design assertions or exceptions.
- Test implementation
- Each class has at least one test case. It resides in the same directory as the class does. Test cases which have same namespce form a unique test suit.
- Testers implement test cases for each test suit.
- Test excution
- Automated testing.
- Regression testing.
- Test reporting
- Automatic report generation.
Tools
- NUnit
- NUnit ASP
- Test driven dotnet 2.0
Test Code Allocation
A solution in Vistual Studio 2005 may have several implementation projects created by the implementation team. Each implementation project is associated with a testing project created by the testing team. Inside a testing project, there are serveral testing classes. Each of them is corresponding to a particualr implementation class for testing purpose. Here is a sample test code allocation:
Test Code Convention
// Add your Nunit framework and other necessary namespaces here.
// Please refer to the stand sample code below for more detail.
namespace RiskWizard.Cgt.XX.Test // XX will be toolkit or userApp or dataLayer
{
/// <summary>
/// Summary for this class goes here.
/// </summary>
[TestFixture]
public class XXTest // XX is the class to be tested.
{
// Declear any required variables here for testing.
/// <summary>
/// Your summary.
/// </summary>
[SetUp]
protected void SetUp()
{
}
/// <summary>
/// Your summary.
/// </summary>
[TearDown]
protected void TearDown()
{
}
/// <summary>
/// Your summary.
/// </summary>
///
/// <remarks>
/// Input:
/// * The input for the method for the test.
/// Input includes arguments and states(i.e. relevant global variables).
///
/// Expected Output:
/// * Your expected output.
///
/// NOTE:
/// * Extra infomation goes here if necessary
/// </remarks>
[Test]
public void TestXX() // XX is the method name to be tested.
{
}
/// <summary>
/// Your summary.
/// </summary>
///
/// <remarks>
/// Input:
/// * The input for the method for the test.
/// Input includes arguments and states(i.e. relevant global variables).
///
/// Expected Output:
/// * Your expected output.
///
/// NOTE:
/// * Extra infomation goes here if necessary
/// </remarks>
[Test]
public void TestXX() // XX is the method name to be tested.
{
}
}
}
|
Standard Sample Code
using System;
using NUnit.Framework;
using NUnit.Extensions.Asp;
using NUnit.Extensions.Asp.AspTester;
/// <summary>
/// Summary for this class goes here.
/// </summary>
namespace RiskWizard.Cgt.UnknownClassTest
{
[TestFixture]
// Need to extend WebFormTestCase if testing webpage.
public class GuestBookTest : WebFormTestCase
{
/// <summary>
/// Summary for this tester object goes here.
/// </summary>
private TextBoxTester name;
/// <summary>
/// Summary for this tester object goes here.
/// </summary>
private TextBoxTester comments;
/// <summary>
/// Summary for this tester object goes here.
/// </summary>
private ButtonTester save;
/// <summary>
/// Summary for this tester object goes here.
/// </summary>
private DataGridTester book;
/// <summary>
/// All presetups go here.
/// </summary>
// NUnit asp has used the fixture [SetUp], so we can't use here.
protected override void SetUp()
{
name = new TextBoxTester("name", CurrentWebForm);
comments = new TextBoxTester("comments", CurrentWebForm);
save = new ButtonTester("save", CurrentWebForm);
book = new DataGridTester("book", CurrentWebForm);
Brower.GetPage("http://localhost/RiskWizard/Cgt/SomeTest/GuestBook.aspx");
}
/// <summary>
/// Summary for what are being tested goes here.
/// </summary>
///
/// <remarks>
/// Input:
/// * TextBoxTester name = new TextBoxTester("name", CurrentWebForm);
/// * TextBoxTester comments = new TextBoxTester("comments", CurrentWebForm);
/// * TextBoxTester save = new ButtonTester("save", CurrentWebForm);
/// * DataGridTester book = new DataGridTester("book", CurrentWebForm);
///
/// Expected Output:
/// * TestLayout passes testing.
///
/// NOTE:
/// * Extra infomation goes here if necessary
/// </remarks>
[Test]
public void TestLayout()
{
AssertVisibility(name, true);
AssertVisibility(comments, true);
AssertVisibility(save, true);
AssertVisibility(book, false);
}
/// <summary>
/// Test String UnknownClass.show(String text).
/// </summary>
///
/// <remarks>
/// Input:
/// * TextBoxTester name.Text = unknownObject.Show("name");
/// * TextBoxTester comments .Text = unknownObject.Show("comments");
///
/// Expected Output:
/// * name.Text == ""
/// * comments.Text == ""
/// </remarks>
[Test]
public void TestShow()
{
UnknownClass unknownObject = new UnknownClass();
name.Text = unknownObject.Show("name");
comments.Text = unknownObject.Show("comments");
AssertEquals("name", "name", name.Text);
AssertEquals("comments", "comments", comments.Text);
}
/// <summary>
/// Summary for miscellaneous method goes here.
/// </summary>
///
/// <param name="misc1">
/// Summary for this argument goes here.
/// </param>
///
/// <param name="misc2">
/// Summary for this argument goes here.
/// </param>
///
/// <returns>
/// Summary for this return value goes here.
/// </returns>
private string Miscellaneous(string misc1, string misc2)
{
return "something";
}
/// <summary>
/// All clean-up go here.
/// </summary>
// NUnit asp has used the fixture [TearDown], so we can't use here.
protected override void TearDown()
{
// tear-down code here
}
}
}
|
Integration Testing
Definition
Integration testing is the incremental testing of two or more integrated components. The aim of integration testing is to cause interface defects.
Objectives of Integration Testing
- Cause failures when different components are integrated.
- Report these failures to the implementation team, so that underlying failures can be identified and fixed before moving to system testing.
- Minimize the defects that will prevent effective system testing.
Preconditions of Integration Testing
- All units for integration testing have passed unit testing.
- All units for integration testing are integrated.
Completion Criteria
- A test suite of test cases exists for each interface between components.
- Component integration test suits are completely executed and actual results match expected resuts.
Responsibilities
- Implementation team integrate different components.
- Test team performs integration testing.
Tasks
- Test design
- Design test suits for interfaces.
- Test Implementation
- Implement test cases of test suits for interfaces.
- Test excution
- Automated testing
- Regression testing.
- Test reporting
- Automatic report generation.
Links
http://www.eirikr.net/projects/cgt/wiki/index.php?title=Integration_Testing
System Integration Testing
Definition
System integration testing is to cause integrated system components to produce failures. This is important for software components distributed onto multiple platforms (e.g. client, web server, application server and database server).
Objectives of System Integration Testing
- Cause integrated system components to produce failures.
- Cause failures when system and other applications are integrated.
- Minimize the numbers of defects before moving to system testing.
Preconditions
- All distributed system components have passed integration testing.
- All system components are integrated.
Completion Criteria
- A system integration test suite is designed and implemented for the interfaces among system components.
- System integration test suites are completely executed and actual results match expected resuts.
Responsibilities
- Implementation team integrate different system components.
- Test team performs system integration testing.
Tasks
- Distribute the software components on to the hardware components.
- Integrate the system with other external systems.
- Execute the system integration test suites.
- Report the results of the system integration test suites.
Links
http://www.eirikr.net/projects/cgt/wiki/index.php?title=Integration_Testing
System Testing
Definition
System testing is to test integrated system components, which are treated as a blackbox application against its requirements.
Objectives of System Testing
- Determine whether it fulfills its requirements.
- Identify defects which are not identified in unit testing, integration testing and system integration testing.
- Determine whether the entire applicaiton is ready to launch.
Preconditions
- All systems are integrated.
- All systems have passed system intergation testing.
Completion Criteria
- Pass Functional Testing, which it fulfills all the requirements.
- Pass Performace Testing, which it satisfies the performance requirements.
- Pass Stress Testing, which it still functions reasonably under extreme utilization.
- Pass Robustness Testing, such as it can restart after hardware failure.
- Pass Security Testing, which it satisfies the security requirements.
Responsibilities
- Implementation team integrate different systems.
- Test team performs system testing.
Tasks
- Distribute the entire application in the client's machines.
- Excute the application.
- Report the distribution.
Functional Testing
Definition
Functional testing is to test an integrated application against its funcational requirements.
Objectives OF Functional Testing
- Determine whether the application fulfills the functional requirements partially or completely.
- Cause the application to fail by excuting different functionalities.
- Report these failures to the implementation team so that those defects can be fixed.
Preconditions
- The funcational requirements to be tested have been specified and fully implemented.
- Relevant software components have passed unit testing, integration testing, system integration testing.
Completion Criteria
- The corresponding test suits are executed successfully according to pre-defined schedule.
Responsibilities
- Implementation team release parts of the system or the completed system to test.
- Test suits are built by the testing team.
Tasks
- Release parts or completed system to test.
- Gather feedbacks in order to make further improvements.
Performance Testing
Definition
Performance testing is to test whether the system meets the performance requirements, which are normally specified SRS, in order to identify inefficiencies and bottlenecks.
Objectives Of Performance Testing
- Validate the system fully or partially to determine whether it satisfies its performance requirements.
- Determine four basic performance elements:
- Capacity: Maximum numbers of objects which the system can handle.
- Latency: Maximum time to complete a system operation.
- Response time: Maximum response time of system.
- Throughput: Maximum outputs from the system.
- Report performance failures to the implementation team.
- Reduce hardware cost by:
- Identify minimum hardware necessity to meet the performance requirements.
- Identify optimal configuration to meet the performance requirements.
Preconditions
- The performance requirements to be tested have been specified.
- Components for performance testing have been implemented and passed unit testing.
- Components for performance testing have been integrated and have passed integration testing. (Performance testing can start at this point)
- Systems have been integrated and have passed system integration testing. (Performance testing can start at this point)
Completion Criteria
- Test suits for performance testing exist.
- Those test suits have been executed successfully.
Responsibilities
- Implementation team release parts or full system to be tested.
- Testing team implement the test suits for performance testing.
Tasks
- Implement parts or full system and release it to be tested.
- Implement the test suits and test the released system.
Stress Testing
Definition
Stress testing is to test application when it performances under extreme but valid conditions. For example, extreme utilization, insufficient memory, inadequate hardware and over-utilized shared resources problem.
Objectives Of Stress Testing
- Determine how an application degrades and fails eventually, under extreme conditions.
- Determine the worse conditions for the system.
- Provide these data to implementation team for scaling improvement purpose.
Preconditions
- The scaling requirements to be tested have been specified.
- Unit testing, integration testing, system integration testing have occured.
Completion Criteria
- Test suits for stress testing are implemented and executed successfully.
Responsibilities
- Implementation team releases the system for stress testing.
- Testing team implements the test suits and execute them.
Tasks
- System is released for stress testing.
- Test suits are implemented and executed.
Robustness Testing
Definition
Robustness testing is to test application that how it behaves under invalid conditions. For example, unavailability of dependent applications, hardware failure, and invalid input.
Objectives Of Robustness Testing
- Determine how well the application handles failures and performs rollover.
At the same time, the application should not fail catastrophically when the unexpected occurs.
- Determine extreme exceptional conditions which causes the system fail catestrophically if they exists.
- Provide these data to the implementation team for application improvement.
Preconditions
- The exceptional operational requirements to be tested have been specified.
- Unit testing, integration testing and system integration testing have occured.
Completion Criteria
- Test suits for robustness testing exists.
- Exceptional operations can be detected by the test suits.
Responsibilities
- Implementation team releases the system to perform robustness testing.
- Testing team implement the test suits and execute them successfully.
Tasks
- System is released for robustness testing.
- Test suits are implemented and executed correctly.
Security Testing
Definition
Security testing is to test whether partial or entire system satisfies the security requirements. Security testing can be tested by security tools or performed manually.
Objectives Of Security Testing
- Security testing is aiming to disclose failures in these major areas:
- Identification.
- Authentication.
- Encryption and decryption.
- Authorization.
- Content protection.
- Integrity.
- Intranet client and internet client.
- Data at server sides and client sides.
Preconditions
- Security requirements to be tested have been specified.
- Relevant components have passed unit tesitng and integration testing.
- Test suits are implemented or mechanisms are developed.
Completion Criteria
- Partial or entire system have passed the test suits.
- Partial or entire system satisfies the security testing mechanisms.
- Security vulnerabilities have been reviewed and fixed.
Responsibilities
- Implementation team releases partial or entire system.
- Test team performs the security testing.
Tasks
- Partial or entire system is released.
- Test suits or mechanisms are developed by the test tea.
Regression Testing
Definition
Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes.
Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have reemerged. The actual test cases used in regression testing are no different from any other test cases. The regression test suite is a sample of the functional tests for all of the components.
Objectives of Regression Testing
- Ensuring that new code added to the system does not cause new bugs in the system by running all previous tests through the system.
- Ensuring that new code added to the system does not break existing functionalities of the system.
Preconditions of Regression Testing
- Test code has been written for the system and has been successfully passed.
- New code has been added to the system after the initial testing.
Completion Criteria
- As regression testing is a continuous process, it is deemed to be complete only when all test suites are successful passed at the completion of the system. Therefore during development, regression testing is deemed to be successful when no regression bugs are found in the system.
Responsibilities
- Testing team to combine all test cases and create an automatic schedule to re-run all test cases available to the system.
- Test team is responsible for the correctness and consistency of test suits. Test manager should ensure the testing process is followed.
- Testing team to liaise with the Implementation team when regression bugs are found.
Tasks
Maintaining the Regression Test Suite
- Keep a list of all successful unit test cases and add them to the regression test suite.
- If any unit test case becomes obsolete, the maintainer of that particular unit test case must inform the testing manger. Since the regression test case suite is automatically built from unit test cases, the regression suite will always contain the most up to date unit test cases.
Executing the Regression Test Suite
The testing manager is responsible for executing and reporting the results of regression test suites.
- Regular Regression Testing: Regression Test Suite must be executed three times a week on Monday, Wednesday and Sunday.
- Regression Testing on Request: When there is a major change to existing implementation such as changes in interfaces of methods or adding a new feature to a class, the implementer of the code must request the testing manager to run the regression tests.
Reporting the Results of Regression Testing
- The outcome of regression testing must be stored in Regression Testing Log by the testing manager. The team must also be informed of the status of regression testing.
- If an error occurs, the error must be logged in Mantis and the implementer of the class must be notified of the error.
The implementation team must fix the system for all failed regression test cases. This must take precedence over write new components for the system.
Links
http://www.eirikr.net/projects/cgt/wiki/index.php?title=Regression_Testing_Log
Deployment Testing
Definition
Deployment testing is conducted by the development organization prior to releasing the software to customers for acceptance testing. Where acceptance testing focuses on the functionality of the delivered product, deployment testing covers all the unique system configurations upon which the product is to be deployed.
Objectives of Deployment Testing
- Deployment testing focuses on the interaction between the system and platform-specific libraries, device drivers, and operating systems. During the deployment testing phase, the application's ability to deploy or install itself is also tested.
Preconditions of Deployment Testing
- The system passes system testing and is ready to be deployed on client machines.
Completion Criteria
- The system passes the deployment testing suites.
- The system is successfully deployed on all required range of hardware specifications.
- The system is successfully deployed on all required operation systems.
Responsibilities
- The testing team performs the deployment testing.
- The testing team to report any deployment testing issues.
Tasks
- Organise the hardware required for deployment testing.
- Execute the deployment test suites.
- Report the results of the deployment test suites.
Usability Testing
Usability Testing involves the assessment of the appearance and functionality of the CGT by both the client and the team. Usability of the CGT shall be discussed or demonstrated at client meetings. Results of this testing shall be recorded as per usual in the minutes for that meeting. In later stage, parts of system or beta system will be released in order to gather feedbacks from the client. If the client wishes an aspect of the appearance of the CGT to be changed:
- The Client Liaison officer shall notify the Implementation or Design Manager (as appropriate depending on the current phase of the project) of the requested change.
- The Implementation or Design Manager shall decide whether the change should be made, then notify the Client Liaison officer as such.
- The Client Liaison officer shall report to the client the outcome and related reasons for either accepting or rejecting the change.
Links
http://www.eirikr.net/projects/cgt/wiki/index.php?title=Usability_Testing