Once the test cases have been created, the test environment can be prepared. The test environment is defined as the complete set of steps necessary to execute the test as described in the test plan. The test environment includes initial set up and description of the environment, and the procedures needed for installation and restoration of the environment.
Description - Document the technical environment needed to execute the tests.
Test Schedule - Identify the times during which your testing facilities will be used for a given test. Make sure that other groups that might share these resources are informed of this schedule.
Operational Support - Identify any support needed from other parts of your organization.
Installation Procedures - Outline the procedures necessary to install the application software to be tested.
Restoration Procedures - Finally, outline those procedures needed to restore the test environment to its original state. By doing this, you are ready to re-execute tests or prepare for a different set of tests.
Inputs to the Test Environment Preparation Process
Technical Environment Descriptions
Approved Test Plan
Test Execution Schedules
Resource Allocation Schedule
Application Software to be installed
Test Planning
Careful planning is the key to any successful process. To guarantee the best possible result from
an automated testing program, those evaluating test automation should consider these fundamental planning steps. The time invested in detailed planning significantly improves the benefits resulting from test automation.
Evaluating Business Requirements
Begin the automated testing process by defining exactly what tasks your application software should accomplish in terms of the actual business activities of the end-user. The definition of these tasks, or business requirements, defines the high-level, functional requirements
of the software system in question. These business requirements should be defined in such a way as to make it abundantly clear that the software system correctly (or incorrectly) performs the necessary business functions. For example, a business requirement for a payroll
application might be to calculate a salary, or to print a salary check.
Creating a Test Plan
For the greatest return on automated testing, a testing plan should be created at the same time the software application requirements are defined. This enables the testing team to define the tests, locate and configure test-related hardware and software products and coordinate the human resources required to complete all testing. This plan is very much a “living document”
that should evolve as the application functions become more clearly defined. A good testing plan should be reviewed and approved by the test team, the software development
team, all user groups and the organization’s management. The following items detail the input and output components of the test planning process.
Inputs to the Test Planning Process
Application Requirements - What is the application intended to do? These should be stated in the terms of the business requirements of the end users.
Application Implementation Schedules - When is the scheduled release? When are updates or
enhancements planned? Are there any specific events or actions that are dependent upon the application?
Acceptance Criteria for implementation - What critical actions must the application accomplish before it can be deployed? This information forms the basis for making informed decisions on whether or not the application is ready to deploy.
Test Design and Development
After the test components have been defined, the standardized test cases can be created that will
be used to test the application. The type and number of test cases needed will be dictated by the
testing plan.
A test case identifies the specific input values that will be sent to the application, the procedures
for applying those inputs, and the expected application values for the procedure being tested. A
proper test case will include the following key components:
Test Case Name(s) - Each test case must have a unique name, so that the results of these test
elements can be traced and analyzed.
Test Case Prerequisites - Identify set up or testing criteria that must be established before a test can be successfully executed.
Test Case Execution Order - Specify any relationships, run orders and dependencies that might exist between test cases.
Test Procedures – Identify the application steps necessary to complete the test case.
Input Values - This section of the test case identifies the values to be supplied to the application as input including, if necessary, the action to be completed.
Expected Results - Document all screen identifier(s) and expected value(s) that must be verified as part of the test. These expected results will be used to measure the acceptance criteria, and
therefore the ultimate success of the test.
Test Data Sources - Take note of the sources for extracting test data if it is not included in the test case.
Inputs to the Test Design and Construction Process
Test Case Documentation Standards
Test Case Naming Standards
Approved Test Plan
Business Process Documentation
Business Process Flow
Test Data sources
Outputs from the Test Design and Construction Process
Revised Test Plan
Test Procedures for each Test Case
Test Case(s) for each application function described in the test plan
Procedures for test set up, test execution and restoration
Executing the Test
The test is now ready to be run. This step applies the test cases identified by the test plan, documents the results, and validates those results against expected performance. Specific performance measurements of the test execution phase include:
Application of Test Cases – The test cases previously created are applied to the target software application as described in the testing environment
Documentation - Activities within the test execution are logged and analyzed as follows:
Actual Results achieved during test execution are compared to expected application behavior from the test cases
Test Case completion status (Pass/Fail)
Actual results of the behavior of the technical test environment
Deviations taken from the test plan or test process Inputs to the Test Execution Process
Approved Test Plan
Documented Test Cases
Stabilized, repeatable, test execution environment
Standardized Test Logging Procedures
Outputs from the Test Execution Process
Test Execution Log(s)
Restored test environment
The test execution phase of your software test process will control how the test gets applied to the application. This step of the process can range from very chaotic to very simple and schedule driven. The problems experienced in test execution are usually attributed to not properly performing steps from earlier in the process.
Additionally, there may be several test execution cycles necessary to complete all the
necessary types of testing required for your application. For example, a test execution may be required for the functional testing of an application, and a separate test execution cycle may be required for the stress/volume testing of the same application. A complete and thorough test plan will identify this need and many of the test cases can be used for both test cycles. The secret to a controlled test execution is comprehensive planning. Without an adequate test plan in place to control your entire test process, you may inadvertently cause problems for
subsequent testing.
Measuring the Results
This step evaluates the results of the test as compared to the acceptance criteria set down in the test plan. Specific elements to be measured and analyzed include:
Test Execution Log Review - The Log Review compiles a listing of the activities of all test cases, noting those that passed, failed or were not executed.
Determine Application Status - This step identifies the overall status of the application after testing, for example: ready for release, needs more testing, etc.
Test Execution Statistics - This summary identifies the total number of tests that were executed, the type of test, and the completion status.
Application Defects - This final and very important report identifies potential defects in the software, including application processes that need to be analyzed further.