Quality Management on Software Projects

This is the first in a series of articles about managing the Quality related

activities in a software project, written from the project manager's perspective. The first step the project manager will take should be to plan the Quality activities that are required for the application, web site, or system to meet its goals and objectives. You may need to document the goals, objectives, roles, responsibilities, and other details in a formal Quality Management Plan depending on the size and complexity of your project. If your project is not large or complex enough to require a formal plan, scheduling the work and assigning it to a resource in your WBS may be sufficient.

There are 3 different phases or types of testing required during the build phase of the project:

Developer testing - this is testing that will be done by the developers on the team and will include unit testing, function testing, thread testing, integration testing, and system testing. This testing is sometimes referred to as "white box" testing because the testers have knowledge of how the software works and test accordingly.

Quality Assurance testing - this may be performed by a separate QA team or by the development team. It is sometimes referred to as "black box" testing because the QA team will have no knowledge of how the software was designed to work and will be testing against the business requirements for the software.

User Acceptance testing (UAT) - this testing is performed by the user community, usually that part of the community that will be using the software. This testing will normally consist of the user performing their work on the new system. In some cases it may consist of employees of the performing organization simulating external customers.

The resources that must be brought to bear on quality activities can be divided into 2 categories: materials including hardware and software, and human resources. The materials can be divided into 2 types: platforms and licenses to support the testing and tools to automate it. The types and quantities of materials will have an impact on the quantity of human resources required so the availability of testers and the budget available for them must be considered when deciding on whether to invest in automated test tools. Enterprise Release Management Software

The methods discussed in this article are intended to support the best practices described by the PMBOK 4th Edition. You can learn these by taking a PMP Course, or other PMP Exam Preparation training and passing your PMP exam. The result will be that you are better equipped to manage quality activities in a software project and you will be recognized as such by your peers and sponsors.

Goals and Objectives

Goals and objectives will come from 2 different sources: the Project Charter where the work is to be performed for an internal customer or client, and the Statement of Work (SOW) where the work is to be performed for an external customer. Your plan should support the achievement of these goals within the budget and schedule constraints established for your project. Where the Project Charter or SOW is lacking in specifics your plan needs to support generally stated goals with specific targets such as going to the UAT phase with no more than 2 open severity 1 bugs, going to production with no open severity 1 bugs and no more than 3 open severity 2 bugs, etc. Performance, load, and stress targets should also be specified. For example, log on in 5 seconds or less, support at least 500 active, concurrent sessions, the system should fail with an error message, etc. The tools you use and activities you plan should all enable meeting these objectives.

The goals and objectives should be verified with the sponsors or customers for the project. This is especially important where goals and objectives have been left vague; the only thing worse than believing you are building sufficient quality into your software and fail to, is believing the quality you're building in is required and it isn't. Once you have verified you have the right set of standards and benchmarks you're ready to choose the tools you need to achieve them.

Tool Selection

Your software development environment and Quality Assurance environment may already come stocked with all the testing tools you need. Where that is not the case, such as when a completely new software application, web site, or system is being developed or a new platform is being used, you will have to identify the tools that must be used to achieve your quality targets, and procure them. This article is intended to address planning quality activities only; a future article will cover the selection of testing tools in more depth.

There are a wide variety of testing tools available which will automate tests and tend to reduce the manual effort required to perform testing. The tools available for your use will tend to be constrained by the software being developed and the platform chosen. Test tools that work with C++ won't necessarily work with Java; this is especially true about unit test tools. Determine the platform that will be used for your project, then select the testing tools.

Tools also vary according to which type of testing they support. Automated unit testing tools will be used during the development phase and automated regression test, performance test, load test, and stress test tools will be used in the Quality Assurance phase. User Acceptance Testing is the only testing that does not lend itself to automation as the object is to test the system or application under actual usage conditions.

Continuous Integration (CI) tools such as Hudson are a special type of automated testing tool in that they facilitate testing through an automated build process they call continuous integration. The build process may be triggered by an event, manually, or it may be scheduled. Hudson, and other CI tools will also enable initiation of test scripts when the build is triggered which will automate integration testing, based on the test scripts which must be manually written. Many of these tools are freeware, as is Hudson, so won't impact on your budget.

Automated unit testing tools should create test cases and test harnesses which are where the bulk of the manual effort is spent. Each tool has a different feature set and is adapted to one specific platform which means if you're creating an application or system on a new platform you'll have to buy a new test tool. Quality Assurance tools will automate regression testing, performance, load, and stress testing. These tools tend to require an investment in hardware so investigate their hardware requirements before deciding on which tools to use. One feature you may want to acquire is the ability to measure code coverage. This feature will indicate where more testing is required and also where "dead code" exists (code that will never be exercised because calling conditions can not be met, etc.). If you have, or will have a tool with that feature, plan on setting a threshold for coverage.

Your project will need a bug reporting tool that will track and communicate bugs as they are found during the QA and UAT phases of your project. There are several vendors, such as Altassian, that offer a web based "wiki" engine that will handle project communications. These engines either come with a bug reporting tool or can accommodate one as a plug in. These engines are particularly useful when your team does not have shared access to the internal project environment, such as when some of the development or quality work is being outsourced. Choose a bug reporting tool that will measure all the metrics you need to meet your quality targets (e.g. bug severity). Most of these tools come with some form of reporting engine which will also weigh in your decision on which tool to buy.

You will need at least 3 platforms for your software development effort, and probably more. You need a separate environment for development, including build, deployment and integration/system testing, one for QA testing, and the production environment. You may need a separate environment for User Acceptance Testing, regression testing, performance testing, load testing, and stress testing in addition to these. You also may need to provide additional servers for tools such as your "wiki" or bug reporting tool.

Testing Activities

The more automated tools you use to facilitate your quality activities, the more training you require. You may be using the suite of tools already in place, but unless you will acquire a project team fully trained in that tool suite you must still plan training. New tools may come with training included in their price, but this will not cover the time required to take for the entire team to take the training. Don't forget that any additional resources acquired must also be trained.

Unit testing, thread testing, and function testing are part of developing the software and do not require separately planned activities. These activities are the responsibility of the developer and should be completed before the code is included in a system build. Ensure that sufficient time is allowed for these testing activities when effort and duration are estimated. There are numerous techniques and tools available for the estimation of testing effort but the quickest way is to first estimate the development effort required and then take the testing effort as a percentage of that number. An historical database of forecast and actual effort from previous projects will be helpful when calculating effort. If that's not available, start with a 25 % base and adjust this according to the automated test tools, available to the project. In fact, when a senior programmer is asked to estimate effort required to develop code they will automatically factor in testing effort.

Depending on the tool set and test environment, you may need a "build master" to coordinate build efforts. You may also need a software librarian to keep your source library organized. Resources with the necessary skill sets should be identified for these jobs. Depending on the size of your project, you may not need a full-time resource, but if you determine that a developer can perform these duties in addition to developing code, ensure that you allow sufficient time for both duties to be completed. Re-work must be scheduled during QA and UA testing activities which will detract from the time available for developing new code. In an iterative approach, this is best addressed by treating re-work like other requirements, estimating the effort required and subtracting that effort from the total effort available. Developers will need to be devoted to the work in a waterfall environment. The effort required for this activity will vary based on the size, complexity, and quality of the code. Remember that effort expended in testing during development will reduce testing and re-work effort later.

There are 2 main activities to be planned for the QA group: writing test cases and test scripts and executing them. The number of test cases and scripts that are necessary to exercise the application, web site, or system will be proportional to its size. The effort required to execute the test scripts will be proportional to the time required to write them when the tests are to be executed manually. Test time will be drastically reduced when an automated tool is used, but remember this savings comes at a price and that price is the labor that must be put into writing the automated test cases. Vendors of these tools tend not to dwell on this fact but some of the tools require quite intensive labor to create test scripts.

Plan activities to create and download test data. This is frequently an after-thought when planning testing activities but is actually just as critical as the test cases themselves. Developers will be responsible for providing data that is specific to their software but "system" data, or data that is common to the entire system will need to be produced by a separate effort. Product catalog is a good example of this type of data. This data may come from an operational group or some other source, but will have to be organized, formatted, and tailored to the new system. Your Database Administrator will be responsible for refreshing the various databases but someone else must create the data files containing the data. This activity also requires someone to create the scripts for loading the files. Disaster Recovery Management

Testing activities will also include mining data and creating the reports that will communicate the quality results to the project stakeholders. These reports should demonstrate that the software has met or exceeded all the standards or benchmarks established by the project. The reports you are capable of producing will be constrained by the tools you selected to automate testing and track bugs so choose those tools wisely. These reports will be published at regularly frequencies so a resource must be identified to produce them, and that resource (unless the resource is full-time, or you) will be taking time away from their other duties so factor that effort into their schedule.