Leave a comment

Are We Done Testing?

john_brooks_b360_1by John Brooks, Vice President of Engineering

A question often asked of QA managers is, “Are we done testing yet?”  In reality, what they are likely asking is, “Is it good enough to make our release deadline?”  In an effort to maintain project delivery schedules, there can be tremendous pressure on QA to authorize releases before the software is truly ready for release. This problem is amplified by newer software application architectures and technologies, such as a SaaS application hosted in the cloud where a release can be practically silent and the end users have no idea that an update has occurred.  This scenario can create a less disciplined “what they don’t know won’t hurt them” or “fix it in the next release” kind of approach to software releases, which in turn can harm a company’s reputation (or worse) if critical defects that negatively affect their users are not caught before a release.

So what is a QA organization to do?  How to best deal with this pressure?  Every software project has its constraints – resources, time, and scope – and more often than not it is in the testing phase where shortcuts are taken to try and keep the project within bounds.  But taking shortcuts in quality is high risk—the expense of fixing it later is many times greater, as analysis over the last several years has repeatedly illustrated.

One of the best ways for QA to ensure that adequate testing is completed and an accurate measure of the software quality is provided is to establish gateways between the software development life cycle (SDLC) phases with entrance and exit criteria at the start of a project.  These gateway checkpoints should occur frequently throughout the SDLC, not only at the end immediately before a major release.

How does establishment of the phase gateways help ensure quality software?  Up front recognition and agreement about the test criteria and plans that QA is executing helps project management more objectively measure project status, software quality and readiness.  For example, items that are usually part of the entry criteria include (in order of importance and other phases in the SDLC):

  • Requirements traceability matrix
  • Release notes with defect logs
  • Identified technical and support resources
  • Gateway closure report for previous releases (if applicable)

Common exit criteria include:

  • Key component:  minimum metrics for successful transition through the phase gateway, such as maximum number of Severity 1 and 2 defects and overall quality score
  • Test scenarios and scripts executed, along with (or as a part of) a test-to-requirements matrix (includes a minimum % pass value)
  • List of assumptions and tests that are not applicable
  • Test closure report (includes entry release notes)

In summary, defining the SDLC phase transition criteria for a particular project, if done properly, will enhance both the flow and the quality of a project—not burden it down with excessive “hoops to jump through.”  Phase gateways then become an important tool for the team to recognize where they are in the process and how well that process is working for the project.


Leave a comment

Requirements, really?

Nadine Parmelee

by Nadine Parmelee
Senior Quality Assurance Engineer

Change is hard. Project teams frequently resist the addition of new processes and procedures, and one process frequently fought is the creation and review of requirements. The reaction is understandable—it’s most likely resistance to creating more work in the midst of already tight schedules. On the other hand, some of the consequences of not having requirements or having poorly thought out requirements are:

  • Developers guessing how the application should behave
  • Secondhand guessing: testers guessing how the developers guessed the application should behave
  • Customers not liking or having difficulty with the product
  • Spending time and money reworking a product to remove ambiguities
  • Inability to map test case to requirements, which can lead to not really knowing if the testing is complete

The benefits of creating requirements and making them clear in the first place are numerous. Here are just a few:

  • Understanding the purpose of a new feature
  • Understanding the workflow of a new feature
  • Understanding the options of a new feature
  • Understanding how a new feature integrates with existing functionality

And that’s just the beginning of what ultimately amounts to a little time spent upfront for maximum gain and minimizing stress on a project.

Creating requirements does not have to be an exercise in torture. Even the most basic requirements list is better than no requirements at all. Whenever there is functionality without requirements, developers and testers will make assumptions on how each thinks the functionality should work. Those assumptions may not match the intended behavior. By at least starting a list with the most basic of the requirements, early misunderstandings of what is expected can be avoided.

Additionally, the more people that review the requirements as early as possible, the sooner questions about functional behavior and whether certain functionality is even possible can be raised and potentially save large amounts of time and money. Reworking requirements adds delays and costs to any project, so the better the requirements are to begin with, the more efficiently the project schedule and budget can be maintained. Reworking code over and over and the subsequent testing of that code can add even more costs and delays until the project team can agree on the desired functionality. The earlier everyone has the same understanding of what needs to be done and the desired outcomes, the more quickly a project can progress and the more quickly new personnel can get up to speed on what they need to do to help complete the project.

When requirements are organized well and are succinct and clear, it is easier for developers and testers both to meet those requirements and map specifications and test cases to those requirements and improve the product quality from the beginning. By having better quality early on, the odds of getting a project completed on time and on budget are greatly improved.