Leave a comment

3 High-Octane Tools for Performance Testing

By John Cavazos, Senior Performance Test Engineer

John-Cavazos_cropAt one time, it was very common for companies to use a homegrown solution to performance test their software. This was mainly due to two factors:

  • the lack of available, cost-effective commercial testing tools
  • the complexity of their software

The situation is different today as it is common to find excellent, free, open-source tools online. These high-quality, stable, easy-to-use tools don’t have the inherent maintenance costs associated with using homegrown solutions.

There are now several industry standard tools available that can be used to performance test most, if not all of the features a company might need. What isn’t readily available can typically be added to these tools with minimal effort due to their open-source nature. Here are a few I have used: Continue reading


2 Comments

How a QA Team Fits into an Agile, TDD/BDD World

by Chris Durand, CTO

ChrisDurand-B360Let’s be honest, Quality Assurance is a necessary evil. Many of you have probably wondered why developers can’t just get it right the first time. Why do we pay developers to write bugs, and then pay testers to go and find them?

The answer simply stated is — writing software is hard. Software developers need to be extremely detail oriented, yet also keep the big picture in mind. They must balance short-term tradeoffs with long-term considerations. They often have to learn new tools and technologies with each project while balancing the demands of outside influencers. Oh, you needed that feature or bug fix done yesterday? And you want that application to work on all browsers released in the last 3 years (that’s 24 versions of Google Chrome alone since March 2011), plus mobile? It’s no wonder developers struggle to write perfect code every time.

While it can produce quality results, the old model of throwing code over the wall from the development team to the QA team is inefficient. We that build software have to do more for less, faster, and in a more complex environment than ever before. The rise of automated unit testing and Test- or Behavior-Driven Development (TDD and BDD) is an attempt to realize improvements. By testing at the core (that is, with the developer), we find bugs as early as possible (which yields tremendous savings as discussed in the post https://bridge360blog.com/2013/12/04/5-ways-qa-supports-development/). Automated unit testing also makes it easier to update the application going forward by identifying unexpected broken dependencies in the code before they get in front of a customer. Continue reading


Leave a comment

An Introduction to Property-Based Testing

By Paul Bostrom, Senior Software Engineer

Paul-BostromDo your software testing teams ever discover bugs that seemingly should have been found by the developers’ unit tests? Quite often, the developer actually did unit test the software, but perhaps simply failed to think of scenarios using the problematic inputs. What if we could tell the computer to “think” of all the values used in our unit tests? This is the approach of property-based testing.

Instead of specifying a limited number of inputs and outputs for testing a unit of software, property-based testing specifies properties that a unit of software must hold, and then relies on the computer to generate the test values.

The authoritative library for property-based testing is called QuickCheck (http://en.wikipedia.org/wiki/QuickCheck), created for the Haskell programming language, but implementations of the library exist for many other popular programming languages. To illustrate the differences in these two testing approaches, we will use a simple example — testing a square root function. Continue reading


Leave a comment

How QA Helps Nurture Long-Term Customer Relationships

By Rema Sreedharakurup, Senior Quality Assurance Engineer

Rema_Pic_BlogI have recently facilitated discussions with both peers and clients about the value that software quality assurance brings to a customer’s business.

Although it is clear that a QA engineer’s job is to accurately report product quality, there are opportunities to deliver value beyond what is expected and in the process nurture a long-term customer relationship.

During a client engagement, we select our testing approach based on factors ranging from budget, scope, team dynamics, release schedules, and in many cases alignment with customer demands.

Over time, the selected approach matures and becomes optimized to suit the nuances that exist within the customer’s business model. The time period in which the approach becomes more mature offers our QA team opportunities to engage with the client at a new level, where we help them with their business and with their software delivery process. Once this normalization period is complete, our QA engineers have learned the business and are capable of identifying additional opportunities to increase the value that the QA efforts can bring to the business. Continue reading


Leave a comment

Analyzing and Selecting the Right Software Automation Tool

By Nadine Parmelee, Senior Quality Assurance Engineer

Nadine-Parmelee1-bWith all the tool choices available today, selecting an automation tool is a project all on its own. There are a number of variables to consider when selecting an automation tool, as the upfront cost of the tool is not the only consideration. There are potential costs in tool configuration requirements as well as resource training. Next you have to identify the resources you need and how long they will be needed for your project. Taking the time to do the research for selecting the right tool is essential for both tactical and strategic success. Below is a list of elements to take into consideration when researching automation tools:

  • Is this tool recommended for the programming language used for your product?
  • Initial cost per user of the tool – Is it a “per seat” cost or is there a “site” cost available?
  • What are the maintenance costs of the tool?
  • What are the configuration costs to interact with defect and manual test case tools?
  • How many systems the tool can be utilized on?
  • Are there any training costs?
  • Is customer support provided by the tool vendor?
  • What is the expandability for the future for platforms and environments?
  • What expectations should the tool fulfill for test coverage?

We have helped a number of clients with their tool analysis and have seen firsthand how there is no single solution available for everyone. In some cases clients have wanted to use tools they already had purchased corporate licenses for, and we suggested a different tool because of application compatibility or the amount of time that would be involved to create workarounds to best fit the tool for their situation. Some clients look for “free” open source tools thinking their return on investment will be greater, yet they neglect to account for configuration costs and the costs of training involved in getting the resources up to speed. Moreover, there is the need to provide back to the open source community any changes made with the code that might help others; again a time-consuming effort to comply with while still getting the product tested.

To ensure that your team selects the right tool, make sure they are asking the questions that are most relevant and essential to your requirements. This will advance the effort forward without losing non-essential time exploring too many options. I’d recommend that a BRD (business requirements document) for the tool evaluation effort be created up front. This doesn’t necessarily need to be overkill, but collectively focusing stakeholders on requirements is worth the effort in order to keep the business and technical team in sync with expectations. Having a clear understanding of your automation project needs defined and understood is always a best practice. I also recommend that the selection process include obtaining evaluation copies of the potential tools and give your team time to evaluate them. Some tools might be eliminated from consideration quickly while others may necessarily require more time to evaluate.

In my opinion, one of the best tool evaluation methods is to take one of the most complex tests expected for the automation project and try to create those automated scripts with the evaluation packages. Make sure your project team keeps the primary automation project goals in focus; if data creation is the goal of the automation project, an automation tool that doesn’t work with a specific GUI screen or control may not be out of the running. If user interaction is the primary concern, a tool that works great with the GUI but not so great with your database structure may still be a viable candidate. It is important to analyze the strengths and weaknesses of the tools against your project needs and pick the one that will be the most efficient for your situation. Your automation project requirements may get adjusted along with the tool evaluation process as your team evaluates both together.

Test Automation is essentially a strategy that provides acceleration to market with confidence, if well executed. Taking the time to plan well and select the right tool or tools is critical for that confidence. Remember, once you take steps toward test automation, you should consider this effort as part of your ongoing quality assurance ecosystem. You will want to ensure you keep your automation scripts up to date as your product changes over time.


2 Comments

Top 5 Ways Software Quality Assurance Supports the Development Team

by Chris Durand, CTO

ChrisDurand-B360Are you thinking Development and Quality Assurance are separate, independent activities? Think again. Development and QA go hand in hand, and the better (and earlier) both teams are engaged in solid quality processes, the stronger your software will be. Here are five ways software quality supports your development team.

1. Software Quality Reduces the Cost of Fixing Bugs

At the risk of beating a dead horse, if you are not familiar with the cost of fixing a bug at various stages in the software lifecycle, this graph is crucial:

Adding more to my investment

As you can see, the later in the development cycle a bug is discovered, the more it costs to fix. If you find a problem in the requirements analysis phase, you simply change the requirements document to fix it. If you don’t discover that same issue until you are testing your beta release, you have to change the requirements document and rewrite code and retest. A good QA process will find defects earlier in the development process and reduce the cost of fixing those defects.

2. Software Quality Improves Requirements

Doing requirements well is hard without a strong software quality process in place. In many projects, testing begins too late since testers don’t start writing test cases until they have (hopefully) working code in their hands to break. If this is your process, you are missing out on the benefits of having the QA team engaged early on in the process. Strong QA professionals have an eye for detail and ensure your requirements are clear and testable. If your QA team cannot start writing test cases based on requirements, it is likely you have insufficient detail in your requirements documentation. This means you are leaving it up to a developer to self-determine many details of how your application should work instead of building a customer-driven application. So if your QA team says they cannot start writing test cases until they have a working application to reference, get them involved early and shore up those requirements.

3. Software Quality Improves Predictability of Releases

Predicting time to completion is challenging on poor quality projects with loose processes and parameters. Again, testing early is key. Feature development has a finite list of things to do, but the number of bugs in an application is unknown until you start testing. A strong QA process tests early and often, so at all stages in the development process you have an idea of where you stand. A weak QA process tests too late or not very thoroughly, and you find your dates slipping because you are still finding more and more bugs with no end in sight. High quality software also has fewer customer support issues and therefore your team can stay focused on new features instead of getting randomly pulled off to deal with the latest customer crisis and torpedoing your release schedule.

4. Software Quality Allows You to Refactor with Confidence

Writing software is a lot like gardening. Old plants must be removed, new plants added, existing plants trimmed and dead growth removed. Failing to keep your software well-groomed results in buggy software that is difficult to understand and expensive to maintain. You find you can’t easily add a simple feature because it breaks something elsewhere. To avoid this you must constantly keep your software pruned, and restructure parts that no longer make sense. Strong software quality allows you to do this with confidence since you know how the software was supposed to work before you made changes, and you can verify the software still works after your changes. If you have an automated unit or functional test infrastructure in place, that’s even better.

5. Software Quality Improves Team Morale

No one likes working on projects that have a bad reputation within the company or with customers.  High quality software improves the morale of everyone from the sales team to the support team. The sales team has confidence that the product won’t blow up in their faces during a demo and that they won’t be damaging future sales opportunities by selling the customer a “lemon”. Developers feel more pride in their work and perform better since they want to maintain the good reputation for the project or team. The support team does not dread yet another call from an upset customer due to issues that clearly should have been caught during the development process.

In summary, if you don’t have strong quality practices in place, your development team (and the rest of the company) is missing out! Get your QA team involved early and often in your development process and watch your development costs shrink, schedules become shorter and more predictable, and customer satisfaction soar. Happy testing!


Leave a comment

Understanding Build Server and Configuration Management – Part 2

Implementing Configuration Management: How to Start the Process

by Morgan McCollough, Senior Software Engineer

Morgan McCollough Bridge360In my last post we discussed the benefits of creating a development environment with continuous integration and configuration management using a centralized build server. We have had a number of clients that hesitated to start the process of building such an environment because they were too busy and thought it would take too much time and too many resources to make meaningful progress. After all, if you have a team of smart developers, they should be able to handle the build process on their own without any trouble, right? Other teams simply lack the leadership or buy-in from management to put the proper processes in place.

The following are a few basic steps your team can take to start the process of configuration management. Each step solves specific problems and will be invaluable in bringing more consistency and stability to the development process.

1.  Establish a source control branching strategy and stick to it

The first rule of branching is don’t. The goal here is to establish a small number of branches in source control to help you manage the process of merging and releasing code to test and production. Avoid the common situation where multiple and nearly identical branches are created for each deployment environment. Merge operations can be a nightmare in any environment. The number of branches should be kept to a minimum with new branches created only when something needs to be versioned separately. This even applies with distributed source control systems like Git. There may be many disparate development branches but QA and release branches should be kept to an absolute minimum.

2. Establish basic code merging and release management practices

The goal is to establish this process along with the branching strategy in order to foster a more stable code base whose state is known at all times. There are many resources available that give examples of which kinds of strategies can be used for environments with different levels of complexity. Avoid the situation where features for a given release are on multiple different branches and integration has to be done at the last minute and involves multiple merges.

3. Create and document an automated build process

Make sure the entire source tree can be built using a single automated process and document it. Avoid the situation where only one person knows all the dependencies and environmental requirements for a production build. We have seen all too often cases where it takes too much time and money to reverse engineer and reproduce a build when the point person is not available, or when critical steps of the manual process are forgotten or performed incorrectly.

4. Create and configure a build server using off the shelf components

There are many packages available for putting together your own build server and continuous integration environment. TeamCity is one of the more popular of these.

5. Establish continuous integration on the central development branch

Designate a central development branch for continuous integration. The first step is simply to initiate the automated build process every time someone checks in code. Packages like TeamCity make this very easy and include notification systems to provide instant feedback on build results. Avoid the situation where one team member checks in code that breaks the build and leaves other developers to debug the problem before a release.

6.  Create and document an automated deployment process

Avoid the inevitable mistakes of a manual deployment process or the huge waste of time to manually reconfigure a deployment for a different environment: Create automated scripts to make sure a build can be easily deployed to any environment.

7. Create unit tests and add them to continuous integration

Avoid nasty system integration problems before they happen. Also, avoid situations where changes in one area break basic functionality in another by establishing a process and standard for developers to create automated unit tests using a common framework like JUnit or NUnit. These tests should be configured to run as part of the continuous integration process and if any of them fail, the entire build process fails, and the whole team is notified.

Finally, remember that software development is a very complicated process. Even with a team staffed with geniuses, people will make mistakes, undocumented branching and build processes will confuse and confound the team, and there will always be turnover in any organization. With even the most basic processes put in place for software configuration management, any organization can avoid the horror of weeks of time spent debugging code integration problems or even worse the situation where no one is quite sure how to reproduce the production version of an application in order to find and fix a critical bug. No one is going to jump into this and reach CMM level 5 in a few weeks, but starting small and establishing meaningful and attainable milestones can get any team a long way towards the goal of a stable, reproducible, and flexible build and deployment environment.