by Troy Rudolph, Senior Software Engineer
We’ve been assisting one of our clients in the development of a large web application facing many large customers and working with several types of devices for data gathering. As a result, we’ve built a lot of automated tests at all levels: acceptance, functional, integration and unit.
Originally, our acceptance testing environment consisted of:
- Codeception, primarily the WebDriver module,
- Selenium, as a browser manager, e.g. to start/stop browsers as needed, and
- PhantomJS, a headless browser.
We selected PhantomJS because it is a headless browser, e.g. it requires no physical display to run the application and thus has a reputation for speed. For whatever reason, we were unable to make PhantomJS work effectively for us and set out to find another solution.
Unfortunately, there aren’t any other headless browsers of note. So, we started looking for ways to run an existing browser as if it were headless. We settled on Firefox largely because it is more compatible with our testing environment.
To run in a headless environment, Firefox still needs a place to render. In a Unix/Linux environment, the X-Windows Virtual Frame Buffer (Xvfb) provides a memory-based X-terminal and thus no physical display is needed.
Thus, our new acceptance testing environment consists of:
Although it took a lot of experimentation, it turns out that this is an easy thing to set up. First, we installed Xvfb on Linux environment. On our Centos system, this can be done as follows.
yum install Xvfb
Next, we created a new service for Xfvb, by adding a new service script to /etc/init.d/ folder. In the Xfvb service script, starting the service is easy. In the start() function, this command below will execute Xvfb as a detached process and create window :4444.
/usr/bin/Xvfb :4444 -ac &>/dev/null &
Next, we installed Firefox. We downloaded the package from https://download-installer.cdn.mozilla.net/pub/firefox/releases/37.0/linux-x86_64/en-US/ and installed it manually using good old TAR.
Finally, the script for our Selenium service, also in /etc/init.d, was modified to establish window :4444 as the default window, and to make Firefox available in the path. Now when Selenium launches Firefox, it will render in window :4444 which exists nowhere but virtual memory.
export DISPLAY=:4444 export PATH=/testautomation/firefox java -jar /ta/selenium-server-standalone-2.45.0.jar -port 4444 &>/dev/null &
Having established the services, we updated the Codeception configuration file for our acceptance suite (acceptance.suite.yml) to request an instance of Firefox for each test executed.
modules: enabled: - WebDriver: url: 'http://your.app.com/' browser: firefox
The results from this effort were very good. The switch to Firefox caused a ten to fifteen percent increase in the elapsed time for our tests. This “cost”, however, is well worth it. The reliability of our acceptance suites is greatly improved. This gives us much better feedback on the health of the application.
Note: Some experimentation was required with releases of Selenium Standalone Server and Firefox to arrive at a consistently reliable combination. For us, Selenium Standalone Server 2.45.0 and FireFox 37 worked the best.