In earlier blogs in this series we talked about configuring and tuning Unix and Linux based servers for high performance systems. Now, I will talk about configuring Windows servers. Windows servers are traditionally not used as servers anymore, but there are some applications where Windows is still the only option. Unix, and primarily Linux, have become the go-to platforms for servers due to their security, relatively low setup and maintenance costs and high performance output. Windows servers aren’t as easy to maintain, have a large memory footprint and until more recently have been problematic when it comes to security. They do often make great simulator machines especially when you have some old commodity Windows boxes lying around.
Database structure changes can be very stressful especially in production. It is a pain to migrate databases without having a formalized solution. We all are aware that keeping track of the schema changes is important and always very beneficial to reduce the overall cost of the change.
Database migration primarily involves changing the database structure at a point in time to another, resulting in a new version. Prevention of possible errors in the process, caused by any human factor, can be critical in the maintenance of database servers running different schema versions. Importantly, we should apply one schema change only once and in cases when running two or more operations at same time, we need to make sure the changes are free of race conditions.
While there are other database migrations tools available, we preferred alembic as an automated migration tool for a client application with multiple databases. We use SQLAlchemy as an Object Relation Mapper (ORM) in the application and alembic is a lightweight database migration tool for usage with SQLAlchemy. Also, alembic is developed and maintained by Mike Bayer, the author of SQLAlchemy.
Alembic can be useful for:
Multiple database support
Automatic scripts generation
Avoidance of race conditions
Scripts encapsulation in a single file
Backward or downgrade compatibility
Offline mode support
Typical alembic environment structure looks like:
project/ alembic.ini alembic/ env.py script.py.mako versions/ generated_migration_scripts.py
Alembic can be configured by providing the database driver information in the alembic.ini file or by customizing the database config in env.py so that the configurations are in sync between application and migration scripts. Alembic supports the multidb configuration to have the versions for multiple databases defined in the same environment. In our application for databases with separate schema definitions we created two different alembic environment directories, each holding their own configuration and versioned migration scripts. This way it helped us to maintain the versions separately and avoid probable schema conflicts while running them. One way or the other, it seems to be well customizable and easy to adapt as needed.
Our client application is hosted in Amazon’s Elastic Compute Cloud (EC2) with the database in Amazon’s Relational Database Server (RDS). We configured our Amazon CloudFormation script adding a new task definition to run the database migration script as a standalone task. We included alembic upgrade commands in a bash script, which are triggered by the defined AWS task. The task can be run using ‘Run Task’ operation in Amazon ECS cluster console or using AWS Command Line Interface as,
aws ecs run-task --cluster <ECSCluster_identifier>\ --task-definition <alembic taskdefinition identifier>
The setup process seems to be straightforward and well documented. Using SQLAlchemy and alembic together, we can automatically generate and customize the migration scripts and we can identify the differences between existing database tables and the defined model in code. There are some limitations of the autogenerate option that are mentioned in the documentation. Each migration is called a revision and knows what order to be run in because each revision is given a down_revision to identify its parent. The revisions range from base to head, base being the initial or stamped revision and head as the latest revision.
Adopting the simpler workflows of alembic has helped us reduce complexity and risks, save implementation time, and increase development throughput. It has provided an automated database refactoring technique which certainly gives us more control over the release process of new changes and enables continuous delivery of our product.
Over the years the software industry has developed many solutions to producing quality software to meet business needs. Software, however, is an ever-changing industry, and our tried methods are failing to keep up with modern development practices. Quality Engineering has made waves in the industry for a few years and is often associated with iterative or agile development processes, as a new way of ensuring quality software. How does Quality Engineering fit in to traditional Quality Assurance to get us working software?
In Java performance testing, one of the most common and sometimes most frustrating issues is the memory leak. Even the most experienced engineer can slip a memory leak into their code. It is important to know how to spot one and debug the issue. Even if you can’t find the source of the issue without a developer’s help, they will always need data to work with so being able to provide useful information is crucial. Therefore, for a Java memory leak, the first thing you will always be asked for is a heap dump.
by Paul Cooper, Senior Localization Engineer
Any time software is destined for the global market or for a diverse audience within a region, it is well worth planning the localization effort. Seemingly small errors in wording or presentation can ruin the professional aspects of a product, leaving instead an impression of amateurism or cultural indifference. The following steps are a guide to the localization process to produce successful translation packages.
The new employee selection and hiring process is important for any business, but it is uniquely important, and uniquely difficult, for software development organizations. Many organizations conduct a hiring process that overemphasizes the assessment of some of the candidate’s qualities, while ignoring others that may be more important. A balanced approach, and more reasonable expectations, can lead to a more efficient hiring process. Continue reading
by Larry Van Sickle, Senior Software Engineer
AngularJS is a versatile toolset for building browser-based applications.
How can a developer make the elements of a chart clickable using NVD3 in an Angular application?
by Gary Ormsby, Senior Software Engineer
Bamboo, Atlassian‘s Continuous Integration (CI) product, can be an attractive option for those utilizing Atlassian’s other products, as we do. We started using its cloud version recently for a client’s turn-key web app that we are developing. Historically, we’ve used self-hosted servers for our infrastructure but here we wanted to explore an online solution. The initial setup to integrate these systems was substantial and we have found that their maintenance has required spikes in time as well.
by Karel Gonzalez, Senior Software Engineer
A few weeks ago, I had the opportunity to attend the Lonestar Application Security Conference here in Austin. Security is something I have always been mindful of during my development, but I still felt a sense of futility about it. I ask myself on a fairly regular basis “I’m doing something, but am I doing enough?” Continue reading