In this blog post I’m going to talk about the technical details of deploying software. I have worked at a few companies that started out small and out grew their deployment strategy. Many of these companies didn’t really have a deployment strategy. In this post, I’m going to describe the process from development to version control to deployment and testing.
The first thing you need to realize is that your company’s source code is an expensive resource. The price of that source code is measured in man-hours of labor that it takes to develop the software. Loss of such software has the potential of bankrupting a company and there are such simple and cheap ways to ensure against such a catastrophe that I am dumbfounded when I see a company setup that doesn’t take these steps.
The first step is to create a repository (aka repo). This is a place where your developers check in and out changes. This is a major step up from just backing up a common source code directory. By having a version controlled repository, you can look at any version and roll back or undo changes made by a programmer gone rogue (intentionally or not). It also aids in troubleshooting bugs that suddenly appear.
If your a small company or you have limited funds you can setup a GitHub or BitBucket account. These accounts have free accounts and cheap accounts. For GitHub you’ll have to pay in order to keep your repository private. BitBucket advertises free unlimited private repos. My company uses BitBucket and I also have my own BitBucket account for personal use. I also have a GitHub account. GitHub currently charges $7 a month for 5 private repositories. BitBucket has a pricing scheme for number of developers. You can get up to 5 uses/developers for free and the pricing starts at 10 users for $10 per month.
You’ll have to do some research on each of these repositories before you make your choice. There are other products that can be used with these repositories and I’m going to talk about those next.
You absolutely have to have version control. As soon as you add your second developer you will begin to lose productivity if you are not using a version control system. There are plenty of free systems available and they all have their quirks and difficulties. My company uses Mercurial for version control and it has a steep learning curve. Mercurial works with BitBucket but not with GitHub. Git is another version control system that I have not had the time to use yet. Git works with both GitHub and BitBucket. If you’d like to know more about the differences between GitHub and BitBucket, there is an excellent article here: Bitbucket vs. GitHub: Which projcect host has the most? I would recommend reading this article in addition to my blog post.
Mercurial is a distributed version control system which is far superior to the old clunky version control systems such as Visual Source Safe, Microsoft Team Foundation Server and CVS. These older systems usually allow a user to check out a file and keep it locked until they check it back in. The problem with this method of version control is that only one developer can work on the files at a time and if they check it out at one location (say at home) and they are at another location, they can’t unlock the file. Mercurial allows users to check out the whole repository to their local computer. They work on their changes, then they can either “merge” their changes into the master repo or they can create a branch. If the merge has conflict then the developer must resolve such conflicts right away. Mercurial has tools for handling complex situations and the developer can roll-back versions to undo major snafus.
Branching is the best invention since sliced bread was introduced to the toaster. Let’s pretend for a moment that you have a dozen developers working on the same software. They are working on different enhancements that could cause them to change the same classes and methods for different purposes. Now let’s also pretend that there is a deployed version. If there is only one branch, then the deployment must be carefully controlled so that a partially completed project doesn’t get deployed before it is ready. Also, every project being worked on must be completed and debugged before the next release can go out. To make matters worse, a bug fix would be next to impossible once the first enhancement changes have been checked in. It is also recommended that developers check in their changes at least every night (in case of an overnight disaster).
Branches are created for each new enhancement and are separated from each other so each enhancement can be completed at any time and then re-merged with the default or master branch. The default branch would be the branch that is always available for deployment. To keep things very clean the default branch should never contain un-reviewed code. The reason the default branch should remain clean is in case an emergency crops up and a bug must be fixed immediately. If bugs that have been fixed but not reviewed remain on their own branch, then developers can go home on the weekend and an emergency call to fix something will not require someone to hurry up and review the other bugs that are not ready for release.
A staging server is usually a server that is identical to the production server or system. You can also point your staging system to the production database if your staging tests are read-only. With a staging web server and a staging database you can run comparison tests between stage and production web sites. Simple web crawling software can match web pages between your stage web site and production web sites (assuming your comparing with the same database server). You can also build software to test features of your system like form data entry web pages. These tests might be scripted and take hours to run, but you can run those scripts at night and deploy first thing in the morning (after reviewing your test results).
My company performs code reviews. That means that we have another developer double-check the code written by the first developer. The feature or bug is tested and the code is looked over and then it is checked off. We use Jira to track the progress of a task and one of the steps is the code review. Jira integrates into BitBucket and our version control system is linked right to the ticket so the code reviewer can quickly find the branch and changes that were made. Once the code is reviewed and the ticket is dragged into the ready for release swim lane, then the branch is merged. Sometimes the reviewer merges the branch, sometimes the developer, that’s entirely up to you. The branch gets merged into default and is ready for release. Many times our company will merge several changes into the default branch and then a release of several versions will go out at once.
Releasing and Deploying
We use TeamCity for our deployment server. This server uses powershell scripts to check out versions of software from the repository, compile the software, run unit tests and prepare the package for deployment. Then the developer can run testing scripts against the software in a staging or test environment. TeamCity is a very powerful product and it integrates with BitBucket and provides roll-back capabilities. When your developers deploy software, no matter how well the software is tested in staging, it might utterly fail when it is deployed on the production system. When that occurs, you need to roll back as quick as possible to reduce your customer down-time. At that point the version that was unsuccessfully deployed must be analyzed and properly fixed. There is also the possibility that you need to add testing to catch such problems in the future.
BitBucket also has a product called Bamboo. If I was starting a new system, I would investigate Bamboo and see if it fits your needs better than TeamCity. With that said, TeamCity is a really good product. You can use the deployment capabilities of Visual Studio if your system is used only internal to your company or your website isn’t used by many people. However, if you have a web site or a web farm, you’ll need a proper deployment server. The deployment server can deploy to multiple servers and it can be configured to signal servers for shutdown (i.e. iis servers) before switching to a new version and then signaled to startup. We currently deploy our new version of software into a new directory on the web server (naming the directory with the version number) and then the iis server is stopped, the iis server is pointed to the new directory and then started back up. If something happens and we need to go back, we can manually point the iis server to the previous directory and not wait for the previous version to be re-copied to the web server. If you have a web farm, it is too tedious to deploy to each server without an automated or scripted method. TeamCity has the ability to deploy to multiple servers making it a one-click operation.
One other aspect of development to be aware of: Logging. Use a logging system such as Log4Net. This software allows your developers to create log files and email error messages to a central email in-box. Once this is setup and logging is added to every try/catch block (as well as anything in your code you wish to monitor), you will be able to open the email in-box and watch errors come in when you deploy. If you suddenly begin to get log errors about web page crashes, then you need to roll-back quickly. Without logging, you might not even realize that your customers are experiencing page crashes.
One aspect of software development that I see neglected too often is an isolated development system that matches your production system. I’m not advocating that you spend a lot of money on this system, but it needs to be able to be used by your development team to test software as they are developing it. No developer should ever test software against a production system. Such an act is inviting disaster. At my company we have two MS SQL servers and a development web server (as well as other special purpose servers like ftp). Most of our development is done on our PC (such as the web application which we normally test with our local iis server). Keeping a development database in synch with a production database is a massive undertaking. Our system uses scripts to copy backup files from the production database server and restore to the development servers. These servers are alternated giving us an alternate database in case the script fails during the night. If you are designing your first system, give some thought about the size of your customer databases and how you are going to handle your development database.
Another process we have in place is to use deployment scripts. These are MS SQL query scripts that can modify tables, stored procedures, views, etc. to match changes on a branch. When the branch is merged with default and deployed, the database update scripts must be run (usually before the deployment). Some deployments require a clean-up script that must be run after deployment. These same scripts are checked into a separate repository with its own version control and branches (though we try to name the branch to match the software that they belong to). The scripts are tested on the test servers and we usually try to provide a roll-back script if the deployment is hazardous (like changes to stored procedures).
Everything in this blog post can be done on the cheap. If your a small start-up company or you have a limited development budget, you can setup an entire version control and deployment system to protect your investment and reduce your down-time. By setting up such a system early on (like the day you hire your first programmer), you will avoid the headaches I have seen over the years when scores of programmers are performing dangerous tasks on a production system, or software versions get lost because they don’t have a proper repository setup. Not to mention downtime and debug time due to the lack of testing tools or versions to compare with.