Agile Programming with Kanban

Summary

In this post I’m going to discuss the differences between waterfall and agile methods of programming.  Then I’m going to focus more on Kanban and what advantages and disadvantages it has in comparison to Scrum.  I will also be discussing the business case for using Kanban and how it can improve the performance of developing software and reducing waste.

Producing Software

Producing software costs money.  That is the bottom line.  The largest cost to creating software is the cost of man-hours or how much money is paid to the analysts, developers and quality people to create the software.  After the software has been written and delivered, then the primary cost becomes the operating costs.  Operating costs can break down into licensing fees for database and operating systems as well as hardware or hosting fees.  There will also be labor expenses tied up in support such as IT personnel, help-desk and training personnel.  I’m going to focus on the cost it takes to create the software.  As a seasoned software architect, my job is to analyze and design the software with an eye for reducing the cost of operating the software after it is delivered.  That is a subject for another blog post.  In this post I’m concerned about the process of creating the software with an eye toward reducing wasted time and money.

Waterfall Method

Everyone has heard of the waterfall method.  It’s the most obvious way to design and build software.  Basically a team of software analysts communicate with the customer to determine a list of features required/desired for the software to be built.  They will then turn this into a requirements document.  Once the requirements document has been agreed upon, a massive design effort is launched.  This can take months and for grand projects it can take years.  The project is designed down to the smallest detail so that it can be estimated.  Estimating the amount of man-hours it takes to perform the programming can be done by analysts or, if the developers are already working for the company performing the work, they can perform the estimation (it’s preferable to obtain estimates from the developers that will actually work on the project because then you can get buy-in).  Next, the negotiations start with the customer.  The customer will most certainly decide that they can’t afford the “magic wand”1 version of the software that they described and will begin to reduce the features to get the price down to something reasonable.

Once a contract is signed, then the specifications are thrown over the wall (figuratively) to the developers.  I’ve seen companies that use GAANT2 charts with critical paths detailing when each feature will be scheduled, but most companies just hand over the spec with the idea that developers can just divide-and-conquer.  Then months and probably years are spent developing the software.  There are usually demonstrations of what has been created to keep the customer from cancelling the project.  There is nothing worse than paying for a product and not hearing anything about the product until a year later when the whole thing is delivered.  The customer will want progress reports.  Is it on schedule?  Hopefully, for the customer’s sake, they paid a flat-rate price for the software and will not have to shell out more money if the developers cannot produce the software by the deadline indicated in the estimates.  Otherwise, more than likely, the project will come in over budget and late.

Once the developers have completed their tasks, quality can start testing the whole project.  QA should be building test scripts while the developers are creating the code.  Otherwise, there should be no QA people on the project until they are about ready to start testing.  Once the QA people have completed their tasks and the developers have fixed any bugs found, then it’s time for the release.

This is the waterfall method in a nutshell.

Problems with Waterfall

The biggest problem with the waterfall method is that it is wasteful.  Many projects get cancelled before completion and the customer is left with nothing.  If the tasks are not scheduled to ensure that sections of the software are workable, then there is no way to cut off development and deliver a working product with fewer features.  Also, if a project is cut because of cost overruns, then there is still QA work to be done.  Finally, the software analysts and designers must be paid for designing the entire project before any software is built.

Next, there are months where no software is being written.  This is the analysis, design and estimation phase.  The whole thing must be estimated before an agreement can be signed.  That means that time is burning while this whole process is going on and development doesn’t start until all of this work is completed.

Usability is difficult to perform before the software is built.  Usability is expensive to fix after the whole project is complete (designed, built and QA’d).  The best method of ensuring usability is cheap and effective is to test the usability as the software is being built.  This is not something that waterfall can accommodate without difficulty.  It would require very detailed scheduling and changes would loop back to a lot of changes in the design that has already been completed.  In practice, waterfall does not support effective usability testing.

The Agile Method

The theory behind the agile method is that the software is only roughly specified up front.  Then the most critical and important parts of the software are designed and estimated first.  Usually only a week or month’s worth of software is designed at a time.  Then the design is estimated by the developers who will start work immediately.  Once a few pieces of the software (usually called stories) are completed, they are QA’d and then deployed to a limited audience (the customer).  The customer will then review what is being demonstrated and normally they get hands-on access to test what has been created.  If the piece of software is large enough to be useful, the customer can start using the software.  A critique of the software can be fed back to the analysts to have some of the usability problems fixed in a future release.  Meanwhile the design team is working on the next specifications or stories to be put in the hopper.  At the same time the developers are working on the previously designed and estimated stories.  QA is working on the previously finished stories and the quality checked pieces are collected up for the next deployment.  This continues like a factory.

The stories need to be produced in the order that the customer demands.  So they can put off minor enhancements to the end and have working software as early as possible.  If the customer decides that they have spent enough money and the product is good enough, then they can cut off the project and walk away with a product that is usable.  This reduces waste.

Benefits over Waterfall

It’s almost obvious what the benefits are:

  • Working software is delivered right away.
  • There is a short startup time when the designers create the first stories.  The developers start working right away instead of waiting months or years to start.
  • The customer is more involved in the creation of the product.  Instant feed-back about usability problems can help fix the problem before the developers have forgotten what they’ve worked on.
  • The customer can cut off the project at any time and walk away with a functioning product.
  • The customer, theoretically, could start using their product before it is finished.
  • Re-prioritizing features is quick and easy since developers don’t just grab any story at any time.  The customer has full control on when features are developed.

Scrum vs. Kanban

There are several methods of running an agile development team, but two are near the top: Kanban and Scrum.  Scrum is a method where there is a fixed amount of time that developers will work on a set of stories.  This is called the sprint and it may last two, three or four weeks.   The sprint is usually a fixed time frame that is repeated throughout the project, and they are usually numbered.  For example: The analysts/designers will group stories into sprint 1 until they have filled two weeks worth of work for the team that will work on the software.  Then they will start to fill in sprint 2, etc.  In a Scrum, there are daily-standup meetings where the “team” discusses the progress since the previous standup.  The idea for a standup is that everyone stands and the meeting is limited to reporting progress, issues and blockers.  If a one-on-one discussion lasts more than a few minutes, it must be taken off-line because it is wasting the time of the entire team to sort out a problem that can be solved by two people.  Scrums provide an environment where the team knows what everyone else is working on for each sprint.

Stories that are completed are immediately passed on to QA during the sprint.  The QA personnel are included as part of the “team” and they attend the daily standup meetings as well.  At the end of the sprint the stories that are complete and QA’d are deployed and demonstrated to the customer who can give feed-back.  Any changes can be put into the next sprint or a future sprint.

Kanban is a bit different.  Kanban is broken down like a factory.  There are “lanes” where the stories will go when they are moved from the design phase to the deployment phase, like a pipeline.  Analysts/designers will put the stories into the backlog lane.  Once the stories are in the backlog, developers can immediately pick one story up and work on it.  The developer will move the story to their development lane.  When the story is complete, it is moved into the “To be QA’d” lane.  Then a QA person can pull a story out of that lane and put it in their “QA” lane.  After the QA person has completed their task, the story can be placed into the “To be Deployed” lane.  When enough stories are in that lane, they can be deployed to be reviewed by the customer.  Technically, each story can and should be deployed immediately.  This can be accomplished by an automated deployment system.

In the case where QA discovers a bug, the story must be moved back into the developer’s lane.  The developer must fix the bug as soon as he/she can and get it back into the “To be QA’d” lane.  There can be limits set to each lane to reduce the amount of allowed work in progress or WIP.  The WIP count controls the flow of stories going through the pipeline.  As you can see, Kanban is setup as a just-in-time delivery system.  The customer can literally end a project as soon as they decide they have enough features and the whole assembly line can stop without much waste in the system.  Scrum can have up to two-weeks (or three or four depending on the sprint size) worth of waste in the system when a cut-off occurs.  Keep in mind that scrum is still extremely efficient compared to waterfall and we are only splitting hairs over how much waste exists if a Kanban or Scrum project is cancelled.

Kanban

I’m going to focus on potential issues that can occur with Kanban.  First, it can be difficult to determine when a feature will be delivered.  This happens for items that are sitting in the backlog lane.  Items in the backlog can be re-prioritized until they are picked up and work begins.  Once work begins, the story must be completed (unless there is some circumstance that warrants the stoppage of work on a particular story).  If a story is at the top of the back-log an estimate of when it will get completed can only be determined when it is picked up by a developer.  Adjusting the WIP count to be low can make it easier to estimate when the story will go through the pipeline, but assumptions must be made about the timing of existing stories.  If the WIP count is high, then developers might have several stories in their lane at one time.  Now you’re probably scratching your head and thinking “why would a developer have two or more stories in their lane at a time?”  This situation usually happens when there is something that is blocking the completion of a story.  Maybe there is a question about how the story will be implemented.  The developer is waiting for an analyst to make a decision on how to proceed.  In such an instance, it is best for the developer to pickup the next story and start working on that one.  Developers are responsible for clearing their lane before picking up another story unless there is something blocking a story.  In other words, no non-blocked stories can sit in the developer’s lane.

Straight Kanban assumes that all developers are minimally qualified to work on every task.  That is not usually realistic and there are work-arounds for shops that have specialized developers.  First, Kanban can be setup with multiple backlog lanes.  Let’s pretend that your shop has back-end developers and front-end developers3.  As stories are created by analysts/designers, they can be divided and organized by front-end work vs. back-end work and placed in the appropriate backlog lane.  Back-end developers will pull stories from the back-end backlog lane and so on.  Of course there is the scheduling problem where now back-end developers must finish an API before a front-end programmer can consume the API for their interface.  This can be mitigated by breaking off a story that creates the API “shell” program with dummy data that responds to requests from the front-end.  Then the front-end developer can start consuming the API data before the back-end developer has completed the API.  Both lanes must be monitored to ensure that back-end programming lines up with front-end programming.  Otherwise, there could be situations where the front-end programmers have no API to consume and the software to be QA’d and deployed is not usable.  For Scrum, the back-end programming and front-end can be staggered in different sprints to ensure that the APIs are completed before the front-end programmers start programming.  This technique can also be used in Kanban by starting the back-end group ahead of the front-end group of programmers.

As you can tell there is usually no daily standup for Kanban.  There’s no need.  Each individual developer can meet with the person they need to in order to complete their work.  Meetings can still be held for kick-offs or for retrospectives.  I would recommend a retrospective for every project completed, Scrum or Kanban.

One last feature of Kanban that is more accommodating is the idea of throwing more resources at the problem.  When there is a team working on a problem there are multiple communication paths to consider.  Each team member must be made aware of what everyone else is working on.  In Kanban the idea is to design the stories to be independent of each other.  If a developer can focus on one story at a time, then adding a new developer to the mix is easy.  The new developer will just pickup a story and start working on it4.

Pitfalls to Watch For

Here’s a big pitfall with the agile method that must be headed off early on:  In an agile workshop the entire project is not estimated up front.  The customer wants to know what it would cost for the whole product.  In fact, most customers want a cafeteria style estimate of what every feature will cost so they can pick and choose what they want.  What the customer does not want is to pay for a month’s worth of work and then wonder how many more months it will take to get to a viable product.  It would be nice to know ahead of time how long it will take and how much it will cost.  To accommodate this, agile shops must be able to give a rough estimate of the entire project without designing the entire project.  In fact the product does not have to be designed down to the nth degree to get an estimate.  Also, an estimate on a product that is designed to the tiniest feature is not more accurate than an over-all rough estimate.  Confidence level is something that should always be taken into consideration in software design.  As projects get larger, the confidence level drops lower and lower.  The confidence level does not increase just because the product is designed down to the detail.  Don’t believe me?  Search for any large government software project that was cancelled or over-budget an you’ll discover that these projects missed their marks by 100 to 200% or more.  They are always underestimated.  Those projects are designed to the intimate detail.  The problem with software design is that there are always so many unknowns.

Create a rough design.  List the features and give a rough estimate for each feature.  Add some time to features that are really gray in definition.  Tighten your estimates for features that you know can be built in “x” amount of time.  This estimate can be used for a contract to “not exceed…” “x” amount of months or “x” amount of resources.  When the project is about to run up against the end, the customer must be made aware of the short-fall (if there is any).  Normally a shortfall will occur because a customer thinks up a feature that they need while the project is in progress.  This is additional work that can be inserted into the work-flow and preempt one of the lower priority features or the customer can agree to an extension of the project.  Sometimes a feature takes longer than the estimate and the customer should be notified of each feature that went over budget.

Customers can also be A.D.D. when it comes to deciding which stories to stuff in the backlog.  The backlog queue can churn like a cauldron of stories causing the scheduling of features to be delivered to be unknown.  If the customer is OK with the unknown delivery time, then the churn does not effect the development staff.  However, if stories are pulled out of the work lanes, then problems can start.  Shelving unfinished code can be hazardous.  Especially if a story is shelved for a month and then put back in play.  By that time the un-shelved code my not work with the current code-base and must be reworked, causing the estimate for the story to go long.

Documentation

I would recommend a wiki for the project.  The wiki should contain the design specifications and changes as they are made.  If you are using a product such as Confluence and Jira, you can use the forum feature to add questions to a story and follow up answers.  This becomes your documentation for the software.  If you add developers, they can read through the notes on what is going on and get a good idea of why the software was built the way it was built.  This documentation should be maintained as long as the software is in production.  Future development teams could use this documentation to see what ideas went into the original design.  When an enhancement is added, the notes for the enhancement should be appended to this documentation for future developers to refer to.  This documentation can also provide witness testimony for any disputes that occur between the customer and the entity developing the software.

Notes

  1. The term “Magic Wand” refers to the idea of: What would the customer want if they had a “Magic Wand” and could have every feature right now for free.
  2. GANNT charts and critical path methodology is used in physical construction projects.  Many people try to visualize software development as a “construction” project, like building a house.  Unfortunately, the methodology does not fit software design because every software project is like inventing something new, where building a house is so methodical that there are books full of estimates for each task to be performed.  GANNT charts are used for home construction, assembly line theory fits software development more accurately.
  3. A typical shop with a large number of developers will contain experts in database design, front-end advanced developers, entry-level front-end developers, back-end developers (which are usually API experts) and other specialized developers.  In such a situation scheduling can get a bit dicey, but the same work-arounds apply.
  4. In practice this technique should always work.  In the real-world there are pieces of the puzzle that are dependent on other pieces that are already completed.  A new developer will need some ramp-up time to get into the flow of what is being built.  This can also slow down existing developers who must explain what is going on.

 

 

Automated Deployment with .Net Core 2.0 Unit Tests

If you’re using an automated deployment system or continuous integration, you’ll need to get good at compiling and running your unit tests from the command line.  One of the issues I found with .Net Core was the difficulty in making xUnit work with Jenkins.  Jenkins has plug-ins for different types of unit testing modules and support for MSTest is easy to implement.  There is no plug-in that makes xUnit work in Jenkins for .Net Core 1.  There is a plug-in for nUnit that works with the xUnit output if you convert the xml tags to match what is expected by the plug-in.  That’s where this powershell script becomes necessary:

https://blog.dangl.me/archive/unit-testing-and-code-coverage-with-jenkins-and-net-core/

If you’re attempting to use .Net Core 1 projects, follow the instructions at the link to make it work properly.

For .Net Core 2.0, there is an easier solution.  There is a logger switch that allows you to output the correct xml formatted result file that can be used by the MSTest report runner in Jenkins.  You’ll need to be in the directory containing the project file for the unit tests you want to run, then execute the following:

dotnet test --logger "trx;LogFileName=mytests.trx"

Run this command for each unit test project you have in your solution and then use the MSTest runner:

This will pickup any trx files and display the familiar unit test line chart.

The dotnet-test command will run xUnit as well as MSTest so you can mix and match test projects in your solution.  Both will produce the same formatted xml output trx file for consumption by Jenkins.

One note about the powershell script provided Georg Dangl:

There are environment variables in the script that are only created when executed from Jenkins.  So you can’t test this script from outside of the Jenkins environment (unless you fake out all the variables before executing the script).  I would recommend modifying the script to convert all the $ENV variables into a parameter passed into the script.  From Jenkins the variable names would be the same as they are in the script (like $ENV:WORKSPACE), but you can pass in a workspace url to the script if you want to tests this script on your desktop.  Often times I’ll test my scripts on my desktop/laptop first to make sure the script works correctly.  Then I might test it on the Jenkins server under my user account.  After that I test from the Jenkins job itself.  Otherwise, it could take a lot of man-hours to fix a powershell script from re-running a Jenkins job just to test the script.

 

 

Deploying Software

Story

I’ve worked for a lot of different companies.  Most of them small.  Several of the companies that I have worked for have had some serious growth in their user base.  Every company I have worked for seem to follow same path from start-up to mid-sized company.  Start-ups usually staffed by amateur programmers who know how to write a small program and get it working.  Inevitably the software becomes so large that they are overwhelmed and have no clue how to solve their deployment problems.  Here are the problems that they run into:

  1. The customers become numerous and bugs are reported faster than they can fix them.
  2. Deployments become lengthy and difficult.  Usually causing outages after deployment nights.
  3. Regression testing becomes an overwhelming task.
  4. Deployments cause the system to overload.
  5. Keeping environments in-sync becomes overwhelming.

Solutions

This is where continuous integration techniques come into play.  The first problem can be tackled by making sure there is proper logging of system crashes.  If there is no log of what is going on in your production system, then you have a very big problem.

Problem number two is one that can be easy to solve if it is tackled early in the software development phase.  This problem can only be solved by ensuring everyone is on-board with the solution.  Many companies seem to double-down on manual deployments and do incredibly naive things like throwing more people at the problem.  The issue is not the labor, the issue is time.  As your software grows, it becomes more complex and takes more time to test new enhancements.  Performing a scheduled deployment at night is a bad customer experience.  The proper way to deploy production is to do it in the background.

One method of performing this task is to create new servers to deploy the software to and test the software before hooking the servers into your load-balancer.  The idea is to automate the web server creation process, install the new software on the new servers and then add them to the load-balancer with the new features turned off.  The new software needs to be setup to behave identical to the old software when the new features are not turned on.  Once the new servers are deployed, the old servers are removed from load-balancing one at a time until they have been replaced.  During this phase, the load of your servers need to be monitored (including your database servers).  If something doesn’t look right, you have the option to stop the process and roll-back.

Database changes can be the challenging part.  You’ll need to design your software to work properly with any old table, view, stored procedure designs as well as the new ones.  Once the feature has been rolled out and turned on, a future clean-up version can be rolled out (possibly with the next feature release) to remove the code that recognizes the old tables, views, stored procedures.  This can also be tested when new web servers are created and before they are added to the web farm.

Once everything has been tested and properly deployed the announcement that a new feature will be released can be made, followed by the switch-on of the new feature.  Remember, everything should be tested and deployed by the time the new feature is switched on.  If you are running a large web farm with tens of thousands (or more) of customers, you may want to do a canary release.  A canary release can be treated like a beta release, but it doesn’t have to.  You randomly choose 5% of your customers and switch on the feature on early in the day that the feature is to be released.  Give it an hour to monitor and see what happens.  If everything looks good, add another 5% or 10% of your customers.  By the time you switch on 20% of your customers you should feel confident enough to up it to 50%, then follow that by 100%.  All customers can be switched on within a 4 hour period.  This allows enough time to monitor and give a go or no-go on proceeding.  If your bug tracking logs are reporting an uptick in bugs when you switched on the first 5%, then turn it back off and analyze the issue.  Fix the problem and proceed again.

I’ve heard the complaint that canary release is like a beta program.  The first 5% are beta testing your software.  My answer to that is: If you are releasing 100% of your customers at the same time, doesn’t that mean that all your customers are beta testers?  Let’s face the facts, the choice is not between different versions of the software.  The choice is between how many people will experience the software you are releasing, 5% or 100%.  That’s why I advocate random customer selection.  The best scenario rotates the customers each release so that each customer will be in the first 5% only one it twenty releases.  That means that every customer shares the pain 1/20th of the time instead of a 100% release where every customer feels the pain every time.

Regression Testing

Regression testing is something that needs to be considered early in your software design.  Current technology provides developers with the tools to build this right into the software.  Unit testing, which I am a big advocate of, is something that needs to be done for every feature released.  The unit tests must be designed with the software and you must have adequate code coverage.  When a bug is found and reported, a unit test must be created to simulate this bug and then the bug is fixed.  This gives you regression testing ability.  It also gives a developer instant feed-back.  The faster a bug is reported, the cheaper it is to fix.

I have worked in many environments where there is a team of QA (Quality Assurance) workers who manually find bugs and report them back to the developer assigned to the enhancement causing the bug.  The problem with this work flow is that the developer is usually on to the next task and is “in-the-zone” of the next difficult coding problem.  If that developer needs to switch gears, shelve their changes, fix a bug and deploy it back to the QA environment, it causes a slowdown in the work flow.  If the developer checks in their software and the build server catches a unit test bug and reports it immediately, then that developer will still have the task in mind and be able to fix it right there.  No task switching is necessary.  Technically many unit test bugs are found locally if the developer runs the unit tests before check-in or if the system has a gated check-in that prevents bad builds from being checked in (then they are forced to fix their error before they can continue).

Load Testing

When your software becomes large and the number of customers accessing your system is large, you’ll need to perform load testing.  Load testing can be expensive, so young companies are not going to perform this task.  My experience with load testing is that it is never performed until after a load-related software deployment disaster occurs.  Then load testing seems “cheap” compared to hordes of angry customers threatening lawsuits and cancellations.  To determine when your company should start load-testing, keep an eye on your web farm and database performances.  You’ll need to keep track of your base-line performances as well as the peaks.  Over time you’ll see your server CPU and memory usage go up.  Keep yourself a large buffer to protect from a bad database query.  Eventually your customer size will get to a point where you need to load test before deployments because unpredictable customer behavior will overwhelm your servers in an unexpected manner.  Your normal load will ride around 50% one day, and then, because of year-end reporting, you wake up and all your servers are maxed out.  If it’s a web server load problem, that is easy to fix: Add more servers to the farm (keep track of what your load-balancer can handle).  If it’s a database server problem, you’re in deep trouble.  Moving a large database is not an easy task.

For database operations, you’ll need to balance your databases between server instances.  You might also need to increase memory or CPUs per instance.  If you are maxed out on the number of CPUs or memory per instance, then you are left with only one choice: Moving databases.  I could write a book on this problem alone and I’m not a full-time database person.

Environments

One issue I see is that companies grow and they build environments by hand.  This is a bad thing to do.  There are a lot of tools available to replicate servers and stand up a system automatically.  What inevitably happens is that the development, QA, staging and production environments get out of sync.  Sometimes shortcuts are taken for development and QA environments and that can cause software to perform differently that in production.  This guarantees that deployments will go poorly.  Configure environments automatically.  Refresh your environments at regular intervals.  Companies I have worked for don’t do this enough and it always causes deployment issues.  If you are able to built a web-farm with the click of a button, then you can perform this task for any environment.  By guaranteeing each environment is identical to production (except on a smaller scale), then you can find environment specific bugs early in the development phase and ensure that your software will perform as expected when it is deployed to your production environment.

Databases need to be synchronized as well.  There are tools to sync the database structure.  This task needs to be automated as much as possible.  If your development database can be synced up once a week, then you’ll be able to purge any bad data that has occurred during the week.  Developers need to alter their work-flow to account for this process.  If there are database structure changes (tables, views, functions, stored procedures, etc.) then they need to be checked into version control just like code and the automated process needs to pickup these changes and apply them after the base database is synced down.

Why spend the time to automate this process?  If your company doesn’t automate this step, you’ll end up with a database that has sat un-refreshed for years.  It might have the right changes, it might not.  The database instance becomes the wild west.  It will also become full of test data that causes your development processes to slow down.  Many developer hours will be wasted trying to “fix” an issue caused by a bad database change that was not properly rolled back.  Imagine a database where the constraints are out of sync.  Once the software is working on the development database, it will probably fail in QA.  At that point, it’s more wasted troubleshooting time.  If your QA database is out of sync?  Yes, your developers start fixing environment related issues all the way up the line until the software is deployed and crashes on the production system.  Now the development process is expensive.

Other Sources You Should Read

Educate yourself on deployment techniques early in the software design phase.  Design your software to be easy and safe to deploy.  If you can head off the beast before it becomes a nightmare, you can save yourself a lot of time and money.  Amazon has designed their system around microservices.  Their philosophy is to keep each software package small.  This makes it quick and easy to deploy.  Amazon deploys continuously at a rate that averages more than one deployment per second (50 million per year):

http://www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html

Facebook uses PHP, but they have designed and built a compiler to improve the efficiency of their software by a significant margin.  Then they deploy a 1.5 gigabyte package using BitTorrent.  Facebook does daily deployments using this technique:

https://arstechnica.com/information-technology/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/

I stumbled across this blogger who used to work for GitHub.  He has a lengthy but detailed blog post describing how to make deployments boring.  I would recommend all developers read this article and begin to understand the process of deploying software:

https://zachholman.com/posts/deploying-software

Finally…

Believe it or not, your deployment process is the largest factor determining your customer experience.  If your deployments require you to shut down your system in the wee-hours of the morning to avoid the system-wide outage from affecting customers, then you’ll find it difficult to fix bugs that might affect only a hand-full of customers.  If you can smoothly deploy a version of your software in the middle of the day, you can fix a minor bug and run the deployment process without your customers being affected at all.  Ultimately, there will be bugs.  How quickly you can fix the bugs and how smoothly you get that fix deployed will determine the customer experience.

 

 

 

Creating POCOs in .Net Core 2.0

Summary

I’ve shown how to generate POCOs (Plain Old C# Objects) using the scaffold tool for .Net Core 1 in an earlier post.  Now I’m going to show how to do it in Visual Studio 2017 with Core 2.0.

Install NuGet Packages

First, you’ll need to install the right NuGet Packages.  I prefer to use the command line because I’ve been doing this so long that my fingers type the command without me thinking about it.  If you’re not comfortable with the command line NuGet window, you can use the NuGet Package Manager Settings window under the project you want to create your POCOs in.  If you want, you can copy the commands here and paste them into the NuGet Package Manager Console window.  Follow these instructions:

  1. Create a .Net Core 2.0 library project in Visual Studio 2017.
  2. Type or copy and paste the following NuGet commands into the Nuget Package Manager Console window:
install-package Microsoft.EntityFrameworkCore.SqlServer
install-package Microsoft.EntityFrameworkCore.Tools
install-package Microsoft.EntityFrameworkCore.Tools.DotNet

If you open up your NuGet Dependencies treeview, you should see the following:

Execute the Scaffold Command

In the same package manager console window use the following command to generate your POCOs:

Scaffold-DbContext "Data Source=YOURSQLINSTANCE;Initial Catalog=DATABASENAME;Integrated Security=True" Microsoft.EntityFrameworkCore.SqlServer -OutputDir POCODirectory

You’ll need to update the datasource and initial catalog to point to your database.  If the command executes without error, then you’ll see a directory named “POCODirectory” that contains cs files for each table in the database you just converted.  There will also be a context that contains all the model builder entity mappings.  You can use this file “as-is” or you can split the mappings into individual files.

My process consists of generating these files in a temporary project, followed by copying each table POCO that I want to use in my project.  Then I copy the model builder mappings for each table that I use in my project.

What This Does not Cover

Any views, stored procedures or functions that you want to access with Entity Framework will not show up with this tool.  You’ll still need to create the result POCO for views, stored procedures and functions by hand (or find a custom tool).  Using EF with stored procedures is not recommended.  Anyone who has to deal with legacy code and legacy database will run into a situation where they will need to interface with an existing stored procedure.

 

Using Scripts

Summary

In this post I’m going to show how you can improve developer productivity by steering developers to use scripts where it makes sense.

Setting up IIS

As a back-end developer, I spend a lot of time standing up and configuring new APIs.  One of the tools I use to reduce the amount of man-hours it takes me to get an API up and running is PowerShell.  Personally, the world “PowerShell” makes my skin crawl.  Why?  Because it’s a scripting language that has a syntax that feels like something built by Dr. Frankenstein.  To get beyond my lack of memorizing each and every syntax nuance of PowerShell, I use a lot of Google searches.  Fortunately, after several years of use, I’ve become familiar with some of the capabilities of PowerShell and I can save a lot of time when I create IIS sites.

Now you’re probably wondering where I save my time, since the script has to be written and the site only needs to be setup once.  The time saving comes when I have to change something minor or I have to establish the site on another environment.  In the case of another environment, I can change the path name or url to match the destination environment and run my script to create all the pieces necessary to run my API.

Before I get into the script, I’m going to go through the steps to create an IIS site for WebApi for .Net Core 2.0.

Step 1: Setup the Application Pool.

  • Open IIS and navigate to the Application Pool node.
  • Right-click and add.
  • Give your app pool a name that matches your site, so you can identify it quickly.  This will save you troubleshooting time.
  • For .Net Core, you need to set the .Net Framework Version to “No Managed Code”

Step 2: Setup IIS site.

  • Right-click on the “Sites” node and “Add Web Site”
  • I usually name my site the same as the URL or at least the sub-domain of the URL so I can find it quick.  Again, this name is not used by the system, it is only used when I have to troubleshoot and saving time troubleshooting is the number one priority.
  • Set the path to point to the root of your publish directory (make sure you have done a local publish from Visual Studio before performing this step).
  • Type in the host name.  This is the URL of your site.  If you are just testing locally, you can make up a URL that you’ll need to add to the Hosts file.
  • Select the Application Pool that you created earlier.

Step 3: Optional, setup Hosts file.  Use this step if you are setting up a local website for testing purposes only.

  • Navigate to C:\Windows\System32\drivers\etc
  • Edit “Hosts” file.  You might have to edit with Administrator rights.
  • Add your URL to the hosts file: “127.0.0.1       MyDotNetWebApi.com”

Now try to visualize performing this process for each environment that your company uses.  For me, that comes out to be about half a dozen environments.  In addition to this, each developer that will need your API setup on their PC will need to configure this.  Here’s where the time-saving comes in.  Create the PowerShell script first, and test the script.  Never create the site by hand.  Then use the script for each environment.  Provide the script for other developers to setup their own local copy.  This can be accomplished by posting the script on a wiki page or checking the script into your version control system with the code.

Here’s what an example PowerShell script would look like:

# if you get an error when executing this script, comment the line below to exclude the WebAdministration module
Import-Module WebAdministration

#setup all IIS sites here
$iisAppList = 
    "MyDotNetWebApi,MyDotNetWebApi.franksurl.com,c:\myapicodedirectory,", # use "v4.0" for non-core apps
    "testsite2,testsite2.franksurl.com,c:\temp,v4.0"
    


# setup the app pools and main iis websites
foreach ($appItem in $iisAppList)
{
    $temp = $appItem.split(',')
    
    $iisAppName = $temp[0]
    $iisUrl = $temp[1]
    $iisDirectoryPath = $temp[2]
    $dotNetVersion = $temp[3]
    
    #navigate to the app pools root
    cd IIS:\AppPools\

    if (!(Test-Path $iisAppName -pathType container))
    {
        #create the app pool
        $appPool = New-Item $iisAppName
        $appPool | Set-ItemProperty -Name "managedRuntimeVersion" -Value $dotNetVersion
    }
    
    
    #navigate to the sites root
    cd IIS:\Sites\
    
    if (!(Test-Path $iisAppName -pathType container))
    {
        #create the site
        $iisApp = New-Item $iisAppName -bindings @{protocol="http";bindingInformation=":80:" + $iisUrl} -physicalPath $iisDirectoryPath
        $iisApp | Set-ItemProperty -Name "applicationPool" -Value $iisAppName
        
        Write-Host $iisAppName "completed."
    }
}

c:

You can change the sites listed in the list of sites at the top of the script.  The app pool is setup first, followed by the IIS web site.  Each section will test to see if the app pool or site is already setup (in which is skips).  So you can run the PowerShell script again without causing errors.  Keep the script in a safe location, then you can add to the list and re-run the PowerShell script.  If you need to recreate your environment, you can create all sites with one script.

If you delete all your IIS sites and app pools you might run into the following error:

New-Item : Index was outside the bounds of the array.

To fix this “issue” create a temporary web site in IIS (just use a dummy name like “test”).  Run the script, then delete the dummy site and it’s app pool.  The error is caused by a bug where IIS is trying to create a new site ID.

Setting a Directory to an Application

There are time when you need to convert a directory in your website into it’s own application.  To do this in IIS, you would perform the following steps:

  • Expand the website node
  • Right-click on the directory that will be converted and select “Convert to Application”
  • Click “OK”

To perform this operation automatically in a script, add the following code after creating your IIS sites above (just before the “c:” line of code):

$iisAppList = 
    "MyDotNetWebApi,MyAppDirectory,MyAppDirectory,MyDotNetWebApi\MyAppDirectory,"

foreach ($appItem in $iisAppList)
{
    $temp = $appItem.split(',')

    $iisSiteName = $temp[0]
    $iisAppName = $temp[1]
    $iisPoolName = $temp[2]
    $iisPath = $temp[3]
    $dotNetVersion = $temp[4]

    cd IIS:\AppPools\

    if (!(Test-Path $iisPoolName -pathType container))
    {
        #create the app pool
        $appPool = New-Item $iisPoolName
        $appPool | Set-ItemProperty -Name "managedRuntimeVersion" -Value $dotNetVersion
    }

    cd IIS:\Sites\
    
    # remove and re-apply any IIS applications
    if (Get-WebApplication -Site $iisSiteName -Name $iisAppName)
    {
        Remove-WebApplication -Site $iisSiteName -Name $iisAppName
    }

    ConvertTo-WebApplication -PSPath $iisPath -ApplicationPool $iisPoolName
}

Now add any applications to the list.  The first parameter is the name of the IIS site.  The second parameter is the application name.  The third parameter is the pool name (this script will create a new pool for the application).  The fourth parameter is the path to the folder.  The last parameter is the .Net version (use v4.0 if this application is not a .Net Core project).

For the above script to run, you’ll need to create a blank directory called: C:\myapicodedirectory\MyAppDirectory

Now execute the script and notice that the MyAppDirectory has been turned into an application:

You can add as many applications to each IIS website as you need by adding to the list.

What the code above does is it creates an application pool first (if it doesn’t exist already).  Then it removes the application from the site followed by converting a directory to an application for a specific site.  This script can also be executed multiple times without causing duplicates or errors.

If you run into problems executing your script, you might have to run under an Administrator.  I usually startup powershell in Administrator mode.  Then I navigate to the directory containing the script.  Last, I execute the script.  This allows me to see any errors in the console window.  If you right-click on the ps1 file and run with powershell, your script could fail and exit before you can read the error message.

Feel free to copy the scripts from above and build your own automated installation scripts.

 

XML Serialization

Summary

In this post I’m going to demonstrate the proper way to serialize XML and setup unit tests using xUnit and .Net Core.  I will also be using Visual Studio 2017.

Generating XML

JSON is rapidly taking over as the data encoding standard of choice.  Unfortunately, government agencies are decades behind the technology curve and XML is going to be around for a long time to come.  One of the largest industries industries still using XML for a majority of their data transfer encoding is the medical industry.  Documents required by meaningful use are mostly encoded in XML.  I’m not going to jump into the gory details of generating a CCD.  Instead, I’m going to keep this really simple.

First, I’m going to show a method of generating XML that I’ve seen many times.  Usually coded by a programmer with little or no formal education in Computer Science.  Sometimes programmers just take a short-cut because it appears to be the simplest way to get the product out the door.  So I’ll show the technique and then I’ll explain why it turns out that this is a very poor way of designing an XML generator.

Let’s say for instance we wanted to generate XML representing a house.  First we’ll define the house as a record that can contain square footage.  That will be the only data point assigned to the house record (I mentioned this was going to be simple right).  Inside of the house record will be lists of walls and lists of roofs (assume a house could have two or more roofs like a tri-level configuration).  Next, I’m going to make a list of windows for the walls.  The window block will have a “Type” that is a free-form string input and the roof block will also have a “Type” that is a free-form string.  That is the whole definition.

public class House
{
  public List Walls = new List();
  public List Roofs = new List();
  public int Size { get; set; }
}

public class Wall
{
  public List Windows { get; set; }
}

public class Window
{
  public string Type { get; set; }
}

public class Roof
{
  public string Type { get; set; }
}

The “easy” way to create XML from this is to use the StringBuilder and just build XML tags around the data in your structure.  Here’s a sample of the possible code that a programmer might use:

public class House
{
  public List<Wall> Walls = new List<Wall>();
  public List<Roof> Roofs = new List<Roof>();
  public int Size { get; set; }

  public string Serialize()
  {
    var @out = new StringBuilder();

    @out.Append("<?xml version=\"1.0\" encoding=\"utf-8\"?>");
    @out.Append("<House xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">");

    foreach (var wall in Walls)
    {
      wall.Serialize(ref @out);
    }

    foreach (var roof in Roofs)
    {
      roof.Serialize(ref @out);
    }

    @out.Append("<size>");
    @out.Append(Size);
    @out.Append("</size>");

    @out.Append("</House>");

    return @out.ToString();
  }
}

public class Wall
{
  public List<Window> Windows { get; set; }

  public void Serialize(ref StringBuilder @out)
  {
    if (Windows == null || Windows.Count == 0)
    {
      @out.Append("<wall />");
      return;
    }

    @out.Append("<wall>");
    foreach (var window in Windows)
    {
      window.Serialize(ref @out);
    }
    @out.Append("</wall>");
  }
}

public class Window
{
  public string Type { get; set; }

  public void Serialize(ref StringBuilder @out)
  {
    @out.Append("<window>");
    @out.Append("<Type>");
    @out.Append(Type);
    @out.Append("</Type>");
    @out.Append("</window>");
  }
}

public class Roof
{
  public string Type { get; set; }

  public void Serialize(ref StringBuilder @out)
  {
    @out.Append("<roof>");
    @out.Append("<Type>");
    @out.Append(Type);
    @out.Append("</Type>");
    @out.Append("</roof>");
  }
}

The example I’ve given is a rather clean example.  I have seen XML generated with much uglier code.  This is the manual method of serializing XML.  One almost obvious weakness is that the output produced is a straight line of XML, which is not human-readable.  In order to allow human readable XML output to be produced with an on/off switch, extra logic will need to be incorporated that would append the newline and add tabs for indents.  Another problem with this method is that it contains a lot of code that is unnecessary.  One typo and the XML is incorrect.  Future editing is hazardous because tags might not match up if code is inserted in the middle and care is not taken to test such conditions.  Unit testing something like this is an absolute must.

The easy method is to use the XML serializer.  To produce the correct output, it is sometimes necessary to add attributes to properties in objects to be serialized.  Here is the object definition that produces the same output:

public class House
{
  [XmlElement(ElementName = "wall")]
  public List Walls = new List();

  [XmlElement(ElementName = "roof")]
  public List Roofs = new List();

  [XmlElement(ElementName = "size")]
  public int Size { get; set; }
}

public class Wall
{
  [XmlElement(ElementName = "window")]
  public List Windows { get; set; }

  public bool ShouldSerializenullable()
  {
    return Windows == null;
  }
}

public class Window
{
  public string Type { get; set; }
}

public class Roof
{
  public string Type { get; set; }
}

In order to serialize the above objects into XML, you use the XMLSerializer object:

public static class CreateXMLData
{
  public static string Serialize(this House house)
  {
    var xmlSerializer = new XmlSerializer(typeof(House));

    var settings = new XmlWriterSettings
    {
      NewLineHandling = NewLineHandling.Entitize,
      IndentChars = "\t",
      Indent = true
    };

    using (var stringWriter = new Utf8StringWriter())
    {
      var writer = XmlWriter.Create(stringWriter, settings);
      xmlSerializer.Serialize(writer, house);

      return stringWriter.GetStringBuilder().ToString();
    }
  }
}

You’ll also need to create a Utf8StringWriter Class:

public class Utf8StringWriter : StringWriter
{
  public override Encoding Encoding
  {
    get { return Encoding.UTF8; }
  }
}

Unit Testing

I would recommend unit testing each section of your XML.  Test with sections empty as well as containing one or more items.  You want to make sure you capture instances of null lists or empty items that should not generate XML output.  If there are any special attributes, make sure that the XML generated matches the specification.  For my unit testing, I stripped newlines and tabs to compare with a sample XML file that is stored in my unit test project.  As a first-attempt, I created a helper for my unit tests:

public static class XmlResultCompare
{
  public static string ReadExpectedXml(string expectedDataFile)
  {
    var assembly = Assembly.GetExecutingAssembly();
    using (var stream = assembly.GetManifestResourceStream(expectedDataFile))
    {
      using (var reader = new StreamReader(stream))
      {
        return reader.ReadToEnd().RemoveWhiteSpace();
      }
    }
  }

  public static string RemoveWhiteSpace(this string s)
  {
    s = s.Replace("\t", "");
    s = s.Replace("\r", "");
    s = s.Replace("\n", "");
  return s;
  }
}

If you look carefully, I ‘m compiling my xml test data right into the unit test dll.  Why am I doing that?  The company that I work for as well as most serious companies use continuous integration tools such as a build server.  The problem with a build server is that your files might not make it to the same directory location on the build server that they are on your PC.  To ensure that the test files are there, compile them into the dll and reference them from the namespace using Assembly.GetExecutingAssembly().  To make this work, you’ll have to mark your xml test files as an Embedded Resource (click on the xml file and change the Build Action property to Embedded Resource).  To access the files, which are contained in a virtual directory called “TestData”, you’ll need to use the name space, the virtual directory and the full file name:

XMLCreatorTests.TestData.XMLHouseOneWallOneWindow.xml

Now for a sample unit test:

[Fact]
public void TestOneWallNoWindow()
{
  // one wall, no windows
  var house = new House { Size = 2000 };
  house.Walls.Add(new Wall());

  Assert.Equal(XmlResultCompare.ReadExpectedXml("XMLCreatorTests.TestData.XMLHouseOneWallNoWindow.xml"), house.Serialize().RemoveWhiteSpace());
}

Notice how I filled in the house object with the size and added one wall.  The ReadExpectedXml() method will remove whitespaces automatically, so it’s important to remove them off the serialized version of house in order to match.

Where to Get the Code

As always you can go to my GitHub account and download the sample application (click here).  I would recommend downloading the application and modifying it as a test to see how all the piece work.  Add a unit test to see if you can match your expected xml with the xml serializer.

 

 

 

Environment Configuration

Introduction

We’ve all modified and saved variables in the web.config and app.config files of our applications.  I’m going to discuss issues that occur when a company maintains multiple environments.

Environments

When I talk about environments, I’m talking about entire systems that mimic the production system of a company.  An example is a development environment, a quality environment and sometimes a staging environment.  There may be other environments as well for purposes such as load testing and regression testing.  Typically a development or QA environment is smaller than the production environment, but all the web, database, etc. servers should be represented.

The purpose of each environment is to provide the ability to test your software during different stages of development.  For this reason it’s best to think of software development like a factory with a pipe-line of steps that must be performed before delivering the final product.  The software developer might create software on his/her local machine and then check in changes to a version control system (like Git or TFS).  Then the software is installed on the development environment using a continuous integration/deployment system like TeamCity, BuildMaster or Jenkins.  Once the software is installed on the development system, then developer-level integration testing can begin.  Does it work with other parts of the software?  Does it work with real servers (like IIS)?

Once a feature is complete and it works in the development environment, it can be scheduled to be quality checked or QA’d.  The feature branch can be merged and deployed to the QA environment and tested by QA personnel and regression scripts can be run.  If the software is QA complete, then it can be merged up to the next level.  Load testing is common in larger environments.  A determination must be made if the new feature causes a higher load on the system than the previous version.  This may be accomplished through the use of an environment to contain the previous version of code.  Baseline load numbers may also be maintained to be used as a comparison for future load testing.  It’s always best to keep a previous system because hardware can be upgraded and that will change the baseline numbers.

Once everything checks out, then your software is ready for deployment.  I’m not going to go into deployment techniques in this blog post (there are so many possibilities).  I’ll leave that for another post.  For this post, I’m going to dig into configuration of environments.

Setting up the physical or virtual hardware should be done in a fashion that mimics the production system as close as possible.  If the development, QA and production systems are all different, then it’ll be difficult to get software working for each environment.  This is a waste of resources and needs to be avoided at all costs.  Most systems today use virtual servers on a host machine, making it easy to setup an identical environment.  The goal in a virtual environment is to automate the setup and tear-down of servers so environments can be created fresh when needed.

The Web.Config and App.Config

One issue with multiple environments is the configuration parameters in the web.config and app.config files (and now the appsettings.json for .net core apps).  There are other config files that might exist, like the nlog.config for nlog setup, but they all fall into the same category: They are specific for each environment.

There are many solutions available.  BuildMaster provides variable injection.  Basically, a template web.config is setup in BuildMaster and variables contain the data to be inserted for each deployment type.  There is a capability in Visual Studio called web.config transformation.  Several web.config files can be setup in addition to a common web.config to be merged when different configurations are built.  Powershell can be used to replace text in a web.config.  One powershell script per environment, or a web.config can have sections commented and powershell can remove the comment lines for the section that applies for the environment being deployed to.

These are all creative ways of dealing with this problem, but they all lack a level of security that is needed if your company has isolation between your developers and the production environment.  At some point your production system becomes so large that you’ll need an IT department to maintain the live system.  That’s the point where you need to keep developers from tweaking production code whenever they feel the need.  Some restrictions must come into play.  One restriction is the passwords used by the databases.  Production databases should be accessible for only those in charge of the production system.

What this means is that the config parameters cannot be checked into your version control system.  They’ll need to be applied when the software is deployed.  One method would be to insert parameters in for the values that will be applied at the end of deployment.  Then a list of variables and their respective parameters can exist on a server in each environment.  That list would be specific to the environment and can be used by powershell to replace variables in the web.config file by the deployment server.

There is a system called Zookeeper that can contain all of your configuration parameters and centrally accessed.  The downside to this is that you’ll need a cluster of servers to provide the throughput for a large system, plus another potential central point of failure.  The complexity of your environment just increased for the sole purpose of keeping track of configuration parameters.

Local Environments

Local environments are a special case problem.  Each software developer should have their own local environment to use for their development efforts.  By creating a miniature environment on the software developer’s desktop/laptop system, the developer has full flexibility to test code without the worry of destroying data that is used by other developers.  An automated method of refreshing the local environment is necessary.  Next comes the issue of configuring the local environment.  How do you handle local configuration files?

Solution 1: Manually alter local config files.

The developer needs to exclude the config files from check-in otherwise the local changes could end up in the version control software for one developer.

Solution 2: Manually alter local config files and exclude from check-in by adding to ignore settings.

If there are any automatic updates to the config by Visual Studio, those changes will not be checked into your version control software.

Solution 3: Create config files with replaceable parameters and include a script to replace them as a post build operation.

Same issue as solution 1, the files could get checked-in to version control and that would wipe out the changes.

Solution 4: Move all config settings to a different config file.  Only maintain automatic settings in web.config and app.config.

This is a cleaner solution because the local config file can be excluded from version control.  A location (like a wiki page or the version control) must contain a copy of the local config file with the parameters to be replaced.  Then a script must be run the first time to populate the parameters or the developer must manually replace the parameters to match their system.  The web.config and app.config would only contain automatic parameters.

One issue with this solution is that it would be difficult to convert legacy code to use this method.  A search and replace for each parameter must be performed, or you can override the ConfigurationManager object and implement the AppSettings method to store the values in the custom config file (ditto for the database connection settings).

Why Use Config Files at all?

One of the questions I see a lot is the question about using the config file in the first place.  Variables used by your software can be stored in a database.  If your configuration data is specific to the application, then a table can be setup in your database to store application config data.  This can be cached and read all at once by your application.  There is only one catch: What about your database connection strings?  At least one connection string will be needed and probably a connection string to your caching system (if you’re using Redis or Memcached or something).  That connection string will be used to read the application config variables, like other database connection strings, etc.  Keep this method in mind if you hope to keep your configuration rats-nest to something manageable.  Each environment would read it’s config variables from the database that belongs to it.

The issues listed in the local environment are still valid.  In other words, each environment would need it’s own connection and the production environment (and possibly other environments) connection would need to be kept secret from developers.

Custom Solutions

There are other solutions to the config replacement problem.  The web.config can be deserialized by a custom program and then each config parameter value can be replaced by a list of replacement values using the key (for appSettings as an example).  Then you can maintain a localized config settings file for each environment.  This technique has the added bonus of not breaking if someone checks in their local copy into the version control software.

Here is a code snippet to parse the appSettings section of the web.config file:

// read the web.config file
var document = new XmlDocument();
document.Load(fileName);

var result = document.SelectNodes("//configuration/appSettings");

// note: you might also need to check for "//configuration/location/appSettings"

foreach (XmlNode childNodes in result)
{
    foreach (XmlNode keyItem in childNodes)
    {
        if (keyItem.Attributes != null)
        {
            if (keyItem.Attributes["key"] != null && keyItem.Attributes["value"] != null)
            {
                var key = keyItem.Attributes["key"].Value;

                // replace your value here
                keyItem.Attributes["value"].Value = LookupValue(key);
            }
        }
    }
}

// save back the web.config file
document.Save(fileName);

The same technique can be used for .json files to read, deserialize, alter then save.

Conclusion

Most aspects of system deployment have industry-wide standards.  Managing configuration information isn’t one of them.  I am not sure why a standard has not been established.  Microsoft provides a method of setting up multiple environment configuration files, but it does not solve the issue of securing a production environment from the developers.  It also does not work when the developer executes their program directly from Visual Studio.  The transform operation only occurs when a project is published.

BuildMaster has a good technique.  That product can be configured to deploy different config files for each environment.  This product does not deploy to the developer’s local environment, so that is still an issue.  The final issue with this method is that automatically added parameters will not show up and must be added by hand.

Jenkins doesn’t seem to have any capability to handle config files (if you know of a plug-in for Jenkins, leave a comment, I’d like to try it out).  Jenkins leaves the dev-ops person with the task of setting up scripts to modify the config parameters during deployment/promotion operations.

If you have other ideas that work better than these, please leave a comment or drop me an email.  I’d like to hear about your experience.

 

Ransomware

My wife and I recently took a vacation in Nevada so we could hike trails in parks surrounding Las Vegas.  To get a sense of what we did on vacation you can check out the pictures on my hiking blog by clicking here (I lined up the post dates to be the date when we visited each park).  This article is not about the fun we had on our vacation.  This is about a ransomware attack that occurred while I was on vacation.

I’m a software engineer.  That means that I know how to hack a computer and I also know how to protect myself from hackers.  But I’m not immune to making mistakes.  The mistake I made was that I have passwords that I haven’t changed in forever.  Like all large companies, Apple had been hacked about a year ago and their entire database of passwords were obtained and distributed throughout the hacking community (yeah, “community”, that’s what they are).  The Apple hack involved their cloud service, which I didn’t pay much attention to, because I don’t use their cloud storage.  What I didn’t pay attention to was that their cloud services play a part in some of the iPhone and iPad security.

If you clicked the link above and started looking through my pictures, you’ll notice that the first place we visited was Death Valley.  I discovered that it was supposed to be a record 120 degrees that day and I wanted to know what it would feel like to stand outside in 120 degree weather!  Yeah, I’m crazy like that.  As it turned out it got up to 123 degrees according to the thermometer built into the Jeep that we rented.  Dry heat or not, 123 degrees was HOT!

I use my Canon Rebel xti for the photos that I post on my hiking blog, but I also take pictures with my iPhone in order to get something on my Facebook account as well as get a few panoramas.  When we arrived at the sand dunes it was just before noon local time or before 3PM Eastern time.  My photos indicate they were taken around 2:07, but the camera time is off by more than an hour and the camera is on Eastern time.

I took a few pictures of the dunes and then I pulled out my iPhone and was about to take pictures when I got a warning that my phone is locked and I need to send an email to get instructions on how to pay for access to my iPhone.  So I used my usual pin number to unlock my iPhone and it worked correctly.  I was annoyed, but I moved on.  I thought it was some “clever” advertisement or spam notification.

When we returned to the resort, I sat down on the couch and started to use my iPad to do some reading.  My iPad had the same message on the front and a pin number was set up.  Unfortunately, for me, I never set a pin number on my iPad because I only use it at home to surf the web, read a book and maybe play a game.  What the hackers did was setup a pin number on my iPad.  What an annoyance.  Ransomware on any of my devices is no more worrisome than a rainy day.  It’s more of an irritation than anything.  I have off-site backups for my desktop machine and I know how to restore the entire machine in a few hours.  Hacking my iPad while I was on vacation (and the second day of vacation to boot), was really annoying.  Primarily because I don’t have access to all of my adapters, computers and tools.  My wife has a tiny laptop that we use for minor stuff.  It has a grand total of 64 gigabytes of storage space.  So she installed iTunes on it (with some difficulty) and we restored the iPad and got everything back to normal.

After returning from vacation, I cleaned out all of my spam emails for the past couple of weeks and discovered these emails:

It appears that someone manually logged into my iCloud account, enabled lost mode and put in a message, for both of my devices.  The iPhone was first, which was locked by pin number, so they couldn’t change that.  The iPad, however, was not setup with a pin number, so they went in and set their own.  Or so I assumed, when I saw it was asking for a 6-digit pin.  Apparently, the pin that shows up is the pin that is set when the device is first setup.  My pin was not the same for the iPad as I used on my iPhone (which was what I tried when I first saw it appear).

My wife and I changed the password on our iCloud accounts when we were at the resort and she set the two-factor on for the iCloud.  Of course, that is a bit of a problem if I lose my iPhone, but it prevents anyone from hacking my iCloud account.

One thing that makes me wonder… how was Apple storing my password?  Are the passwords stored in clear text?  Are they encrypted with an algorithm that allows the password to be decrypted?  That seems foolish.  Maybe Apple was using something like a weak MD5 hash and the hacked database was decrypted using a brute force method like this: 25-GPU cluster cracks every standard Windows password in <6 hours.  I know that the correct password was used to login to the iCloud using a browser.  The notification sent to my email account proves it.

How to Protect Yourself

The first level of protection that I have is that I assume I will get hacked.  From that assumption, I have plans in place to reduce any damages that can occur.  First, I have an off-site backup system that backs up everything I can’t replace on my desktop computer.  Pictures, documents, etc.  They are all backed up.  Some of my software is on GitHub so I don’t worry about backing up my local repository directory.  I have backup systems in place on my blogs and my website.

Next in line is the two-factor system.  This is probably one of the best ways to protect yourself.  Use your phone as your second factor and protect your phone from theft.  If someone steals your phone, they probably don’t have your passwords.  If someone has your passwords, they don’t have your phone.  If you see messages arrive at your phone with a second factor pin number, then you need to change the password for the account that requested it.

Next, you should turn on notifications of when someone logs into your account (if the feature is available).  Like the notifications about my iCloud being used in the emails above, I can see that someone accessed my account when I wasn’t around.  If someone is silently logging into your account, a lot more damage can be done before you figure out what is going on.

If you’re using email as your second factor, you need to protect your email account as though it was made of gold.  Change your email password often, in case the provider has been hacked.  Your email account is most likely used as a method of resetting your password on other sites.  So if a hacker gets into your email account, they can guess at other sites that you might have accounts and reset your password to get in.  I have my own urls and hosts so I create and maintain my own email system.  If my email system gets hacked it’s 100% my fault.

Disable unused accounts.  If you’re like me, you have hundreds of web accounts for stores and sites that you signed up for.  Hey, they were “free” right?  Unfortunately, your passwords are out there and any one site can get hacked.  You can’t keep track of which sites got hacked last week.  Keep a list of sites that you have accounts on.  Review that list at least annually and delete accounts on sites you no longer use.  If the site doesn’t allow you to delete your account, then go in and change the password to something that is completely random and long (like 20 characters or more depending on what the site will allow).

Use a long password if possible.  Just because the minimum password is 8 characters doesn’t mean you need to come up with an 8 character password.  If sites allow you to use 30 characters, then make something up.  There is an excellent XKCD comic demonstrating password strengths: click here.  For companies providing websites with security, I would recommend you allow at least 256 characters for passwords.  Allow your customers to create a really strong password.  Storage is cheap.  Stolen information is expensive.

Don’t use the same password for everything.  That’s a bit obvious, but people do crazy things all the time.  The problem with one password for all is that any site that gets hacked means a hacker can get into everything you have access to.  It also means you need to change all of your passwords.  If you use different passwords or some sort of theme (don’t make the theme obvious), then you can change your most important passwords often and the passwords to useless sites less often.

Last but not Least…

Don’t pay the ransom!  If you pay money, what happens if you don’t get the unlock key?  What happens if you unlock your computer and it gets re-ransomed again?  Plan for this contingency now.  Paying ransom only funds a criminal organization.  The more money they make performing these “services” the more likely they will continue the practice.  I like to think of these people as telemarketers, if nobody paid, then they would all be out of work.  Since telemarketing continues to this day, someone, somewhere is buying something.  Don’t keep the ransomware cycle going.

 

Documentation

Why is there No Documentation?

I’m surprised at the number of developers who don’t create any documentation.  There are ways to self-document code and there are packages to add automatic help to an API to document the services.  Unfortunately, that’s not enough.  I’ve heard all the arguments:

  • It chews up programming time.
  • It rapidly becomes obsolete.
  • The code should explain itself.
  • As a programmer, I’m more valuable if I keep the knowledge to myself.

Let me explain what I mean by documentation.  What should you document outside of your code?  Every time you create a program you usually create a design for it.  If it’s just drawings on a napkin, then it needs to go someplace where other developers can access it if they need to work on your software.  Configuration parameters need documentation.  Not a lot, just enough to describe what the purpose is.  Installation?  If there is some trick to making your program work in a production, QA or staging environment, then you should document it.  Where is the source code located?  Is there a deployment package?  Was there a debate on the use of one technology over another?

So what happens when you have no documentation?  First, you need to find the source code.  Hopefully it’s properly named and resides in the current repository.  Otherwise, you may be forced to dig through directories on old servers or in some cases the source might not be available.  If the source is not available your options are limited: De-compile, rewrite or work with what is built.  Looking at source code written by a programmer that no longer exists at your company is a common occurrence (did that programmer think not documenting made him/her valuable?).  Usually such code is tightly coupled, contains poorly named methods and variables with no comments.  So here are the arguments of why you should do documentation:

  • Reverse engineering software chews up programming time.
  • Most undocumented code is not written to be self-explanatory.
  • Attempting to figure out why a programmer wrote a program the way he/she did can be difficult and sometimes impossible.
  • Programmers come and go no matter how little documentation exists.

It’s easy to go overboard with documentation.  This can be another trap.  Try to keep your documentation to just the facts.  Don’t write long-winded literature.  Keep it technical.  Start with lists of notes.  Expand as needed.  Remove any obsolete documentation.

Getting Your Documentation Started

The first step to getting your documentation started is to decide on a place to store it.  The best option is a wiki of some sort.  I prefer Confluence or GitHub.  They both have clean formatting and are easy to edit and drop in pictures/screenshots.

So you have a wiki setup and it’s empty.  Next, create some subjects.  If you have several software projects in progress, start with those.  Create a subject for each project and load up all the design specifications.  If your development team is performing a retrospective, type it directly into the wiki.  If there is a debate or committee meeting to discuss a change or some nuance with the software, type it into the wiki.  They can just be raw historical notes.

Next, add “documentation” as a story point to your project, or add it to each story.  This should be a mandatory process.  Make documentation part of the development process.  Developers can just add a few notes, or they can dig in and do a brain-dump.  Somewhere down the road a developer not involved in the project will need to add an enhancement or fix a bug.  That developer will have a starting point.

Another way to seed the wiki is to create subjects for each section of your existing legacy code and just do a dump of notes in each section.  Simple information off the top of everyone’s head is good enough.  The wiki can be reworked at a later date to make things more organized.  Divide an conquer.  If a developer has fixed a bug in a subsystem that nobody understands, that developer should input their knowledge into the wiki.  This will save a lot of time when another developer has to fix a bug in that system and it will prevent your developers from becoming siloed.

You Have Documentation – Now What?

One of the purposes of your technical documentation is to train new people.  This is something that is overlooked a lot.  When a new developer is hired, that person can get up to speed faster if they can just browse a wiki full of technical notes.  With this purpose in mind, you should expand your wiki to include instructions on how to setup a desktop/laptop for a development environment.  You can also add educational material to get a developer up to speed.  This doesn’t mean that you need to type in subjects on how to write an MVC application.  You should be able to link to articles that can be used by new developers to hone their skills.  By doing this, you can keep a new person busy while you coordinate your day to day tasks, instead of being tied down to an all-day training session to get that person up to speed.

Your documentation should also contain a subject on your company development standards.  What frameworks are acceptable?  What processes must be followed before introducing new technologies to the system?  Coding standards?  Languages that can be used?  Maybe a statement of goals that have been laid down.  What is the intended architecture of your system?  If your company has committees to decide what the goal of the department is, then maybe the meeting minutes would be handy.

Who is Responsible for Your Documentation

Everyone should be responsible.  Everyone should participate.  Make sure you keep backups in case something goes wrong or someone makes a mistake.  Most wiki software is capable of tracking revisions.  Documentation should be treated like version control.  Don’t delete anything!  If you want to hide subjects that have been deprecated, then create a subject at the bottom for all your obsolete projects.  When a project is deprecated, move the wiki subject to that folder.  Someday, someone might ask a question about a feature that used to exist.  You can dig up the old subject and present what used to exist if necessary.  This is especially handy if you deprecated a subsystem that was a problem-child due to it’s design.  If someone wants to create that same type of mess, they can read what experience was learned from the obsolete subsystem.  The answer can be: “We tried that, it didn’t work and here’s why…” or it can be: “We did that before and here’s how it used to work…”

If you make the task of documenting part of the work that is performed, developers can add documentation as software is created.  Developers can modify documentation when bugs are fixed or enhancements are made.  Developers can remove or archive documentation when software is torn down or replaced.  The documentation should be part of the software maintenance cycle.  This will prevent the documentation from getting out of sync with your software.

 

Three Tier Architecture

There is a lot of information on the Internet about the three-tier architecture, three-tier structure and other names for the concept of breaking a program into tiers.  The current system design paradigm is to break your software into APIs and the three-tier architecture still applies.  I’m going to try and explain the three-tier architecture from the point of practicality and explain the benefits of following this structure.  But first, I have to explain what happens when a hand-full of inexperienced programmers run in and build a system from the ground up…

Bad System Design

Every seasoned developer knows what I’m talking about.  It’s the organically grown, not very well planned system.  Programming is easy.  Once a person learns the basic syntax, the world is their oyster!  Until the system get really big.  There’s a tipping point where tightly-coupled monolithic systems become progressively more difficult and time consuming to enhance.  Many systems that I have worked on were well beyond that point when I started working on them.  Let me show a diagram of what I’m talking about:

Technically, there is no firm division between front-end and back-end code.  The HTML/JavaScript is usually embedded in back-end code and there is typically business code scattered between the HTML.  Sometimes systems like this contain a lot of stored procedures which does nothing more than marry you to the database that was first used to build the application.  Burying business code in stored procedures also has the additional financial burden of ensuring you are performing your processing on the product with the most expensive licensing fees.  When your company grows, you’ll be forced to purchase more and more licenses for the database in order to keep up.  This proceeds exponentially and you’ll come to a point where any tiny change in a stored procedure, function or table causes your user traffic to overwhelm the hardware that your database runs on.

I often-times joke about how I would like to build a time machine for no other reason than to go back in time, sit down with the developers of the system I am working on and tell them what they should do.  To head it off before it becomes a very large mess.  I suspect that this craziness occurs because companies are started by non-programmers and they hook-up with some young and energetic programmer with little real-world experience who can make magic happen.  Any programmer can get the seed started.  Poor programming practices and bad system design doesn’t show up right away.  A startup company might only have a few hundred users at first.  Hardware is cheap, SQL server licenses seem reasonable, everything is working as expected.  I also suspect that those developers move on when the system becomes too difficult to manager.  They move on to another “new” project that they can start bad.  Either that, or they learn their lesson and the next company they work at is lucky to get a programmer with knowledge of how not to write a program.

Once the software gets to the point that I’ve described, then it takes programmers like me to fix it.  Sometimes it takes a lot of programmers with my kind of knowledge to fix it.  Fixing a system like this is expensive and takes time.  It’s a lot like repairing a jet while in flight.  The jet must stay flying while you shut down one engine and upgrade it to a new one.  Sound like fun?  Sometimes it is, usually it’s not.

Three-Tier Basics

In case you’re completely unfamiliar with the three-tier system, here is the simplified diagram:

It looks simple, but the design is a bit nuanced.  First of all, the HTML, JavaScript, front-end frameworks, etc. must be contained in the front-end box.  You need isolation from the back-end or middle-tier.  The whole purpose of the front-end is to handle the presentation or human interface part of your system.  The back-end or middle-tier is all the business logic.  It all needs to be contained in this section.  It must be loosely coupled and unit tested.  Preferably with an IOC container like AutoFac.  The database must be nothing more than a container for your data.  Reduce special features as much as possible.  Your caching system is also located in this layer.

The connection between the front-end and back-end is usually an API connection using REST.  You can pass data back and forth between these two layers using JSON or XML or just perform “get”, “post”, “delete” and “put” operations.  If you treat your front-end as a system that communicates with another system called your back-end, you’ll have a successful implementation.  You’ll still have hardware challenges (like network bandwidth and server instances), but those can be solved much quicker and cheaper than rewriting software.

The connection between the back-end and database has another purpose.  Your goal should be to make sure your back-end is database technology independent as much as possible.  You want the option of switching to a database with cheap licensing costs.  If you work hard up-front, you’ll get a pay-back down the road when your company expands to a respectable size and the database licensing cost starts to look ugly.

What About APIs?

The above diagram looks like a monolithic program at first glance.  If you follow the rules I already laid out, you’ll end up with one large monolithic program.  So there’s one more level of separation you must be aware of.  You need to logically divide your system into independent APIs.  You can split your system into a handful of large APIs or hundreds of smaller APIs.  It’s better to build a lot of smaller APIs, but that can depend on what type of system is being built and how many logical boxes you can divide it into.  Here’s an example of a very simple system divided into APIs:

This is not a typical way to divide your APIs.  Typically, an API can share a database with another API and the front-end can be separate from the API itself.  For now, let’s talk about the advantages of this design as I’ve shown.

  1. Each section of your system is independent.  If a user decides to consume a lot of resources by executing a long-running task, it won’t affect any other section of your system.  You can contain the resource problem.  In the monolithic design, any long-running process will kill the entire system and all users will experience the slow-down.
  2. If one section of your system requires heavy resources, then you can allocate more resources for that one section and leave all other sections the same.  In other words, you can expand one API to be hosted by multiple servers, while other APIs are each on one server.
  3. Deployment is easy.  You only deploy the APIs that are updated.  If your front-end is well isolated, then you can deploy a back-end piece without the need for deployment of your front-end.
  4. Your technology can be mixed.  You can use different database technologies for each section.  You can use different programming languages for each back-end or different frameworks for each front-end.  This also means that you have a means to convert some or all of your system to a Unix hosted system.  A new API can be built using Python or PHP and that API can be hosted on a Linux virtual box.  Notice how the front-end should require no redesign as well as your database.  Just the back-end software for one subsection of your system.

Converting a Legacy System

If you have a legacy system built around the monolithic design pattern, you’re going to want to take steps as soon as possible to get into a three-tier architecture.  You’ll also want to build any new parts using an API design pattern.  Usually it takes multiple iterations to remove the stored procedures and replace the code with decoupled front-end and back-end code.  You’ll probably start with something like this:

In this diagram the database is shared between the new API and the legacy system, which is still just a monolithic program.  Notice how stored procedures are avoided by the API on the right side.  All the business logic must be contained in the back-end so it can be unit tested.  Eventually, you’ll end up with something like this:

Some of your APIs can have their own data while others rely on the main database.  The monolithic section of your system should start to shrink.  The number of stored procedures should shrink.  This system is already easier to maintain than the complete monolithic system.  You’re still saddled with the monolithic section and the gigantic database with stored procedures.  However, you now have sections of your system that is independent and easy to maintain and deploy.  Another possibility is to do this:

In this instance the front-end is consistent.  One framework with common JavaScript can be contained as a single piece of your system.  This is OK because your front-end should not contain any business logic.

The Front-End

I need to explain a little more about the front-end that many programmers are not aware of.  Your system design goal for the front-end is to assume that your company will grow so large that you’re going to have front-end specialists.  These people should be artists who work with HTML, CSS and other front-end languages.  The front-end designer is concerned with usability and aesthetics.  The back-end designer is concerned about accuracy and speed.  These are two different skill sets.  The front-end person should be more of a graphic designer while the back-end person should be a programmer with knowledge of scalability and system performance.  Small companies will hire a programmer to perform both tasks, but a large company must begin to divide their personnel into distinct skill-sets to maximize the quality of their product.

Another overlooked aspect of the front-end is that it is going to become stale.  Somewhere down the road your front-end is going to be ugly compared to the competition.  If your front-end code is nothing more than HTML, CSS, JavaScript and maybe some frameworks, you can change the look and feel of the user interface with minimum disruption.  If you have HTML and JavaScript mixed into your business logic, you’ve got an uphill battle to try and upgrade the look and feel of your system.

The Back-End

When you connect to a database the common and simple method is to use something like ODBC or ADO.  Then SQL statements are sent as strings with parameters to the database directly.  There are many issues with this approach and the current solution is to use an ORM like Entity Framework, NHibernate or even Dapper.  Here’s a list of the advantages of an ORM:

  1. The queries are in LINQ and most errors can be found at compile time.
  2. The context can be easily changed to point to another database technology.
  3. If you including mappings that match your database, you can detect many database problems at compile time, like child to parent relationship issues (attempt to insert a child record with no parent).
  4. An ORM can break dependency with the database and provide an easy method of unit testing.

As I mentioned earlier, you must avoid stored procedures, functions and any other database technology specific features.  Don’t back yourself into a corner because MS SQL server had a feature that made it easy to use as an enhancement.  If your system is built around a set of stored procedures, you’ll be in trouble if you want to switch from MS SQL to MySQL, or from MS SQL to Oracle.

Summary

I’m hoping that this blog post is read by a lot of entry-level programmers.  You might have seen the three-tier architecture mentioned in your school book or on a website and didn’t realize what it was all about.  Many articles get into the technical details of how to implement a three-tier architecture using C# or some other language, glossing over the big picture of “why” it’s done this way.  Be aware that there are also other multi-tier architectures that can be employed.  Which technique you use doesn’t really matter as long as you know why it’s done that way.  When you build a real system, you have to be aware of what the implications of your design are going to be five or ten years from now.  If you’re just going to write some code and put it into production, you’ll run into a brick wall before long.  Keep these techniques in mind when you’re building your first system.  It will pay dividends down the road when you can enhance your software just be modifying a small API and tweak some front-end code.