In this post I’m going to discuss the differences between waterfall and agile methods of programming. Then I’m going to focus more on Kanban and what advantages and disadvantages it has in comparison to Scrum. I will also be discussing the business case for using Kanban and how it can improve the performance of developing software and reducing waste.
Producing software costs money. That is the bottom line. The largest cost to creating software is the cost of man-hours or how much money is paid to the analysts, developers and quality people to create the software. After the software has been written and delivered, then the primary cost becomes the operating costs. Operating costs can break down into licensing fees for database and operating systems as well as hardware or hosting fees. There will also be labor expenses tied up in support such as IT personnel, help-desk and training personnel. I’m going to focus on the cost it takes to create the software. As a seasoned software architect, my job is to analyze and design the software with an eye for reducing the cost of operating the software after it is delivered. That is a subject for another blog post. In this post I’m concerned about the process of creating the software with an eye toward reducing wasted time and money.
Everyone has heard of the waterfall method. It’s the most obvious way to design and build software. Basically a team of software analysts communicate with the customer to determine a list of features required/desired for the software to be built. They will then turn this into a requirements document. Once the requirements document has been agreed upon, a massive design effort is launched. This can take months and for grand projects it can take years. The project is designed down to the smallest detail so that it can be estimated. Estimating the amount of man-hours it takes to perform the programming can be done by analysts or, if the developers are already working for the company performing the work, they can perform the estimation (it’s preferable to obtain estimates from the developers that will actually work on the project because then you can get buy-in). Next, the negotiations start with the customer. The customer will most certainly decide that they can’t afford the “magic wand”1 version of the software that they described and will begin to reduce the features to get the price down to something reasonable.
Once a contract is signed, then the specifications are thrown over the wall (figuratively) to the developers. I’ve seen companies that use GAANT2 charts with critical paths detailing when each feature will be scheduled, but most companies just hand over the spec with the idea that developers can just divide-and-conquer. Then months and probably years are spent developing the software. There are usually demonstrations of what has been created to keep the customer from cancelling the project. There is nothing worse than paying for a product and not hearing anything about the product until a year later when the whole thing is delivered. The customer will want progress reports. Is it on schedule? Hopefully, for the customer’s sake, they paid a flat-rate price for the software and will not have to shell out more money if the developers cannot produce the software by the deadline indicated in the estimates. Otherwise, more than likely, the project will come in over budget and late.
Once the developers have completed their tasks, quality can start testing the whole project. QA should be building test scripts while the developers are creating the code. Otherwise, there should be no QA people on the project until they are about ready to start testing. Once the QA people have completed their tasks and the developers have fixed any bugs found, then it’s time for the release.
This is the waterfall method in a nutshell.
Problems with Waterfall
The biggest problem with the waterfall method is that it is wasteful. Many projects get cancelled before completion and the customer is left with nothing. If the tasks are not scheduled to ensure that sections of the software are workable, then there is no way to cut off development and deliver a working product with fewer features. Also, if a project is cut because of cost overruns, then there is still QA work to be done. Finally, the software analysts and designers must be paid for designing the entire project before any software is built.
Next, there are months where no software is being written. This is the analysis, design and estimation phase. The whole thing must be estimated before an agreement can be signed. That means that time is burning while this whole process is going on and development doesn’t start until all of this work is completed.
Usability is difficult to perform before the software is built. Usability is expensive to fix after the whole project is complete (designed, built and QA’d). The best method of ensuring usability is cheap and effective is to test the usability as the software is being built. This is not something that waterfall can accommodate without difficulty. It would require very detailed scheduling and changes would loop back to a lot of changes in the design that has already been completed. In practice, waterfall does not support effective usability testing.
The Agile Method
The theory behind the agile method is that the software is only roughly specified up front. Then the most critical and important parts of the software are designed and estimated first. Usually only a week or month’s worth of software is designed at a time. Then the design is estimated by the developers who will start work immediately. Once a few pieces of the software (usually called stories) are completed, they are QA’d and then deployed to a limited audience (the customer). The customer will then review what is being demonstrated and normally they get hands-on access to test what has been created. If the piece of software is large enough to be useful, the customer can start using the software. A critique of the software can be fed back to the analysts to have some of the usability problems fixed in a future release. Meanwhile the design team is working on the next specifications or stories to be put in the hopper. At the same time the developers are working on the previously designed and estimated stories. QA is working on the previously finished stories and the quality checked pieces are collected up for the next deployment. This continues like a factory.
The stories need to be produced in the order that the customer demands. So they can put off minor enhancements to the end and have working software as early as possible. If the customer decides that they have spent enough money and the product is good enough, then they can cut off the project and walk away with a product that is usable. This reduces waste.
Benefits over Waterfall
It’s almost obvious what the benefits are:
- Working software is delivered right away.
- There is a short startup time when the designers create the first stories. The developers start working right away instead of waiting months or years to start.
- The customer is more involved in the creation of the product. Instant feed-back about usability problems can help fix the problem before the developers have forgotten what they’ve worked on.
- The customer can cut off the project at any time and walk away with a functioning product.
- The customer, theoretically, could start using their product before it is finished.
- Re-prioritizing features is quick and easy since developers don’t just grab any story at any time. The customer has full control on when features are developed.
Scrum vs. Kanban
There are several methods of running an agile development team, but two are near the top: Kanban and Scrum. Scrum is a method where there is a fixed amount of time that developers will work on a set of stories. This is called the sprint and it may last two, three or four weeks. The sprint is usually a fixed time frame that is repeated throughout the project, and they are usually numbered. For example: The analysts/designers will group stories into sprint 1 until they have filled two weeks worth of work for the team that will work on the software. Then they will start to fill in sprint 2, etc. In a Scrum, there are daily-standup meetings where the “team” discusses the progress since the previous standup. The idea for a standup is that everyone stands and the meeting is limited to reporting progress, issues and blockers. If a one-on-one discussion lasts more than a few minutes, it must be taken off-line because it is wasting the time of the entire team to sort out a problem that can be solved by two people. Scrums provide an environment where the team knows what everyone else is working on for each sprint.
Stories that are completed are immediately passed on to QA during the sprint. The QA personnel are included as part of the “team” and they attend the daily standup meetings as well. At the end of the sprint the stories that are complete and QA’d are deployed and demonstrated to the customer who can give feed-back. Any changes can be put into the next sprint or a future sprint.
Kanban is a bit different. Kanban is broken down like a factory. There are “lanes” where the stories will go when they are moved from the design phase to the deployment phase, like a pipeline. Analysts/designers will put the stories into the backlog lane. Once the stories are in the backlog, developers can immediately pick one story up and work on it. The developer will move the story to their development lane. When the story is complete, it is moved into the “To be QA’d” lane. Then a QA person can pull a story out of that lane and put it in their “QA” lane. After the QA person has completed their task, the story can be placed into the “To be Deployed” lane. When enough stories are in that lane, they can be deployed to be reviewed by the customer. Technically, each story can and should be deployed immediately. This can be accomplished by an automated deployment system.
In the case where QA discovers a bug, the story must be moved back into the developer’s lane. The developer must fix the bug as soon as he/she can and get it back into the “To be QA’d” lane. There can be limits set to each lane to reduce the amount of allowed work in progress or WIP. The WIP count controls the flow of stories going through the pipeline. As you can see, Kanban is setup as a just-in-time delivery system. The customer can literally end a project as soon as they decide they have enough features and the whole assembly line can stop without much waste in the system. Scrum can have up to two-weeks (or three or four depending on the sprint size) worth of waste in the system when a cut-off occurs. Keep in mind that scrum is still extremely efficient compared to waterfall and we are only splitting hairs over how much waste exists if a Kanban or Scrum project is cancelled.
I’m going to focus on potential issues that can occur with Kanban. First, it can be difficult to determine when a feature will be delivered. This happens for items that are sitting in the backlog lane. Items in the backlog can be re-prioritized until they are picked up and work begins. Once work begins, the story must be completed (unless there is some circumstance that warrants the stoppage of work on a particular story). If a story is at the top of the back-log an estimate of when it will get completed can only be determined when it is picked up by a developer. Adjusting the WIP count to be low can make it easier to estimate when the story will go through the pipeline, but assumptions must be made about the timing of existing stories. If the WIP count is high, then developers might have several stories in their lane at one time. Now you’re probably scratching your head and thinking “why would a developer have two or more stories in their lane at a time?” This situation usually happens when there is something that is blocking the completion of a story. Maybe there is a question about how the story will be implemented. The developer is waiting for an analyst to make a decision on how to proceed. In such an instance, it is best for the developer to pickup the next story and start working on that one. Developers are responsible for clearing their lane before picking up another story unless there is something blocking a story. In other words, no non-blocked stories can sit in the developer’s lane.
Straight Kanban assumes that all developers are minimally qualified to work on every task. That is not usually realistic and there are work-arounds for shops that have specialized developers. First, Kanban can be setup with multiple backlog lanes. Let’s pretend that your shop has back-end developers and front-end developers3. As stories are created by analysts/designers, they can be divided and organized by front-end work vs. back-end work and placed in the appropriate backlog lane. Back-end developers will pull stories from the back-end backlog lane and so on. Of course there is the scheduling problem where now back-end developers must finish an API before a front-end programmer can consume the API for their interface. This can be mitigated by breaking off a story that creates the API “shell” program with dummy data that responds to requests from the front-end. Then the front-end developer can start consuming the API data before the back-end developer has completed the API. Both lanes must be monitored to ensure that back-end programming lines up with front-end programming. Otherwise, there could be situations where the front-end programmers have no API to consume and the software to be QA’d and deployed is not usable. For Scrum, the back-end programming and front-end can be staggered in different sprints to ensure that the APIs are completed before the front-end programmers start programming. This technique can also be used in Kanban by starting the back-end group ahead of the front-end group of programmers.
As you can tell there is usually no daily standup for Kanban. There’s no need. Each individual developer can meet with the person they need to in order to complete their work. Meetings can still be held for kick-offs or for retrospectives. I would recommend a retrospective for every project completed, Scrum or Kanban.
One last feature of Kanban that is more accommodating is the idea of throwing more resources at the problem. When there is a team working on a problem there are multiple communication paths to consider. Each team member must be made aware of what everyone else is working on. In Kanban the idea is to design the stories to be independent of each other. If a developer can focus on one story at a time, then adding a new developer to the mix is easy. The new developer will just pickup a story and start working on it4.
Pitfalls to Watch For
Here’s a big pitfall with the agile method that must be headed off early on: In an agile workshop the entire project is not estimated up front. The customer wants to know what it would cost for the whole product. In fact, most customers want a cafeteria style estimate of what every feature will cost so they can pick and choose what they want. What the customer does not want is to pay for a month’s worth of work and then wonder how many more months it will take to get to a viable product. It would be nice to know ahead of time how long it will take and how much it will cost. To accommodate this, agile shops must be able to give a rough estimate of the entire project without designing the entire project. In fact the product does not have to be designed down to the nth degree to get an estimate. Also, an estimate on a product that is designed to the tiniest feature is not more accurate than an over-all rough estimate. Confidence level is something that should always be taken into consideration in software design. As projects get larger, the confidence level drops lower and lower. The confidence level does not increase just because the product is designed down to the detail. Don’t believe me? Search for any large government software project that was cancelled or over-budget an you’ll discover that these projects missed their marks by 100 to 200% or more. They are always underestimated. Those projects are designed to the intimate detail. The problem with software design is that there are always so many unknowns.
Create a rough design. List the features and give a rough estimate for each feature. Add some time to features that are really gray in definition. Tighten your estimates for features that you know can be built in “x” amount of time. This estimate can be used for a contract to “not exceed…” “x” amount of months or “x” amount of resources. When the project is about to run up against the end, the customer must be made aware of the short-fall (if there is any). Normally a shortfall will occur because a customer thinks up a feature that they need while the project is in progress. This is additional work that can be inserted into the work-flow and preempt one of the lower priority features or the customer can agree to an extension of the project. Sometimes a feature takes longer than the estimate and the customer should be notified of each feature that went over budget.
Customers can also be A.D.D. when it comes to deciding which stories to stuff in the backlog. The backlog queue can churn like a cauldron of stories causing the scheduling of features to be delivered to be unknown. If the customer is OK with the unknown delivery time, then the churn does not effect the development staff. However, if stories are pulled out of the work lanes, then problems can start. Shelving unfinished code can be hazardous. Especially if a story is shelved for a month and then put back in play. By that time the un-shelved code my not work with the current code-base and must be reworked, causing the estimate for the story to go long.
I would recommend a wiki for the project. The wiki should contain the design specifications and changes as they are made. If you are using a product such as Confluence and Jira, you can use the forum feature to add questions to a story and follow up answers. This becomes your documentation for the software. If you add developers, they can read through the notes on what is going on and get a good idea of why the software was built the way it was built. This documentation should be maintained as long as the software is in production. Future development teams could use this documentation to see what ideas went into the original design. When an enhancement is added, the notes for the enhancement should be appended to this documentation for future developers to refer to. This documentation can also provide witness testimony for any disputes that occur between the customer and the entity developing the software.
- The term “Magic Wand” refers to the idea of: What would the customer want if they had a “Magic Wand” and could have every feature right now for free.
- GANNT charts and critical path methodology is used in physical construction projects. Many people try to visualize software development as a “construction” project, like building a house. Unfortunately, the methodology does not fit software design because every software project is like inventing something new, where building a house is so methodical that there are books full of estimates for each task to be performed. GANNT charts are used for home construction, assembly line theory fits software development more accurately.
- A typical shop with a large number of developers will contain experts in database design, front-end advanced developers, entry-level front-end developers, back-end developers (which are usually API experts) and other specialized developers. In such a situation scheduling can get a bit dicey, but the same work-arounds apply.
- In practice this technique should always work. In the real-world there are pieces of the puzzle that are dependent on other pieces that are already completed. A new developer will need some ramp-up time to get into the flow of what is being built. This can also slow down existing developers who must explain what is going on.