Culture of Failure
We’ve all worked at a place where the deadlines are tight, the feature set large, and pressure is everywhere. In this compressed environment, failure is the norm. Quality is the first thing to go on the relentless march to the next release date. Given the needs of the business, how can a company change from this culture of failure to one of success?
Software development is a tricky business. We’ve all heard (nay lived) the Iron Triangle and the pressures it creates: given the three choices of fast, cheap, and good, only two are attainable. If you then consider scope (aka features), it complicates the picture:
Each of these three constraints affect quality; it is the challenge of company to find the right balance, oftentimes as a result of knowing your customer. If your product is for the enterprise, how often do you need a full product release? Will your existing customers upgrade once a quarter? Once a year? How often can your sales team successfully sell a new version?
Cultivating Failure
No one wants to fail. Events, left unchecked, will conspire against us, failure is the
most likely outcome. Success comes to those who will work for it with conscious effort, intelligent thought, and careful planning.
Let’s Wing it
Let’s face it, many developers hate process. Those who have been around long enough, however, have learned to embrace it. A process adds predictability to the product life cycle, establishes expectations, and enforces constraints. Not all processes are created equal: each business must perform a certain amount of self discovery and analysis to find out just how much process their business and their customers demand.
I have a teaser on my resume that states that I offer “experience building appropriate quality software” that also applies to process. Your company needs just enough process — an appropriate level — to perform at a high level. Even a micro-isv needs some process:
- How much design do you do up front?
- When do you freeze your requirements?
- What do you do when a customer reports an issue?
- How do you test?
Your answers to these (and other) questions define your process.
Roadmap? We don’t need direction!
Your product has a lifespan. It evolves over time, sometimes into something that barely resembles your initial vision. Your customers and their requirements change as well; to continue to make a product that is successful you’ll need a vision, and a roadmap. A roadmap allows you to plan for now and more importantly, for the future. Each release of a product now can contain enabling technology for the next version, the next new feature, the next big thing.
Scheduling
One thing to remember when you’re scheduling your project is that those little boxes labeled resources actually refer to people. This is important, so it bears repeating: resources are people - not cogs, not replaceable parts but people. Treat them with care and respect. Most people will gladly work extra hours during crunch time; respect and reward this extra effort — it doesn’t come cheaply. Never assume that as the employer, your workers owe you after hours work. Remember that they have a life outside of the workplace. There is no quicker way to build ill-will among your team than insist they work more hours than you pay them for on a regular basis.
Leverage experience
So you’ve built your team and have a few good experienced developers. They can mentor your junior staff, much like a veteran baseball player helps develop a rookie. These experienced developers are a goldmine of knowledge that transcends mere coding. They’ve labored in the trenches, they know how to work, what works, and more importantly, what won’t work.
When their scoping estimates blow out your schedule, ask them why. Sometimes it’s the gut feeling — the unknown unknowns — that adds time to the estimates. It’s the years of experience that allows them to know that here, in this feature, there be dragons.
When planning your next release, make sure you ask your teams (dev, qa, support) what parts of the code or the product need work. Ask them what they think needs to be done. Prioritize and address some of these issues in every release. Nothing is worse than knowing where the broken bits are and not being given time to fix them.
Listen to them. They’ve been there. That’s why you hired them, right?
Quality is for the other guys
Testing software is an art form; some would say it’s a black art. Good QA/QE people are hard to find and if you are lucky enough to have some on your staff, be prepared to do whatever is necessary to keep them. Your customers and developers will thank you.
You’ll want to automate as many test cases as you possibly can and run them often. Run them every build if possible. Every failure must be resolved, either by fixing the test, the code, or the environment. One place I worked had a test that failed intermittently only on the official build machines; never on a developer’s workstation. It’s been failing for over a year now, with no resolution in sight. It’s almost a rite of passage that each new hire gets a crack at fixing it. So far, none have succeeded.
Give your QA department the resources that it needs. Hardware is cheap, happy customers are not, and angry customers can be very expensive indeed! Make sure you test with legacy data. Make sure you have datasets that have gone through the upgrade transformations so that you know your current product can handle the buggy data created by your upgrade path. Your QA department should have this data readily available from previous testing cycles; there should always be at least one installation of your product that has been upgraded from a previous version. You do test your upgrades, don’t you?
It’s Not Our Fault
Ok, so you failed. Now what? How do you prevent it from happening again? Run a postmortem analysis of the project. Encourage everyone to be candid and listen in a nonjudgmental manner. Address the issues now, before the next release cycle is too far along to withstand change. Some of the issues may bring about disruptive change, such as increased process.
You’re ahead of schedule?
Ahh, success! Learn to recognize it, embrace it, reward it! Make sure your teams see it as well. There’s no shame in self-promotion, so make sure the whole company is aware of your team’s success. Now what? Well, do what you did in the failure case — have a postmortem but this time focus on what went right. Strengthen the things that went well, and address your weaknesses again. This is the iterative success model.
Filed under: Editorial
Great article Jeff. I’ve been thinking about this lately since you mentioned it and I have one more thing to add to the list and that is the internal battleground that I perceive as existing within a company over the triangle. For my example I’m going to deviate slightly from your model and use the Triangle of Features, Time, and Quality.
The internal battle I’m referring to is waged between groups in a company and even between management and their staff. It is my opinion that developers leans towards Features and Quality. Most want to develop good code by taking the correct steps towards fixing a broken system and then implement interesting new features on top of a better off code base. However if we move over to Sales you’ll see that they’d rather have more Features in less Time. These two forces will always be in conflict with each other and that’s where we come up with the schedule. I don’t argue that schedules are a bad thing, heck if developers don’t stop to say it’s in a state we can ship and be happy we’d never ship anything because there’s always more work (so long as there’s a roadmap.. but I’ll get to this in a second.) The big problem in this conflict is in which direction is it skewed and on which side is the management in charge of engineering. When the engineering managers are on the side of Quality, they will fight tooth and nail against any unreasonable release dates and rather push them out than insult a costumer with poor quality. When the managers are fighting for Sales, they have now become the person who looks for a way to sweep something under the rug and to ‘bring things in’. Often you hear the phrase “Is it really going to take that long,” “Do we really need to do that,” “Customers won’t do that,” “That isn’t a typical use case,” or even worse, an admission of failure, “We’ll fix that in a service release.” We you hear these phrases from your managers and you’re in engineering, you’ve just been set up for failure. Management in engineering should /never/ be given incentives to ship on time above having a product of the highest quality and they should be willing to fight for this.
Now on a tangent, I’ve thought a bit lately about what a Roadmap means. I’m sure this has come to many already but a new developer must pause to think about why it is called a Roadmap. A Roadmap isn’t just a list of things you need to do or plan to do. It is that list of things in a two dimensional list, your map, and their distance from each other represents how related they are in the code and feature set. Now, often companies elicit feedback from their customers which will prioritize these features. If you were to superimpose this onto your map by using circles and those with a higher priority are darker and have a larger radius, you can start to see what customers are calling for and what all the features in the vicinity affected by the demands are. The goal of product management, as I see it now given what I view a roadmap as, is to play the traveling salesman. They should look at the map and start around the highest priority item and then figure out what is the shortest distance they need to travel to get all of these things implemented? By minimizing distance he will save development time because this helps to group like features together which means less context switching on the part of development. There is no later ‘coming’ back to figure out how something worked again. This reduction of context switching will also lead to better quality since all the gotchas a developer discovered while in the code the first time will still be fresh in their mind as they move to the next related feature. Another nice thing about this view of the Roadmap is you can see the hot areas and when you implement all the things in a hot area you haven’t just released a feature, you’ve released a feature set which will more serve the customer in what they were trying to to. So perhaps an example of what I’m saying here is a feature to send messages to customers is in high demand. A related feature around this is that you have a notification system in your product but it really isn’t that good, maybe bug ridden or just flakey. These two would be on the roadmap and somewhat close together and while customers are not calling for fixing your current notification system the overlap of the demand for messaging will dragged notification in as a necessity.
Anyways, this is my brain dump for now. I think I might have exceeded my quota for thinking today.
~ Rod
Thanks for the comment, Rod. You make some good points but I need to point out that while Sales wants more Features in Less TIme but at a high Quality; the only way that cam be accomplished is by adding Resources.
I’ve also worked with many developers who just want a paycheck, and only care as much as they have to. These people are the bane of good engineers everywhere.
The Bullet Hole methodology you propose for creating a roadmap is solved, at most companies by Drunkard’s Walk because the Travelling Salesman is just too damn NP-Hard!.
Hi Jeff,
Great article! I tend to use a pyramid, with the axis of features (scope), time, and quality. I tend to lump the current team, their abilities and their tools together and just call it the team. The team is able to slice a plane through the pyramid. The area they can cover is their combined abilities (tools, talent, and cohesiveness).
Management can tilt the slice anyway they want, but the area of the intersection cannot be changed. But that’s just the way I think of the work pyramid.
Good ideas!
Take care,
mike
[...] on August 30th, 2007 by Jeff So you’re discovered that the company you work for has a culture of failure, and there’s no light at the end of the tunnel. You’ve finally realized that [...]
ugtd3bcynrai1sdp