Estimation of Outsourced Projects

  1. Posted on June 6, 2007 7:59:AM by Steve McConnell to 10x Software Development
  2. Technique, estimation, Management

A question we sometimes hear from our clients is, "My company does outsource software development for other companies. Is there anything special about estimating in that context?" There actually are some distinctive aspects to estimating in the context of preparing a bid or price quote, and I don't discuss that in my book Software Estimation.

Estimation in a Time & Materials Context

Creating estimates to support time and materials bids (i.e., charging by the hour), is only barely a special case because the very structure of T&M implies some variability in the outcome, same as my recommendations for estimating in-house development work. The only real difference if you can even call it a difference is that you have to make doubly sure that you're setting expectations clearly: "This is an estimate. We can't know the outcome with 100% certainty. Actual results will depend on exact details of what you end up requiring and how different issues get prioritized throughout the project," and so on.

Estimation in a Fixed-Price Context

In contrast, estimation in a fixed-price context is very much a special case. If your estimate causes you to bid too high, you won't get the work. If it causes you to bid too low, you will lose money. Both of these are undesirable outcomes! In other circumstances I usually find myself recommending that people back away from really elaborate estimation approaches because there's so much inherent variability in software projects that the accuracy of your estimates is inherently limited, and you reach the point of diminishing estimation accuracy after you've put in even a little bit of effort. But a fixed price environment, at proposal time, is one of the few circumstances I've run encountered in which an elaborate estimation approach is warranted. And so my first recommendation is, If your business depends on creating fixed price bids, focus on estimation skills as a core competency and treat estimation work as a business-critical function. That means, read Software Estimation, take my company's estimation class, and read other people's estimation books.

My second recommendation is similar to my general recommendation that you separate the "estimate" from the "target." In a fixed price bid context, separate estimation from pricing. The estimate informs the price you'll charge, of course, but there isn't any necessary relationship between the two. You can price a bid at the "unlikely" end of the estimation range if it's really important to you to win the work, and you're willing to lose money on it. Or you can price it way above the estimation range if you think you have an approach that allows you to perform the work at low cost to you and that delivers a higher value to the client.

We've seen lots of companies wrap themselves around the axle when the sales staff insists on lowering the "estimate" to get the work, when really what needs to be lowered is the price. This creates confusion throughout the project. Giving everyone permission to keep estimates and prices separate increases accountability on the sales side (they have to own up to the fact that they're pricing something on the low end of the estimation range and get buy in to do that), and it improves planning on the dev side -- if there's a big gap between the price and the estimate, the project needs to be treated as a higher risk project than if there isn't a large gap. When estimation and pricing are merged into one concept and called "estimation" (even though it isn't really estimation), the project planners can lose the important risk information that arises from the relationship between the price and the estimates.

Try not to do the "commitment/pricing" estimate until later in the cone of uncertainty. Of course this is the holy grail, but most companies can't do this with any regularity because they feel that the competitive pressures require them to submit bids in the wider part of the cone.

Bid smaller amounts of work when you can, i.e., be more iterative. One of the great benefits of iterative development is the ability to generate project-level data on early iterations that can be used to estimate later iterations with really good accuracy. The companies we've worked with have settled in on 3 iterations as the number needed to calibrate a project team's productivity. Interestingly enough, it doesn't seem to matter whether the iterations are 1 week or 1 month or longer -- it still takes 3 iterations.

Consider creating two-phase bids when you can. You can call the first phase "preliminary work", "exploratory phase," "proof of concept phase," "design phase", "Phase 1", etc. The purpose of this phase is to attack all the sources of high variability feeding into the estimates and ultimately deliver a bid for the second part of the project after the cone has been narrowed considerably. We've seen many companies use this approach successfully, although I can't think of any companies we've seen that have been able to use it for the majority of the work they bid on. Again, competitive pressures seem to lead to their using this approach only selectively.

Two phase bids can be structured either more "waterfallish" or more "agile". The description above assumes a more linear development approach in which you're trying to get most or all of the requirements defined up front and then bid the whole project. In a more agile approach, you can treat "Phase 1" as an actual design-build-deliver cycle, but structure it into 3 iterations so that you can get good project-level calibration data that you can then use as the basis for bidding the remainder of the project.

Collect historical data on your estimates at proposal time vs. the eventual outcomes so that you can build your own cone of uncertainty. The better records you keep about what materials fed into your estimate, the more meaningful your cone will be. For example, you might have really specific requirements for one bid and pretty vague requirements for another. In one sense, if they're both "proposal time" estimates, you might treat them similarly. But if one was supported by significantly more detail in the requirements that implies you're at a different location in the cone, and you'd want to account for that.

Non-Estimation Recommendations

On this particular topic, several of the most powerful recommendations aren't specifically about estimation; they're about project control.

Go highly iterative as early as you can, regardless of whether the bid is structured into one or two phases. Even if you're working to a single-stage bid, there's value to getting project-level calibration data sooner rather than later. If you discover 10% of the way into the project that you've under bid it by a factor of 2, you can go back to the customer sooner and reset their expectations, you can give the customer options that you still have time to act on, you can implement functionality in strict priority order, you can identify the project as a high risk project and manage it accordingly, etc. But if you don't have the project level data that tells you your initial estimates were way off, you'll just run the project as "business as usual", which is really the last thing you want to do.

Document assumptions at the contract level, spell them out in as much detail as you can, and then *contain* them. If you build a house, your building contractor might let you specify the kitchen cabinets, but there will be a line item in the contract budget for cabinets. If you end up choosing cabinets that are more expensive, you pay the difference. You typically would have line items for all kinds of things: lighting, landscaping, carpet, flooring, countertops, etc. The areas that are more certain (e.g., roofing, siding, foundation, plumbing) are simply specified. In software projects we also typically have areas that we can specify in detail and other areas that we don't know enough about at contract time to specify in detail. So in software contracts you can include clauses like, "The exact work required in the XYZ module has been budgeted at 40 staff hours. If work on XYZ exceeds its budget, the contract price will be increased correspondingly." I'm not an attorney so I am not recommending this as specific contract language, but hopefully this gives you a general idea about the general kind of clause you would ask your attorney to include in a contract.

Manage your set of projects/bids as an investment portfolio, accepting that some will "win" and some will "lose." From a theoretical point of view, if you're estimating early in the cone there just isn't a good answer to improving the accuracy of your estimates on a project-by-project basis. The fact is, your estimates will be off to varying degrees, and when you happen to get one that's pretty accurate it will be a matter of luck, not skill, because of the inherent limits of the Cone. On the other hand, assuming there isn't any bias in the early-in-the-cone estimates (which can be a huge assumption), you can essentially punt on the question of project-by-project profitability and instead focus on portfolio level profitability. Solving the problem of estimating accurately in the wide part of the cone for an individual project isn't even theoretically solvable. But solving the problem of estimating a collection of projects in the wide part of the cone IS solveable. The key to solving that problem is rooting out any systemic bias in those estimates so that the error tendency is neutral. Then with that set of neutral estimates you simply increase each estimate by the amount you'd like your profit margin to be. If you want it to be 10%, you bid 10% higher than your neutral estimate. This will result in your actual project cost coming in higher than some of your estimates and lower than others, but on balance, assuming no systemic bias, you should make a 10% profit on your portfolio of projects.

Of course this requires that you have several projects in your portfolio, and that there aren't just one or two huge projects whose estimation errors could drown out whatever error was contributed by the smaller projects, and that you can afford to take a loss on some percentage of your projects. And those are big assumptions that might not be true in your specific case.

Bottom Line on Estimating in a Fixed-Price Bidding Context

The bottom line on this particular question is that it isn't possible to solve this particular problem purely using estimation practices. You have to change when you're estimating (later in the Cone), or what you're estimating (e.g., portfolios vs. individual projects), or how many times you estimate (e.g., two-phase bids). And project-control responses (as opposed to estimation responses) and even contract-level responses will probably turn out to be at least as useful as estimation responses.

Resources

Jerry Deville said:

June 6, 2007 9:09:AM

One of the big assumptions to document is the budget for anticipated change requests.  Industry average is 1.5% - 4% change per month.  However, you should use your historical data to estimate and *contain* the change budget.  When using fix price contracts, you need to clearly document the penalties for exceeding the budget (i.e. extra payments).  You should also include incentives for not exceeded the budget (i.e. spit the difference for any unused budget).

links for 2007-06-07 « John’s musing said:

June 7, 2007 5:25:PM

Pingback from  links for 2007-06-07 « John’s musing

Maksym Shostak said:

June 13, 2007 1:40:PM

As the citizen of country with mostly outsourcing companies, Ukraine, I would like to add some assumptions.

Some of our companies (or decision-makers there) are not fucusing on the quality of the outsourced software they create or maintain.

When "your companies" do estimation in a time & materials context -  "our companies" sometimes just follow the objective to have more money adding more people to the project...

But, developers here want to innovate, improve their producivity and skills, experience in developing healthy software... not just making money.

So, it would be great if "your companies" with outsoucing their software will also bring the attitude to the job of constructing software.

P.S. So it would be easy for them to estimate.

John Rusk said:

July 10, 2007 2:45:AM

Steve,

You wrote that the key to early-cone estimation is "rooting out any systemic bias in those estimates so that the error tendency is neutral".  I'd be interested in your thoughts on one particular source of systemic bias, a source which is not under the control of the company doing the estimation: in a competive situation, the price quoted influences whether or not you get the work.  So, while you may be able to produce neutral estimates for the set of all projects you _bid_ for, I suggest that you cannot produce neutral estimates for the set of all projects that you _win_.  

Imagine 5 suppliers competing for the same piece of work.  Even if all suppliers estimate neutrally across their respective portfolios, for this particular job some will have over estimated and some will have underestimated.  The customer will usually consider price when awarding the contract, and thus the contract is somewhat more likely to go one of the companies which estimated low.  

The end result is that market forces systematically bias the _winning_ bid, even when the set of _all_ bids is not biased at all.

That's my theory.  What do you think?

Steve McConnell said:

July 10, 2007 11:55:AM

To John Rusk's comment ....

I don't think this is pointing to a dynamic that's any different from what I described in my original post. I think it's providing detail that elaborates on my original post. There are lots of forces that apply pressure both downward and upward on prices in an outsourced context. John describes one of the downward forces.

I would still be careful to differentiate between estimation and pricing. John makes a good point that estimates and prices are correlated, and so when bids with lower prices are chosen, that probably (but not necesssarily) implies that those bids were also based on lower estimates.

To John's specific question, does this make it impossible to root out bias in the winning estimates, I think the logic is interesting and appealing but reality says that that CAN'T be true. If it were true -- i.e., happening consistently for a company -- then that company would go bankrupt because they would engage in a steady stream of projects that were underestimated and underbid and lost money. Undoubtedly that has happened to some companies, but there are other companies that have been doing software outsourcing for decades without going out of business, and so there have to be other dynamics in play.

John Rusk said:

July 11, 2007 3:33:AM

Steve,

Yes, I think there are other dynamics at play, but I wonder if the other dynamics are more about providing _other_ sources of revenue to the companies in question, rather than actually making their "early-cone" wins profitable.  In my experience, companies in this sector do not derive all their revenue from early-cone contracts.  In many cases they derive much of their revenue from three other sources:

(a) Change requests and follow-on projects, following on from the initial early-cone project. These are not subject to the market forces I mention (and their estimates are better because they can use real data from the initial project, as you suggest in your book).  The company may stuggle financially on the initial project, but do just fine on the follow-on work.

(b) Competitive bids with commitments later in the cone, as you suggest above (e.g. two-phase bids)

(c) Projects which never went through the competivie bid process, and so were never subject to the market force I mentioned.

I believe it is possible that the market force I suggested really does exist, but that the companies earn the bulk of their profits from these three other sources.  That would explain why they don't go bust.

An interesting question, which I have no way of answering, would be to find out what percentage of outsourcing profits are derived from early-cone commitments, and what percentage are derived from these three other sources?  (Profits, not revenue, that is).

I note that selecting the lowest bid has been blamed for creating unsustainable markets in traditional (construction) engineering.  In fact, price-based selection has actually been banned for many years in federally-funded US construction projects.  (I blogged about this here www.agilekiwi.com/beware_lowest_price.htm )

To me, the experience of the construction industry offers some indication that price-based selection can indeed skew the winning bids downwards, to the detriment of an entire industry.  If it is documented in traditional construction, perhaps it possible in IT too.  Perhaps it hurts all outsourcers, but they cope by getting enough business from other sources...

Post a Comment:

 
 

Steve McConnell

Steve McConnell is CEO and Chief Software Engineer at Construx Software where he consults to a broad range of industries, teaches seminars, and oversees Construx’s software development practices. In 1998, readers of Software Development magazine named Steve one of the three most influential people in the software industry along with Bill Gates and Linus Torvalds.

Steve is the author of Software Estimation: Demystifying the Black Art (2006), Code Complete (1993, 2004), Rapid Development (1996), Software Project Survival Guide (1998), and Professional Software Development (2004). His books twice won Software Development magazine's Jolt Excellence award for outstanding software development book of the year.

Steve has served as Editor in Chief of IEEE Software magazine, on the Panel of Experts of the SWEBOK project, and as Chair of the IEEE Computer Society’s Professional Practices Committee.

Steve received a Bachelor’s degree from Whitman College, graduating Magna Cum Laude, Phi Beta Kappa, and earned a Master’s degree in software engineering from Seattle University.
Contact Steve