I have read Agile Estimation and planning http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp... but I'm not at a high enough level in my career for software planning.
Little tangent.
It's unfortunate that so much of the literature on relative estimation/story points/velocity/planning poker ends up intertwined in agile-development-consultantese, so sometimes reading some of these things, you have to take it with a serious grain of salt and weed out all the BS and dogma to get to the useful and important bits. The important bits there are really pretty simple: * Estimate in terms of points, not days, and estimate tasks relative to one another * Use planning poker (or something like it) within the team to get more accurate estimates * Empirically measure how many points you do in a given period of time to come up with your "velocity". To do that, you have to be pretty honest about things being "done done" before you move on to the next task; otherwise it's too easy to fudge and your numbers are worthless. "Oh, yeah, we did 10 points of work, well, except for writing the tests or doing the styling . . ."
Remember that velocity is NOT a productivity measure, it'll change over time, and it'll change if the team members change or if other external factors intervene, like an increase in support requests or something. So this technique kind of only works if your organization isn't already dysfunctional: as soon as velocity is seen as a productivity measurement, you're pretty screwed.
That's pretty much it. The relative estimates let you prioritize work appropriately (i.e. "I'd rather have these five 1-point stories than this one 5-point story, so we'll do those first"), and the velocity lets you track how fast you're actually going and about when you'll be done with a given chunk of work, so you can adjust plans as needed.
Note that relative estimation doesn't work nearly so well for large-scale estimation, or for greenfield development where you don't know what you're doing yet. For large-scale planning, my company will generally just give a SWAG in terms of points (we think this feature is 100 points) to give us at least some initial idea of expected relative sizes of things, then we'll compare that initial SWAG to the actual points as we break things out into more bite-sized stories that we can estimate more accurately. If we feel like we're half way through the feature and we've already done 70 points of work, that's a signal that we need to up our estimate for the remainder of the work. Steve McConnell's book is good as well, though honestly we don't really do much in the way of confidence-based estimates at this point. My experience is that out of every 10 projects we do, 8 or 9 will be within 20% of our initial SWAG and 1 or 2 will blow out to 2 or 3x the initial estimate. Of course, we never know which ones will blow out, we just know something will. In other words, we don't bother with confidence intervals at the individual feature level, we just do it at the macro level. So if a team has a velocity of 10 and we have 26 weeks to deliver a release, giving us a hypothetical total capacity of 260 points, we'll "commit" to maybe 2/3 of that. So we say "Okay, this first 170 points we're pretty sure we can get done. Anything after that will be a bonus."
http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp...
But the simple version of the problem, in my experience, is related to the 80/20 rule. No matter how many times a developer goes through the process of estimation -> development -> slippage, whether on a big scope or small, we will inevitably estimate the very next project as if only the clean 80% exists and the 20% will magically disappear this time.
Let's back up. In my experience the 80/20 rule (or Pareto Principle) applied to software estimation means that you will spend 80% of your time writing 20% of your features. Usually this has to do with various technical blocks, edge cases, framework incompatibilities -- the list goes on and varies wildly from one feature or application to the next.
You will spend 20% of your time working on 80% of the feature(s). This is the stuff you have done before: basic CRUD, ORM-fu, REST actions -- these are the clean, easily understood software tasks that you see right in front of you as the happy yellow brick road to feature completion.
And no matter how many times we wash-rinse-and-repeat this cycle, the next time a PM or colleague asks us "how long this will take," we will stare at that beautiful, pristine part of the path that we understand and know and have done before, and conveniently elide the fact that our development process has never been that clean and there is a nasty 20% (what? my code won't connect to the staging server because of some weird certificate problem? what do you mean my gems are incompatible? I can't get 'make' to build this gcc lib!! IE is doing what?? Why do our servers treat HTTP basic auth in such a non-standard way?) just waiting to jump out, muss your hair, laugh in your face and dance like a hyperactive monkey all over your keyboard.
* Mike Cohn resources were the most helpful to me:
- free presentations around estimating and planning (eg: http://www.mountaingoatsoftware.com/presentations/131-agile-... and others on the same site)
- the "Agile Estimating and Planning" book (http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp...)
- Mike Cohn "Agile Estimating and Planning" course (2 days) was worth taking
* Software estimation: demystifying the black art (http://www.amazon.com/Software-Estimation-Demystifying-Pract...) is a very useful book too
* For lightweight project management tool: I use a mixture of Acunote (for estimating and burndown chart - I even generate my quotes out of it using Ruby and Prawn) and Freckle to monitor the time spent.
I tried many other tools (including Mingle, Pivotal, etc...) but Acunote is the only one that works well for us.
Having that said, there has been a lot of controversy on the value of estimates recently:
Estimation is Evil[2]
Purpose of Estimation[3]
I believe their TL;DR comes from the second one:
For me, estimation is valuable when it helps you make a significant decision
[1] http://www.amazon.com/Agile-Estimating-Planning-Mike-Cohn/dp...
[2] http://pragprog.com/magazines/2013-02/estimation-is-evil
[3] http://martinfowler.com/bliki/PurposeOfEstimation.html