Software Estimation - why is it so darn tough?

When I talk to my software industry friends and colleagues, one thing almost all of us seem to agree upon is the difficulty in assessing the effort and time required for any particular task. Most of the times, we are not sure why this happens, but in general, we almost always end up taking more time than committed. More time may not necessarily mean delaying deadlines, in most cases, it ends up eating into personal time and late hours at work.

Why is estimation so darn tough??

To begin with, in most cases, we have no clue as to the work that we will be doing. Let me take an example: Add News Content from a Third Party to an existing site. Generally, our approach would be:
A) Have a gut feel. Somehow. 3 weeks. Does it sound reasonable? Umm, yeah, why not..maybe its a bit off the mark..let me look further
B) Well, we need to make changes to the front end, new pages have to be created. See how many pages to be created and the time it generally takes to create one on an average. 1 week max
C) Changes to the middle tier - some coding will have to be done to get the data..Maybe parsing it etc. Another week
D) Overall, few changes here and there..Yeah, I think I can complete in 2.5 weeks, but let me add some buffer just to be safe. 3 weeks look good

One week down the line, we realize 3 weeks is way too less..what went wrong?A quick checklist to avoid the pitfalls of hasty estimation
A) Ask the right questions and scoping correctly. We need to be absolutely sure about the requirements
B) Question the interfaces. What kind of support will we get from the third party? What status are their APIs in? The more the interfaces, the more time it will take
C) Go into details and specifics. How many new classes/ methods/changes to existing classes and methods? Any new stylesheets required? Any database changes? To what effect?
D) How will we test? What all needs to be tested
E) We will need time to document design and code
F) Time will be spent in code and design reviews
G) Time is taken up in follow up with team members
H) Time will be spent in code control, build scripts, release depolyments
I) Time is required for preparing test beds
J) Time is required for the creation of development environment
K) Do we need to test all that is fully functional already? Impact on existing system. Whereall does it hit? We need time to reconfigure the earleir working version so that doesnt break with new added functionality
L) Time is also taken up in organizational status reporting and monitoring
M) There may be sick time - plan some unexpected time off for team members
N) Some trainings and holidays may impact the project schedule
O) Last but not the least, review the estimate. If we can list all the major and monir activities we will be doing and the time taken to do them, we can not be too way off from the actuals. However, if we miss some activities completely, they will really hit hard on the estimates.

Eldeco, NOIDA, marketing et al

I booked a flat in Eldeco Ananda, hoping I will save myself from the travails of constructing my own house. Though the prices were exorbitant, I was expecting that I will get service asI am opting for top-end premium stuff and paying the price for it. How wrong could I be!

1. Before booking the flat, the marketing executive told us that it would be possible to replace the wooden flooring in the bedrooms with marble, just like the rest if the house. No cost would have to be borne by us. As soon as we decided to book and communicated the same to him, he said that the flooring will be changed subject to a very nominal amount. After booking, I was told that flooring can not be changed in all the rooms, only the flooring in one of the rooms can be changed. The cost was to be about 15k, which is even more than the flooring cost of a similar sized room. To say the least, if I want to get it done, Eldeco expects me to pay more than double the cost, one which I have already paid in the base amount, and second is the extra charges for "alteration"

2. The company asked me to pay them the registration amount along with the miscellaneous amount and only then they would start thehalnding over of possession process. Registration is to be done in NOIDA authority and the company, being the seller, is also a party to the process. However, Eldeco mandates that registration be done through them, and they charge a hefty fee for that. Its been more than two months, I have been following up with Eldeco like crazy, and there is no reply. They say it is in progress, whereas I know ATS and other builders are queuing up at NOIDA office and getting their stuff done easily. Even individuals are able to get registration done at the authority within 2-3 days, so I am wondering what makes Eldeco spend that much time sitting on my money..?

3. At the time of booking, I was told I could opt out of the power backup(12k per kva!!) if I wanted. Ofcourse, now the reps tell me that they could NEVER EVER have told me that, how could they when it is just NOT possible? Ofcourse, some miscommunication, if at all..But I wonder why all the miscommunication seems to fit their ends, not mine...?