Software Estimation - why is it so darn tough?

When I talk to my software industry friends and colleagues, one thing almost all of us seem to agree upon is the difficulty in assessing the effort and time required for any particular task. Most of the times, we are not sure why this happens, but in general, we almost always end up taking more time than committed. More time may not necessarily mean delaying deadlines, in most cases, it ends up eating into personal time and late hours at work.

Why is estimation so darn tough??

To begin with, in most cases, we have no clue as to the work that we will be doing. Let me take an example: Add News Content from a Third Party to an existing site. Generally, our approach would be:
A) Have a gut feel. Somehow. 3 weeks. Does it sound reasonable? Umm, yeah, why not..maybe its a bit off the mark..let me look further
B) Well, we need to make changes to the front end, new pages have to be created. See how many pages to be created and the time it generally takes to create one on an average. 1 week max
C) Changes to the middle tier - some coding will have to be done to get the data..Maybe parsing it etc. Another week
D) Overall, few changes here and there..Yeah, I think I can complete in 2.5 weeks, but let me add some buffer just to be safe. 3 weeks look good

One week down the line, we realize 3 weeks is way too less..what went wrong?A quick checklist to avoid the pitfalls of hasty estimation
A) Ask the right questions and scoping correctly. We need to be absolutely sure about the requirements
B) Question the interfaces. What kind of support will we get from the third party? What status are their APIs in? The more the interfaces, the more time it will take
C) Go into details and specifics. How many new classes/ methods/changes to existing classes and methods? Any new stylesheets required? Any database changes? To what effect?
D) How will we test? What all needs to be tested
E) We will need time to document design and code
F) Time will be spent in code and design reviews
G) Time is taken up in follow up with team members
H) Time will be spent in code control, build scripts, release depolyments
I) Time is required for preparing test beds
J) Time is required for the creation of development environment
K) Do we need to test all that is fully functional already? Impact on existing system. Whereall does it hit? We need time to reconfigure the earleir working version so that doesnt break with new added functionality
L) Time is also taken up in organizational status reporting and monitoring
M) There may be sick time - plan some unexpected time off for team members
N) Some trainings and holidays may impact the project schedule
O) Last but not the least, review the estimate. If we can list all the major and monir activities we will be doing and the time taken to do them, we can not be too way off from the actuals. However, if we miss some activities completely, they will really hit hard on the estimates.