Thanks for the comment! I think I wasn't clear enough in describing what I meant by the generating process, it's not just for(random int) but rather having a random recursion depth. I was hoping to show that with the diagram but maybe my words didn't quite match the picture!
I should probably try coding up some toy models like you did to how different sorts of variables ("try again probability", "sub-task branching factor", etc) alter the curve shape. Would be interesting!
Nice article. You have found a pattern quite similar to what I found comparing project estimates. When I researched further I found that this pattern was quite common. p10 at 1x, p50 at 2x, and p90 at 4x. Your pattern is a bit different as you have p50 at 1x and p90 at about 4x, but then I expect differences as you are at the task level and I measured at the project. As you say, lognormals add together and stay lognormal, but the standard deviation decreases. As tasks add up to be projects the range becomes smaller. As always one of the big challenges is that data is not readily available. Can you make the data available in some form?
Thank you for the comment! Every time I discuss this with others who have collected real data on task completion, I hear the same. I wish I could share the data! Unfortunately, while the company has given their blessing to me writing this article, they've declined to make the data available for the time being. If they change their mind, I will provide an update. Data of such high quality is scarce at the task level, and at the project level even scarcer.
Thank you for the article. I was sent here by a team member.
My intuition and what I've used in my career is the following:
1) Estimate the tasks
2) Think about everything that can go wrong, add to the estimates.
3) Satisfied with the results? Multiply by 2.
Using the log-normal distribution above, it looks like this results in roughly P75. In other words, with this estimate, my tasks will be properly estimated or over-estimated 3 times out of 4. That seems like a good cost balance for project success. (3 projects on-time, 1 project late).
That does sound like a good balance, and by all means keep using it! Especially if it's working well for you so far. I think the only other takeaway I'd emphasize is it's also okay to adjust that threshold according to the risk tolerance of your audience or situation. For example you might use times 2 for an estimate you give to your team, but times 5 for an estimate that gets used when bidding for a fixed-price contract, even if it's the same work in both cases.
ESR sharing this article on X brought me here, but one of the comments on his post most jumped out at me as how this might be useful:
"It also suggests a visual scale for PMs and execs to track estimate accuracy: lognormal = reasonable, normal = random guesses, lumpier right than left = you are being lied to."
Excellent article. Fits entirely with my experience and has parallels outside of software engineering.
For example, years ago I used to lead analytical teams in hospitals. We would get ask questions the essence of which is “how many beds do I need to (to accommodate unplanned admissions from A&E)?”. The correct answer to give, after analysis, was the answer to a slightly different question: how many beds do I need such that the likelihood it’s not enough is less than 5% (or some other risk threshold).
As an aside, people _really_ struggle to understand why that does not result in a 95% average bed occupancy.
I used to be the product guy asking for the commit date. My dev team (this was a semiconductor company) consistently gave me nothing-will-go-wrong dates, so I eventually took their swags (they certainly weren't commits - they were absolutely not held to them) and padded it to show at least 2 iterations before a chip was ready to sample - let alone volume production. This meant showing customers a date >6 months beyond their estimates.
What made this interesting was having a dev team that avoided confrontation. They were a bunch of 50+ YO hardware engineers from Taiwan & Japan, friends with the CEO, who didn't know how to handle an American who didn't give a rat's ass about seniority. Fun times.
The kicker came when my boss and a developer told me I was being too conservative when <ding> the devs announced another delay - literally in the middle of the argument. It was a really good feeling TBH.
> So completing a technical task can involve a "for" loop with a random number of iterations, which can quickly generate a lognormal curve
A for loop with random stuff would generate a normal
function getResult(){
result = 0
for (i=0, i<n, i++) {
result += get_random_factor(i)
}
return result
}
I think what would generate a lognormal is
function getResult2() {
result = 0
for(i=0, i<getDifficultyOfStage(0), i++){
for(i=0, i<getDifficultyOfStage(1), i++){
for(i=0, i<getDifficultyOfStage(2), i++){
for(i=0, i<getDifficultyOfStage(3), i++){
result += delta
}
}
}
}
}
return result
}
where the different stages correspond to e.g., solve problem, check with management, check with clients, fight upper management, etc etc.
Thanks for the comment! I think I wasn't clear enough in describing what I meant by the generating process, it's not just for(random int) but rather having a random recursion depth. I was hoping to show that with the diagram but maybe my words didn't quite match the picture!
I should probably try coding up some toy models like you did to how different sorts of variables ("try again probability", "sub-task branching factor", etc) alter the curve shape. Would be interesting!
Ah, random recursion depth makes more sense than constant recursion depth but variable iterations for each loop!
Substack replaces whitespace, so here is a pastebin: https://pastebin.com/TnKhFvm1
Nice article. You have found a pattern quite similar to what I found comparing project estimates. When I researched further I found that this pattern was quite common. p10 at 1x, p50 at 2x, and p90 at 4x. Your pattern is a bit different as you have p50 at 1x and p90 at about 4x, but then I expect differences as you are at the task level and I measured at the project. As you say, lognormals add together and stay lognormal, but the standard deviation decreases. As tasks add up to be projects the range becomes smaller. As always one of the big challenges is that data is not readily available. Can you make the data available in some form?
Thank you for the comment! Every time I discuss this with others who have collected real data on task completion, I hear the same. I wish I could share the data! Unfortunately, while the company has given their blessing to me writing this article, they've declined to make the data available for the time being. If they change their mind, I will provide an update. Data of such high quality is scarce at the task level, and at the project level even scarcer.
Thank you for the article. I was sent here by a team member.
My intuition and what I've used in my career is the following:
1) Estimate the tasks
2) Think about everything that can go wrong, add to the estimates.
3) Satisfied with the results? Multiply by 2.
Using the log-normal distribution above, it looks like this results in roughly P75. In other words, with this estimate, my tasks will be properly estimated or over-estimated 3 times out of 4. That seems like a good cost balance for project success. (3 projects on-time, 1 project late).
I will keep using the "times 2" formula.
That does sound like a good balance, and by all means keep using it! Especially if it's working well for you so far. I think the only other takeaway I'd emphasize is it's also okay to adjust that threshold according to the risk tolerance of your audience or situation. For example you might use times 2 for an estimate you give to your team, but times 5 for an estimate that gets used when bidding for a fixed-price contract, even if it's the same work in both cases.
This is a superb article -- and that’s coming from somebody who doesn’t know much about maths or probability. Thank you! 🙏🏻
ESR sharing this article on X brought me here, but one of the comments on his post most jumped out at me as how this might be useful:
"It also suggests a visual scale for PMs and execs to track estimate accuracy: lognormal = reasonable, normal = random guesses, lumpier right than left = you are being lied to."
https://x.com/skibidiquasar/status/1919913356863144379
Excellent article. Fits entirely with my experience and has parallels outside of software engineering.
For example, years ago I used to lead analytical teams in hospitals. We would get ask questions the essence of which is “how many beds do I need to (to accommodate unplanned admissions from A&E)?”. The correct answer to give, after analysis, was the answer to a slightly different question: how many beds do I need such that the likelihood it’s not enough is less than 5% (or some other risk threshold).
As an aside, people _really_ struggle to understand why that does not result in a 95% average bed occupancy.
I used to be the product guy asking for the commit date. My dev team (this was a semiconductor company) consistently gave me nothing-will-go-wrong dates, so I eventually took their swags (they certainly weren't commits - they were absolutely not held to them) and padded it to show at least 2 iterations before a chip was ready to sample - let alone volume production. This meant showing customers a date >6 months beyond their estimates.
What made this interesting was having a dev team that avoided confrontation. They were a bunch of 50+ YO hardware engineers from Taiwan & Japan, friends with the CEO, who didn't know how to handle an American who didn't give a rat's ass about seniority. Fun times.
The kicker came when my boss and a developer told me I was being too conservative when <ding> the devs announced another delay - literally in the middle of the argument. It was a really good feeling TBH.