Correct vs Good
Part 2: Can we have it both ways?
In part 1, I talked about the difference between being correct (meets a spec) and good (meets a need).
Conflicts Between Correct & Good
So obviously it is better to be both correct and good but there are conflicts between them.
To be correct:
Go carefully
Long release cycles
Exhaustive testing
Culture of perfection
Don’t test at customer sites
To be good:
Go fast
Fast iterations
Fast testing
Culture of shipping
Must test at customer sites
You can choose for yourself and your company where you want to be on each of these axes. And the right spot to be will change with the life of your product and company.
“Goodness” is something related to the product fit and can grow infinitely, while “correctness” is about how well implemented it is, and tops out at 1.0 (100% correct). I’m going to multiply them together to estimate the total value of a product.1 Using this model explains why startups can move so fast and big companies get slower and slower over time.
You can see that at the big company, they have been working for a long time and have built a lot of goodness and correctness. Adding new goodness is still possible, but each step is a smaller percentage. Even a very small drop in correctness has the chance to wipe out hundreds of good features’ worth of value. Big companies are paranoid about correctness, and rightly so. They have a ton of value to defend.
Startups, on the other hand, don’t have a ton of goodness or correctness. That means that a single new good feature is a much larger percentage improvement to the startup. In addition, the consequence of a drop in correctness is smaller.2
Being correct slows you down
Being more correct encourages things like exhaustive testing, a culture of perfection, compliance review, process driven release, etc. All of those things help protect correctness at the cost of slowing down iteration, because the easiest way to make it hard for things to regress is to make it hard for things to change.
We often have an emotional bias towards correctness.
Engineers like correctness. We are people who deal with systems and rules and valuing correctness has been trained into us.
Engineers coming from mature projects to early/experimental ones often try to bring big company best practices. They’ve seen smart people succeeding by doing things a certain way and it feels right to copy them. And maybe they've seen a small drop in correctness cause a huge drop in a big company’s value. But if you are early in discovering the shape of your product, the last thing you want to do is carve it out of stone.
Lastly, if you are not correct enough it is embarrassing. Customers seeing bugs is embarrassing. Shipping code that isn’t quite right is embarrassing. It is very hard to look at your boss and say “Yeah, I shipped a bug to the customer, and I still think I made the right tradeoff between speed and correctness”.
Consequences
Obviously it is best to get exactly the perfect balance between being correct and good, but what happens when you get it wrong?
If you aren’t correct enough then customers see bugs. If those bugs are bad enough they might fire you, but most of the time the consequence for a startup is embarrassment.
On the other hand if you aren’t good enough then you will find customers who order a small pilot, usually through their innovation team. They will use your robots occasionally, perhaps only when giving tours to VIPs. They won’t expand their deployment or will do so very slowly. Sales will be slow and expensive. You will always be ‘almost there’ in terms of scale. You will have a bunch of unused robots sitting around (that you paid for, couldn’t sell, and are now getting slowly outdated) and you can’t raise more venture capital because you don’t have enough traction and you run out of money and die.
This is how 99% of robotics startups die.
So, while you can die from lack of correctness,3 you are much more likely to die because your product isn’t good enough. That means you want to pick your sweet spot for ‘correct enough’ given your maturity and focus 1000% of the rest of your time and attention on getting good as fast as you can.
Build process and infrastructure to verify, as fast as possible, that you are at your minimum correctness bar.
Get great at release rollout and rollback so you reduce the consequences of shipping bugs.
Get robots into customer hands as fast as possible.
Remove tech debt that slows down iteration cycles.4
Do good product development: (guess → try & observe → iterate)
And if want to keep reading things like this I recommend Part 1 of Correct vs Good, or my article on Tech Debt.
I don’t have a particularly rigorous reason why multiplying these things together gives total value, but it explains the different behavior of successful big companies vs successful startups really nicely. It's really satisfying when a model explains an observed phenomenon.
If a startup introduces a 1 in 10,000 bug that requires one of their engineers to intervene they might not even hit it. And the odds of that feature being removed or completely rewritten before anyone ever hits it are reasonably high. If Google introduced a 1 in 10,000 bug that required manual intervention it would be a catastrophe.
Cruise arguably died from lack of correctness: robots that can kill people have a much much higher minimum correctness bar.
It could be that your iteration cycles are slow because you keep finding regressions you’ve introduced. That is a sign that you are undershooting your minimum correctness bar, and should add more testing.