When people start working on General Purpose Robots there is a tendency to try to make sure that everything they do is Fully General Purpose And Future Proof. They have grand visions and world changing ambitions. Those make it seem like they should be building grand software, to match. This pushes folks really hard towards a really expensive kind of tech debt. Here’s some sniff tests to keep an eye out for:
Something comes through to me as contradictory. In "What is tech debt?" you enumerate type 1: fast, cut corners, and type 2: grand framework. In "Yagni vs. DRY: Fight" you confirm the enumeration and assert type 1 (fast, cut corners) is cheap to remove, doesn't hamper careers, whereas the 2nd flavor is expensive to acquire and remove. But then in note 6 you say your team estimated the cut-corner debt to be 5x more expensive to remove. This seems to contradict your assertion that it's fast and cheap to remove. What am I missing?
Great series - I've read them all & really enjoy them. I feel the pain and have lived it. Thanks!
One thing I've learned to embrace along these lines is to start with spaghetti-code, and organize from there. Starting from the assumption that you won't get it right until you've tested and iterated a bunch of times, a lot of up-front "framework design" is just going to be wasted or worse, getting you over-committed to bad design decisions. The bad thing about spaghetti-code (i.e., more or less one big function that does a bunch of stuff) is mostly that if you come back to it after some time, it becomes impossible to make heads or tails of it. But freshly-written spaghetti-code is really flexible and easy to change (e.g., swapping code blocks around, adding new variables or config values, etc.). So, a process I sometimes recommend is to write the quick-and-dirty spaghetti-code that works, start immediately testing it out and iterating, and as it stabilizes, you organize the code. The abstractions or APIs needed tend to emerge naturally (e.g., what group of variables or configs are often used together, what data is config / state / scratch-space / IO, etc.). One counter point I've heard is "what if you can't test or integrate it right now?", to which I would reply, "if so, then, why are you writing it now?".
On the Grand Framework thing, a common thing to hear people say is: "we might not need this feature right now, but if I don't design the framework with this feature, it will be impossible to add it later". Generally, IMHO, that just points to design issues with the framework, usually a lack of composability or transparency. And as you said, it can often be a DRY thing too. A simple example, say, you need a single-produce-single-consumer (SPSC) thread-safe queue, but you could write it to be more general, and knowing that if you needed an MPMC queue at some point, you'll probably end up having to write a new class (to avoid changing your heavily used SPSC queue) and "repeat-yourself" a lot. It's almost a no-brainer that a very simple SPSC is preferred to a much harder MPMC, and if you design things in a reasonably flexible way, adding or switching to an MPMC later should not be that hard.
I guess another way to put it is that imaginary features should have imaginary implementations, not concrete ones. I think it's reasonable to have TODOs and other comments in the code that discuss pieces of code or API decisions that are as they are in case an imaginary feature is needed, so it remains easy to add or to find where changes are needed. In other words, there is an imaginary implementation for an imaginary feature. What is serious tech debt is when you have concrete implementations for features that aren't used (or barely used), especially when it encumbers a lot of the details of the code. A good framework is flexible, i.e., it allows for many imaginary features, while a grand bloated framework has a bunch of features baked in that are never quite the right mix for any user and with too much overhead (performance, maintenance, cognitive load, etc.).
Fantastic post. As someone who codes mostly against microcontrollers mostly for prototyping, your perspective on tech debt ring really true.
I "grew up" coding in .NET where resources were infinite and abstraction was street cred. But MCUs are unimpressed by hifalutin architecture. My code may not conform to traditional beauty standards, but it runs just fine in the finite space of the microcontroller's operating conditions.
Thanks for giving me some great vocabulary to pin to this philosophy!
Something comes through to me as contradictory. In "What is tech debt?" you enumerate type 1: fast, cut corners, and type 2: grand framework. In "Yagni vs. DRY: Fight" you confirm the enumeration and assert type 1 (fast, cut corners) is cheap to remove, doesn't hamper careers, whereas the 2nd flavor is expensive to acquire and remove. But then in note 6 you say your team estimated the cut-corner debt to be 5x more expensive to remove. This seems to contradict your assertion that it's fast and cheap to remove. What am I missing?
Great series - I've read them all & really enjoy them. I feel the pain and have lived it. Thanks!
Great post as usual Benjie!
One thing I've learned to embrace along these lines is to start with spaghetti-code, and organize from there. Starting from the assumption that you won't get it right until you've tested and iterated a bunch of times, a lot of up-front "framework design" is just going to be wasted or worse, getting you over-committed to bad design decisions. The bad thing about spaghetti-code (i.e., more or less one big function that does a bunch of stuff) is mostly that if you come back to it after some time, it becomes impossible to make heads or tails of it. But freshly-written spaghetti-code is really flexible and easy to change (e.g., swapping code blocks around, adding new variables or config values, etc.). So, a process I sometimes recommend is to write the quick-and-dirty spaghetti-code that works, start immediately testing it out and iterating, and as it stabilizes, you organize the code. The abstractions or APIs needed tend to emerge naturally (e.g., what group of variables or configs are often used together, what data is config / state / scratch-space / IO, etc.). One counter point I've heard is "what if you can't test or integrate it right now?", to which I would reply, "if so, then, why are you writing it now?".
On the Grand Framework thing, a common thing to hear people say is: "we might not need this feature right now, but if I don't design the framework with this feature, it will be impossible to add it later". Generally, IMHO, that just points to design issues with the framework, usually a lack of composability or transparency. And as you said, it can often be a DRY thing too. A simple example, say, you need a single-produce-single-consumer (SPSC) thread-safe queue, but you could write it to be more general, and knowing that if you needed an MPMC queue at some point, you'll probably end up having to write a new class (to avoid changing your heavily used SPSC queue) and "repeat-yourself" a lot. It's almost a no-brainer that a very simple SPSC is preferred to a much harder MPMC, and if you design things in a reasonably flexible way, adding or switching to an MPMC later should not be that hard.
I guess another way to put it is that imaginary features should have imaginary implementations, not concrete ones. I think it's reasonable to have TODOs and other comments in the code that discuss pieces of code or API decisions that are as they are in case an imaginary feature is needed, so it remains easy to add or to find where changes are needed. In other words, there is an imaginary implementation for an imaginary feature. What is serious tech debt is when you have concrete implementations for features that aren't used (or barely used), especially when it encumbers a lot of the details of the code. A good framework is flexible, i.e., it allows for many imaginary features, while a grand bloated framework has a bunch of features baked in that are never quite the right mix for any user and with too much overhead (performance, maintenance, cognitive load, etc.).
Fantastic post. As someone who codes mostly against microcontrollers mostly for prototyping, your perspective on tech debt ring really true.
I "grew up" coding in .NET where resources were infinite and abstraction was street cred. But MCUs are unimpressed by hifalutin architecture. My code may not conform to traditional beauty standards, but it runs just fine in the finite space of the microcontroller's operating conditions.
Thanks for giving me some great vocabulary to pin to this philosophy!
Solid article 👍🏻