Iterative is Not Incremental

Correlation is not causation, is something that you'll hear often in statistics. Similarly, I think Agile deserves one, Iterative is not Incremental.

Before we get into the waterfall vs. agile flame war, let's define what we mean by waterfall. Loosely, it is said that waterfall is akin to long-term planning while agile is responding to changes as they happen. But that's an overly simplistic definition of each.

Any endeavour will have a long-term vision and a set of short-term projects to support that vision. If you want to be a better engineer five years from today, you have different options; be it mastering algorithms, or learning TLA+ to improve your chops for distributed systems, or designing the next Rust.

Thus, long-term and short-term thinking in themselves are not the differentiating characteristics between waterfall and agile.

Planning happens at many levels. Just like a current flows through resistance, "heading" for a destination in the face of random obstructions, so does the execution of our plans. For example, the exact actions that'd contribute to your 5-year goal of becoming a better engineer could keep changing. It could be learning Rust today, prompt engineering tomorrow, and systems engineering the day after. Although your goal is still the same, your immediate actions depend on the immediate circumstances.

Navigating a goal is much like navigating traffic. You may go straight, take detours, or even go against the direction of a goal temporarily because the "road" was closed. But your destination is still the same. It's as if the goal was an abstract class and your immediate actions the most viable implementations.

So it's not long-term vs. short-term when we talk about waterfall and agile. It is the rigidity of the details. Earlier waterfall models insisted on having all the details, even the tiniest one, up-front. Which is akin to you deciding which specific courses you'll do for the next 5 years. But maybe by year 2, there is a better course for learning Rust, or the one you had in mind got outdated.

Or maybe your goal isn't "learning Rust." It's learning something cutting-edge. It happens to be Rust (and many other things) today. Tomorrow, when you actually sit down to do something new, it very well could be something else.

Thus, agile's response to changes has to do with the details, not the goal itself. Waterfall would be like a predefined plan, whereas Agile would be a policy that's applied to a changing landscape like a reinforcement-learning agent.

However, there is a way to take even agile too far. One can much as easily assume that having any kind of long-term goal is NOT agile because it locks you in. And we should be willing to change long-term goals too. True, that can sometimes happen. But you wouldn't want to work at a startup that was creating a database in the morning, a game in the afternoon, and finally pivoted to being a todo-list app in the evening.

Agile doesn't mean you shouldn't have long-term goals. But what it means is in the face of a policy biased toward some long-term goal, you should act intelligently on the actual circumstances rather than what you wished the circumstances were.

Waterfall tries to build a plan that can be greedily followed. Agile keeps the option open for maximising long-term outcomes, even if you make u-turns in the short term. And that's where the distinguishing characteristics of Iterative not being Incremental comes in.

Incremental is a greedy strategy. You had 5 lines of code yesterday, today you'll have 3 more, and tomorrow you'll have 4 more. They all add up to 14 lines of code.

Iterative looks a bit different. You have 5 lines of code today, which exposed some problems with the original design. Tomorrow, you have a different 4 lines of code as you threw out the 5 you had written previously. And so on. In the end, you end up with 3 lines of code that solved your problem, but only after you threw away 34 that didn't.

If we don't throw things away, then we'd be resistant to doing experiments to learn new things. And the iterative strategy often has us take detours, u-turns, for the long-term good, whereas incremental insists on short-term gains. If you have a feeling you're not making a lot of progress, you may benefit from viewing it from an iterative lens rather than an incremental lens.