The reason agile came to prominence 20 years ago was that responding to change was more valuable than sticking to a plan.
And yet, this feels more relevant than ever. Agile has been thoroughly taken over by management and consultants. We might have some new tools and phrases, but behind the facade it’s all waterfall. And that comes with sticking to the plan with the caveat that certain special people absolutely can change the plan, in some cases very quickly and dramatically. Having something empirical to point to that maybe that reduces productivity is absolutely a positive.
That is to be expected, because "waterfall" or pipeline is the general structure of work. At the very least you have design, production and verification steps.
Agile is not about removal of waterfall, but rather applying 1. waterfalls to small chunks of work 2. multiple "tiers" of waterfall. The mythical waterfall that many pseudo-agile proponents argue against is simply one giant waterfall applied to the whole project. If you start building image manipulation software you generally do not pivot to building a presentation software without redefining or scrapping the "master" waterfall. Stages of the whole project generally do have dependencies on other stages, there is no way around that.
As long as you have at least some visibility into the project you are going to have stages and checkpoints and in the end a "waterfall" will surface somewhere. Waterfall is at the core a way to represent staged/checkpointed work.
I'm just going to grab this tiny part and focus on it. I've seen Agile fail many times for this exact reason. In start-up land, one might pivot and massively change an end goal. In corporate land that's a lot less off a thing.
I've worked on some big projects in government and fortune 500, all of them pretended to be Agile and all of them had predefined scopes, deadlines and budgets. In all of those projects, Agile was more of a disturbing factor than anything else. The only flexible part was the scope of work, because of renewed insights or lack of specification while planning. This always ends with teams getting pressured to make deadlines (because those are never flexible), so they cut corners and work extra long hours.
I think it would be much better for these types of projects (and the sanity of devs) not to pretend to be doing Agile, but just admit they want a more Prince2 led Waterfall approach. This won't happen though, because that puts the pressure with management in stead of the dev teams.
I've done agile projects where the client was a big corporation. There was a fixed budget and deadline, but we specifically reserved some of that time/money for unspecified work at the start of the project. In the latter parts of the project we could then prioritise the work that was deemed most important by the client once they'd already seen the work in progress.
I suspect it took quite a bit of work by our project team to convince them to work this way. But it's really great if you make it happen.
> I suspect it took quite a bit of work by our project team to convince them to work this way.
It probably did. I've seen a couple of organizations that did this well, but the majority did not, unfortunately. In the end the sales guy will usually have received their bonus far before the project is over. I'm mostly amazed that companies like this can keep running for decades, but they do.
My argument is that the things that waterfall specializes in, which is managing stages of work and dependencies between work is actually insignificant to the success of projects people care about.
Waterfall is good when you are building structures. But value exclusively comes from the integration, verification, validation, and deployment of structures. With these types of tasks, the variance in estimates exceeds the mean of those estimates. Effective workers constantly probe different pathways to success and this makes your project network, with its delineated stages and strict dependencies, totally divorced from reality.
The outcome of knowledge work, is knowledge. Not story points allocated, not dollars spent, not time used. The best hope of having a project management approach with valid predictive power is to track knowledge earned and not EVMS metrics satisfied.
>The mythical waterfall that many pseudo-agile proponents argue against is simply one giant waterfall applied to the whole project.
It's much rarer than it used to be but it is far from mythical.
Companies that are used to planning projects like this (e.g. construction) often have managers who apply the same mindset to software and are frequently almost incapable of considering that there is another way.
In my many years of software, I've come to the following conclusion: The "worst" process followed consistently across all levels of a company will always outperform the "best" process followed haphazardly with no real buy-in.
The reason agile fails at so many companies is that Product and Leadership don't want to do the hard work of keeping an up to date rank ordered list of groomed and deconstructed work that provides the greatest value to the business. Agile is about progress and learning and using that new information to decide what to do next.
At my current company we are using SAFe, where you plan 3 months of work at a time which is the complete opposite of being agile. Again, it asks the wrong questions. It asks "What are we committing to during this PI?" instead of "What is the most important and business value adding tasks we should be working on?" Leadership only wants the answer to when it will be done but that is up to them. It will get done as soon as they make it the highest priority thing.
SAFe doesn't require planning in 3-month program increments. The methodology is flexible and allows much shorter planning horizons when appropriate such as when the team is small and requirements are in rapid flux.
I think everyone understands that running long program increments isn't ideal from an agile development perspective. However, sometimes it is the least bad option for the enterprise as a whole. There are often a variety of program stakeholders outside the core team who can provide valuable input but aren't always available. It's a lot more practical to pull in everyone for two days of focused PI planning than to try to chase them down and schedule meetings on demand.
Managers with half knowledge and semi sincere attitudes have tried to pull "agile means you need to be flexible" on my team. To force a scope change and brand new stories in the middle of a 2-week sprint. The same folks would not accept any explanation whatsoever on why some stories didn't make the sprint delivery -- even if the blockers were completely in their court.
These buzz words just become intimidation tools at times. A junior team member cannot say it to their face that Agile doesnt mean what they think it means and should learn things before they speak.
Back when I was a software manager, I had to sequence work into sprints or everyone (up and down) would freak out. The part I relaxed the team on: I didn't care at all if work spilled over into the next sprint. This alone removed the stress of adding mid-sprint activities (which obviously needs to be done from time to time for at minimum reactionary purposes) and allowed more thoughtful medium-term analysis and execution. I was still on-time for delivery (at least above par on delivery timelines compared to sister teams), but with much less burn-out and generally more favorable ratings and well-constructed systems.
The thought that any meaningful work (including components of meaningful work) would naturally fall within a 2 week period of time has always baffled me. That said, I have always been very good at identifying critical paths, minimal requirements, and estimated effort; I'm not sure if this strategy would work well without sufficient planning.
> The thought that any meaningful work (including components of meaningful work) would naturally fall within a 2 week period of time has always baffled me.
Sprint duraration is supposed to be flexible and adjusted to each project type and yet everywhere I’ve been workint at it’s always 2 weeks
It's very true, but to get agile to work well, I found you need quality senior people at every levels. Otherwise, gateways come in place to make up for the lack of experience... and you wind back in the waterfall hell scape.
Also a lot of people have taken in agile and bastardised it. They kept the keywords and love it in business speak, but won't let a sprint run without putting their hands on everything and everyone, destroying the actual velocity of the said team. And that is I think why so many people now hate agile. We went from increased engagement and transparency to impossibility to concentrate.
I had one assignment where we had a Scrum Manager. He was a project manager who was also the scrum master. Luckily he was aware of the conflict in these roles, so he took my criticism well and was later replaced by an actual scrum master.
Agile has nothing to do with it. That just sounds like lack of general vision and direction. Or complete misunderstanding of Agile. "We have only two weeks, so we can't undertake anything big that spans multiple sprints!"
Plenty of managers do lack vision and direction (often feels like the norm) so it's not surprising that they lack creativity when it comes to assigning work to be completed within a single sprint. If they were free of that self-imposed creative barrier, they may be better positioned to develop an overall strategy.
I was once tasked to build an integration while working directly under the CTO. They designed their own API and claimed there was no specification in the ecosystem that we needed to follow.
I noticed that there was a specification and I informed them that we might want to change our system to match it. The request was ignored and instead I tried to make incremental suggestions to their design to make it more compliant with the existing specification.
We release the system and the first feedback we got was backlash for not following the spec.
We are now 9 months later and the CEO has requested that we follow the specification and I now get to rewrite the system.
I hardly find any motivation to rewrite the system, I haven't talked to the CTO for a few months and now suddenly they offload a bunch of their tasks on us by personally assigning us work and just invites our manager to listen in lol
I’ve been at my current job for 10 months. I’ve had three instances where I proposed a solution implementation to a problem, was denied, built the implementation I was told to build, had it break, and then told to build it the way I initially proposed (without any credit to the fact that I initially suggested it, of course).
I had a manager for 5 months, was without a manager for 5 months, and now have a new manager.
I checked out a few months back and haven’t been able to motivate myself to work at all. I would quit but I’m locked in for a year with golden handcuffs and don’t want to repay my starting bonus. I can’t wait to quit this shit job.
(This isn’t even half of the shit I’ve gone through since starting here, but only the most relevant pieces to the topic at hand. I could go on and on.)
As a developer you need to understand your customer and user. The more you understand them, the more able you are you guess where and what requirements will change and thus design for those changes in advance. Some requirements will never change, and the more you understand the problem the better you are able to predict which and then you can design those parts to not change.
Note that customer and user are different! If you are lucky they are the same people, but they are still different and have different needs you need to understand. You need to make the customer buy your product and continue to buy it and that sometimes means features that will never actually be used - these need to look nice in the demo, but not need to be high quality. If the user is not happy eventually they will find a way to get something else, you can milk a bad user product for a few years, but at sometime a user will get promoted to a level where they are able to make their displeasure unknown.
>> These events cause developers to feel fatigued, frustrated, and angry.
This type of thinking is the biggest assumption in the whole article, and something I often see pronounced as though it were fact by some default. Humans can have a strong degree of control over their emotions, and do not have to react to any events with any particular emotion. The reaction is often associated as 'being caused by' such and such thing, but when a feeling arises within, one can allow the feeling and not become overwhelmed/consumed with the effects of an unchecked emotional reaction. This allowing is the action of acting, rather than reacting. We do not have to behave as we always have, and get the results we have always gotten from our 'usual' states of being. We do not have to place the blame for our internal state on the external world, but benefit greatly by taking the responsibility for our own state.
On a personal level I agree, but the manager responsible for dealing with changing requirements will not get very far by telling people to manage their emotions better. It will come off as condescending and likely make morale even worse. If you're looking to change the way people think, know that it is incredibly difficult (sometimes impossible) and can only be done bit by bit over time by helping the individual to see a different way of looking at things.
That attitude can help. (In fact, I think it's pretty close to essential for maintaining your mental health.)
But I still prefer that my code be used, not just thrown away. I want my work to matter for something, for someone, not be just a paycheck. In the short run, sure, things get thrown away. Over a long time, I'll gravitate to situations where my work is used.
> At the end of the day you should shrug, warn of missing deadlines (if that doesn't change their mind) and start again on the new requirements.
Be sure you warn in a way that leaves a paper trail. You're going to need it to protect you from the higher-ups trying to blame you for missing the deadlines.
Oh I completely empathize with you, but I also believe that part of the high pay in tech has partly to do with the 'management overhead premium'.
Like I'd love to work for Google or Amazon on cutting edge shit - but I'd rather make half the pay to avoid the clown-brained managers, useless politics, and soul-crushing waste.
My solution btw is a simple one: managers should be forced to find executive coaches outside their org. Leadership and management can be learned from a good coach – and BAD leadership/management is, IMO, responsible for most problems inside an org. Not shifting market conditions, not complex tech, not incompetent engineers – but bad leaders.
Measure it with peer rating systems. Simple questions like "Would you follow this leader if they moved to a different company?" should make that manager's skills or progress pretty clear, pretty quick.
> Like I'd love to work for Google or Amazon on cutting edge shit - but I'd rather make half the pay to avoid the clown-brained managers, useless politics, and soul-crushing waste.
I work at a “big tech” company working on cutting edge shit but I’m with you. So done with this bullshit. It’s not worth the extra money.
If I do a weeks work on getting a feature ready, and then product decide to change all the requirements around and it needs redone. Why do I care ? I still get paid for that weeks work...
Next week I'll rework the requirements, and I'll get paid for that weeks work too!
More often than not, part of the issue is the real needs begin to be discovered after the first version ships.
When possible I always encourage creating a first version disposable of a working prototype that is disposable. Maybe it's built in a low/no-code tech to minimize any emotional attachment to throwing it away.
It's also important to remember that especially in the beginning, software is a vehicle for learning and experiments on what's needed. If it's too unstructured, or too structured, it can introduce constraints that limit the ability to learn and solve the problems wherever they are ultimately discovered.
> RCs are introduced when developers are already in the midst of the implementation or testing phases
Yeah, if you're still doing waterfall after all the decades of experience confirming Royce's observation, "the implementation described above is risky and invites failure"[1], then you're going to have a bad day. Many bad days, really.
Trying to 'kill the old ways' without understanding them is how we get junk such as Scrum.
Agile is just fast Waterfall with iteration.
Waterfall is naturally inherent in the process of going from idea -> working software that a user can use - Agile takes those inherent factors and makes the developer more responsible for them, in smaller, tighter, more Jira-friendly chunks.
If you can do Waterfall properly, you are ready to do Agile properly. If you can't do Waterfall properly, do not pass go, do not collect $200 Agile Dollars: learn Waterfall first. Agile fails because Waterfall is not understood well by newbies and grey-beards, alike.
You will see Waterfall in every Agile cycle - Waterfall describes the natural condition of software. Agile provides tools to rapidly manifest those conditions.
Disclaimer: 40 years of software development. I've seen some shit. Waterfall is real. Be agile about it if you must.
Hot-take: Agile is for service work, where the customer can't explain the problem very well yet, and you can take the reputation damage of showing half-work. Product work should have some amount of discovery, but derisked by user research (and thus is more waterfall) and thus will be released polished.
It's been well-demonstrated that users can't articulate what they really want. I don't recall the source, but I read one observation, "users will ask for a toaster in their car's dashboard when their real problem is that they don't have time to make make breakfast in the morning".
You can do all the research you want on in-car toasters to get the optimal product, but you still aren't really solving the right problem.
I agree with both the sibling comments. However, in my opinion, in a scenario where the customer is not clear about what they want without something in front of them, it is better to create a high fidelity click through design prototype, rather than actual code.
Finally, I'm personally very averse to finalise requirements where the customer is suggesting or dictating UI. Many times, there's a better way to fulfill their requirements by identifying what they want, in textual format, rather than asking them from the UI perspective.
Many if not most people are better at describing "what should be different" given a concrete thing than they are at describing "what should be" given a blank sheet. The idea is to build that fact into the work plan.
> Trying to 'kill the old ways' without understanding them is how we get junk such as Scrum.
You can hardly do better in understanding the "old ways" than reading Royce's paper. And the difference between Waterfall and Agile is not just iteration, it's rapid feedback. As noted in Royce's paper, Waterfall is inherently one way, hence the name. You don't see water flowing uphill, at least not without a lot of external force. Agility means feedback comes at every step of the way, not just at the end, when testing.
This is also the difference between what's been called "the Toyota way" vs prior automobile manufacturing. In the traditional automobile factory, the car was put together and completed, then a team of engineers would exercise it to find and fix defects. The Toyota was was to find and fix defects along the assembly line, and perhaps more importantly, learn from defects found during assembly to prevent or reduce defects before they appear.
The same applies to Waterfall vs. Agile. Instead of completing the entire system and then testing it and fixing defects, as waterfall would have, even when done iteratively, agility works out the defects in every aspect and uses what's found to reduce the rate of defect creation.
And yet, this feels more relevant than ever. Agile has been thoroughly taken over by management and consultants. We might have some new tools and phrases, but behind the facade it’s all waterfall. And that comes with sticking to the plan with the caveat that certain special people absolutely can change the plan, in some cases very quickly and dramatically. Having something empirical to point to that maybe that reduces productivity is absolutely a positive.