The customer doesn't know what they want
Agile assumes as a first principal that including the customer throughout the development process will align the building team and the customer to come to the same conclusion.
This is almost never actually true. Having built and managed a lot of products I can state fairly confidently that the amount of time the customer has, and clarity about the problem they want solved is orders of magnitude more ambiguous than is required to actually build and deliver something to their satisfaction.
In my opinion successful products, and I mean ones with significant flywheels with customers and viral growth and stickiness are with few exceptions - accidents. It was someone who had an interesting thought and a lot of people for whom the product was good enough for their desire that they often didn't even know they had
You can't write down how to do that, it's like asking a nobel prize winner how they came up with their discovery. It's not repeatable. Some people have better intuitions than others and these are the people who are repeatably successful with products. But it doesn't generalize.
That said, the fundamental problem isn't wrong in theory, it's just naiive in practice.
This is not what the article is about. If you look at the Agile Manifesto, it says e.g.
- Individuals and interactions over processes and tools - Responding to change over following a plan
What you get in quite a few big companies following "Agile" with the air quotes is the opposite:
- Processes and tools over individuals and interactions - Following (and making) a plan over responding to change
Because that's what makes middle management happy.
So this perversion of "Agile", which is actually the opposite of Agile, is the FUNDAMENTAL problem.
I once had it used against me to justify not writing tests ("processes and tools!"). We had meetings about bugs instead. Seriously. Complain all you like about perversion, that was a perfectly valid interpretation coz those hallowed commandments are, frankly, ill-defined bollocks.
The principles are better but still suffer the same problem. If the most efficient way to convey information in a dev team is face to face why don't you come over here and I'll whisper my pull request in your ear. Jesus.
There's no point complaining about the perversion of a thing that was never very specific about what it wanted to be in the first place.
Once people (especially managers) get specific about what agile means to them it turns out it means very different things to different people. This is entirely the fault of the originators.
One benefit of Scrum (the catholic church to agile's christian sect) was that it was proscriptive and was specific. Unfortunately it's also a bit shit and its practitioner-priests LOVE to tell you that if it isnt working, well, You Probably Just Weren't Doing It Correctly.
(1) The Manifesto is written in non-actionable terms.
(2) Many of the same people that initially attached themselves to the manifesto were already pitching canned processes, that may have been the outcome of agile process where they originated but did not put the critical feedback mechanisms more prominently than the low-level processes. So, those became the actionable versions of “Agile”.
It would be so much better if instead of the “X over Y” language, the Manifesto has spent a few words talking about how the less important things were subordinated to the more important ones.
The weird thing is that continuous, bottom-up, empirically-based process improvement wasn't an unknown concept at the time the Manifesto was written. It wouldn't have been hard for it to reference more concrete, actionable, concepts and practices.
Agile is Dead - https://m.youtube.com/watch?v=a-BOSpxYJ9M
Here’s a blog post by him: https://pragdave.me/blog/2014/03/04/time-to-kill-agile.html
If that’s not following a plan over responding to change, i don’t know what is.
Agile tries to turn software development into a product design process rather than a product manufacturing process. Together with the client, iteratively you try to find the right solution and design for the customer problem.
By being easy to adopt, software lends itself to an iterative design process, where you adopt it multiple times based on the iterative feedback of the customer. But this means that at the end of the design process, you can skip the manufacturing part, as you've already built the software during design.
Alternatively you could argue that this is too expensive, inefficient and slow, and you need some design before you begin to manufacture software. Modeling tools could help here, e.g, low code tooling like CAD tools in fysical product design.
Oh, man. I think you hit the head of the nail. One succinct phrase that describes the entire problem.
Early in my career somebody said something like "make sure you distinguish between <<what they (customers) want>> and <<what they need>>.". This thought stuck with me for years and in my opinion has been reposible for success I witnessed.
It's important to unwrap "what they want" so you, as developer, can apply more first principle thinking to arrive at "what they need".
BAs have tendency to project their own solutions, however when unwrapped so you see the problem that needs solving, you may arrive at solution that is simpler (like orders of magnitue is not that rare in my experience), reflects reality better, takes advantage of somehow hidden from BAs PoV system internals etc.
Agile enouraged dialogue is the platform where those ideas can surface.
Usually what happens is that you get a customer team, that is supposed to voice how the organization wants the software to be.
So the same mismatch is bound to happen anyway.
And when they do involve the people from the field, depending on the company culture, they might even be quite positive on the demos (cause being negative isn't good) and then completely find it unusable on final delivery.
Agile misses that many engineering teams lack people skills to actually navigate and avoid these scenarios.
EDIT: several typos
"I thought this was your job" is something I've heard a few times from clients when we have sent over wire frames or beta applications.
Especially in B2B and technical realm it is feasible to actually plan in every sense of the word "plan" a new product - business, tech, etc. Of course the plan must be malleable on the facts on ground. And probably is incorrect on a few places. But never the less, a plan that will get a business from place a (no product, but strong understanding of customer needs and business plus strong technical savvy) to place b (final product, customers, revenue etc).
I've seen this many times, both in startups with a new product and new customers (automotive software) and established companies that just expand their existing portfolio (cad software).
I'd say the "not an accident" is feasible if these facts are in place:
- enough A-team players (tech,business,sales)
- healthy culture (that facilitates long term growth and can focus on the product without too much politics)
- understanding of customer business
- identified an OBVIOUS need for the customer
- strong engineering org (can be just a few coders that get things done or larger)
So I agree, you can't train for these requirements like you can train airplane pilots.
But I do claim you can have enough understanding that is the current org in possession of these qualities. And if it is, it's a very reasonable risky investment to try to create a product for the need.
For internal customers or in B2B settings getting the actual users and outcome owners involved early is vital - but they are often poorly incentivized to get into the action as there are layers of business mangers etc in place that are supposed to stand in (but lack the detailed knowledge).
What was the 'standard' way to develop systems when the Agile Manifesto was written: the same as for building a bridge.
You gather and freeze requirements. You write a massive plan and plenty of specs. You code accordingly. You deliver.
Time to working software is long and a lot of the effort that goes into specs is wasted, especially for design specs.
Here's the introduction of Steve McConnell's "Rapid Development" from 1994 - well before the Agile Manifesto! - Chapter 1, page 1:
> The product manager told me he wanted to build a product right for a change. He wanted to pay attention to quality, prevent feature creep, control the schedule, and have a predictable ship date.
> When the time came to actually do the project, it became clear tha getting the product to market quickly was the only real priority. Usability? We don't have time. Performance? It can wait. Maintainability? Next project. Testing? Our users want the product now. Just get it out the door.
> This particular product manager wasn't the manager on just one product. He could have been almost any product manager I've worked for. This pattern is repeated day after day, state by state, all across the country. ...
This is as far from building a bridge as you can get. The next page of the book starts:
> [1.1] What is Rapid Development?
> To some people, rapid development consists of the application of a single pet tool or method. To the hacker, rapid development is coding for 36 hours at a stretch. To the information engineer, it's RAD - a combination of CASE tools, intensive user involvement, and tight timeboxes. To the vertical-market programmer, it's rapid prototyping using the latest version of Microsoft Visual Basic or Delphi. To the manager desperate to shorten a schedule, it's whatever practice was highlighted in the most recent issue of Business Week.
Agile drew from earlier methods ("intensive user involvement", "rapid prototyping") which were pretty common even before Agile.
The objective function is some function of the quality of the product, as measured by the customer, and the work put in. The gradient evaluation is then the process of determining which feature gives the most bang for the buck, implementing it, and repeating.
But gradient descent has problems with local minima. So if that's what Agile is, no surprise it doesn't do surprising projects - even if the customer does know what they want. If the programmers are too inexperienced and it's not just a CRUD, then you'll end up with something like Jeffries' TDD Sudoku solver.
Lean is.
Agile is “It would be nice to do what works for your team in your particular challenges, not some pre-packaged one-size-fits-all processes sold by consultants. Now, here’s a bunch of pre-packaged one-size-fits-all processes, and an army of consultants selling them.”
You push decisions as far out as possible and gather feedback from individuals every step of the way.
This is what allows the greatest amount of flexibility.
The biggest problem with agile is that it assumes all problems can be put off to the last minute.
I'm frequently finding myself in situations where it is 100x better to hammock program it out for a few weeks before pushing any code. Those systems last ten years. Those systems are able to be modified simply and quickly when business needs change. Those systems do build up legacy code, but it's at a slower rate than other systems. Those systems have fewer hacks over time and fewer bugs as a result.
This isn't a dig at agile, it's a dig at using agile as a design process, for which it is utterly inept. Agile is great at pushing products, terrible at pushing thoughtful system design. (On average, sometimes working a problem works too but more often than not you don't get the chance to redo middle sprint 5.)
I don't disagree, though I firmly believe Agile does a much better job of dealing with the problem you state than any other software development methodology that has been attempted before.
Is it perfect? of course not. As a methodology, is it better for building software than methodologies that have been used to build enormous physical structures for generations? Undoubtedly yes.
As far as I'm aware, there currently isn't anything better for building complex software with changing requirements (i.e. the real world).
The only thing that is less likely to have the building team and customer come to the same conclusion is to add many more layers in between.
I’ve seen where that ends. It’s not a good place.
If that were true the customer never drop you for the competition. The customer will drop you in a heartbeat.
The customer cannot express what they want in a language you can articulate which is something amazingly different. As a product manager you need to account for that.
It's a way to meeting actual business software needs, much more suited to custom development for one customer than product development, which necessarily requires some additional insight into what the market wants.
That's pretty much all you can do. How you organize it only matters insofar as it makes your team effective - sprints, waterfall, XP whatever... is kind of a toss up and context dependent.
The Agile Manifesto is a failure at trying to make Agile happen because it can't tell you how to make it happen, because it varies wildly. No manager can read a book on how to "Agile-ify" their org, they have to apply their brains and figure out how their specific version of "Agile" will work. But the skill-set required to do this is not a Managerial skill, it is a lower-level-worker skill. But it's also a very advanced lower-level-worker skill.
And that's why things like "Agile", "DevOps", etc will fail. People at the higher end have no clue how to make it happen, and people at the lower end who have an idea how to make it happen don't have the power to make the organizational changes to do so. You need a way for the lower-end people to tell the higher-end people what to do, and have the higher-end people listen to them, and make the changes happen. This is very hard in a traditional organizational hierarchy, because higher-end people have big egos and bigger concerns over things like politics.
Maybe this isn't the norm?
The best I have seen were mini-silos that managed to de-couple themselves from the org chart.
However I also have seen that it hardly lasts more than a couple of years, because as soon as someone noticed the success of the business unit they were re-integrated so that they could teach the reasons for their success to others.
You can imagine how well did they usually fare afterwards.
Wich makes it all a glorious Catch 22.
These companies, that rely completely on software, are unable to make the required leadership changes. It’s cultural.
I completely disagree. Most team leads and technical managers where I've worked get promoted up from within a highly technical position. I wouldn't have any respect for my team lead or my PM if they didn't know what they were talking about.
If my PM is going to try to tell me that I should work on this feature over this other feature or I should implement it in this way over this other way do you really think I'm going to listen to them if it's clear they have no idea what the implementation details are let alone the tools? Of course not.
I'm making DevOps happen and I do that by knowing the tools and practices. I'm being given the power I need to make it happen. I earn the respect every day I come in and write the code or identify the weaknesses we need to shore up to move faster.
In the end if you are difficult you become easy to replace.
How much of each dollar you make for the company with your hard work do you take home?
That doesn't mean you need points or tshirt sizes or constant status meetings. You just need channels of communication with the right people and a chance to adapt to what they say.
And it’s something that good devs will already be trying to accomplish anyway. There’s also the part about breaking stuff down into small parts so that small cycles of meaningful dev can be accomplished. But that’s mainly to be able to release small and often - another thing you should already be doing to incrementally update your app avoiding big releases. It also stands in the face of waterfall development, though I see lots of teams doing test at the end still, making it waterfally.
I also see too many agile teams that have iterations that never finish, get feedback in prod, create half complete features that are weird or don’t gel - all in the name of agile.
I think the kids that get it were already doing it as it’s common sense. The rest end up sticking an agile badge on their practices when it’s all but.
The patron wants more, the creator doesn't want to create more, so you need to force a detente so the patron will continue to pay the creator, and the creator will continue to produce.
You get t-shirt sizes and points when the patron feels the creators aren't doing what they said they'd do, so in a way it's a communication problem, yes, but the cause of the communication problem is that there is almost never an alignment between the two groups, and so you need some kind of shared language where expectations can translate, and accountability can exist at all, usually in some flawed form.
The problem, in my experience, is many development contracts are still structured in such a way that this becomes too difficult. Management fearing early exposure to the customer will lead to scope creep and cost growth.
Tech companies can look at the agile manifesto and use it as a heuristic guide, because engineers are already kind of on the same page about it. You don't need heavyweight process etc.
Non tech-companies need the window dressing of tech companies to retain their best engineers, but ultimately agile is kinda telling them to turn everything upside down and also threatening to make them superfluous. Nobody is really up for that.
So agile in non-tech companies is Kabuki theater and engenders cynicism, and agile in tech companies is basically superfluous "water is wet" advice no one even bothers to comment about.
You'll notice a lot of these blogs about how agile has failed are coming from consultants, who are basically brought into traditional companies that are struggling with some part of this process, not tech companies.
After the buy-out we were told to undergo a thorough transformation into this brand new unified top-down software development process the company had some consultants design for them. Complete with a baffling array of buzzword driven "agile" development practices and project/squad/team/chapter manager/lead/head roles to be filled. The more you kept inquiring what exactly those roles should entail, the more conflicting and vaguer the answers got until you realized that no one had the faintest idea how any of this was supposed to actually mesh together in practice. The license packages for the expensive project planning software we were to use where long paid however.
Raised in a christian tradition, and somewhat a student of history of christianity, I was fascinated by parallels that unfolded in months` time what took place over centuries in christianity.
5 years after the movement began, I found that it felt as watered down, abstracted, and diverse as christianity spent 2000 years becoming. I could bond with a fellow programmer as an "agilist" and at a very vague level, there was some abstract similarity (e.g as christianity might be to "be kind", agile might be to "be lightweight"). Everyone took the parts they wanted and formulated what fit for them. Which was/is both a good thing and a bad thing.
Perhaps we could call it, "social virtue mythology".
What is a social virtue mythology? It answers the following questions:
* Why is there evil in the world? (ie., why arent things perfect)
* How do you overcome evil? (ie., become perfect)
* Who is good? Who is evil?
* In what or who should I place my trust?
* etc.
Today we can see many such competing virtue mythologies....
Why is there evil in the world? New Atheists: Religion; Feminism: Patriarchy; Socialists: Capitalism; Idenity-Woke: Essentialism.
Personally I regard this as wholely "mythological" because the questions these social-virtue systems are designed to answer "take place" in a utopian/dystopian reality.
Ie., many people eventually discover that there is no need of this type of mythology: reality is itself flawed. We aren't "owed" a utopia, and thus there is no question to answer.
Here, in agile then we see these patterns even in this thread. Some commenters are still trying to answer the question, "why isnt software design/construction perfect?" as-if there can be a virtue/vice answer, ie., "because we have failed to...".
Rather, no. These problems are irresolvable. The reality of software is intrinsically broken: there is no customer to give you the requirements, there is no habit which elicits them, there is no mechanism to build reliable software, and so on.
Many are extremely loath to confront this reality, especially as adolescents; ie., that they are powerless to build their utopia. The really-existing world has irresolvable contradictions that admit no virtuous resolution.
The "scrum master" was a dedicated role, filled by a technically able person, who sometimes helped a little bit with the coding. This individual had read several books and taken a course on Agile. They were sincerely passionate about Agile and wanted to implement it effectively.
The "project manager", a DIFFERENT PERSON was part of the "business", and interfaced primarily with the scrum master, but they were with us for several hours of both planning and retro.
Our sprints were 2-3 weeks. At the end of the sprint, we spent AN ENTIRE DAY on retro. Before the next sprint started, we spent AN ENTIRE DAY on planning. We used a physical board which was matched by the tracker. They were kept in sync by individuals and validated by the DEDICATED DOCUMENTATION PERSON.
We broke down the tasks relentlessly, until there were almost no 3-point tasks, maybe one or two 5-point tasks, and everything else was 1-pointers. I think this one is the most bang for your buck, lowest hanging fruit, easiest to implement winner. Someone has to enforce it vigilantly until everyone is used to it.
Those three things are, IMNSHO, the keys to getting halfway decent Agile going.
Edit: Was one of the least soul-crushing and nicest experiences in professional software development. The planning meetings were a serious effort, but also a pleasure because I liked everyone on the team, and the large time windows made it not seem rushed, so we had time to joke around and shoot the shit.
Example: You want to improve performance. Somebody will have to fire up a profiler and measure things, and then figure out which hotspots are worth optimizing.
I totally get breaking it down to "set up profiling tool", "do profiling runs for workflow X/Y/Z", and "identify low-hanging fruit". Presumably these tasks are not going to take the entire sprint.
How do you plan the next step in task format? Do you not do any further work on the topic until the next sprint, where you then plan out concrete "optimize function X" tasks?
If not, i.e. whoever does profiling also gets to optimize code during the same sprint, how do you account for that in planning? Three dummy tasks "optimize a hotspot" without any actual idea what it will involve? Maybe it requires rewriting a component to use different data structures? Maybe it's just a single JVM config tweak? "Could be a 1 or it could take my whole life?"
Maybe it's just less common in certain domains, but this kind of "we need to investigate this issue and then decide what changes to make based on the investigation (and good engineering judgement)" - which is bread and butter in my part of the org - does not seem to fit into the "tasks with points" concept at all. Well, unless you want to always spread the process of "investigate issue", "identify problem and solution(s)" and "implement solution" over a whole month. That seems soul-crushing to me.
To be clear, I'm very interested in seeing how to make this mesh with Agile. But it feels like trying to write a binary search in a language that has no indirection or conditionals.
Given the "about one hour of work, give or take" target point value we used, there weren't many tasks which had more points than that.
It isn't.
Soul-crushing is doing "Agile" where the "scrum master" tells you how long each ticket will take, where the client is invisible and never involved, and where retro is 1 hour at the end of the sprint and focused on blame.
The time spent in those retrospectives and planning sessions are about the technical aspects - how talks should be broken down, how overall design of interacting components should look like. It's 100% actual work.
I'd take these over another bs "status sync" any day.
I'm getting ill just thinking about this. Two whole days wasted every sprint!
And it was not "16+ hours of meetings", but two days of taking a break from coding to do more social work, also having a 2-hour lunch together in the middle :)
I will fight, in any team I'm a part from now on, to avoid the interminable scrum rituals. They are soul crushing for most good engineers I know. I never ever want to do them again.
Also the pattern I’ve seen is over time businesses want more and more fine grained control over the engineering process, as if treating the team like replaceable workers doing well defined tasks (like Ray Croc I guess) leads to more efficiency.
This doesn’t work when the burger and chips has zero cost of replication and your staff are not frying and serving chips, they are inventing new cuisines every day.
Agile is a risk management method.
You won't build things faster with agile in general. It's aim is to decrease the chances of spending 80 manyears on a project and then realising at the end that you've built the wrong thing.
It was refreshing to hear their take on project managers, as it matches how I’ve been feeling recently. They schedule 7 checkpoint meetings per week and I’ve given up on attending, I just can’t do it anymore.
I hope that we can move past agile into a new world that is free of the baggage and process of yesteryear. Even if the core principles are the same, let’s call it something different so we can start fresh and impress upper management.
In fact, all agile needs is good retrospectives, and all the rest you can decide there.
It’s like shaping a system of people. I think of it as 50% of the development endeavor at larger shops.
Retros are used to identify ”technical organization debt” starting to build up, in a way.
[0] A couple of said theory tweets, from memory
And here lies the difference: agile includes retrospectives. I It's the only thing you need for agile. And if you don't have that, you don't have agile.
I would put things differently: micromanagers leveraged the agile and scrum buzzwords to justify and convey an aura of legitimacy to micromanaging.
I mean, how many orgs forced daily meetings where workers are forced to enumerate what they did and delivered in a short time frame, but mysteriously left out the part where a feedback cycle exists to revise plans other than performance evaluation plans?
It seems to me, that all these buzzwords and meetings are just a way to always keep you "busy" (or at least looking busy), to always have some kind of fire below your feet so you don't "slack off" (or have a bit of free time to actually gain or sharpen the skills you need for the same company...). I'm just tired of this. In a way, good teams will be good, and bad teams will be bad, doesn't really depend on whatever process they have.
Agile is very open to attack when it veers from a list of useful practices to try out in a given context and instead turns into dogma.
This is one example how processes supercede people even though that is against the original agile manifesto.
Often it seems that people talk about the 'micromanagement' and 'wasted time' as if they would not enjoy the daily. To me this is a warning, that something within the setup is not right. In general, micromanagement requires hierarchy, but I don't recall from the Scrum Guide [1] that a manager should be present or that directions should be given. So who is managing who?
If done right, there should be absolutely no reason why there should be negative emotions involved: The devs get together and talk about their work and the challenges they want to share. It is about fostering communication.
Because the daily standup can only be X minutes, someone always gets cut off, causing negative emotions.
On the one hand, it is the reason why Scrum is so successful in corporate environments, on the other hand it is also somewhat non-agile. That is why I tend to call Scrum the least agile Agile Methodology and a good starting point for Agile transformations.
Just don't forget to keep moving on as your Retros lead you the way.
For example, teams running Agile are very reluctant to give both a delivery date and a fixed set of features. They're willing to promise one or the other, but not both. And that's a problem, because the whole rest of the organization really wants to know when they can announce the release to customers. Planning for releases tends to be a really big deal, the pressure to make promises of both functionality and delivery dates is very strong.
Also, Agile methodologies tend to assume you are in close contact with the customer, who can tell you what they actually want and decide how things should work. But this is rarely the case; typically you have a PM or something like that who is in touch with the customers, and is supposed to understand what they want. But this person is rarely senior enough to actually make decisions; important decisions are made by a dev manager or a project lead or someone like that, with the PM just providing input and perspective.
Finally, Agile has this notion that everything is supposed to be handled informally, verbally, person to person. And that's fine for small tweaks to the product. But as soon as you start building something big enough that it takes months and crosses team boundaries, it gets really useful to have an actual document explaining how something is supposed to work. Such a document is an invaluable record on what has actually been decided, and anyone joining the project or wanting to contribute to it absolutely should read it. But Agile tends to discourage creating such documents in the first place.
Given that Agile clashes so hard with how actual organizations get things done, it rarely gets adopted in anything approaching a pure form. The practices that tend to get picked up might be called Agile-light: sprints, daily meetings, and task estimation in points.
Businesses don't "get things done" by operating as if they can have both. That's why projects fail, businesses cut corners, and then inevitably ship broken products or don't ship at all.
Most of the "agile methodologies" are nonsense dreamt up by borderline con artists attempting to sell their services. The fact that it's an either / or choice - and that nothing can get around that choice - is inherent in the nature of the work.
The best I can come up with is the notion of a double contingency plan. Engineering agrees to a set of functionality to be delivered and a delivery date. This is inevitably going to be a bit optimistic, because people consistently overestimate themselves.
To deal with that, the first contingency plan addresses the question of what should be done if things are not converging to the ship date. The plan here is to keep the ship date, but ask hard questions about what bits of functionality actually need to be kept. What are the actual P0 - MUST HAVE features?
The second contingency plan addresses what is to be done if the first one fails. At this point engineering has already done all they can. They have pushed as hard as they can, and they have deferred every feature that is deferrable. They are down to the actual MUST HAVEs. Now the rest of the organization has to figure out what to do with a product that is inevitably going to be late. What is the alternate ship date? What customers are going to be really unhappy. And so on.
It seems to me any large engineering project should think out these contingency plans in advance. What will they do if making the deadline starts to look daunting? And what will they do if making the deadline turns out to be impossible?
Yes. Kinda :) But the issue isn't want. The friction is actual make-an-impact business needs.
It's easy to sit around a conference table and spitball wants. But trying to bin that person or group down to specifics as well as resolve disconnect and inconsistencies between the ideas and faces go blank.
Customers want the luxury to spit out wants without having to take the time to knuckle down and do the hard work of defining needs. They have a "you figure it out" attitude. That leads to assumptions. And assumptions lead to dysfunctional product / features, for which the engineers get blamed. Again.
Agile is a worthy solution to this problem, but when IT is treated as a service provider and not an equal partner the tool (i.e., agile) gets mutated to a point it's not truly agile anymore.
Extreme Programming suggests having a customer on site. In Scrum the Product Owner is merely a substitute for the Customer.
The agile manifesto values working software more than comprehensive documentation, but does not forbid documentation.
So why do companies and their devs misunderstand this?
Pretty much wish this was a defined part of agile, most of the time all my boss/pm is good for is maybe reminding me of meetings. They never understand what I’m working on to help, but they have power to make giant decisions and elevate problems I have been waiting for the client on for some time. Idk if this is a common theme.
I read a comment here a while back I agree with: your project manager, a good one, should remove obstacles that could slow you down, and anticipate them. They will only really interact when they notice bad patterns; like you being too slow. And most of the time let you be unless you introduce to new team members or elevate issues. Focus on your major tasks, allow them to manager the small mundane.
I hate being micro managed as much as the next guy, but I've also had way too many coworkers over the past decade who can't seem to just sit down and get the job done. They fiddle and tinker and explore way too much. It's helpful to do that at times but many lack the discipline to produce software without a PM to answer to. I wish it were different, but it isn't.
My current PM is great and if I found out the management wanted to get rid of him, I'd vouch him and go to war if I had to. Good PMs are indispensable, in my opinion.
It is team that are working intelligently with trust between members. Each member does it job, feels responsible and autonomous. Enough to do by itself everything needed for the project to succeed and with other members trusting him to be able to do what he needs to do without micro management.
The tough part for me has been experiencing a team like that, then moving and having no idea how to help create a similar environment with my new team. Any tips?
Also, in my experience, most of cases like this happened in places were the management was kind of "absent" or "defective" in some way. For example when you have the next level of management hierarchy in another place/country. In such a case, often, the dev/engineers are considered autonomous and they are just trusted and expected to "take care" of all the technical things, with little supervision.
If you are already inside a team, I think that there is little to do to create a similar environment, except trying to get the maximum "power"/"position"/"space" from the company, and using that to organize yourself with the persons you are interacting with in your team.
Trust is the key element. But other members have to earn your trust, and on the other side, you should also ensure to worth the trust of your company by being reliable and deliver. Most of the time, even management will let you a lot of "space" if you are use to reliabily take over "loads" off of their shoulders.
The worst case scenario is when there are persons in management that have "bullshit" jobs like "scrum masters" where they have to do "something" to justify their work.
Otherwise, if you are in the lucky position of creating a team, the best thing is to try to have a multi disciplinary team with most of the people with a specific responsibility on some components of your system.
This is the exact opposite of saying that you have a an agile pizza team with all members that should be equal on all subjects and easily be interchangeable at will on whatever subject...
There is also then the wildly misunderstood aspect of estimation. In my experience the only value beyond saying a feature is easy or hard is that it does spark discussion among engineers, the irony being if said discussion goes on for too long it usually gets shut down. Adding estimates to work you are going to do within the next 2 weeks seems of limited value, it changes no behaviour you will still do the work.
I feel we would be better served by reinforcing the principle that all of our purposes within the organisation is to deliver a quality product for the customer.
Product development is responsible for taking ideas and generating specifications for product features while software engineering are responsible for taking specifications and implementing them within a software product. This simple process should be monitored and systematically improved to deliver better on the core principle.
I feel like they had a great idea, gave birth to something, but then failed to nurture it and we are still evolving a solution. Ok we got scrum but we all know that’s a complete shit show.
What agile is really missing is that it doesn’t touch on leadership. We like to blame management and say they should let us do what we know, but we don’t lead. Decentralised control only works when everyone leads.
Let’s add a line to the manifesto and fix it:
“Lead from the bottom over centralised control”
1. Big Ball of Mud: work without a plan
2. Waterfall and alike: make a very elaborate plan and stick to it
3. Agile: work without a plan, but promptly refactor issues caused by lack of foresight
Waterfall worked well but is too slow to respond to change, so Agile replaces one-off planning with iterative refining. So far, great idea.
However, if the backlog is never short of business requests, and business requests are always higher priority than refactoring, then when will refactoring ever be done?
If we don't have architects anymore, but also don't designate time to refactor, we are de-facto doing Big Ball of Mud with stand-ups.
I could just as easily do this in a notebook, but it wouldn't be indexable or searchable, is all.
To me that's an indicator that decision makers in those companies have no understanding about software development.
> The important piece that gets forgotten is that Agile was openly, militantly anti-management in the beginning.
Yet here we are: The process just became an endless rabbit hole of filling jira tickets, workday schedules, and a bunch of other systems(including a shared google sheets table as it was in a company I used to work for with 5000+ employees) which easily eat up a day of your week. Detrimental if you are aiming for a deadline. And while I embraced wfh and felt insanely comfortable with it, it was used as an excuse for management to further push those things further down everyone's throats. The end result was that, at least from an engineering prespective, work became more of a marketing team meeting, filled with absurd abbreviations and processes which hold no value and only drag you down. Another problem is when you get an email from jira that someone assigned you a ticket called "Implement X in Y" with no further explanation when you click the link. It's absurd that in large companies(think 250+ people) your value is determined by how many tickets you've ticked off. "Implement X in Y" seems like just another one, but in many case those 4 words end up being 8 separate things, each of which can take several days but no one cares even if you described those in detail in the comments. You might think that story points solve that but they often don't. Imagine the following scenario: you are tasked to hook up a system to an s3 bucket to store data but you have restricted access to the aws console so you can't even create a bucket on your own. Then it turns out that the vpn is set up wrong and you can't even resolve the host if you are connected to the vpn. Then it turns out that the library you use has some bug in it or it's incompatible with some of the other libraries you use so you end up upgrading a bunch of libraries, patch up a large codebase and only then be able to get on with the s3 bucket thing. At first glance what you would assume is a 2 hour task tops ends up being a week and a half of your life. And it can be insanely demotivating when you have the same thing to report that you are working on the s3 thing in your daily scrum for a week and a half. Which is even worse when your managers have little to no tech skills beyond excel for instance. The point of my rant is that agile became the thing it was supposed to destroy - a time tracking system for the assembly line.
Edit: those things don't seem to apply for small companies/startups because... Well they are small and you get to know everyone on day one.
One subtle, but necessary, effect of the original scrum sprint planning is that it splits accountability between product owner, team, and master.
The effect is that, upon failures, there is a way that developers can rightfully blame the product owner or master. Part of the point of time-boxing increments is checkpoints of accountability, so that once the "contract" of a sprint has been fulfilled, the devs can not be held accountable for its product value.
If this mechanism is not in place, it means the product owner/master (or whatever title) is just another name for plain old manager with mandate to overrule anything, and in the worst cases with influence over careers of the devs. This is the high entropy state most organizations converge to. In these cases, all product failures will be blamed on the devs (in effect, if not verbally.)
In an effective agile organization, there can't exist mechanisms where blame/consequences for failures can be shifted downwards postfact. But this is also the basis for most old organizations. Hence, implementing agile in older organizations will never work.
The main idea is that (as a programmer):
"You don’t give a fuck about process name, until you have full rights to modify and change the process."
[1] https://medium.com/fibery/post-agile-process-agnosticism-d4a...
The customer had a problem, he knew he wanted a computer in the solution. The first person he contacted to solve the problem hired me to write the solution. He hated the hardware chosen, and went off to make is own choices.
Having chosen acceptable hardware, he hired a different person to solve the problem. They also hired me, we pitched a solution. It took about 2 months to code.
The program worked as promised, met all the requirements. Up to this point, it was essentially waterfall. We agreed that a wider sale/deployment would be contingent on me working directly with the customer on-site in an agile fashion.
I wrote code, he brought a random power plant worker in to try it out (in the days before everyone knew how to run a PC), and we iterated. Everyone was quite happy with the product for years, until networks, Windows and other reporting systems made it obsolete.
Nobody knows the details of what is useful until there is a prototype to work with, and get the feel of. Agile is the way people have been solving problems forever.
There seem to be a lot of "No True Scotsman" threads here. Since I did this before "agile" was a thing, I don't have to worry about it. 8)
It is important to recognise this, and carefully manage the necessary give and take. Finding the right balance is tricky and higly context-specific.
Customers may think that "Agile" teams are stacked with experts all working in a state of grace, effortlessly pulling off technological miracles.
Behind the scenes, however, most of the work may have to be delivered by junior devs under the supervision of stressed-out micromanagers who use top-down bureaucratic control to progress the work.
This is what we do at the consultancy I work at - billing is simply based on time & materials. We work from our customer's premises (when there isn't a global pandemic) and we have people from the customer integrated into our team, so customer collaboration is constant and continuous. Feature creep doesn't really happen, because all parties know what the budget is and how much our time costs.
In my experience, the customer collaboration happens through a proxy called the product manager. It is their job to hold meetings with customers to figure out priorities.
You don’t “prevent” feature creep ever. It is part and parcel of development. You can, however, manage it by restricting when the goalposts change. In Scrum, that’s at sprint boundaries. The most important bit is that the customer chooses priority but the developers choose velocity.
> Responding to change over following a plan
“Responding to change” is treated as a right to bust WIP at any moment in time. All engineers I have worked with want a roadmap and a plan that has enough room and flexibility to be changed due to some external circumstances (bespoke “change”).
> Working software over comprehensive documentation
This statement puts two characteristics of a software product (be it SDK, library, source code package, binary, whatever) into contradiction. Working software is accompanied with comprehensive documentation.
> Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
Late requirements change is what all engineers hate… And as stated in a number of comments here customers want no more than a faster horse. Best products ever made were not made based upon focus group feedback.
> The best architectures, requirements, and designs emerge from self-organizing teams.
Are scrum masters an antipattern then or just a sign that the team in question is not self organized and immature?
Bottom line: any “agile” methodology I’ve come across and experienced myself for the last several years is a mix of cargo cults, religious ceremonies and abusing the word “agile” to justify chaos, incompetence, ignorance and arrogance.
What in essence all the agile methodologies are meant to be is the domain specific extension of the PDCA cycle.
- tech companies want to push decent features live every 2 weeks (the usual sprint duration). They follow the mantra “move fast and break things”. Technical debt is fine for them as long as money flows. They want to beat their competitors. This makes totally sense from a business perspective, of course
- certain tech people (in my experience) don’t want to move fast and break things. They don’t want to break things, they want to produce products of quality, stable, with minimal defects... if that comes fast that’s fine, but usually it takes time to produce something of quality (at least for average developers—-I am one, and the vast majority of developers are). This makes sense from the point of view of someone who cares about their craft
That’s the mismatch. Companies love Agile (even if it’s done wrongly); tech employees usually hate the very notion of estimates, sprints (are we even in a rush?l, story points, etc.
Since no one does pure waterfall, everyone does Agile.
Even this essay encourages that interpretation: "Everyone in this group had deep experience writing software, and they were all looking for an alternative to documentation-driven, heavyweight software development processes that ruled the day."
Did that "everyone" include Mac development in the 1980s? Or Amiga development? Not from what I've read of the history.
Did it rule the day for the development of gcc, the Linux kernel, Perl, or CPython? No.
I pulled out my 1996 copy of Steve McConnell's "Rapid Development." Page 271 says:
> In addition to providing explicit support for morale, Microsoft gladly trades other factors to keep morale high, sometimes trading theme in ways that would make other companies shudder (Zachary 1994). I've seen then trade methodological purity, programming discipline, control over the product specification, control over the schedule, management visibility -- almost anything to benefit morale.
The Zachary 1994 citation is the book "Show-stopper! : the breakneck race to create Windows NT and the next generation at Microsoft" at https://archive.org/details/showstopperbreak00zach . A quick read (search for 'documentation') shows Windows NT development did not use a 'documentation-driven, heavyweight software development processes'.
And of course oodles of small software projects in the microcomputer and early web era were written with no formal software development processes at all.
This "rule the day" statement must therefore be interpreted within a narrow context.
Was "Everyone" doing it wrong? Mathematically improbable. Were the vast majority, the mainstream, stuck in a dysfunctional waterfall system? Oh hell yes.
And the usual answer to the failure of up-front planning was "Clearly we didn't plan up-front enough, we need plan even longer". It's a self-reinforcing pathology. So is the use of JIRA, but that's another story, post-agile, not pre-agile.
Utter nonsense. They were consultants looking to sell consulting services. Just in the next paragraph:
> Many of these people already had a methodology they had created and/or were proselytizing.
When “Agile” ideas are applied poorly, they often lead to more interference with developers, less time to do the work, higher pressure, and demands to “go faster”. This is bad for the developers, and, ultimately, bad for the enterprise as well, because doing “Agile” poorly will result, more often than not, in far more defects and much slower progress than could be attained. Often, good developers leave such organizations, resulting in a less effective enterprise than prior to installing “Agile”.
I can totally relate with this experience in a previous company I've worked with. They effectively starting the transition to Agile, but the execution was so poor and the general culture was so immature with a lot of chain of command and top-down that it ended up being a total toxic nightmare. I end up blaming Agile for this at first, but now i'm realizing that was not the tool.
I tend to work that way, now.
Myself.
Things get difficult in teams. Especially cross-disciplinary teams, and aggregate efforts.
Much as we like to denigrate managers (DISCLAIMER: I was one), they are necessary; and not as a “necessary evil.” Good managers can be amazingly effective and well worth it.
Unfortunately, like in any vocation, the good ones are rare. The same goes for good software developers.
The Agile Manifesto was written by a group of working software engineers, with decades of experience, at the top of their game.
Precisely the type of engineer that today’s hiring process tends to filter out. They don’t really represent the current field of practitioners.
Like so many “perfect world” scenarios, the Agile Manifesto was designed for a “perfect audience.” As the article points out, when the manifesto hit the real (imperfect) world, the wheels fell off. To make matters worse, the designers tend to get huffy, and blame the people making a hash of it; which doesn’t make friends. They may be technically correct, but humans are messy, chaotic beings, that tend to have highly individual worldviews, abilities, intelligence, education, experience, workflows, and motivations. Most folks expected to actually implement an objective bear little resemblance to the ones that developed the plan. If the planner fails to account for this, then (in my opinion, as a planner) it’s the planner’s fault.
That kind of sums up human history. Groups of elite thinkers develop a Grand Plan, then it gets bloodied and bruised, once it hits the proletariat. Sometimes, with horrendous results (Cultural Revolution, anyone?).
As someone that actually designed a successful system for the proles[0], I can report that designing real-world solutions for a distributed, heterodox, self-driven, target audience is difficult, messy, unintuitive, humbling, frustrating, and, quite often, absolutely infuriating. Not the kind of work that folks at the top of their game like to do. Frankly, it sucks, but I think that it’s also the best way to make something that actually has a chance.
I wanted to add that follow-through is also important. When I designed the system I referenced, I did it with an understanding that it would be a years-long commitment. It's like having children: Making them is fun. Having them is easy. Raising them, however, is not fun, and not easy.
But we need to do it. For myself, I spent ten years, traveling around, giving presentations, classes, gladhanding, answering questions that I thought "should have been obvious," suffering some withering attacks, and fine-tuning the system, as I realized I screwed something up. I also went around, looking for successors. I wanted to become obsolete (which I have).
But that’s just my experience. YMMV.
Good Xs (where X = developers, managers, designers, etc.) can be amazingly effective and well worth it. This is obvious. The vast majority of managers out there are average, and so they are a necessary evil. Compare this to the situation: the vast majority of developers out there are average (I include myself in this group )... and this is fine, they are not seen as a “necessary evil”.
So my next task, obviously, was to define the content of all the sprints and write them down in the proposal. I was well aware of the absurdity of the whole thing but did it anyway, carefully planning out what would be done in each sprint. Sure, it sounds stupid, but just look at how cute it is! It's like a whole bunch of little waterfalls! People pay good money to see such nice waterfalls, you know!
Of course I'm assuming that they'll kick us off next week. Right now the project is in the "let's delay starting until missing the deadline is guaranteed" stage. One more week, and then my careful planning will have become officially impossible and it will be panic all around, and demands that I work days, nights, and weekends to make up for the delay they themselves caused.
Do you want to guess who isn't allowed to go on holiday because of this incredibly urgent project? :-(
Delivering sooner, not faster.
To do this, decision makers must be close to the team, and ideally a part of the team. Either managers are part of the team, or the team a given the authority to make product decisions.
Full time managers lose power and influence either way, and I think this is the real reason Big Corp never really bought into it except on paper.
Many of Agile’s problems arise when you begin to apply them to long-running products that undergo continuous cycles of improvements - stretching years rather than months.
And what we got from that through corporate senior management involvement is some sort of “scalable agile framework” or a minimal surrogate of “agile scrum”.
Might not be ideal, but it still generously beats the waterfall with quarterly “slave galley-style” weeks of overtime to make it to “the release” and to be able to deploy things that actually already became irrelevant 5 weeks into the release cycle.
Similarly, consultancies need to sell "products", i.e. implementable processes and change, so again principles are a problem.
Lastly, in a lot of places the actual users or product owners from the business side are only involved via representatives, so there is no closed loop to run.
Arbitrarily chosen pig and chicken link: https://helpingimprove.com/agile-commitment-scrum-pig-chicke...
http://darkagilemanifesto.org/dark-side-of-agile-janes-succi...
It’s the sad combination of not being open to test other approaches and condemnation of people questioning it.
Most common is the manager or consultant pushing Scrum who doesn't even know what the Agile manifesto is.
Dissident manifestos tend to share a the same failure point. They're heavy on critique of the system that they want to overthrow, hand-wavy on the system that they want to enact.
The Communist Manifesto, to pick on the obvious example, is mostly about capitalism, other (wrong) versions of socialism.. notably reactionary socialism. It's also about the struggle to bring about revolution. There's surprisingly few paragraphs about what communism will do differently, once the revolution takes place. Liberal revolutionary manifestos was similar, with French and american revolutionaries settling (or not) on a system of government only after overthrowing monarchy.
The agile manifesto does outline a lot of how agile works, but the most compelling part was its description of "waterfall" and all its ills.
Liberalism and communism, in their idyllic literature, never really contended with the fact that these were alternative methods of "ruling over a population." Agile never contented with Agile was as a people management system. Well... Agile is a way of telling developers what to do. It is a people management system.
Where its completely unlike the political analogy is that agile was derived from working systems. It wasn't just an idealization. Ironically though, it ends up in the same place. What worked well for particular teams became the ideals.
I think the ultimate stinker though, was the "political economy" of agile. A company where everything works great isn't going to tear out their process and bring in consultants to implement a new management system. It's companies that are failing to produce software well that do this.
The problem is: the manifesto states clear dichotomous preferences without nuance, and it certainly not for example states that it embraces well-maintained and useful documentation but that "[we value] working software over comprehensive documentation".
See? It is written "comprehensive".
Not "shitty".
And I even agree that maybe "comprehensive documentation" can be a dangerous concept if taken too far, but that's pretty much true for anything taken too far, and actually I have never seen a problem in this direction, but pretty much all the time and extremely often in the opposite way (I could say : a comprehensive lack of documentation...). So maybe de-emphasizing "documentation" was not that smart...
Either it was very poorly written or there is an attempt at rewriting history. At the end of the day, you can't just hand wave the shortcomings originating directly from the text by just stating "oh but we did not mean that, wasn't it obvious?" -- well... no! it was not!
Likewise for the other "values". They seem to have been written because of a perceived lack of balance (and maybe there are even again some adjustments to be made and they are not exactly intended to mean what is written...) in the personal experiences of the signatories, however there is no provision against balancing too much in the other direction, or against inappropriate applications in specific contexts (the "preferred values" are actually extremely context and project dependent)
So maybe people actually got what was written, or maybe the whole thing was so vague that they got random outcome from a mostly cargo cult application. And in a few case, intelligent practice in a good context, so good outcomes; but when that happens, would the successful team have done poorly with other preferences of "values" or with other stereotypical day-to-day practices (weird btw when processes are to be deemphased)?
> What do you mean you’re going to start building the software before you’ve gathered all of the requirements and estimated every piece of functionality? That’s insane!
Like, the significant software from the 80 or 90 was written like that (in inappropriate contexts)?
And when it was (again, in inappropriate cases), was the solution to settle on the four stated value preferences? Why? From where does that even come from? Where is the evidence? How can it be applied? Why can't I write precisely the opposite and say "see? here is my solution!" -- if not just because of the obviousness of some outcome: yeah not valuing working software enough will probably not lead to better working software. My point, however, is that stating that you value working software more cannot yield to actionable things in not completely dysfunctional environment, because whatever their current practices are everybody will say they want that and what they are currently doing has value to achieve that goal. And the other value preferences are highly contextual.
There is no silver bullet. Not in technical areas. Not in project management. Especially not when project management is not interested in the technic; that is: in the project, if the project is of a technical nature...