But the author's criticisms of the incentives of Scrum are on point I think. Because the stories are always articulated in terms of user facing features they encourage developers to hack things together in the most expedient way possible and completely fail to capture the need to address cross cutting concerns, serious consideration of architecture, and refactoring.
This is how you can get two years into a project and have managers and clients that think that things are going well when the actual code is an increasingly unmaintainable rats' nest. Good devs confronted with this kind of mess will eventually burn out on sticking their necks out defending necessary but opaque refactoring tasks and move on to greener pastures.
There's some development work to be able to capture these things, but it gives visibility and power to Tech to inject some maintenance work into normal sprint development. I'd love to hear what other people are doing.
This is where I suspect manager-types and developers have a vigorous divergence in values.
Professionals routinely encounter situations where something is wrong and needs to be "actioned" but its wrongness isn't effectively measured by any metric (other than the opinions of the experienced people looking at it).
There is a certain species of manager who has taken Taylorism a bit too far and says that anything not reflected in the KPIs is not real and not getting acted upon. I hope this isn't what you're expressing here, but oh man is that mindset frustrating.
We talk about organisations applying Scrum but what we're really referring to are groups of people, and depending on those people, their mindset, their history and their current role requirements - they will use the Scrum model in their own idiosyncractic way.
Take the point about technical debt and running so fast with an Agile development workflow that you never get to refactor code or properly document etc.
Even if you just took that in reference to one specific company, the value of, cost of, and importance of these things could be different at different times.
When a startup is running to get MVP out to get customer feedback, it's usually much more efficient to set the basic expectation that the first version of your product will get thrown away once you've learned all the important aspects of what you need to deliver. With that in mind - startup development is a completely different beast than when your product is established and you have paying customers with expectations of up-to-date documentation etc.
Some managers are do not have good people skills and manage by just holding developers to account based on the expectations set in their Scrum planning day. Discussions about whether refactoring should be done yet or not may not even be something they want to know/talk about and they rely on engineers to factor that into their sprint estimates.
Depending on whether the company is being driven by sales/demo opportunities, or feature roll-out timescales etc. then the call about whether to do something 'quick' or 'right' may drop either way.
For developers it's important to understand the dynamics that are driving the business and what their role is (sometimes you have to 'JFDI'). If you have a good engineering team with a strong leader then these things shouldn't be that visible to anyone else. It's also important to be able to meet the expectations you set. If you keep delivering late then you'll also struggle to get people to let you do more than the minimum required at the time.
For the business as a whole it's all about the big picture and understanding the decisions you make and what their impact (short and medium term) is.
If you want quick releases and don't have the resources for that to allow for good documentation and/or scalable code, then you need to understand that technical debt is building up and at some point it will need to be paid.
Utterly dysfunctional of course, but if it's do-or-die, I'd prefer to 'do'.
"Oh, since we're adding new email features, we'll need to clean up some of the old email code. That will add additional complexity to this task, so we will estimate it higher."
It works, allows us to still track the impact on velocity, keeps everyone informed, and makes technical debt clear and trackable.
I'll add that some subsystems are a mess but stable. It's difficult to make refactorings with other changes when there aren't other changes to make. I see this in particular with old but stable subsystems that will need to be ported to new environments someday, but it's never a good day to lay the groundwork for that inevitable change.
If the organization is not sane, that's another thing of course (and often the case, sadly).
The fault is almost all on the management side because, ipso facto, they're the ones making the decisions that drive this outcome.
In the real world, there are outcomes that are much worse than this. But this does have the effect of neutralizing the value of scrum: Management gets what implementers decide they get, and scrum is just the way middle management is told the story that they pass up the chain.
It's the only way that seemed sane without twisting user stories into weird refactoring or architecture stories and keeping team from writing garbage code waiting for these "tech stories" to clean it up.
The PO should try to balance this with feature impementation and the team + SM should make it clear when it's needed. This seems to work fine in an environment based on collaboration between PO,SM and dev team, but will probably fail in an adversarial one.
If I had asked beforehand to invest time for this, the response would be "let's postpone this (forever)".
Let's just remember that Scrum is just a tool that is trying to replicate some practices of the toyota production system (TPS), and an important principle of the TPS is continuous improvement, and Scrum has this through restrospectives. Another principle is quality built in.
Now, how to avoid having a rat's nest after 2 years?
First of all, you have to remove velocity as a goal for your team. It's easy to game, and it de-incentivizes finding a _predictable_ velocity, which is the goal of that tool.
The goal of velocity, is to know what's the "cost" of building an healthy system.
i.e.: Your team ships an arbitrarily 10 points per sprint by doing what they think is right. The same team could ship an arbitrarily 20 points per week. 10 is the _true_ cost of building a healthy system. There is not a lot more to do for good team, honestly, at that point. Management can't say anything, because you bring them predictability, and that's mainly what they want. They might think you're slow, but hey, all of this is relative.
Suddenly, what does that mean when your team only ships 5 arbitrary points? That they had to push extra. That there was an unexpected problem. That things were on fire and you had to stop working on features entirely.
Basically, that there is something to retro on, find the root cause and anticipate for next sprints, and use to go back to the healthy level of points.
I used this at several places. It demands trust, but it always pays off.
With some teams I worked with, we set up a technical prioritization planning meeting every week, where basically decided what was going to be prioritized for that week. Each team was picking 2/3 things to do on top of the features.
You don't tell anyone. You do it.
Unfortunately this has become a cliché whenever someone criticizes scrum, Agile in general, etc.: it's never the fault of the idea/process, it's the fault of the manager/team for not understanding it properly or not doing it right.
Even if these philosophies and processes are so amazingly great when properly understood and implemented, the fact that hardly anyone seems to be able to properly understand and implement them would be a fatal flaw.
In my experience, it means some developers failed to game the system that particular time.
But you can be sure there were many other problematic moments in previous sprints, but the developer could hack together an ugly mess of a code to avoid the shame of failing in front of everyone in a meeting, and dragging the team's points down.
Because the unwritten thing about points is that they are used to shame people. In public. Sometimes not explicitly, but the feeling is there. It's always there.
This sort of thing is only useful for managers who don't understand what is going on. Since they are unable to look at code and understand it, 'points' give them a semi-opaque substitute.
If management trusts the team, it is not necessary.
Mostly joking but plenty of places do work like that.
Why can't we make some sub-component this sprint then the UI bit the next?
I tried various ways of reframing it such that the developer of the UI be the "user" but it didn't wash.
What if you finally hook up UI to the sub-component and the customer/stakeholder decides they don't like any of it? You could have known about it earlier.
Are you surprised? The developer isn't the user of the UI, product wise, so no wonder it didn't get very far.
The trick in this case is to break the story down. So the original story isn't do-able in one sprint? Ok, so what is actually the MVP of that Story? What's the first block that builds the overall feature? Take a 5 point Story and make it three 2 point Stories or something. That's what the refinement step is for in Scrum.
Unless you're doing abnormally short sprints there should be some part of that Story that can be abstracted into a smaller Story that fits into a sprint.
Because that's how you get bad UI. The user-facing design needs to drive the API interface, not the other way around.
Build Quality In
Optimize the Whole
I've never practiced it and my only exposure has been reading some of the book "Lean Architecture". But it certainly seems like an improvement over Scrum which seems blind to some of the most critical things in developing complex innovative systems. In fact I'm convinced that Scrum was designed to work with basic information systems where a feature consists of essentially adding a new data entry form or report for a database (e.g. a lot of web stuff)
> if a user story requires a refactor or architectural change then that's just fine
https://en.wikipedia.org/wiki/Concentration_(game)
As though a useful, usable product design (architecture) could be divined thru piecemeal revelation.
The debt dial should be from 0% (spend all the time refactoring) to 100% (spend zero time refactoring, get it out at all costs).
Management should have complete control over the dial. Developers should have control over what kind of refactoring they do (ideally the retro should have a question "what debt/tooling issues caused you the most pain this sprint? and a parallel track for debt/tooling stories").
I'm not sure I ever found myself in the position of writing bad code just for the sake of speed - I surely wrote tons of bad code because I didn't know how to do it better, or because I didn't have the requirements clear from the start, or because of bad design and planning. But to get a feature out quicker, no. If anything, it seems to me that writing bad code requires more time than writing clean and elegant code.
In the end, "technical debt" becomes a way to shift the blame from your own (the team's) inadequacy at planning, designing and developing, towards supposed time constraints that always lie outside of the team's responsibility.
The other interpretation might be a neat solution. Allow developers to indicate publicly what proportion of the time their last tasks took, they should have taken had there been no technical debt. It would handle the communication from developer into business-speak pretty effectively.
Reminder: the "User" doesn't have to be an external person to the team. It can be internal for tools, or a tech debt task for internal issues like the ones you mention.
Thinking Scrum is just about end/paying User stories is mistake #2 I see when people do Scrum (#1 is the classic "We're going to not actually do Scrum but call it Scrum" that leads to a lot of "Scrum doesn't work" articles itself).
This is one of our biggest challenges. Much of the work we're doing isn't confined just to a user feature, so square-peg-round-hole syndrome affects us a lot.
If youre working on a project where it is important that you have as-accurate-as-is-realistic an idea of the size of the project, or more specifically your progress through that project, then I can't see how a methodology could be any simpler.
If having a good idea of the size of your project over time and your progress through that project are not very important from a management perspective, the Scrum artefacts will seem like, and will probably in fact be, needless overhead.
Scrum is not opinionated about the actual development methodology so claims about how it affects the code that is written are themselves a bad smell IMO.
Scrum is actually part of the problem, IMO. I've seen many teams turn scrum into a hammer and treat all future problems as nails.
Example problem: The foobar story has failed failed for the third sprint in a row.
Likely discussed in retrospective (plausibly good ideas, mind you):
- We need to break down stories more before we estimate them.
- Or we need to stop underestimating foobar stories.
- Or we need to focus on unblocking subtasks related to foobar stories.
Probably unconsidered:
- The foobar code is a mess and needs to be refactored.
- Or the foobar subsystem is too coupled to the Fizzbuzz subsystem.
- Or the need for some developer tools to increase productivity in the foobar ecosystem.
Since scrum is methodology oriented, methodology is the first tool teams reach for when a problem is encountered. And I see this after team leads make it explicitly OK to discuss technical subjects in retrospectives.
I'm not a psychologist, so I can't describe why this phenomenon happens, but I see it regularly.
Pretty much every kind of deadline driven development ramps up technical debt. Scrum certainly isn't the worst in this respect (developers make their own deadlines, and conscientious ones will build the time in), but the emphasis on commitment and the pressure to deliver at the end of the sprint puts pressure on developers to cut corners.
The worst part though, is that the product owner is usually non-technical and will deprioritize stories to clean up technical debt as a result.
IMO for any kind of development methodology to work it must have an opinion on technical debt. Scrum doesn't.
One of the few defining characteristics of scrum is that the developers define how much they can achieve, and this estimation is improved over time. If this is not happening there is something else wrong with the culture and Scrum is being used as a scapegoat.
Scrum is complex and not always possible to follow exactly, so this is to be expected but it makes me wonder, how many successful projects are out there that are following the true Scrum methodology?
My guess is that it's a few more than the classic waterfall but I still seem to see far more failure than success stories.
Regarding success stories, it might be that process doesn't play such a critical role as long as solid engineering techniques are used and the team is competent.
It remains to be seen if big Scrum engineering projects where Scrum is actually applied even exist. I can't even think about one on the top of my head. I'm not even sure Scrum is that well defined for us to be able to judge if is correctly applied or not. And it's yet another story to judge if they are successful or not.
In the end it does not matter much. The theoretical vision that nobody ever uses has almost no interest if you are concerned with real world efficiencies.
Regarding development: My main point is that Scrum leans towards agile methods such as XP (testing, CI etc), but it also sucks the time necessary to do those things well. The time Scrum takes off of the devs' working hours could much better be spent on those.
There's slightly more to it than that: it also encodes an assumption that you're working with a single fairly-tightly-integrated group (with synchronisation points at least daily). It's possible that this helps with estimation and scheduling -- it's a lot less clear that it helps get the best outcome in other respects.
More like wandering in the desert, hoping you find the promised land.
Been thru scrum master training 3 times, been on many "agile" teams. I've never heard this rationalization. Rather, a common justification for "agile" was you always have a working product. Which might be nice if things worked out that way.
Also, PMI style critical path worked just fine for figuring out that "straight line".
Scrum and "agile" democratized project management, empowering every poseur to claim expertise and ability. Whereas PMI required real effort to learn and master, Scrum flavored "self help" books can be flipped thru before you finish your coffee and then safely stored in plain sight on a book shelf, never to be touched again, allowing said poseur to claim the daily mutant chaotic dysfunctional mismanagement that they've always done is now "agile".
If you are objecting to defining the scope as small tasks and measuring your progress through that over time, then continually re-evaluating this scope as requirements change, then I think you are not working in an environment that would benefit from this kind of tool.
Its just a pragmatic set of guidelines, and objecting to it with such ridiculous vitriol makes you sound as foolish as the people I think you're objecting to.
if a software engineer is more than a couple years out of college, i don't think she/he should put up with it either.
I do a bit of planning poker with my team every couple of weeks, despite being a pretty experienced engineer, and it's great. It's been good at exposing assumptions or concerns that members of the team have, or about confirming that everybody is on the same page wrt. what the complexity of a particular task is likely to be.
I reckon that more groups should be doing this, and not fewer.
More fool them if so. I think pg said something along the lines of "unprofessional is what you say when you don't like something but don't have any real criticism".
An alternative would be to explain what the biases are and address them in each estimation round. But who does that really?
We've settled on playing games, which can seem childish at times.
it always takes 20% longer than expected, even when taking into account it takes 20% longer.
Most industries (don't know about bank analytics in particular) are more personality driven and/or more repeatable. There are few industries that:
- make entirely new things on a regular basis
- are mysterious to laypeople
- cannot ship 50% of a solution and get 50% of the value
Maybe software is reasonably different in that there are a lot of management level people with little technical knowledge of how the work gets done? I don't know how that compares to other similarly technical industries.
Points, for example, the argument is based on the premise that teams are obsessed with points. What if the teams use points as a framework to discuss complexity? I have worked in scrum groups where if something was given a large point value it would be questioned and "split" to break the job into component tasks that could be done by separate people simultaneously (or some now some later).
Long meeting times with the wrong people in the meeting? That's a managerial issue and has nothing to do with scrum.
Writing stories is the main art of scrum and without good stories it is pointless. A story can be written to encourage a developers to improve quality of a codebase, share knowledge with another team, or take time to learn themselves. A really good story would encourage these things and also deliver customer value.
Any system for managing software development hinges on good communication. SCRUM's main advantage to me is that it provides continuous opportunities for face to face communication. If you fail to take advantage or engage with those opportunities then it won't work, but I bet another "framework" wouldn't either.
Show that to a people unfamiliar with the Scrum church and it will say that sentence is meaningless. And well, it is, in an absolute way. A story is just that: a story. Maybe it can help to put your son to sleep, but I doubt you need to invent a brand new vocabulary to discuss about software architecture and technical debt, and I doubt you can do anything brillant by only using a few examples in a field that is even more driven by pure logic than maths.
We are grown up. I don't want to work anywhere where I'm told stories and must get some points done by the end of the week. And this is not anecdotal: Scrum is a derivative of a manufacturing framework. I'm doing engineering.
Maybe waterfall with unusually good specifications can use cog-like devs, but the few times I've seen that done it didn't turn out very well.
Planning/grooming we do in one hour. You good story writers is all. I once had a planning last 12 hours because the product management was so awful. If your team sucks, no process will save you.
Again, it all comes down to communication. If nobody wants to talk to each other, SCRUM is not going to help.
As with all of these posts, they're extremely lacking in alternatives. Of course SCRUM won't work for every team, or even every project for the same team. But it provides a way for the team to evaluate and adjust how to best accomplish their goals in a straightforward and (if done right) low friction way.
I think it's taken for granted how much effort is alleviated by adopting SCRUM. Just think how difficult it would be to develop an alternative process for every team, every project, every new person to join one of those teams or projects. Everyone has their own way of doing things, standardizing on a few universal goals aint a bad thing.
If scrum tends to lead with long meeting times with the wrong people then scrum certainly should include safeguards against that. Otherwise we need "x+scrum" framework which includes these safeguards. I don't think scrum should fix every problem "in management", but "meeting management" should be a core scrum issue to deal with.
Even with two-week sprints the majority of your last day is entirely meetings between the review and retrospective. In my experience one-month sprints are more common, especially in large enterprises and government, and you will literally have 7 hours of meetings on that final day if you follow the Scrum handbook.
Long meetings are built into Scrum.
I've never been on a team that does pure Scrum - even ones which intended to do so always ended up with what we termed "Scrum-ish": taking the ideas from the base methodology, but adding a whole level of 'house rules' aiming to patch up obvious gaps. For instance, always making time for one refactoring or technical debt task per sprint, tracking accuracy of estimated time versus actual time elapsed (deeply uncomfortable but very useful!), the hundreds of different rules around trying to make standups shorter and more useful.
I am a big fan of well-run retrospectives, though: they can be a really nice way to feel empowered as a developer, especially when you have one retrospective identifying that Thing A keeps causing everyone pain, and the next retrospective having everyone say 'Hey, Thing A is so much better now!' Never realized they weren't 'meant' to be about technical matters, though: in our Scrum-ish teams, they were always open for all topics, and I think that's a very good idea.
Of course, the fun thing about Scrum-ish teams is now you have a whole new level of debate that can happen: "We're failing because we're not doing Scrum rigorously enough!" vs "We're failing because we're doing Scrum too rigorously, and what we need is more house rules!" ;)
Of course, feedback is solicited, but it is an unspoken rule that criticism of project management is verboten. However, criticism of self and others on the dev team is absolutely allowed and encouraged, and so the brown-nosers amongst the team use the opportunity to make themselves known.
I would even argue that in the same way the "brown nosers" are trying to make some positive impact for themselves, as risky as it is, you can do better by fighting bad product management. If you feel the pain, your team does, and likely, your manager might as well. (if it's also a separate entity from the PMs.) The loyalty and trust you can build by defending your devs and being a force for good can be absolutely invaluable as your career goes on.
By and by, although I acknowledge the risk, if you shape your rebuttals well, you can find yourself bringing PMs to your side (a recent "spirited" discussion in which a PM was refusing to institute KPIs to track their features ended with their other PM peers questioning their resistance and backing the eng push for better telemetry, because when it came to "how can we justify how well we're serving clients if we _don't know_" this speaks even across lines.)
We've certainly done them, identified problem points and then solved them (and it does feel good to do that..) but it doesn't actually seem to make things better.
Lets compare it to say, personal estimations:
When you estimate, execute and reflect, you can tangibly improve your estimation process.
You can quantitatively observe an improvement in estimations on tasks when people go through this process.
Previously; estimated 20 hours for (task). Took 10 hours. Repeat... soon, your estimates are for 10 hours, and you're quantitatively, objectively able to make consistently better estimates.
Retrospectives in my experience don't do that.
You can sit through 50 retrospectives and each one identify a problem area and then fix it and yes that does feel good, but objectively when I reflect over the defect rate as a result of the process, I feel like retrospectives make zero impact on the rate at which technical debt accumulates.
There's something missing in the way they work; all you (well, all we, I suppose, this being my personal experience...) ever do is find things that are wrong and fix them. Objectively when you look at it, there's no closing of the loop where the defect rate drops.
There's no process improvement that generates less problems in the future... all it ever is is band-aiding to prevent technical debt spiraling totally out of control and devastating the project.
There must be a better way, where you somehow measure how technical debt was created and work to incrementally prevent it happening... but I've never seen that actually happen in practice.
It should only be uncomfortable if the team means 'commitments' when they say 'estimates'.
This is part of the reason stories are generally pointed with (more or less) triangular numbers. So that teams stop fretting about missing estimates by an order of magnitude or less.
Does this even mean anything? How would it work in practice?
My impression is that he means something like identifying a part of the codebase that anytime a story requires working in it, it sucks. I remember we had some old UI object written in some arcane javascript and everyone felt like dying when they had to work in it. But in my experience, the pointing process naturally included that as a consideration. I believe my brain already processes that algebra of complexity in it's native operations when I estimate a point value. I'm not sure that attempting to materialize this "algebra" into some system of equations would be very helpful.
The daily standup, IMO, should be only to remove impediments, and if you have none, then a sentence or two will suffice. I see the DS as the most useful meeting, as you are aware of what your workmates are doing.
And if Scrum is still a pain in the back, then you have Kanban, which is sort of Scrum without the straitjackets.
Yes you can do it (and probably should when you understand the purpose of each Scrummy practice), but don't expect to be praised by Scrummers.
I'm extremely happy to discuss the reasons I want to change something, but all to often the only counter-argument is "that isn't Scrum". I'm 100% fine with that, but I'm not going to do something if the only reason for it is religion.
If communication is a problem, having a daily standup just pretends the problem doesn't exist, rather than solve it.
100% this. The only thing a synchronous team-wide meeting is useful for is revealing a significant issue and getting a prompt and definite acknowledgement from the team. And then, if it's a priority, some help with the issue. But given the proper tool, even that feature can be made asynchronous.
It gets worse if the team is actually 2-4 different teams with not much overlap (because companies tend to adopt these agile methodologies without much thought and it just keeps growing and including more people because.. it's nice, right?). Then you're ignoring (or not having a clue) about 90% of the meeting, and it's _daily_.
- Obsession with points
We don't have that obsession, sometimes we even don't assign story points, just hour estimates
- Meeting extravaganza -
Again, for example remote people don't need to attend all meetings, sometimes we just clarify the work items outside the meetings.
- The sprint has its own backlog, which can be changed only in agreement with the team and the product owner.
We also don't have this... if I finished my tasks, I just move new item from backlog to sprint and work on it.
- Refactoring, reading code, researching a topic in detail are all seen as "not working on actual story points, which is what you are paid to do".
This has some truth to it... regarding research, we do this outside the Scrum.
The last point has been the biggest one for me. Across multiple organizations working in the scrum process I've occasionally felt like just getting some time to talk to developers for future exploratory work is like pulling teeth because the scrum process relies so heavily on tangible tasks. I think that's a time management thing that has to be established with teams ahead of time — like "hey 80% of your week are these tasks, but you have 20% set aside for research/discussion'
Usually the 20% is reserved for bug fixing tasks which can't be estimated and therefore can't be included in the Scrum framework.
One of my projects has a "technical backlog" which is actually just a Jira task with multiple subtasks which gets dragged from sprint to sprint. As you can imagine this doesn't work so great from a process/tool perspective, but it gets the job done (i.e. Major refactorings can be and have been performed).
The thing with Scrum is that a lot depends on the PO and SM. If they're open minded and experienced you can get something decent. If they're dogmatic and pray at the altar of Jira, prepare for suffering.
This is why sprints have a fixed duration, the sprint backlog should not be changed, stories are awarded the full points or 0 during review, commitments to finish the sprint, etc.
However their article is criticizing pure unadulterated scrum.
There is little to criticise about pure unadulterated Scrum because there is surprisingly little prescribed and what little there is they openly tell you that you can change it in the retrospective. The real problem is that management, agile coaches (and unfortunately sometimes also some developers) often want to religiously apply every little thing they ever read in a book and seemed like a good idea. I think that solutions to all the problems the author mentioned are possible without deviating significantly or at all from scrum -- should that be important for whatever reason. The only prerequisite is that everyone involved is open to change.
Obviously adding something to the Spring is going to hide the fact that you fucked up at Spring Planning. However, as the author said, very few people actually care about task estimate. ( and often team don't even care about breaking stories into task at all, using story == task ) And in any case, unless you are using postits and manual tracking, any tool will just track the estimate alright for you, the information is there if you want to process it.
Velocity is supposed to change from sprint to sprint, getting better as you are better at assigning story point. You can lose SP when you fuck up a story, you can gain extra when you put additional stuff from the backlog. Velocity once more or less stable is supposed to give an idea of a release date.
However considering that you don't break down big stories and epic into smaller stories until you are about ready to implement them, the uncertainty on the release date is more like 6 months vs 2 years vs 10 years. Management make a big deal of it but really that's how you budget most of the time anyway: I need "3 FTE for 6 months or 3 FTE for the year, to be reevaluated at budget cycle per budget cycle".
Most of OP problem actually started really when Scrum Master became a job. I remember something like 10 years ago being all exited how Jira could make very precise report of velocity and estimate and my very experimented coach looked at me as if she had seen the Demon. Scrum is about keeping a supertanker on track in a tempest and those amazing reports gives the illusion of a nimble sport car racing on a track.
We also had the general expectation that one story should generally take no more than about two weeks. Before starting a story, if you think going in that it will be too big, then you try to limit the scope or defer parts of it into new stories until you're confident it can be done in two weeks.
Once a week we tracked how long stories were in the "In progress" column, and once it'd been up there for three or four weeks, people started asking how they can help wrap it up. I think the longest I remember a story being in progress was about 6-7 weeks, and that was real uncomfortable for us. Typical times were 1-3 weeks.
So we had a weekly cadence for demos, and product owners liked that they would see steady, regular progress rather than being inundated with sudden large dumps at sprint boundaries.
* Individuals and interactions over processes and tools * Responding to change over following a plan
Now, I cannot say that Scum abstractly is a bad approach to attempting to be more agile. Many of us have employed some form of it with varying degrees of success, but, the expectations on software developers from a business perspective are more rigid for business development and tracability purposes, which can,on occasion, omit the consideration of resource demands and complexities.
Often, i believe, it is that a lack of understanding the purpose of the agile manifesto and the poor implementation of a method at the top level of the developer and managment ladder which is the underlying cause.
In the post, the author cites 3 hour or more meeting where he feels there are too many people there. This would be an item for the retrospective feeding the next iteration, but he only says "Blergh" about the needed feedback.
He states that the standup is more ritualistic, but the point is to ensure each member understands where the sprint stands, inspire collaboration, but really take ownership of the code. It could be that the team size is too big, or that the team really has not taken ownership of their code.
When all is said and done, you have to buy into the Scrum process, the process has to be flexible, and the stakeholders have to not believe it is a magical development method that fixes all the issues of the software development process.
My team hasn't tried anything like that, but I think it could really be useful. It encourages better communication throughout the day as well, as it gets people using Slack. Plus, it gives managers the ability to keep track of what people say each day.
At this place, where there are lots of different code bases of various quality and approaches to how the features are implemented has shifted over time, I tend to have questions most days.
I have worked at companies where we're only really working on one thing and I'm familiar enough with it that I really didn't have to ask any questions (they also didn't care much on how it was implemented, either, just that it worked) and daily status meetings were pretty much a waste of our time and ended up becoming more like once every other month.
I haven't used slack in anger. When you need help, how do you tell the difference between a teammate being busy, unavailable, or just too distracted to respond? Part of the point of the in-person standup is to remove ambiguities about this at least once a day.
In the real world, there are externalities that have an impact and require some estimation. Maybe we have to provide new support materials to customers, or finish a contract with a third-party data provider, or change our infrastructure. It's exceptionally hard to do some of these things without being able to make commitments of some kind. Estimates allow us to create a guideline, and subsequently alter either the thing we are going to deliver (i.e. dropping lower-priority features) or the time we are going to deliver (i.e. missing the deadline).
If I hired a builder, and he told me that he refused to estimate the amount of money or time it would take to construct my property, you can be sure I'd move on. I have no idea why we'd consider it appropriate for an engineer to do this.
When every builder (i.e. engineering team) anyone ever hired everywhere blew through their estimate ~70% of the time, perhaps it's time to reconsider that stance. See the Standish Group's annual CHAOS Report for an example of software project success statistics, e.g.: https://www.infoq.com/articles/standish-chaos-2015
Estimates are just mutually agreed upon lies that engineering teams tell themselves.
Some of the better aspects of Agile are aimed at providing an analogous level of obviousness: E.g. don't move on until a story is completely implemented. This means you see schedule risk as it happens. This also avoids the way waterfall projects used to die, like the one I saw at Lotus while consulting on Mac porting, back when using conventional project management tools for software was a progressive management practice: A GANTT chart with hundreds of lines corresponding to tasks. Each progress bar partly filled, most of them 70-80%. Yet the project was dead because it was obvious it would not be completed in a relevant time-frame.
It is far too easy to game completion tracking in big, complex projects if you don't decompose them into small sequential projects. BUT, even though each smaller project has schedule risk that is reasonable, when you add up the schedule risk across all the sub-projects, if you are honest, it will be large enough that an estimate is going to have low reliability.
In summary, he suggests eliminating the review, planning and stand-up meetings (replacing them with asynchronous & as-required meetings), increasing the frequency of retrospectives, and spending time analyzing the backlog to see what the underlying causes are and tackling those.
Scrum becomes useful when you are required to plan(or attempt to) schedules for certain features far in advance.
Start from Kanban and add stuff until you get a process that matches the critieria of the organisation.
This requires a lot of skill though, so many teams are stuck with one-size-fits-all tools like Scrum, especially now that it's become a management buzzword. And it does a not too bad job at delivering somehing of passable quality not too late. :)
Estimating work is a nice idea, but using 'story points' for it just feels like a bad API decision - I have never seen a new Scrum team member (including me) getting it at 1st try.
And maybe it's just me, but having a full time, dedicated Scrum Master, who serves solely as a facilitator (without having any actual role in product delivery) is a nightmare, especially for teams that are used to Scrum practices.
The SM is major piece of the Scrum puzzle and a lot of practices lose their value if the SM is not paying attention.
This over-reliance on the SM is a minus for Scrum IMO.
> What about contributing to open-source software? Reading the code of an important external dependency, such as the web framework your team uses, and working on bugs or feature requests to get a better understanding was not part of any Scrum backlog I've ever seen.
Working at various startups, I have developed a methodology of contributing back to open-source projects we use without accounting for it in sprints or the ticketing system. It involves getting to the office an hour early, over-estimating on my other tasks so I have time for something extra, and a sprinkle of office politics.
And yet, it is some of the most valuable work I have done. Not for me, but for the companies I've worked for.
To get an in-depth understanding of that one Django or Express.js feature you use, or even better, to find and fix a bug that affected the business or may have affected it in the future, just gives you that much more of an edge over your competitors. Say goodbye to that nasty workaround you had to use to get around the bug--now it Just Works exactly how you need it to!
What's more, it's attractive to engineering candidates when you get to tell them the story of when you fixed a big bug in Socket.io.
The best engineering managers I have had have been receptive to the idea that this type of research and/or contribution to other projects should be considered "work" that provides value to the business.
That's a really good point. I'd be excited to see what a team could do if each developer wrote a one-page memo about what he did and what he was going to do once an iteration, and a few sentences for each day. Throw 'em in a log, and they might even aid the retrospective.
Maybe we could just go back to .plan files, like Carmack.
Yes the issue with standups/scrum cereomonies is if people ignore all the 'other' info they are receiving and for what ever reason choose to ignore it.
The real key for Scrum for me has to be not only teams that want to write good code, but teams that WANT to get better at working together. That takes the whole team. If the team aren't bought in to this, then I'm not sure which project management tool will work for them, but it sure isn't scrum!
For example the only purpose for points and velocity is to give product owner idea of how much time will have to pass before a thing might even have a chance of being done. It tells him that you, for sure won't get this thing and that thing in next 2 weeks.
The only utility of points for the developers is when doing planning poker. If something gets a lot of points then this means people are not sure how to do it and the thing needs to be discussed and broken down.
After that, points have no use for the team and team shouldn't even be aware of its velocity and how much they burned so far this sprint. Those are the tools just for the product owner and scrum master.
If product owner is satisfied with "it's going to be ready when it's going to be ready" then points after planning poker can be forgotten and you don't even need to calculate velocity. It's still worth to do the planning poker for the sake of the team discussion and so the product owner can deprioritize task that are hard but not crucial.
But turning points and velocity and performance measure is dumbest thing you can get, because then it get's screwed and loses all utility. Same thing in estimations. If manager is going to negotiate estimates then you can as well not do it at all because they no longer carry any real information and are becoming just the reflection of managers hope slightly skewed towards developers hope (which was already overly optimistic).
So Engineers get herded into little incremental projects that management can swallow as having 'business value'. And the herd marches toward the cliff as the code base wanders around the solution space but never gets fundamentally sound. Anyway, that's my jaded (from experience) view.
If you have a culture of trust between departments, you should be able to have honest conversations.
Scrum was "designed" by Ken Schwaber, a software project manager at the time looking for a better way to control software development processes.
Schwaber discovered he could use an empirical process, rather than a defined one, to control software projects. He created Scrum following this principles.
I'll stand by the last two sentences of that comment, though.
While I'm far from a Scrum fan, it's not necessarily the worst thing in the world. For instance, it's probably better than "just do what the project manager says", especially if he tends towards micromanagement. And double-especially for people who thrive in group discussions but have trouble managing 1:1s.
I suspect the trades also look different depending on level of experience. Scrum, at least in some incarnations, tends to be quite egalitarian, so I can see it appealing to more junior devs in environments where they feel the seniors get to do all the interesting stuff.
Of course, if you already have meeting overload, standup is going to feel like the worst.
That always bothers me, too. The changes you can make continuously are hardly worth bothering to worry about, because they're so small and quick and decent developers will do them automatically. It's the changes that take several days or a week that really make the difference in the long term.
https://news.ycombinator.com/item?id=11548334
To summarise, the issue Dave Thomas has with agile is that it has become a prescriptive framework, 'Agile' (with a capital 'A'), rather than what it originally started out to be, which is a means of being agile (small 'a').
I really liked this talk, and agree with both him and the OP of this thread that some parts of the 'Scrum' as a prescriptive framework may not work for you.
I find it a bit strange that the OP singled out the retrospective as being a negative- for me this is the best part about 'Scrum', as long as you're not too prescriptive about it.
Really the retrospective is a chance to improve yourselves as team each sprint, and it's a chance for people to get stuff off their chests. It shouldn't be just about scrum, it can be anything - adopting new approaches to development eg: TDD, discussing the state of the code, anything. That's what makes them (for me anyway and I think(!) my teams) enjoyable & useful meetings.
I am guessing that such an environment is quite rare, hence the percieved lack of utility of the retrospective.
I can speak from experience that this is very hard to do. I expect that I will have to deal with a brewing technical conflict in the team soon and I'm not sure what's the best way to do it. I've also seen what happens if such coflicts are sweeped under the carpet and it's not nice.
Majority of professionals I've met appear to lack any ability to improvise and literally just follow the book on what to do without any critical analysis of how their situation fits with the defaults provided.
Basically, my issue with scrum is its community.
But how to fix cargo cult depends on the root cause and there are many:
1) Authoritarianism, my boss said read this book, this book said XYZ, therefore XYZ is true. How could the book possibly be wrong? Or for book, insert rockstar ninja consultant or training class or whatever.
2) Religious belief, sure none of this makes logical sense but if you have blind faith and are not one of those apostates, it'll work just fine. Just relax and perform the ritual.
3) Deep game, where we actually run on anarchy or waterfall but someone needs XYZ on their resume so superficially we've skinned it as XYZ. Dig enough and you'll find what really runs the place is something else.
4) Blame games, a large part of management is how blame is distributed, not just distributing work, and when a subgroup who thinks they're agile-ing to distribute work comes in contact with a group using agile mostly to distribute blame or to stall for time or just to goof off on company time, its kinda like matter-antimatter.
Most of the criticisms in this piece can be resolved by team members adapting processes so they work for the team.
I've also encountered most of those problems in non-agile, non-scrum teams as well. Instead of obsessing over points, it's hours.
Meeting hell can happen anywhere. Good employees will tell their managers when meetings become an impediment. Good managers work to fix it (and pro-actively try to prevent it).
I always find it odd when somebody complains that sprint goals need team buy-in to change. Why is that a bad thing? Adjust and move on. Sometimes things don't go as planned, sometimes they go well. Hopefully, it all averages out at the end.
Anyways, I guess I view scrum (or any other methodology) as a loose set of guidelines, not strict rules that must be enforced at all costs. If management is forcing the rules, even when they are becoming impediments, that's a sign of a management problem that is unlikely to be cured by a change in methodology.
It regularly seems to me that people think that there's Agile/Scrum and there's Waterfall and that's it. The first year engineering design course (and the first year programming course) in my department cover a number of different approaches.
That said I prefer Agile to Waterfall by a lot, and Agile developed in reaction to Waterfall so I think it was a success.
Story points are a measure of "relative amounts of effort". A 2 point story is twice as hard, as far as effort to implement it than a 1 point one.
> Scrum meetings (aka rituals) have been among the most miserable hours of my life...
Would you rather have many meetings almost every day, or a single big ol' meeting every few weeks so you can focus on coding without interruptions?
> The review meeting causes utterly unnecessary anxiety (Oh my god, will my feature work?)
It should not cause anxiety because it MUST work. The review meeting is to show stakeholders a "potentially shippable product increment". Everything you show on the review meeting should have been extensively tested already.
> Why estimate stories that you are going to break down anyway?
You only break down stories if they are too big. You are not supposed to work every detail on a planning meeting.
> but in Scrum, the retro is explicitly supposed to be about the Scrum process itself, not about the codebase
The retro is not about the scrum process but about how it is working for the team. All these issues you have already complained about should be reserved for this meeting where the whole team can decide which adjustments to make.
> The main goal of Scrum is to minimize risk and make sure the developers do not deviate from the plan. I will come back to "Scrum controlmania" later.
It is regretful that, like all the other "why scrum sucks" posts I've seen on HN, it all boils down to a bad scrummaster not managing expectations and taking the time to explain where the process really comes from.
When asked to estimate, the answer is "I'll let you know when I'm getting into it". Then you're made to do a 'spike', which is indistinguishable from doing the task. Except do it in a day now. So the Engineer thrashes around trying to figure it all out in a day, and comes up with some number. The planning phase is now over, so they're supposed to just execute, regardless of what (bogus) number they found. So they start. At the end of the sprint, they've done less than if they'd just started the task the first day. They get 'measured' as a low performer. Of course its the process that's performing badly, not the Engineer. They get frustrated, resentful and stop cooperating with the scrum master.
I've seen this so many times at so many places its just exhausting. No amount of discussion can convince the evangelical Scrum experts that something is wrong with the process.
All of a sudden story points disassociated with time. They just became a number. a combination of complexity and understanding.
A lesser, but still important shift, came with our changes to backlog grooming. When part of that grooming included poker planning to estimate points. This Spread that chore out, and our Sprint planning is now approximately half an hour as the team goes down the prioritized list and commits to stories.
It also just felt unproductive - it became something we did to feel like we had "best practices" when in reality it was a waste of everyone's time.
I got the sense that everyone would essentially forget what they said/heard at the end of the meeting (or was simply tuned out).
I ended up launching a tool to solve this problem: (https://jell.com).
We're a lot happier showing a "todo list" with each other and still have a place to write out the challenges/progress we're making - all while respecting each others time.
> First of all, what are story points? Are they measures of time it takes to complete a story? If yes, then why are they not in terms of time?
I don't think I've ever read the official scrum guide, but ever since I started using scrum (around 2004) the concept of story points was clear: They were "perfect days", i.e. days where you don't have any distractions and nothing goes wrong and everything works first try.
As a fairly experienced developer, I can easily estimate how many of those perfect days will take me to do a task (at least if it's a technology I know). When estimating, the whole team estimates all tasks until there's a consensus (and you are right, ideally, all the stories should require similar effort).
After that, the "perfect day" thing is forgotten, the estimates have no unit and the velocity starts playing it's role.
And here's exactly why I like points: The link between real days and points is lost, it doesn't matter. All that matters is that you completed X points in a specific sprint. Chances are that you'll complete X points in the next sprint. Iterate through a few sprints and you'll have a pretty good idea of what will be finished when.
I wrote about this here: "The Agile process of software development is often perverted by sick politics"
http://www.smashcompany.com/business/the-agile-process-of-so...
In such an environment, developers prefer to drastically over test their code, and to undertake work in manageable sprints that let management claim success and understanding even when neither exist.
If you already have a very strong product market fit, and you need to hire developers whose judgement you don't trust, or if there are extrinsic sources of timeline pressure (like investors or non-technical management who think developers are lazy... essentially anyone other than users or customers), then Scrum is perfect for your organization.
The other constituency that seems to love Scrum is product managers who either have no vision for the product or no control of the vision, and are essentially being asked to be cat herders and manage engineers without earning their respect or having any authority over them.
Scrum is a system of managing people, not a software development methodology.
It's about transforming programmers into cogs and gently forcing them to obey certain rituals every day, until they slowly give up their individual creativity and initiative and become good 'team players' .
And when someone says 'team player', I hear 'you belong to us now'.
I disliked scrum since the moment it was decreed upon our team.
And that's because pretty soon our team became obsessed with points and respecting the religion rather than doing the actual work.
The final drop came when I refactored some code, made it twice as fast using half as much memory (the proverbial 'much better'), only to have to fight the team to accept the changes, because that wasn't in the backlog.
Methodology is only good when it helps you achieve your goals easier and faster, safer, etc.
But when the methodology becomes the goal, then your job changes into satisfying the methodology, rather then being the best at what you like and enjoy.
And this is the exact status quo that larger organisations love - people focused on small irrelevant tasks, while the 'grand scheme' is determined by management.
Not for me. I use certain parts of it in my work today (develop in sprints, demo at the end of sprint, planning, backlog and current tasks), but if I see "we use scrum" in the job description, then I'm not your man.
I've had this happen my times while I was leading the project - it bugs me. I've ended up reducing daily standups and replacing with walking around (physically and virtually) and chatting with people and getting similar results. Of course, "similar results" here is that I heard what someone had to say but nobody else did - so a key to this walking around approach is to tell each person what other people are doing in order to connect the dots.
On the surface this might seem terribly inefficient, but there are a lot of positives, biggest of which is building good relationships with team members and learning about all the other little things going on in life.
On the down side, for people who are only part time on my team (not uncommon in my bigco setting) there is extra scheduling effort and discipline required to make these chats happen frequently.
https://www.youtube.com/watch?v=ZWYnVt-umSA
You need to rent this video and show it to the team. (and rent "More Bloody Meetings").
Scrum is a management technique, not related to programming. In fact, it is a "micro-management" technique. However, those who lead don't seem to know how to conduct meetings. I don't see scrum training that even hints at the fundamentals of management, even the fundamentals of meetings.
I agree with a lot of what the author is saying. They're describing a process that appears to be relatively broken, and identifying some good reasons why they're broken.
What's missing is flexibility. As per e.g. the Agile Manifesto (http://www.agilemanifesto.org), the entire point of frameworks like Scrum is to provide a loose set of processes that you adjust and tweak to your specific team. If you're blindly following the process, and slavishly adhering to things like how precisely your daily standup MUST work, you're following the letter but not the spirit. You're being "Agile" with a capital A, but your team isn't actually being "agile" (the adjective), which is actually the important bit.
The correct response to "our team is unhealthily obsessed with story points, and yet doesn't get value out of them" (or standups, or retros, or whatever) isn't to say "story points are broken". It's to say "this bit of process isn't helping us right now. What can we do to modify the process to provide more value to our team?"
During my training the analogy was to categories of dogs based on size. So like a Chihuahua is maybe a 1 and a Great Dane is maybe an 8 (and there's 7 categories of dog size between them.) But a Malamute might also be an 8. Even though it's shorter than a Great Dane, it weighs maybe the same because it is more thickly muscled. But neither of those dogs is equal to 8 Chihuahuas.
The point being that if you categorize stories like this, treating point values as categories rather than as 1 point = N of X, then that makes pointing a lot easier and a lot faster as your team builds up a methodology (what categories your team uses and what they mean is up to you, it doesn't matter as long as you are consistent.)
This leads to meetings taking less time, and leads to a more consistent velocity than if you do something like 1 point = 2 hours or 1 point = something else rigid.
If SCRUM doesn't work for your team by all means you should abandon it. But, to me, it sounds like his team is using it badly. It sounds like they are being really rigid, and it sounds like a few adjustments could lead to a lot more happiness. But maybe not.
But with our new Silver-Bullet Development Methodology™, it's a completely new world where none of your decades of experience apply! This changes the nature of development forever!
p.s.: we sell certifications!
(repeat for a new Silver Bullet roughly each decade)
I found this to be a really good article about standups: http://www.yegor256.com/2015/01/08/morning-standup-meetings....
Practice and experience may have led to this conclusion, but it will be good to know that the (original, Scrum-defining) Scrum Guide, http://www.scrumguides.org/scrum-guide.html, doesn't even mention nor defines "story points". Obviously, there is planning and estimation involved in Scrum, and story points can be a metric used for "amount of work"... but it's not a defining feature.
> So, in summary, Scrum
* wastes too much of the developers' time for management
* does not lead to good quality code
* is a control freak which does not leave room for new ideas and innovation.
Scrum, however, is much closer to the problem solving approach, where analysis (breaking down a problem, and reassembling the solution) is the organizational tool. In order for an interpretive community to emerge, an organization needs ambiguity, open-ended conversations, and alternative perceptions. All of this, Scrum leaves to something else, whatever it is.
But then goes on as though Scrum precludes any kind of thoughtful design or experimental innovation. Nonsense. Nothing in Scrum suggests that a team mindlessly go through sprint after sprint without ever pausing to have design sessions or build prototypes. And there is scant mention of the product owner at all; it's as though the author believes that software is designed by developers, for developers.
Scrum is a way to implement a software design, flexibly and with the understanding that the design will change along the way. How you decide what to implement, how you arrive at that initial design, is up to you. If your organization doesn't understand how to create and manage the abstractions that make up a software application to begin with, then Scrum isn't going to save you.
First, the author refers to Scrum as a methodology. It is not. Scrum is a framework, and being based in agile principles, is much more about a way of thinking and working. It should not be used as a purely do-this-then-that prescriptive approach to getting software built. As we now can see, there many organizations that tout being "agile," when their true behaviors and output are merely agile window dressing. Also known as "fragile" instead of "agile."
I disagree that Scrum is not useful to many organizations and teams. In particular, organizations that basically lack any process (trust me, I've been in a number of them) and it's kind of a free-for-all or based on whichever executive screams the loudest gets what they want when they want it.
What is missing here is why Scrum can provide useful, especially in terms of velocity. A team's velocity can serve as an underlying basis for projecting how long things can take. Like it or not, internal "users" or "stakeholders" in an organization as well as many external ones expect to get some idea of when things are going to start happening and when things are going to be done happening, for particular features or commitments. As expected, this involves being able to at least intelligently (and based on historical data) make a reasonable prediction at dates.
Entire books have been written about software estimation and no framework gets it right.
For me, Scrum has always been nothing more than an “Agile Bootstrap”. The core idea within it is very simple: try something and examine the results. Stick with what works and try alternatives for what doesn’t. Repeat. And, above all, learn.
So Scrum starts you off with a list of practices to get going with. The whole litany of backlogs, story points, sprints, daily stand ups and retrospectives is nothing more than a vocabulary - a pattern language for stakeholders. It’s got pros and cons - a good way to bring people together but it needs to be recognised as a crutch and ditched quickly before the team comes to treat it as an article of faith to be defended at all costs.
And this is where so many agile coaches fail in my (limited) experience. When I was undergoing scrum training, it was hugely disappointing to hear colleagues asking questions of the trainer along the lines of “Can a scrum master multi-task among teams like an anaesthetist in a hospital?”. Or “What does Scrum advocate when a production incident requires someone in the team to drop out and assist?”. Or “Can we have scrums of scrums?”.
The answer in every case is the same: try something - anything. See how it goes and decide whether to apply that approach again in future.
Kind of like democracy.
If you don't like meetings, planning, or giving estimates, you probably don't like working on software for a living, because I simply don't know how to build software without meeting with other people, coming up with an idea, and telling the people giving me money when I hope to be done.
I would argue with that.
So all in all I still think that Scrum & Kanban can make your life easier.
You should not be a fan of a tool, but use the right tool for the job. If Scrum is the right tool, use Scrum. If Kanban is the right tool, use Kanban. If Six Sigma is the right tool, use Six Sigma. If waterfall is the right tool, use waterfall.
Second, if you can't use a tool, don't use it (I've seen companies using Scrum with excellent requirements engineering, rearchitecture/refactoring, headroom, ...)
Scrum Meeting:
- Follow the list of topics for today
... (may include)
... Show progress and new features
... Get feedback
... Discuss blocking points and resources needs
... Get update on supplier status (if buying/selling anything to 3rd parties)
- Decide what are the priorities for this week
- Schedule the meeting for the next week (send the list of topics you'll talk about)
That's overly trivial. It mostly comes down to schedule a meeting every 1-2 week, show/say current status, repeat...
No need to call that scrum and put fancy names on it.
I thought 1 week was the default and it certainly works for my team. The various scrum meetings are short and snappy and have high value.
Oh, and from working side-by-side with the best practitioners in the business at ThoughtWorks for three years. But the book is sufficient.
First, as background, I'm a software engineer, not a "scrum coach" or anything, but I've been on a Scrum team for nine years and nine months. (I know, right? Our first Scrum project was Nov 2006. The set of people has fluctuated over the years but it's still a pretty tight ecosystem.) Just this morning we were requested to make videos about our team(s) to explain why we work so well, so this is pretty apropos.
Second, I really did read the article through and thought it was well thought-out. Notably, I felt they came from a different mindset than mine, a different workplace, and I am a fan of Scrum, so here are my feedback points--I hope they're considered positive, helpful and constructive:
TL;DR: I feel like the author is not empowered in his workplace. Time to upgrade your scrum team mindset.
* Remember that a scrum team is self-organizing and self-managing. Specifically, I'm replying to: "The daily standup is in my opinion a manifestation of a significant but unspoken component of Scrum: Control" implies you don't know or don't practice this and is likely the actual source of your problems with Scrum.
* You mention disliking pointing stories because they never match up with the points on tasks. Points on stories are done because you haven't created tasks yet, so you can't use task hours to add up to a story, and you need to figure out relative complexity before you start.
* Getting points (RE: "I don't get the exact response that was expected, so no points for you.") - Here, the team decides whether it gets points, not a user in a review. That sounds weird, but so does a user saying "No points for you! (soup nazi voice)" in a review. Stick with me here: If you get feedback that says you need to do significant rework, it's clear there was a misunderstanding between your product owner and the business. Make a new backlog item, point it, prioritize it, move on. (If this happens more than once consider how far apart your vision is from your user and their vision of the project!)
* Demoability as a requirement in Scrum (specifically responding to "How can you demo that your code base has become more habitable?") - Business folk understand the value of "plumbing" (pipes in your house aren't visible, but they sure are handy when you want to take a poo). They don't like showing up to meetings to talk about plumbing, though, so either skip that meeting entirely or tell them what will be possible when said plumbing is complete. Point is, don't die on the hill of demoability just to say "See! Scrum sucks! I can't demo all this plumbing!"
* Meetings every two weeks - If they're painful, you're doing something wrong. If it's not ready, it doesn't go in the demo. Don't kill yourself for a demo. It may be a "sprint" but you're still actually in a marathon, so just pace yourself well.
* You mentioned disconnected users. If you have bored users, get better about grouping the meeting. Split it into two meetings, if need be. We don't; we say "Hey, finance folk, you'll want to pay attention the first 15 minutes of the meeting then you can go. You're welcome to stay, but you'll see app features that won't affect you"
* Daily Standup - For high-performing teams, the daily stand up is training wheels. Anything you learn in a daily standup should make you mad. ("Why did you wait til the daily stand up to tell me you [finished X|were blocked|needed this info|are ready for me to test]!"). Do them if you need to. If you do need them, ask yourself why. e.g. who isn't a communicator? Who would've blown you off were they not in this required meeting? Those guys are your blockade for more than just a daily scrum meeting.
* Sprint durations - You make it sound like you don't have a choice. Your team chooses the duration of the sprint.
* "I find the idea that you should get somewhere by sprinting repeatedly [and the rigidness of items in a sprint] rather weird." The rigidness is there to protect you from outside forces, not prevent your team from getting the job done. e.g. It's there to prevent the VP of Marketing from showing up and saying (psst, hey, could you add a blue link on the homepage?" .. "Oh, that link is wrong, make it into a popout." ... "oh, that popout should be a flash video" ... then suddenly you're missing your deadlines to the VP of Finance. It's there to give you a defense mechanism against folks above you trying to work-around their peers for your valuable time.
* "The Scrum coach will find fifty ways of attacking each and every one of these topics, but all of them will be in the form of one more thing. One more meeting, one more document, one more backlog, one more item in the definition of done. " I don't think so, man. It's all about taking the training wheels off, not adding brakes. Maybe you have people working against you. Working against being in a team. In order to protect you from them you're all being laden with extra BS. For us, every painful thing that slowed us down was cut. Blocades were fired. The world got good.
* "Every story in scrum has to end in customer value. [...] why even bother with refactoring?" The team can be a customer. "As a software engineer, I must refactor my [blah] to facilitate [blah]." No need to jump through artificial end-user-centric hoops. When/if you mention it outside the team, just hand-wave them and call it "plumbing." Everyone understands plumbing (as in, pipes in your house you can't see but appreciate every time you flush.) You still get points for work done, you still get value, you're still being a responsible engineer keeping their house clean.
* "What is the job of a software developer? Writing code? I don't think so. I think it's inventing and customizing machine-executable abstractions" Not to be trite, but I'd say the job of a software engineer is to offer solutions to business problems. Usually this is by way of "I've got a hammer so let me hit that hangnail for you", but really, that's all we are, is problem solvers... but... shrug
Re: Ideas for Alternatives:
* "One way to achieve this might be putting work items through what I would call an algebra of complexity, i.e. an analysis of the sources of complexity in a work item and how they combine to create delays. The team could then study the backlog to locate the compositions that cause the most work and stress, and solve these knots to improve the codebase. The backlog would then resemble a network of equations, instead of a list of items, where solving one equation would simplify the others by replacing unknowns with more precise values." I'd love to see some practical examples of this. It sounds more complicated than the simple off-the-hip-shot estimates we get with points, but the idea has promise.
* "The other proposal I would have is to get rid of the review, planning and stand-up meetings." You should be doing these judiciously anyway, not dogmatically. Free yourself of the chains of dogma and just do these when they have value. The thing is that they ARE good training wheels. If you're not in a high-performing team and you skip straight to "just do it when you need it" then you never get into practice, you never get used to them, you ... never do them. They do have value, and you should do them when you can demonstrate value in them. So... Ascend when you're ready to.
In the post, you sort of dismiss it out of hand because it is supposed to be a discussion limited to scrum, but that is an arbitrary and self-imposed restriction. You are doing it wrong. Remove that restriction and the doors will open up.
The retrospective is about creating time for the team to review what works and what doesn't about the software development process in general (no need to limit to scrum).
I've seen retrospective discussions veer into company culture, the need for faster hardware, testing processes, etc. Anything related to making the software, the software development process, or the team better should be on the table.
A good retrospective enables the team to use its sprints as experiments to try different things, to evolve the practices to better fit the needs of the team and organization. If stand-ups aren't working, go a sprint without them, or doing them differently -- whatever change would address the weakness identified by the team -- then in the next retrospective reflect on whether it actually improved things. Rinse and repeat. Done properly, a good retrospective will enable your team to evolve and get better and better with every sprint.
Points can be frustratingly fuzzy, but they serve a valuable purpose. They give a measure of team productivity (e.g. velocity), even if imperfect. Yes points can be gamed, but a team should learn quickly that gaming only serves to fool themselves. Because of they can be gamed, I'm not a fan of using points as an externally visible "vanity metric". They should only be used or shared within the team. They should not show up in performance reviews or presentations to management. But points can be very useful as an internal metric for the team to measure whether or not tweaks to the process made a positive impact. You need some way to objectively measure the success or failure of your process experiments.
Of course, any metrics used are themselves certainly deserving of scrutiny and fine-tuning, as poor metrics lead to poor decisions. To me the benefit of points (or t-shirt size estimates) over something like hours is that in software development, hour-precision estimates give a false sense of accuracy. It requires more effort to provide more precise estimates, yet they won't be any more accurate (given the inherent non-repetitive nature of software development). Thus, the use of a coarse-grained measure like points serves to re-enforce the notion that the estimate are inherently imprecise and that we don't want to waste effort on greater precision estimates.
All that said, whether or not to use points or any measure of team productivity at all is a team decision. If the measure isn't serving the goal of improving the software and the software dev process, then change it or get rid of it. That's the beauty of the retrospective -- it explicitly encourages this sort of process-hacking and fine-tuning.
Embrace the Retrospective!