Podcast here: http://devnology.nl/nl/podcast/10-content/224-devnology-podcast-032-philippe-kruchten

Via iTunes: http://itunes.apple.com/podcast/devnology-podcast/id335170002?mt=2&subMediaType=Audio

This feels to me like one of these strange agile koans. If you repeat it often and long enough, it will gradually become the truth.
Self-organizing teams have very little to do with the architecture of your system. Self-organizing teams are about: task allocation, collaboration, communication, accountability, … it may have to do with time-boxing and therefore what gets accomplished in a certain time-frame. Architecture is about making decisions (choices) about the structure, composition, organization of the software system. It also feels like “architect” is necessarily not a member of the team. Not my personal experience or recommendation. Architect is a role, not necessarily a person (whose only role is to be an architect, though these exist in large organizations.)

Most systems tackled by small agile teams have a pre-defined stable architecture. So yes, they do not need to have anyone playing the role of architect. For novel complex systems, where architectural decisions need to be made, if they are made “by the team”, it means that the team plays the role of architect, and hopefully they have the knowledge and experience to do so. Like some teams have a scrummaster and a product owner, such teams should have an architecture owner, who drives the discovery of architectural issues and their resolution. Because an architecture is not going to gradually emerge out of weekly refactorings (another agile koan), unless this emergence is guided somehow.

There was a whole issue of IEEE Software magazine last year dedicated to the interplay agility-architecture. Start here: http://www.computer.org/portal/web/computingnow/archive/april2010 . Some of my own prose to be found there.

A group of researchers is looking to issue a Manifesto for Rich Software Process Models. Here’s my position on this topic.

“… perfection is achieved not when there is nothing left to add, but when there is nothing left to take away.”  Antoine de St. Exupéry, Terre des Hommes, 1939, chap.3

Over the last 30 years we have tried very hard  the rich process models, and we have not been extremely successful at it. Maybe we should try lean and mean software process models, rather than making them “richer.” At minimum, we should try to analyze why the rich approaches have not worked; where they failed. Could it be that we were trying to solve the wrong problem? or that the real problems by far overshadow the process model issue? Or maybe the whole construction paradigm we use for software development is not adapted anymore? My position is that we should try the route of very simple software process models, to ensure a wider applicability, greater versatility, and acceptance. Possibly these new process models would be based on other paradigms of software or system development than the “technical-rational” construction idea. I would be wary of richer process models.

The Great Process March (1987-2001)

Many of us (old enough) had a great “aha!” moment when Lee Osterweil told us in 1987 that “Software processes are software too”. We elaborated on this with process languages and models. We tried to implement them in various Integrated Programming Support Environments (IPSE) such as the wonderful and clever Rational Environment^® (1987-1998). We embraced object-oriented modeling to build processes like Objectory [3], then the Rational Unified Process^® (RUP^®)  and many other such processes or process frameworks. We extended this to quasi-standards like the Software Process Engineering Metamodel (SPEM 2.0), and real standards, such as ISO 24744-2007 Software Engineering — Metamodel for Development Methodologies (SEMDM), as well as the tools to go with them. Some of this great work branched out into business processes, with languages and tools to support them. And I am sure I am missing a great body of academic work here, having skipped 6 or 7 years of ICSP(P) installments. Other approaches were brought in, looked at, tried out: software processes as state machines or Petri nets, software processes as collection of interacting agents, software processes as complex adaptive systems, etc., etc.

But what have we really achieved? (by 2010)

In most cases, the process models look at transformations: input-process-output, and therefore have an excessive focus on artifacts, and decomposing these artifacts into elementary constituent elements. Such processes are in practice very hard to configure. They are also rapidly misused. I have seen do half of RUP^® implementations  fail, caving in under the weight of the artifacts that the process “forced” (so they say) developers to create, manage, update. See this tongue-in-cheek paper.

In a great reaction 10 years ago, the Agile manifesto started to hack at the base of the tree trunk by claiming that we should favor “interaction between individuals over processes and tools”.

Should we blame the process models? Are they not rich enough? Detailed enough? Flexible enough? They are. Maybe we are not addressing the right problem? Our fundamental metaphor: “this is a process, it looks like a program, the machine is a group of humans” is the wrong metaphor. Machines are very deterministic, humans are not; programs process well-defined input; all software projects are different; even the same software project would not be done the same way twice.

Have we been modelling the right thing?

Maybe even the construction (and architecture?) metaphor is flawed—at least when we push it too far. It does not account well for the creative, trial-and-error approach, nor for the interactions and collaborations between individuals.

Consider the “nice party metaphor”: you organize a party this Saturday at your home. You invite a group of people, order the food, move the furniture, and you record carefully: who comes, when, who eats what, drinks what, etc., and everyone leaves having had a good time. The next Saturday, to have another great time, you decide to have the very same people, the very same food and drinks, you ask them to dress the same way, arrive at the same time, you introduce them to the same people, bring them the same glasses with the same drinks, make the same jokes,… will you be guaranteed to have a nice party again…? Hadn’t you captured the exact right recipe? (Source unknown)

Are software development projects replicable enough to benefit from repeatable, prescriptive software processes, supported or enforced by tools? The many adopters of lean and agile approaches seem to have voted (“no!”) with their feet, and moved away. The spectrum of software development circumstances and contexts is very vast (see my posts on contextualization).

In search of new paradigms to represent software development

We need very simple models of software development. Models that are simple enough to have very wide —quasi-universal— applicability. Models that also encompass the large number of variability factors, not only technical variability (size, technologies, platforms, tools), but also social, cultural, linguistic, legal, commercial factors, etc. Models that move away from the “process as a program, the team as the machine” paradigm. Simple and robust models that are not based on the construction metaphor, the gradual, progressive transformation of various artifacts. Models that can incorporate creativity, knowledge management,  social interactions, trust, shared mental models.

Probably these simple models can gradually, in steps and in layers, or just in some narrow pockets, be made more and more complicated and rich in details and expressiveness, but we should not start with the complex and detailed and super-expressive before we have identified the right paradigm, and the handful of foundational principles that come with it. Revisit Richard Gabriel’s “worse-is-better is better”.

We need to start from what people do (observation) and how they conceptualize what they do (mental models, shared or not), leading to a descriptive approach, that offers choices, possibilities, and not from what we think a priori they should be doing, the prescriptive technical-rational approach. Jane Jacobs said the same thing about great American cities.

In this direction, years ago (1996), Joel Jeffrey had attempted to look at software development from a different perspective, folding in people and collaboration in his paper “Addressing the essential difficulties of software engineering”. More recently, Paul Ralph at UBC proposed a significantly different paradigm for software development with his Sensemaking-Coevolution-Implementation Framework. I tried myself to develop a conceptual model of software development (the frog and the octopus), around:

• 4 concepts: Intent, Product, Work and People
• 5 attributes: Time, Uncertainty, Quality, Value and Cost
• and a handful of variability factors: Size, Business model, Rate of changes, Geographic distribution, etc. See also Scott Ambler’s Agile Scaling Model.

The tools that successful software projects use today are also very simple and lightweight: scrum or kanban boards, tools allowing people to keep track of activities and bring to the developer information associated with a certain (type of) activity.

Richer process models? Danger!

I would be very concerned that “rich” rapidly means heavy, detailed, complicated; that rich means taking the models we have and enriching them with additional concepts, more entities, more details, more relationships between them. I have the same worry with the SEMAT initiative. Can we have rich (in expressiveness and usefulness) and lean at the same time?

I look forward for the face-to-face debate in May!  I have to confess that I am shooting from the hip, here, not having a clue what the proponents of a “Rich process model” have in mind. Maybe we’ll find ourselves in furious agreement. In the meantime I welcome your comments (here on wordpress). It will help sharpen my arguments.

Note: the nice party metaphor is not mine, but I cannot remember where I heard it… if someone knows the origin, please let me know.

Click here to see the video on Youtube

Video on YouTube: The daily Stand Up Meeting

.

The Daily Stand up by Philippe Kruchten is licensed under a Creative Commons Attribution-NoDerivs 2.5 Canada License.

During the 10 years agile celebration meeting in Snowbird, UT, organized by Alistair Cockburn on February 12, after covering the walls with a couple of hundred issues cards, David Anderson noted that there was “an elephant in the room”, a topic that few are willing to debate in the pen, namely the Agile Alliance (its role, mission, accomplishments, etc.). After the lunch, a small group gathered and identified a few other such elephants in the room, other topics that the agile community is not really willing to tackle for a variety of reasons. We ended up with a long list of about 12 such “undiscussable” topics (or at least not discussable in the open).

Here they are, with an additional sentence to explain what the title is about, based on the best of my recollection (see photograph at bottom):

1. Commercial interests censoring failures
The key players in this community have a direct financial interest and have fear that any negative info would be amplified, distorted, and turned against them
2. Pretending agile is not a business
see above
3. Failure to dampen negative behaviours
We do not face, analyse failures and limitations of our assertions, claims, practices (see above)
4. Context and Contextual applicability (of practices)
We fail to describe the context in which some practice works, or does not work.
5. Context gets in the way of dogma
By evacuating context, we constantly return to dogmas, bigotry, claims of universality
6. Hypocrisy
… is a bit at the root of “elephant in the room” (we actually know all this).
7. Politics
A more systematic and thorough acknowledgment that organizational politics play a big role
8. Anarchism
… in the community itself, preventing a more systematic organization of the body of knowledge
9. Elitism
… as a defense (against failure) we limit our message to the best… and blame failure on the “unenlightened” others…
10. Agile alliance
Divergence of views as to what it role has been, was supposed to be, will be…
11. Certification (the “zombie elephant”)
This massive elephant was reported dead a few times, but seems to reappear…

12. Abdicating responsibility for product success (to others, e.g.,product owners)
13. Business value
… mentioned everywhere, but not clearly defined, or pushed onto others to resolve
14. Managers and management are bad
… as an instance of pushing any failure to others?
15. Culture
… better understanding of the concept of culture, at the organization level and at the national level, and its subtle interaction with practices, an integral part of context.
16. Role of architecture and design
… depending on context
17. Self-organizing teams
18. Scaling naïveté (e.g., scrum of scrum)
19. Technical debt
20. Effective ways of discovering info without writing source code

… these last few (16-20) are practices where a large level of variability exists, and little in-depth analysis has occurred.

This is quite a large herd of elephants. So after a night to sober up, here is my view on the elephants we have in the agile community room.

1. The agile alliance elephant

I’ll put aside this one, since I have nothing to say about it. I noted a small group of former AA board members tackled this elephant late in the day. Maybe it is just one instance of the “Community leadership” elephant which would include other organizations…

2. Failures and limitations of agile practices, & context

While the community has been god at amplifying what works, it often has not been good at dampening what did not work, analyzing why it did not work, or under which conditions it did not work (context!). There is in some cases a certain level of hypocrisy (many interested parties know this, but… pretend it does not exist or return to the official dogma).

This elephant has several causes, themselves elephants (i.e., “undiscussables”):

A) Commercial interests
Many of the key players in this community have a direct financial interest in selling something agile (tool, consulting, training, new idea), and there is this latent fear that potential buyers would be detracted by any hint of negativity, that any bad news would get unduly amplified, that it would in the end turn against them or again a whole community.

B) Decontextualization
In most instance, when practices are described, too little of the actual context is described, leaving an impression of very wide applicability, if this wide applicability is not actually claimed. Sometimes this is just pure dogmatism, or bigotry (see ).

C) No obvious way to make progress
Unlike other disciplines, like medicine, there is very little reporting of failures or limitations in software, and no clear venue or even templates or examplars of such reporting. And again the fear that this would reflect badly on the person reporting, or the “victim” or “culprit” organization. A good template for such report would include a detailed description of the context.

D) Limited objective evidence
There are too few people gathering objective, significative, scientific evidence on our practices, except for a very few that are easy to instrument (pair programming, for example).

E) Reasoning fallacies
such as undue generalization (it worked in two cases, therefore it will in all cases), and cognitive biases: anchoring, golden hammer, cargo cult, etc. Others reasoning fallacies include: non sequitur, and Post ergo propter ergo (correlation implies causation, or “guilt by association”), etc.

The two smaller elephants: elitism and hypocrisy are probably in the shadow of this main one, and the anarchism of our community does not help to organize a body of knowledge.

The practices in question for which there is little evidence or understanding of the role in various contexts are also some of the smaller elephants in the list above, often undiscussable (beyond loud rhetoric and posturing):

• thorough articulation of the concept of business value,
• the role of architecture and design, technical debt,
• self-organizing teams,
• writing code as a priority, etc.

3. Politics and culture elephants

These two elephants would require more investigation. I do have some personal opinions on the impact of both national and organizational culture on agile practices, and some investigation has taken place in the Global Software development community. But I know nothing about politics, neither inside the community or as an impediment in organizational change that could be addressed explicitly by our community.

Elephants in the agile room: cards in Snowbird

Slides:

… on their way

The frog’s view point (=the elements that are common across all software projects I know).

Conceptual model of software project management: commonality

See more in this Frog’s point: conceptual model

The octopus’s viewpoint (the contextual factors that make the spectrum of software projects so vast and interesting):

Contextual model of software project management: variability, context

See more here: Contextualizing software development

And in summary: we need both to really understand software development project.

Note: we love to eat frogs and octopus. The ultimate fusion between French and Japanese food.

• Steve McConnell, Notes on Technical Debt (2007), Blog post
• Martin Fowler, Technical debt quadrant (2009), Blog post
• Useful material on this topic from folks at the Cutter Consortium (I. Gat, J. Highsmith et al.) is unfortunately more hidden. (Actually, they offer the whole special issue for free, for a short while: http://www.cutter.com/offers/technicaldebt.html.)

Technical debt is more a rhetorical concept than a technical, scientific or ontological concept, but it seems to resonate well with the software development community, sometimes with managers and business people.

From a scientific viewpoint, we know little about technical debt. I take part in a little research project sponsored by the Software Engineering Institute (SEI), with colleagues Ipek Ozkaya, Rod Nord and Nanette Brown to investigate technical debt.

1. As part of that effort we have organized a workshop on technical debt on June 2-3, 2010 in Pittsburgh,
2. The results of this workshop are consigned in the paper “Managing Technical Debt in Software-Reliant Systems”, presented at the Future of Software Engineering Research (FoSER) workshop in November 7-8, 2010 (preprint).
3. We also created a little board game to illustrate the concept of technical debt. It is called Hard Choices, and is available to download and play (and improve) under a Creative Commons license. I’ve played it with various audience, architects, business analysits and university students in many places around the world.
4. We will organize a workshop on technical debt in May 2011, at ICSE, in Hawaii and we invite practitioners and researchers to come and share their findings, opinions, methods,  on technical debt. Se the call for contributions here.
5. I made this month several presentations on technical debt while in the Netherlands. See my Talks page, or this link for a copy of the slides.

Amsterdam students playing Hard Choices

A longer academic paper was published here.