"Emergence" by Jack Wolf - 15" X 15" -- Oil pastels on posterboard - Creative Commons - Flickr
I am starting the third edition of my book. The 2nd edition came out in 2011, and already the industry and my understanding has transformed again in significant ways.
DevOps was emerging and I covered it extensively in the 2nd edition; it was clearly going to be transformative. What was less clear was the impact that Agile, Lean and DevOps would have on the practices of enterprise IT. This transformation of enterprise IT will be a primary theme of the third edition.
It's easy to fall into naive, triumphalist thinking that Agile and DevOps will conquer all, that somehow the old world will just fall away. That's not gonna happen. Businesses are being run profitably with sub-optimal IT practices (not everything is a Phoenix Project nightmare). Many senior people with substantial authority are vested in these previous models. And where are the new people going to come from? They will have to be the old people, suitably retrained. We're not going to see a mass retirement of the old guard and a mass onboarding of the new thought. 2.5 million people have been trained in ITIL to date... and what if process frameworks are an unavoidable feature of scale?
Instead, like any significant social transformation, this one will be protracted, complex, contradictory, and experienced along a myriad of fronts.
In understanding how this may play out, I've engaged in personal re-skilling and re-invention, starting with the creation of a Continuous Delivery pipeline simulation. As I was doing this, Peter Brooks asked "How can the requirements register be linked to this so that each part of the skeleton and its behaviour can be confirmed to align to corporate vision, charter and policy?"
I promised further writeup, so here it is.
Establishing line of sight from hands on infrastructure work (e.g. Calavera) all the way to corporate IT strategy is a tall order, and exactly what I hope to do with the InsanIT project. This is a major project and will likely consume the remainder of my career.
In building this line of sight, I am conscious first of the training and educational issues. I'm in my 3rd run of my semester-long IT infrastructure course, and it's been a challenge to educate students on the gamut of material from hands-on infrastructure as code all the way through service portfolio management. Some might say it's an impossible scope, but necessity is the mother of invention.
The progression from hands-on infrastructure through IT portfolio can be seen as a matter of increasing scale, in four stages. The thought experiment I present to my students:
Stage 1: You are a solo practitioner or two in a startup, in a garage. You need a complete pipeline, stripped down to the essentials (e.g. Calavera).
Stage 2: Your organization is big enough to need basic collaboration and operation tools
Stage 3: Your organization is big enough to need a formal process architecture
Stage 4: Your organization is at a scale where long-horizon lifecycles and their emergent dynamics must be managed.
Notice that the stages can apply both to an actual organization (as it grows from startup to established concern), and also to an individual's career journey through an organization.
While I am not fond of maturity models, this does have some comparable aspects. I also see related themes in discussions of Dunbar's number, such as this account of Polyvore and this discussion of startup sizes.
In hindsight, it's surprising that there's been so little systematic exploration of this aspect of IT management in terms of the process frameworks. Most discussions seem to fall into a binary "you're either big or you're not." Certainly, this is a new dimension for my work.
Here (as previously posted) is a visualization of the model:
This is the new structure for the book, starting from the bottom up. This hopefully will make the book more suitable for classroom purposes!
- Inception and collaboration: Cynefin Obvious (but with a big asterisk, because we are dealing with computers)
- Elaboration: Cynefin Complicated
- Maturation: Cynefin Complex
Chaos as defined in Cynefin can emerge at any point...
The architectural implications progress as well:
- Foundational delivery pipeline (without this, the rest is meaningless)
- Simple collaboration capabilities (kanban, basic ticketing, etc)
- Then more differentiated processes (ITSM, but with a skeptical, incremental, and queue-aware approach)
- Then finally the sensing mechanisms (including architecture & analytics) needed for adaptively coping with and directing complexity at scale
As always, boundaries are of architectural interest. Not to mention controversy. The debates are archetypal:
- the "lone wolf" programmer who won't collaborate
- the cohesive Agile team in conflict with enterprise PMO and ITIL processes
- the enterprise architect seeing technical debt and failures of execution management emerging from the process architecture
There are no "solutions" to this emergent complexity, but in order for analytics and sensemaking to work at all, we need traceability through the layers. That will be detailed in further chapters, through examination of critical linkages, e.g.:
- Source control commit
- Work activities
There will be significant updates to the metamodel and associated systems architecture. I have not decided yet as to how I will represent emergence, but I suspect I will have to do a progressive architecture, that is enhanced and refactored for each chapter. Much to be determined... and more to be said on the topic of emergence and how the layers and their relationships will evolve as Agile continues its transformation of the enterprise.