 |  |  |
| Main | |
The Market Software Development Paradigm
"Agile" and "Iterative" techniques have gained favor and even to some extent become part of the software development establishment. But they have not impacted large (team of 50+) information technology projects which gravitate to variants of the "Waterfall" methodology and BDUF (Big Design Up Front). From Wikipedia:
Agile methods are often characterized as being at the opposite end of the spectrum from "plan-driven" or "disciplined" methodologies. This distinction is misleading, as it implies that agile methods are "unplanned" or "undisciplined." A more accurate distinction is to say that methods exist on a continuum from "adaptive" to "predictive."
Developers tend to agree that the more adaptive methodologies don't scale well to large projects. I am proposing the Market Paradigm as a means to scaling these adpative techniques to large projects.
Even when the adaptive terminology is borrowed in the context of a large project it is likely to be subverted just like the "iterative" approach was disingenuously invoked as a last ditch effort in the Death March in an Information Technology Project. Here I talk about the potential of swinging to the opposite end of the spectrum from these predictive monoliths to a laissez-faire approach that brings about accelerated results through marketplace forces.
Market forces
Before talking about specific software methodology, think about how progress is made in the industry at large. In a FORTUNE Magazine article entitled Quoted Often, Followed Rarely, Daniel Roth interviews Fred Brooks who wrote the seminal book The Mythical Man-Month (1975). In talking about open source Fred Brooks says:
A second big advantage is that multiple versions of any particular component gets built, and the marketplace votes on them. In many cases one wins out. In many cases, one doesn't, and different versions circulate. That's confusing, and you get compatibility issues... Would you want to build an air-traffic-control system that way? The answer is no. That task requires a tighter degree of integration. Whereas when the community is building tools for itself, it's advantageous to have different people, with different ideas of what they want to build, build it, and then let the market vote.
I doubt I am alone in hearing occasional remarks about bringing "capitalist" forces to bear in software development. People will talk about how a little competition or meritocracy can help, that the re-use of a component or service can be left to depend on whether others find it useful and want it rather than by mandate from leadership.
The reality of waste
But the main obstacle to an acceptance of any real free market mechanisms within a single large project is the knee-jerk rejection of a model that accepts or expects what appears on the surface to be "waste".
Unfortunately I think that the lesson was not learned by the majority of people involved in the death march IT project I wrote about or others like it. Many will simply conclude that the primary methodologies were not at fault, it was simply a problem of execution. Well yes it was a problem of execution, but I would go a step further and say that these methodologies are prone to the abuse and failure that occurred here, and therefore we need to adapt a new approach that expects certain failures and waste in order to harness more natural human productive potential.
Waste is the Gorilla in the closet (the Mountains of Discarded Software I wrote about 7 years ago). Waste is all around us but generally ignored and explained away. A truly affective approach will recognize the realities and work together with the real human dynamics involved. The 3 laws of human nature in software development are:
- We are better at "hands-on" than design-by-meeting
- We are better at our little corner than comprehending the whole
- We are better at adapting than predicting
I'd be rich if I had a dollar for every time a manager said something like "research it, mock it up, prototype it, but don't actually start implementing anything yet!" The desire to control your work and the fear of wasted effort has put them into a state of denial about how software actually comes about! And I'd like another dollar for every time a prototype grows directly into an end product.
Micro managing independent teams
Large projects generally at least give lip service to the idea of "independent" teams or modules. The understanding is that the overhead of communicating within a team grows exponentially as the size of the team grows. So teams are organized according to areas of tight integration, separating teams where there can be loose integration.
In "Building the Machine That Will Build the Machine" I described the problematic tendency in our industry towards architectural frameworks. This tendency turns the integration effort into a counterproductive effort, meddling in the progress of the various teams for the sake of avoiding reinvention and inconsistencies. You are in trouble when project leaders talk about "independent" modules within their architectural framework, and yet every single thing they do increases the interdependence of the modules in the architectural framework!
The problem is that you simply cannot impose a framework if you're going to have truly independent modules. Despite this truism, most people claim that as long as the architecture is well-designed you will be all right. The hope is that by giving up a measure of independence you gain efficiences. But the better answer is not as far off as you might think. You can offer framework-like features instead of imposing them, and suddenly you've turned the tide in your favor.
Real independence
That's it! With a rather small tweak you change a restrictive micro managing intervention into an empowering opportunity. The problem many will immediately point out is that these framework-like features might not get used, and every team might end up "reinventing the wheel."
When a software component made available by one team is not used by another team, it is rejected because it is not adequate, it is not ready, or it is not worth it. The third one is often the hardest to stomach. Every integration involves jumping through a hoop -- a hoop that needs to be maintained over time. A team that properly takes responsibility for its own progress will weigh the benefits and costs, and the costs of incorporating an externally developed component should not be underestimated. Managers love the word "re-use" but it is costly to force it where it doesn't belong.
With the good kind of team independence comes the good kind of distributed responsibility for productivity. Independent teams will use their ingenuity to come up with a working solution rather than waiting to implement pre-designed pieces sent down from above. And that freedom to create solutions brings unrivaled productivity.
The "Market" masterplan is not a micromanaging one; it grows out of the confidence that five independent teams of 10 people will be an order of magnitude more productive than one team of 50 -- so much more productive that you can stand to throw away half of their stray work and still be much much better off.
A consistent product
But there is another legitimate concern: the concern for a consistent end product. The reality is that certain kinds of consistency are overrated. Consistency of the end product should be envisioned with an eye to the productivity of the end user, but not in a way that restricts the explosion of creativity and progress that the Market methodology can nurture.
Windows Office is safely the most-used suite of software products ever. Yet it never consolidated to one consistent macro/scripting technology. Instead it has had a hodge podge of mostly Visual Basic-like technologies over the years. On a related note, remember when Object Linking and Embedding (OLE) was going to usher in an era of "document-centric" computing where you didn't care if you were in Exel or Word? Yeah right! On the other hand, certain consistencies like similarities in toolbars and menus, cut and paste, are essential to productivity across the suite.
It really irks me that they couldn't consolidate many more things across their suite, but the reality is that certain kinds of consistency are not worth holding back an innovative product and those considerations should lose out to business pragmatism every time. Perfectionism and architectural aesthetics can kill a product more than inconsistencies between modules can.
Unifying what shouldn't be unified
One can't help but wish that the Microsoft Office team had come up with a single scripting technology. It would have been ideal, but it is another matter altogether when you try to find consistencies where they don't belong.
Here is a hypothetical example involving various formats of identifier codes used across several legacy systems. In H89A5673EZL, H means aircraft part, 89 is the model year, A is the model year sequence in case there were multiple models that year, 5673 is the part number, and EZL is the location code. Various legacy systems had ways of interrogating the alphanumeric part identifier codes to extract relevant information. This function was of course duplicated in each of the legacy systems with subtle variations. It seemed desirable to everyone that this should be unified across the whole new system and that it was also a great opportunity to introduce a new common nomenclature for part identifier codes.
You immediately think such thoughts as: the two digit year could eventually lead to a Y2K problem, and we need standardized location codes. Common sense right? But attempting to force a new nomenclature could be catastrophic. Firstly, "if it ain't broke, don't fix it." The legacy system part codes already work and are infused in a myriad of business processes. Secondly, it wasn't on the customer's requirements list, so unifying these codes is not or at least should not be a priority. Thirdly, it is paramount to recognize the significance of those subtle variations, and the likeliness of underestimating the difficulty of resolving them.
I observed a situation much like this and it turned out that consolidating the identifier codes was filed away under "nice-to-haves" because it was simply too difficult. But the real problem was that they were focusing on where they could unify the business logic rather than how to identify small doable projects and empower teams to complete them. Micro managing business logic will never scale to a large effort.
Winner takes all
It is hard to imagine that consistency might come through an unpredictable evolving process rather than by consensus beforehand or by being forced, but that is indeed the case in the larger industry with such things as UI advances from Apple and printer advances from HP where a de facto leader in an area sets the standard.
In the case of identifier codes you might find that as business processes evolve one of the systems might find the need to come up with a unified syntax for its own purposes that would slowly be adopted by the other systems, one by one until all of them use it. This is a natural process allowing old ideosyncracies to die away at their own pace.
In a large project involving multiple user communities and different subsystems, you might fear that user interfaces will be divergent if they are developed by independent teams. But actually if one team comes out with great user interface features, that will create expectations for the other teams to implement similar features, much like I suspect happens among the applications in Microsoft Office.
Ultimately, the exchange between teams should be in the form of "touch and feel," rather than design-by-meeting. In other words, "oooh, look what you got! we want that!" instead of reading documents and having discussions beforehand. I don't use the word "competitive" because it is not an overt competitive attitude, although there are underlying competitive processes going on.
Team independence does not mean the team does not answer to the overall goals of the larger project, or that it would reject advances provided by other teams; each team's success will depend on the utilizing and sharing of components and implementations between teams. If the Excel team at Microsoft announced a new direction on a complete tangent from the other tools in the suite, the Excel team leadership would be replaced tout de suite.
Divide by user community
For some reason in large projects they try to divide the larger problem along functional lines. The worst example is creating a team for user interfaces, another for business logic, another for data schemas, and another for reports. I described a case of this under "Separating DBA, UI, reports into teams" in Death March in an Information Technology Project.
I propose instead that you create teams for each identifiable user community or subcommunity. For example, in a typical Department of Defense scenario you would have a team for a group of users at headquarters and a team for each different type of group in the field. You might still have DBAs and reports specialists, but each team will have its own as it requires. Of course a great deal depends on the details of the project.
I have not spoken much about Agile techniques, but when you are regularly releasing a product to your users it is paramount to have a small well defined user base to develop a close relationship with.
A new perspective on waste
For a waterfall-minded project management, giving teams direct access to customers will undoubtedly be the most troubling aspect of the Market paradigm. It would certainly take strong visionary leadership to really open up the floodgates like this and get away from the traditional controlled requirements preparation of the Waterfall paradigm.
Well there is really no alternative! Eventually a piece of software is placed in front of a user and you get real live feedback on what is actually needed for it to be useful. This is where the rubber meets the road and I find it downright delusional at this stage to find anyone claiming that the years of BDUF leading up to this point were worth it when focused agile techniques could have yielded a similar result in a few weeks.
The principle behind BDUF is that it is much more efficient to resolve conflicts during requirements assessment and design phases than once a large system is implemented. However the human reality is that the sheer volume of details cannot be comprehended nor resolved until people begin to see computer programs actually doing things! In this light, the protracted periods of requirements analysis and design are much more wasteful than a program which does something wrong on the first few iterations.
Decoupling
The key to the Market Software Development Paradigm is in dividing the larger project into smaller projects for independent teams. A small project needs to stand on its own, with its own product management, design, testing, QA, user feedback, and release schedule -- everything. Projects are identified based on dividing lines of loose integration with other parts of the larger project.
If there are no obvious points of loose integration, you need to invent them, i.e. design them into your solution.
Developing a separate product for field and headquarters appears to be a terrible waste on the surface. It is probably the most controversial idea I am proposing, and I am not saying it is always a good idea. But if some task is a very large part of the project where you would be assigning 20 IT people, you might have a better chance of success by dividing it into two teams with 14 IT people each.
A task requiring so many people to develop would have significant data entry, reporting and analysis needs on the field side and you don't want to tie these down to development of the headquarters' module which has a whole different set of concerns. As separate products they would develop their own data schemas and the exchange of information between the field and headquarters would involve data interchange. This is not a bad or wasteful thing if it improves the responsiveness and agility of the development teams to their customers who are indeed very separate.
This is not a comparison between 20 people doing it one way or 28 (14+14) doing it the other way; it is a comparison between getting it done well versus late or never.
By introducing points of decoupled interoperability you create more work, but the work on those separated modules can be done more independently and therefore more quickly. In a large project this can mean the difference between soon and never! The challenge is in devising ways of dividing the large problem into smaller problems with separate user communities, not along functional lines.
Stovepipes
One of the words used to negatively describe legacy systems in the Department of Defense is "stovepipes," i.e. separate systems with very little interoperability. Funny that being "stovepipe" is always considered a bad thing, because it evokes positive aspects of independence, redundancy and security. No matter how intelligently some new age architecture is designed, interconnectivity and uniform design will open vulnerabilities to large scale problems and malicious attacks.
But rather than incrementally improving the interoperability of stovepipe systems (to improve something that already works) their first instinct is to replace it all in one go, much like the "Big Bang" method recently lamented by Microsoft CEO Steve Ballmer.
If you are willing to sacrifice the aesthetic purity of the "Big Bang" approach, you just might reap the rewards of a much faster adaptive incremental approach. Things are often not as they seem from the outset. Development can be greatly accelerated by allowing stovepipe systems to evolve in close relationship with their individual user communities within a larger project sharing features and components between systems. The result may bring you to a whole new, robust and interoperable (even consistent) system much more quickly that you would imagine.
Think simple
Since agile development is all the rage, it has inevitably caught the attention of managers of large projects that are traditionally advocates of BDUF. Often though it is assimilated as meaningless jargon, a mere dressing for the same old anti-productive processes. "Agility" is the new meaningless buzzword. People are still talking about designing and implementing large complex systems and investing in architectural frameworks. But my experience supports Gall's Law, which goes like this:
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
After witnessing failures of monumental proportions, it is clear to me that waste is a part of the process. The Market Software Development Paradigm brings this to a level on which failures are small and leveraged as part of an adaptive process, yielding a much better prediction of success.
| Main | |
