"Frankly if every anonymous post disappeared from the Joel on Software discussion group, the overall quality of the conversation would go up, way up, and the discussion would be way more interesting."

I'm a fan of Joel's writings, but in this case he couldn't be more wrong. Funny thing is how he could start with Dave Winer's argument about personal blog comments (which I whole-heartedly agree with) and then draw the wrong conclusion about forum posts and have all of the resulting discussions fall completely into this thought groove.

All of the discussion following Joel's post centered around "the problem" of anonymous posting, and the higher quality of a discussion limited to logged on posters. They also argued about whether mandated logons would be abused or be inconvenient and so forth, but few glimpsed the yawning chasm of irrational leap under this notion that anonymous posting hurts Joel's forums.

Let's break this down in order to observe the thought groove in action:

1. Anonymous posting sometimes results in more trolls, spam and careless comments
2. These are bad, therefore they hurt these open discussion forums
3. Therefore these forums would be better without anonymous posting

Woops! The last leap is the unfounded one. Due to the sheer volume of comments weighing in and quibbling over trolls and spam, everyone ignored my feeble attempts (and those of a few others) to point out the importance of anonymous posts.

That's groupthink when a contrary yet simple point cannot be heard over the din.

So to get more facts I've gone ahead and taken a sample of 131 Joel On Software Discussion Group posts and broken them down according to whether they were anonymous or not. I used strict criteria to determine whether a poster is anonymous. If a logged out poster provides a protected e-mail link, he is not considered anonymous, even though there is no way a reader can identify him. Anonymous posts are strictly posts where the tagname is empty or plain text, i.e. no link, no protected e-mail button, and no logon checkmark. Feel free to e-mail me for the full data in XML format.

• Total posts examined: 131, total comments: 2242
• Anon posts: 86, comments 1478 (avg 17)
• Identified posts: 45, comments 764 (avg 17)

Wow, significantly more anonymous posts! And based on thread size (number of comments) the anonymous posts generate the same average interest as the others! If I included anonymous posters with only a protected e-mail button, the dominance of anonymous posting would have been even more clear.

The anonymous posts are often generated by people who logon for other posts. Why are they anonymous for some and not for others? Obviously it is because some topics are different from others; some are better kept anonymous. Here are some examples of anonymous posts where I show the title of the post, the anonymous tagname the poster used and the number of comments generated.

A lot of valuable anonymous posts give insights into goings on at work, e.g.

• I am being setup as the fall guy Anony for this, of course (14)
• Stupid interview question, courtesy of Wendy's Just back from lunch (30)
• Pronouncing people's names from other countries/languages Cat's Got My Tongue (for this one) (29)

Another valuable category is those letting off steam or in turmoil:

• Gmail Stupidity... (Receiving someone else's mail) Anon Ranter (27)
• i almost give up family IT person (22)

People who understandably don't want to link themselves to a wish or self-judgement:

• What's a guy gotta do to make some money in this field? Kramer (88)
• Expected Salary Paul (20)
• Best way to convey pattern knowledge on resume Wannabe architect (7)

Also there are posters who want to share sensitive revenue information or Micro-ISV internals, and many other kinds of posts that absolutely should be anonymous so the poster can discuss the subject freely. It is also useful how the tagname can be used to set a frame of reference for the original poster who is effectively moderating the thread, e.g. "family IT person."

You could argue that if you disallowed anonymous posting, then those with something valuable to say would step up and logon even though they might still use an anonymous tagname without a link. Adding inconvenience is a troubling and sensitive hot button issue in web sites, and would it actually serve the declared purpose if they are logged on and yet still essentially anonymous? Messing with such a large majority of the posting activity on the forums is also a big concern. Does a quieter forum continue to attract new readers and first time posters?

For me, and for many like me, chiming in with personal experience is out of the question if I have to attach my identity to it for any google search to pull up! Yet it is a valuable contribution, in fact moreso because it is anonymous.

So it is very interesting that the general consensus was that eliminating anonymous posting would improve the quality of the site's discussions when it really is a questionable conclusion. Now if this had been a discussion about the means of filtering spam and trolls by using logons or captcha to limit contributors, then it might have made sense to focus on anonymous posting as a problem (though that kind of site policy discussion is forbidden on the JOS forums). But it wasn't that at all. It was a discussion based on the idea that a user who wasn't willing to put his identity behind what he said online was not a valuable contributor to the discussion. This ignores the tremendous value of anonymous posting.

The Joel On Software forums provide a free and high quality venue for posting both anonymously and not, depending on the context. It would be absolutely devastating to the forums if anonymous posting was disallowed.

I think that when Joel said his forums would be better without anonymous posts he was failing to see that open forums serve a different purpose than personal blogs. One way to say it is that a blog is moderated by one person providing a certain consistency but also limited by that, while a forum offers a less directed but potentially fruitful gathering of many. Anonymous posting is a key feature of open forums.

I'd like to see Joel acknowledge he was wrong on this, but since he cannot do it anonymously, I doubt he will.
:)

Get this: developing software will just be a matter of plugging intelligent components together.

This claim comes up often in discussions about the current and future state of software development. It is the recurring dream of component-based development where pieces of software can essentially figure out what a programmer wants because other programmers have planned ahead. It is the wishful thinking of architecture astronauts and people who trivialize the hands-on design and redesign stages of development.

Integration problems are not obvious

You cannot design two adjacent pieces of a car in isolation; they are interdependent and you must integrate them. Even if they do not touch they must allow room for each other and account for things emitted like heat or condensation. That is a big challenge because it involves knowing the required capabilities and the needs of each piece. And the more interdependent pieces you have, the bigger the integration matrix.

Everyone can see your cool skinnable app, but no one can see that you had to re-implement your cool spinner control because it was incompatible with your skinning technology.

Integration is often the part of the problem that is not immediately apparent. The part that takes all the time, but for which you get no credit.

Reusing source versus binary

Generally you want to reuse as many pre-existing technologies as possible. Developers always caution against "re-inventing the wheel." To reuse something always involves integration, but there are different levels. One clear distinction in programming is source versus binary. Integrating with a source technology is always less risky, because you can see what you are integrating without depending on documentation, and you can fix it without depending on another party.

A car maker does not design a windshield wiper timing mechanism from scratch, but reuses (borrows or licenses) technologies and inventions into their design not as prefabricated hardware, but as bits of expertise much like source code.

When deciding what to reuse you should consider the integration costs. If you are taking responsibility for making your product work and it depends on an unfamiliar third party binary "black box" component, it is a risk that should only be taken knowingly and if there is no better alternative. There are also many binaries we know and trust, such as Win32.

Doing it yourself can be cheaper

Even though the windshield wiper mechanism (the mechanical part, not the blades) sounds like a great candidate for a "plug-and-play" generic part of a car, every manufacturer still implements its own to integrate with the form and function of their own vehicles. They could outsource this part but the communication and team interdependencies during design iterations and implementation troubleshooting would likely be too costly.

Don't you think the software job title of "Solutions Integration Architect" is curious? I probably even applied for a job with that title once. No doubt the position came into being due to the idea that useful software can be architected by identifying components that will meet requirements and then integrating them together. But it is dangerous if it leads one to assume that integrating pre-designed components is always easier than building custom components.

The #1 dev problem at Microsoft

In The Windows Shutdown crapfest, Moishe Lettvin shows that the interdependence between his team and the shell and kernel teams brought design and implementation of the Start Menu to a standstill.

The problem is not that no one was given the power to make the decision because even if a decision was made, it could eventually be overruled by the realities of the software architecture. It is not necessarily the number or quality of people as Joel seems to indicate, it is the interdependence or intersection of the responsibilities of the teams established by the architectural design they've inherited.

To make matters worse, Microsoft has pursued architectural choices like COM and .NET which escalate integration hurdles into the realm of deployment, rather than making programs more modular and independent.

COM is supposed to offer modular object oriented development and yet it ties your hands with respect to the most sensitive aspects of modularity -- versioning and side-by-side installation. Few if any COM components are shipped bug free, but a tool which ships with a 3rd party registered COM DLL cannot be installed without affecting a separate and unrelated tool installed with an earlier version of the same 3rd party COM DLL.

In .NET improvements for MicroISVs, a Microsoft representative admits the challenges of integrating .NET into your product installation. I've also been involved with trying to deliver a .NET component with other products both non-.NET and from other .NET versions. Imagine asking a user to install 2 versions of the .NET platform from one install! (We didn't, but we have more work to rebuild the component for different versions of .NET.) And in another post, a Microsoft program manager tells developers not to use .NET to develop shell extensions because the .NET version may differ from another loaded for a different handler.

Integration is the primary risk in managing a project

Integration roadblocks often rear their ugly heads late in the game. After the obvious requirements are satisfied you get stuck with putting out the fires of unforseen integration issues. This happens with 3rd party components and libraries such as a migration tool that has install requirements that conflict with another 3rd party tool, or a 3rd party binary that has a bug that shows up after field testing. It also comes up in projects that use multiple frameworks like a web site with Java, Resin, Webwork and Freemarker when a feature of one is undermined by a workaround in another.

All these problems stem from earlier architectural decisions that ignore the primacy of the integration problem. These decisions end up causing developers to spend their time solving problems of integration, rather than bugs and features that end users are concerned with.

There is no more fundamental impact on development productivity than this class of problems, and this was instrumental in the Death March in an Information Technology Project story, and awareness of this problem underpins The Market Software Development Paradigm. Good software architecture's primary concern is setting up team responsibilities and framework/component choices that maximize independence.

Some will insist that a good idea is priceless. But I am talking about business ideas. A business idea creatively connects your real circumstances with a real possibility of turning a profit. Such an idea is valuable only in the sense that the first step in a marathon is essential.

People assume it goes like this:

1. Entrepreneur gets Idea
2. Entrepreneur puts it into action

But this is a terribly inaccurate way of describing the process. To explain why it is misleading, let me go through a few things first.

Keeping the idea a secret

One of the common instincts when you have what you think is a great idea is to keep it secret. If you discussed it in a forum, why then everyone would jump in like a pack of, let me see, not a pack of wolves, that wouldn't be right, this is about creating, not devouring. I want a metaphor that evokes something trained and ready, waiting for that one single chance to build something amazing.

What is that animal in nature that sits for days and days, scanning the landscape, waiting to discover a great idea for what to create? There is no such thing! In nature, animals wait for something to eat, not something to create.

There are people who sit on their butts looking out for that next great idea, but you'll notice that if they do like an idea it is never quite perfectly suited to them and they soon grow disillusioned with it. People who can truly create things are already busy creating things, so busy in fact that they will not likely be able to change course and shift all that momentum because of someone else's bright idea.

Looking for an accomplice

Sometimes you have an idea and you even have something started and you feel you just need someone to understand your dream and put great passion into helping promote it and bring it to reality. Actually I think to some extent everyone hopes for this at some point. But again I don't recall reading a single story where this happened, except perhaps the one where Simon Peter dropped his not unprofitable fishing business to dedicate his life to the new purpose he had just discovered.

Your partners will be partners due to relationships and coinciding circumstances, not due to your idea.

Are you really an entrepreneur?

Creative people are rarely idle. They are trying to accomplish something, tinkering with a car, painting or planning excursions. Creative people are not necessarily entrepreneurial.

I think you know if you're an entrepreneur at heart. When you were a kid you were the one who sold lemonade out front; you advertised to draw signs for a small fee; held a raffle at a family event to help pay for that go-cart. And in each case you had discussions with adults about the monetary logic behind it all. Hey! You were already negotiating with potential board members!

And you rarely created anything you were proud of without wondering if you could make your first million off of it.

A buffet of ideas

Benji Smith challenged himself to blog about 30 ideas in 30 days in an effort to decide which idea to focus on (he felt compelled to explain the thinking behind the ideas to show they weren't flippant, so it took longer than 30 days). The other reason he did it, I suspect, was just to show how many legitimate ideas one could have and how ideas are the easy part. Well, he finally made a choice and later a new choice and who knows where it will lead.

Reading about other peoples' ideas and business stories helps you to recognize what makes for a good idea. Even your own ideas might be ones that someone else could do better than you. Ultimately the best idea for you is going to come from what you're already doing. The key to a good business idea is that you have a unique opportunity to execute on it, and the best place to look is in what you've been doing recently. Yes, you're a software developer, but what specific project captured your interest, had you so fascinated you couldn't stop working on it?

Idea versus execution

If an idea is truly right for you and your circumstances, then you will have the fascination with it necessary to execute on it, and it is unlikely that anyone else will be in a position to hear about it from you and execute on it better than you.

You'll see it discussed a lot, that "it is all in the execution." It is a common response to would-be entrepreneurs who go off every second like firecrackers, they've got the next big idea, the next hot thing, and its just hanging there like ripe fruit for the plucking. Then nothing happens.

I regularly run across what I think are fabulous ideas. So often that it can actually be an obstacle to getting other things done as I consider the possibilities. Someone will get together with me and discuss an idea hoping I will magically find an abundant as yet undiscovered reservoir of time I didn't know I had.

Actually, there is no "Idea" with a big I

Although in some cases there might be what could be considered an "initial" idea, there was probably already something before that. In truth, there is really no "Idea" with a capital I, there is really only a steady stream of execution constantly adjusted according to a steady stream of ideas.

At the top I said this was inaccurate:

1. Entrepreneur gets Idea
2. Entrepreneur puts it into action

The real story goes more like this:

1. Entrepreneur has many projects hobbies and contacts
2. Entrepreneur discovers a genuine prospect in amongst these
3. Entrepreneur puts his full attention on it
4. It changes, perhaps one facet of it becomes the main focus
5. Entrepreneur keeps at it

Ideas are not worthless

The purpose of this is to temper your sense of the value of an idea. Of course an idea is priceless in the sense that so many of the blessings we experience in life are priceless. But for an entrepreneur, business ideas are like a fluid running in and around (and coming out of) the things we do. The milestones are built on actions, not ideas.

Are you tired of smug successful people expanding their fame and success by publishing articles about how to become successful?

And it all comes down to this: be really good at it! How helpful is that, really? Oh yes, they mix it up with lots of catchy turns of phrase, like "Fail your way forward" suggesting that they made lots of mistakes along the way. Did they really? I kind of doubt it. I think their mistakes are just little post-it notes on their unstoppable march to a destiny of success.

Thousands of blog posts tell us how to build a blog into a revenue generator, and yes we eat it up: the best ways to go laughing all the way to the bank on one blog post per day. Can you stand a guy who says having a job is dumb and he's making $40000 per month on his $9 blog while he spends time with his family? Let's face it, I can write, but I can't write like that; my strengths lie elsewhere. And please stop telling us to create income generating systems. It is insulting! Don't you think we will, just as soon as we figure out how?

Hanging on every word of Joel Spolsky (my favorite software industry inspiration) is no better. As he exposes his journey of building a software startup into a success he makes it all sound so sensible and straightforward. Does that help me? No, not a bit. It has been interesting to watch but nothing that he did has worked for me.

Some people might say Joel is an inspiration to them. Knowing about him may indeed be an inspiration, but that is different from following his advice or trying to do what he does. I hope I am failing to be perfectly serious here, but might I suggest that the things that worked for these people were likely specific to their strengths and the specific opportunities they encountered along the way?

What is ridiculous too is the thousands of get rich books and schemes on the market that make the authors richer, complete with TV infomercials full of laughable customers prepped and glamorized to help you make believe they're the next Donald Trump.

Oh yes, and Donald Trump and his advice to "love what you do." Mr. Trump, do I love buying and selling companies and real estate? No I don't. Okay fine, it is valid advice if you read the small print, but my rant is this: who are you, Mr. Trump, to tell anyone how to be successful?

Oh yeah sure. Follow these 10 steps, get organized, be consistent, unlock your potential, blah blah blah. If we could really do this beyond 10 minutes of motivated euphoria, wouldn't we already?

When I was ten I climbed onto the roof of my house (where we weren't allowed) with my bad boy cousin Todd while my parents were away. And we were talking about how to jump down onto the balcony which was an 8 foot drop from the eaves (nothing dangerous). Well we were both hanging from the eavestrough when my parents drove into the driveway. My cousin bravely dropped but I didn't dare and no matter how he tried to convince me I was too petrified to let go. So my dad found out where we'd been (and he "rescued" me too).

But in case you missed the point, yeah my cousin was able to do it; he was successful in dropping to the balcony undetected. But so who was he to try and tell me I could do it? He was the brave one.

What I am saying should be completely obvious at this point (if I could write worth anything): who are successful people to tell you and me how to be successful? They're successful, so in that respect they're not like us. And therefore they can't tell us how to be successful.

In fact the specific thing on which they are lecturing us is precisely the thing on which they have nothing in common with us!

You and I can be successful but let's not look for instructions from the juggernauts of success. Seriously, would you ask the most popular kid in school how you can be popular?

All these success messengers truly defeat the purpose of achieving success. They distract us. They reinforce the fact of what we don't have, they remind us of what we are not: successful.

So I am going to do myself a favor and tune them out. And I invite you to do that too. And yes, I am the right person to suggest you do that because I am just like you, I have been completely unsuccessful at it.

The ABO has an annual schedule that revolves around U.S. budget cycles, and they have both civilian and military personel working together on budget coordination. As I recall there was a staff of about 5 and then about 15 analysts would be there at headquarters during crunch time, and many more people were involved. The coordination in the ABO involved gathering and working with lots of forms filled out by people at the field offices as well as the decisions made in budget meetings.

During certain times of the year it became pretty hectic as they put together the issues and decisions that were the basis for the large budget books that would end up being combined with other branches of the Department of Defense and ultimately with all the programs under the U.S. government.

The software we built to help coordinate this was primarily in Microsoft Word using the language WordBasic.

Word was in many ways an excellent solution because it allowed precise simulation of the age-old forms the Army has been using for decades, and through Word's template feature it could be distributed merely by sending the document via e-mail, since all of the machines already had Word installed. The WordBasic software was able to help calculate and check values in tables, enforce naming conventions, and also run reports and combine and merge documents.

They saw a great improvement over programs they'd seen elsewhere or in the past. Typical solutions had to be installed and expected you to enter information through dialogs that did not closely resemble the standard forms. And the other alternative to dialog-based applications, forms packages, did not have the kinds of page breaking and dynamic row breaking and table column headings that Word had. Some other budget forms were done in Excel though Excel neither supported text entry well nor closely resembled the original forms.

But the Word-based solution was not all perfect. There were limitations with respect to controlling and guiding the input. While it was great to allow users to use their familiar Word processor to change font, bullet and bold text and so forth, there was no user friendly way to limit or standardize this usage. It was also very slow and clunky. Reports involving hundreds of pages would take hours.

And we had quite a scramble when the first Word virus burst onto the scene in the summer of 1995, remember that?

Some people felt that our solution should have done more in terms of the work flow. But I felt that at that stage on a tight IT budget, trying to institute conventional work flow software would have added a rigidity to the system that would have stood in their way. Users e-mailed documents to each other and even sent them on diskette, so that in those cases they were ultimately responsible for ensuring the right versions were submitted. The team also used frequent e-mail notices as well as a bulletin board on the wall to help coordinate activities.

The document-based system was ideal for flexibility. Shared folders on the LAN were used by WordBasic to submit documents where WordBasic tracking reports were run on the administrator's machine to summarize status, and merge pieces of documents. The simple architecture meant that workarounds and ad hoc adjustments were intuitive to the team because they understood the folder organization of documents and could access them with Windows Explorer.

By utilizing Word for Windows, e-mail, and familiar ways of using computers, training was minimized and they were able to concentrate more on the actual work than wrestling with the software. The staff really appreciated the fact that although the system did not automate everything it rarely stood in their way, and this was the key to success in that ad hoc environment.

Agile methods are often characterized as being at the opposite end of the spectrum from "plan-driven" or "disciplined" methodologies. This distinction is misleading, as it implies that agile methods are "unplanned" or "undisciplined." A more accurate distinction is to say that methods exist on a continuum from "adaptive" to "predictive."

Developers tend to agree that the more adaptive methodologies don't scale well to large projects. I am proposing the Market Paradigm as a means to scaling these adpative techniques to large projects.

Even when the adaptive terminology is borrowed in the context of a large project it is likely to be subverted just like the "iterative" approach was disingenuously invoked as a last ditch effort in the Death March in an Information Technology Project. Here I talk about the potential of swinging to the opposite end of the spectrum from these predictive monoliths to a laissez-faire approach that brings about accelerated results through marketplace forces.

Market forces

Before talking about specific software methodology, think about how progress is made in the industry at large. In a FORTUNE Magazine article entitled Quoted Often, Followed Rarely, Daniel Roth interviews Fred Brooks who wrote the seminal book The Mythical Man-Month (1975). In talking about open source Fred Brooks says:

A second big advantage is that multiple versions of any particular component gets built, and the marketplace votes on them. In many cases one wins out. In many cases, one doesn't, and different versions circulate. That's confusing, and you get compatibility issues... Would you want to build an air-traffic-control system that way? The answer is no. That task requires a tighter degree of integration. Whereas when the community is building tools for itself, it's advantageous to have different people, with different ideas of what they want to build, build it, and then let the market vote.

I doubt I am alone in hearing occasional remarks about bringing "capitalist" forces to bear in software development. People will talk about how a little competition or meritocracy can help, that the re-use of a component or service can be left to depend on whether others find it useful and want it rather than by mandate from leadership.

The reality of waste

But the main obstacle to an acceptance of any real free market mechanisms within a single large project is the knee-jerk rejection of a model that accepts or expects what appears on the surface to be "waste".

Unfortunately I think that the lesson was not learned by the majority of people involved in the death march IT project I wrote about or others like it. Many will simply conclude that the primary methodologies were not at fault, it was simply a problem of execution. Well yes it was a problem of execution, but I would go a step further and say that these methodologies are prone to the abuse and failure that occurred here, and therefore we need to adapt a new approach that expects certain failures and waste in order to harness more natural human productive potential.

Waste is the Gorilla in the closet (the Mountains of Discarded Software I wrote about 7 years ago). Waste is all around us but generally ignored and explained away. A truly affective approach will recognize the realities and work together with the real human dynamics involved. The 3 laws of human nature in software development are:

In "Building the Machine That Will Build the Machine" I described the problematic tendency in our industry towards architectural frameworks. This tendency turns the integration effort into a counterproductive effort, meddling in the progress of the various teams for the sake of avoiding reinvention and inconsistencies. You are in trouble when project leaders talk about "independent" modules within their architectural framework, and yet every single thing they do increases the interdependence of the modules in the architectural framework!

The problem is that you simply cannot impose a framework if you're going to have truly independent modules. Despite this truism, most people claim that as long as the architecture is well-designed you will be all right. The hope is that by giving up a measure of independence you gain efficiences. But the better answer is not as far off as you might think. You can offer framework-like features instead of imposing them, and suddenly you've turned the tide in your favor.

Real independence

That's it! With a rather small tweak you change a restrictive micro managing intervention into an empowering opportunity. The problem many will immediately point out is that these framework-like features might not get used, and every team might end up "reinventing the wheel."

When a software component made available by one team is not used by another team, it is rejected because it is not adequate, it is not ready, or it is not worth it. The third one is often the hardest to stomach. Every integration involves jumping through a hoop -- a hoop that needs to be maintained over time. A team that properly takes responsibility for its own progress will weigh the benefits and costs, and the costs of incorporating an externally developed component should not be underestimated. Managers love the word "re-use" but it is costly to force it where it doesn't belong.

With the good kind of team independence comes the good kind of distributed responsibility for productivity. Independent teams will use their ingenuity to come up with a working solution rather than waiting to implement pre-designed pieces sent down from above. And that freedom to create solutions brings unrivaled productivity.

The "Market" masterplan is not a micromanaging one; it grows out of the confidence that five independent teams of 10 people will be an order of magnitude more productive than one team of 50 -- so much more productive that you can stand to throw away half of their stray work and still be much much better off.

A consistent product

But there is another legitimate concern: the concern for a consistent end product. The reality is that certain kinds of consistency are overrated. Consistency of the end product should be envisioned with an eye to the productivity of the end user, but not in a way that restricts the explosion of creativity and progress that the Market methodology can nurture.

Windows Office is safely the most-used suite of software products ever. Yet it never consolidated to one consistent macro/scripting technology. Instead it has had a hodge podge of mostly Visual Basic-like technologies over the years. On a related note, remember when Object Linking and Embedding (OLE) was going to usher in an era of "document-centric" computing where you didn't care if you were in Exel or Word? Yeah right! On the other hand, certain consistencies like similarities in toolbars and menus, cut and paste, are essential to productivity across the suite.

It really irks me that they couldn't consolidate many more things across their suite, but the reality is that certain kinds of consistency are not worth holding back an innovative product and those considerations should lose out to business pragmatism every time. Perfectionism and architectural aesthetics can kill a product more than inconsistencies between modules can.

Unifying what shouldn't be unified

One can't help but wish that the Microsoft Office team had come up with a single scripting technology. It would have been ideal, but it is another matter altogether when you try to find consistencies where they don't belong.

Here is a hypothetical example involving various formats of identifier codes used across several legacy systems. In H89A5673EZL, H means aircraft part, 89 is the model year, A is the model year sequence in case there were multiple models that year, 5673 is the part number, and EZL is the location code. Various legacy systems had ways of interrogating the alphanumeric part identifier codes to extract relevant information. This function was of course duplicated in each of the legacy systems with subtle variations. It seemed desirable to everyone that this should be unified across the whole new system and that it was also a great opportunity to introduce a new common nomenclature for part identifier codes.

You immediately think such thoughts as: the two digit year could eventually lead to a Y2K problem, and we need standardized location codes. Common sense right? But attempting to force a new nomenclature could be catastrophic. Firstly, "if it ain't broke, don't fix it." The legacy system part codes already work and are infused in a myriad of business processes. Secondly, it wasn't on the customer's requirements list, so unifying these codes is not or at least should not be a priority. Thirdly, it is paramount to recognize the significance of those subtle variations, and the likeliness of underestimating the difficulty of resolving them.

I observed a situation much like this and it turned out that consolidating the identifier codes was filed away under "nice-to-haves" because it was simply too difficult. But the real problem was that they were focusing on where they could unify the business logic rather than how to identify small doable projects and empower teams to complete them. Micro managing business logic will never scale to a large effort.

Winner takes all

It is hard to imagine that consistency might come through an unpredictable evolving process rather than by consensus beforehand or by being forced, but that is indeed the case in the larger industry with such things as UI advances from Apple and printer advances from HP where a de facto leader in an area sets the standard.

In the case of identifier codes you might find that as business processes evolve one of the systems might find the need to come up with a unified syntax for its own purposes that would slowly be adopted by the other systems, one by one until all of them use it. This is a natural process allowing old ideosyncracies to die away at their own pace.

In a large project involving multiple user communities and different subsystems, you might fear that user interfaces will be divergent if they are developed by independent teams. But actually if one team comes out with great user interface features, that will create expectations for the other teams to implement similar features, much like I suspect happens among the applications in Microsoft Office.

Ultimately, the exchange between teams should be in the form of "touch and feel," rather than design-by-meeting. In other words, "oooh, look what you got! we want that!" instead of reading documents and having discussions beforehand. I don't use the word "competitive" because it is not an overt competitive attitude, although there are underlying competitive processes going on.

Team independence does not mean the team does not answer to the overall goals of the larger project, or that it would reject advances provided by other teams; each team's success will depend on the utilizing and sharing of components and implementations between teams. If the Excel team at Microsoft announced a new direction on a complete tangent from the other tools in the suite, the Excel team leadership would be replaced tout de suite.

Divide by user community

For some reason in large projects they try to divide the larger problem along functional lines. The worst example is creating a team for user interfaces, another for business logic, another for data schemas, and another for reports. I described a case of this under "Separating DBA, UI, reports into teams" in Death March in an Information Technology Project.

I propose instead that you create teams for each identifiable user community or subcommunity. For example, in a typical Department of Defense scenario you would have a team for a group of users at headquarters and a team for each different type of group in the field. You might still have DBAs and reports specialists, but each team will have its own as it requires. Of course a great deal depends on the details of the project.

I have not spoken much about Agile techniques, but when you are regularly releasing a product to your users it is paramount to have a small well defined user base to develop a close relationship with.

A new perspective on waste

For a waterfall-minded project management, giving teams direct access to customers will undoubtedly be the most troubling aspect of the Market paradigm. It would certainly take strong visionary leadership to really open up the floodgates like this and get away from the traditional controlled requirements preparation of the Waterfall paradigm.

Well there is really no alternative! Eventually a piece of software is placed in front of a user and you get real live feedback on what is actually needed for it to be useful. This is where the rubber meets the road and I find it downright delusional at this stage to find anyone claiming that the years of BDUF leading up to this point were worth it when focused agile techniques could have yielded a similar result in a few weeks.

The principle behind BDUF is that it is much more efficient to resolve conflicts during requirements assessment and design phases than once a large system is implemented. However the human reality is that the sheer volume of details cannot be comprehended nor resolved until people begin to see computer programs actually doing things! In this light, the protracted periods of requirements analysis and design are much more wasteful than a program which does something wrong on the first few iterations.

Decoupling

The key to the Market Software Development Paradigm is in dividing the larger project into smaller projects for independent teams. A small project needs to stand on its own, with its own product management, design, testing, QA, user feedback, and release schedule -- everything. Projects are identified based on dividing lines of loose integration with other parts of the larger project.

If there are no obvious points of loose integration, you need to invent them, i.e. design them into your solution.

Developing a separate product for field and headquarters appears to be a terrible waste on the surface. It is probably the most controversial idea I am proposing, and I am not saying it is always a good idea. But if some task is a very large part of the project where you would be assigning 20 IT people, you might have a better chance of success by dividing it into two teams with 14 IT people each.

A task requiring so many people to develop would have significant data entry, reporting and analysis needs on the field side and you don't want to tie these down to development of the headquarters' module which has a whole different set of concerns. As separate products they would develop their own data schemas and the exchange of information between the field and headquarters would involve data interchange. This is not a bad or wasteful thing if it improves the responsiveness and agility of the development teams to their customers who are indeed very separate.

This is not a comparison between 20 people doing it one way or 28 (14+14) doing it the other way; it is a comparison between getting it done well versus late or never.

By introducing points of decoupled interoperability you create more work, but the work on those separated modules can be done more independently and therefore more quickly. In a large project this can mean the difference between soon and never! The challenge is in devising ways of dividing the large problem into smaller problems with separate user communities, not along functional lines.

Stovepipes

One of the words used to negatively describe legacy systems in the Department of Defense is "stovepipes," i.e. separate systems with very little interoperability. Funny that being "stovepipe" is always considered a bad thing, because it evokes positive aspects of independence, redundancy and security. No matter how intelligently some new age architecture is designed, interconnectivity and uniform design will open vulnerabilities to large scale problems and malicious attacks.

But rather than incrementally improving the interoperability of stovepipe systems (to improve something that already works) their first instinct is to replace it all in one go, much like the "Big Bang" method recently lamented by Microsoft CEO Steve Ballmer.

If you are willing to sacrifice the aesthetic purity of the "Big Bang" approach, you just might reap the rewards of a much faster adaptive incremental approach. Things are often not as they seem from the outset. Development can be greatly accelerated by allowing stovepipe systems to evolve in close relationship with their individual user communities within a larger project sharing features and components between systems. The result may bring you to a whole new, robust and interoperable (even consistent) system much more quickly that you would imagine.

Think simple

Since agile development is all the rage, it has inevitably caught the attention of managers of large projects that are traditionally advocates of BDUF. Often though it is assimilated as meaningless jargon, a mere dressing for the same old anti-productive processes. "Agility" is the new meaningless buzzword. People are still talking about designing and implementing large complex systems and investing in architectural frameworks. But my experience supports Gall's Law, which goes like this:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

After witnessing failures of monumental proportions, it is clear to me that waste is a part of the process. The Market Software Development Paradigm brings this to a level on which failures are small and leveraged as part of an adaptive process, yielding a much better prediction of success.

In this one short article I cover all the essentials of a complete multi-session database design. I haven't studied any database theory since college (1991), and I'm not even sure I did then -- although I do have a Computer Science degree. So I am writing this entirely off the cuff based on experiences with databases and my computer science knowledge.

The rowboat database is a single file with a small peer database engine and client application. I use only established techniques (I am not inventing anything here), but I think my design is unique in its simplicity power speed and versatility.

Challenges

In its simplest form, a database is just a group of tables consisting of rows (records). The term database can be used broadly to mean any data container, but I am talking about the common relational archetype. There are 3 primary challenges in implementing a database:

1. I/O: allowing rows to be created and deleted quickly and efficiently
2. Transactions: allowing grouped "all or nothing" actions and recovery on failure
3. Multithreading: ensuring the integrity of the data model despite concurrent activity

All of these need to be designed together. As rows are created they must be:

1. added without moving around other data on disk or wasting too much space
2. marked so that they can be rolled back if the transaction does not complete
3. marked so they are not included in another thread's query begun beforehand

Similar things need to be considered for update and delete of rows. Those are the primary challenges in a nutshell.

File Blocks

Blocks are the universal underpinning of every computer storage technique. The basic idea is that you set a block size and implement a linked list of blocks. When a block is added, it can be allocated anywhere and logically inserted in between two other blocks without having to move any blocks in the storage medium. When a block is removed it can be added to a linked list of deleted blocks in order to be re-used later.

The file system happens to use blocks too, but we manage our own version of blocking or "paging" within a single file system file. I/O reads and writes always occur in terms of whole blocks, that makes I/O simple. In practice, it does not take longer to write one byte or one block because the primary factor is seek time.

A particular block only holds data from one table (a re-used block can of course be assigned to a different table), because when you query the data of a table, you want to involve as little of the disk as possible.

Column Widths Are For Wimps

Based on assumptions rooted in ancient mainframe database technology, most databases like to be able to know column widths when the table is defined (except for large objects). Heck, FORTRAN originally had a rigid input layout with column-based restrictions. But the column width requirement in databases is one of the chief aspects of inflexibility that should be done away with.

Column widths are where most databases miss the boat. Character widths are even more meaningless with the onset of cases where text fields pass through multiple encodings (that can use different numbers of bytes) like when the client and server text encodings differ.

Although relational in attitude, my design does not involve column widths. Rows (or records) are the basis of relational functionality. Columns are a prism through which we select and modify a particular row, but all activities occur in terms of rows. So you've got a bunch of rows and they're all in the same boat, so we'll call it a rowboat.

For the best efficiency, instead of choosing column widths you should choose a base row width. This row block size is the smallest amount of memory that will be used for the row, but the number of these blocks for a particular row can grow arbitrarily.

Row Blocks

In addition to file blocks, there is another level needed to handle the variable width rows. The row block size can be 128 bytes by default and tailored for every table as long as it divides evenly into the file block size. If a particular row is larger than a row block it occupies multiple linked row blocks. Each file block keeps track of how many row blocks are free in it so that the file block can be deleted if it becomes empty. So:

• A logical table is a linked list of logical rows.
• A physical table is a linked list of file blocks.
• A physical row is a linked list of row blocks.

Multi-user Model

There are two ways to handle multiple processes manipulating a database at the same time. One is to use a lock file that each process uses to synchronize their actions. The second is to have a server through which all actions occur.

This is over simplified, but the lock file mechanism depends on the file system's file open command where if two processes simultaneously try to obtain exclusive access to the lock file, one of them is denied or held up until access is granted. The nice part is that the lock file can support multi-user access between processes running on different machines on a network with visibility to a shared file system.

The server mechanism strikes me as easier to conceive and implement since it depends on direct inter-process communication where a single application instance can coordinate all client requests. This is the model I pursue here. A pool of threads can service the queue of data access requests arriving via TCP/IP.

Locking

You can't just add a plain row into a table because until it is successfully committed it is only visible to the session that is inserting it. Similarly when you update a row you cannot just replace the row in the table because it must retain its old value for other sessions until it is committed. But still, a new row is added to the end of the linked list of rows. An updated row is inserted into the linked list of rows adjacent to the row it is replacing.

Different sessions need ways of accessing rows that allow them to determine the state of each row as it pertains to them. To satisfy this requirement, rows are marked as mid-transaction insertion or deletion. In the case of an update the replaced row is marked for deletion. A mid-transaction insertion row is ignored by every session other than the one that obtained the lock, while a mid-transaction deletion row is treated as if it exists by other sessions.

Table locking means that any number of sessions can be reading a table simultaneously, only one can be writing at a time. Table locks are maintained in memory by the server process.

Recovery

For the sake of crash recovery, we need to write any modified block to disk twice. Once in a transaction recovery block, and then overwriting the actual block of the table. A linked list of recovery blocks is maintained in the file. When the commit is successful, the recovery blocks are released back to the pool of deleted blocks.

If the failure occurs while parts of the transaction are being written to the recovery block, the transaction will be discarded. If the failure occurs while parts are being written to the data file, the transaction will be completed during recovery to restore integrity to the data file. Whenever the data file is opened, recovery is performed automatically.

Data Types

Notice that I've said nothing about data types. Each row value is treated by the core database as an opaque string of bytes.

A row (i.e. a record) is generally a structured group of fields. In most tables this will be a simple list of columns implemented for efficiency with either a prefix table of integer offsets or a persisted parsed XML fragment. Data types and row implementations are not integral to the core database design, but are useful in providing the constructs such as indexes, queries and stored procedures.

Indexes are a means of speeding up access by maintaining an ordered or mapped list of keys for a table. An index is an internally maintained table containing links into the indexed table. The database doesn't need to know how to compute the key from a row, but it does need to provide the infrastructure for maintaining the index on table modifications.

Plan of Action

My goal would be to implement this complete database technology in 3 classes in this order:

1. A sockets class to manage thread pools, queue requests, and negotiate peer to peer relations
2. A data core class to provide all the linked lists, file I/O, locking, transactions and recovery
3. An access class to provide the row schemas, data access language and SQL subset

The sockets class would not understand the content of requests but it would send and receive them and delegate threads to usher received requests through the access class. The sockets class could be tested for scalability and throughput by generating requests from multiple peers to each other and performing some random file I/O to represent database processing. The classes would be developed inside a test harness dialog application used to interactively generate requests.

The data core and access classes would actually go hand in hand to some extent. The access class would act as a front end to the data core class but in the beginning would only provide some stubbed out simple methods to service requests from the sockets class. All of the database functionality could be fleshed out while maintaining only rudimentary CRUD methods in the access class. The data core class would not "know" about the access class but would provide hooks for the access class to generate index keys and service triggers.

The access class would also have some client methods used to package requests to be sent, in addition to the server methods that call the data core methods. I would leave both client and server aspects of the access class together because they involve the same knowledge, plus the architecture is peer to peer so a database can be on both sides. Finally, the access class would be fleshed out with data types, row schemas, and full blown language functionality.

I Repeat, No Column Widths

I watched all the excitement about "XML databases" and "Object Oriented Databases" come and go. The hype went away with a resounding thud, although few seemed to notice. We're left with artifacts of the craze such as shallow "object" features in Oracle schemas. The relational model still underpins all the bulk of database programming, while extreme XML databases are more akin to file systems with supporting methods.

But the real question people should have asked is: why do I have to specify a column width at all? We use CLOB fields to store XML documents. Why the complexity? It is just text! Why does my database schema restrict my username length when that should be a business logic or UI decision? If my average username is 8 characters but I reserve 32 for every one of them, does anyone else see the inefficiency here?

I'm trying to rock the boat! "No column widths" is the key to the next real revolution in database programming.

As someone who grew up with the moderately cold winters of southern Ontario, Canada, I always thought it would be great to live on a hot tropical beach somewhere. I'd been working as a C/C++ programmer in Virginia for 5 years after college when the opportunity presented itself.

In 1996, a co-worker at the office circulated a posting he had found on the Internet for a C++ programming job in the Carribean. At first, like everyone else I said "wouldn't that be amazing!" but did not seriously entertain the idea, even though I was one of the few people who was unmarried and relatively free to make such a major move. But a week later the idea was mentioned again and I decided to shoot off an e-mail. That led to an interview with someone from the Holland-based company, and then an expense paid week on the island of Curaçao to check it out!

When you emerge from the plane in Curaçao you are invariably greeted with a warm blast of air. Before you descend the steps to the tarmac you see the small airport backed by a bluff -- a cliff or escarpment covered in green bushes and topped by cactii. People watch and wave from the upper level of the terminal as the passengers make their way inside. I remember looking up and wondering if one of those people was my future employer.

In fact my future boss Ruud did meet me there, and he organized a wirlwind tour of the island that week. I stayed at a hotel in Otra Banda which happened to have some sort of modeling convention going on so there were gorgeous women constantly coming and going or being photographed by the pool. The software company office had floor to ceiling glass on two sides overlooking palm tree filled residential neighborhoods. Needless to say I was convinced the island life was for me.

Back home in Virginia, people were pretty amazed that I was going through with it. There were a lot of preparations. I had to disentangle myself from my job, ship my car, end my rental lease. I could barely believe it was going to happen.

For the first month in Curaçao, I shared a nice new condo with the other new hire, Attila. The place had it's own incredible swimming pool. We had loaner cars from the company and we began getting acquainted with all the things to do on the island, not to mention our new jobs. In the meantime the customs bureaucracy with our cars being shipped to Curaçao from the US turned out to be really awful. It tied up our cars for several months in endless wrangling and numerous fees that eventually involved what in hindsight must have been a bribe to finally get our cars out.

Probably the most important skill for survival on the island is the ability to relax and go with the flow. For me at least, this is way easier said than done! But we managed :)

Attila posted a lot of great information and pictures (some of them mine) of this charming little paradise island just off the coast of South America at Attila's Curaçao Page.

Attila and I eventually got our own places. I found a nice roomy apartment; he rented a great house and we planted a row of palm trees in his yard. I'm not going to tell the whole story of my time in Curacao, but I wanted to tell something about it after reading a couple of discussions: Living "The Life" on a tropical island? and See the world while working in IT? From there I came across this loud warning:

Bali, Thailand or anywhere in the tropics is fecund, incredibly fecund. Things grow. Fungus grows - everywhere. Bugs grow everywhere. Everything is green. Computers are not green. Modern appliances are not green. All the conveniences we enjoy in the middle latitudes rust or fail in the tropics. Expect many difficulties maintaining your equipment and lifestyle.

Not true of Curaçao! I have lived in places like that too, but Curaçao is basically a desert island. Things do grow there, but not like in the wet tropics. The rain comes in short drenching showers, one minute it is dry and sunny with only a few clouds, then there is pouring rain for 20 minutes, then it is sunny again and everything is dry in 10 minutes! The ants will find any food you drop or leave out, but in general you are not overwhelmed by critters and growing things like you can be near a rainforest.

Though it is not your lush tropical look, the dryness of the climate has benefits. But there are other detractors to keep in mind:

• like Europe, stores are not open at convenient hours
• neighborhood dogs bark all night
• banking and exchange rates can be a pain
• imported items (sunblock, even fruit) are more expensive than in the U.S.
• occassional electrical outages

But the little inconveniences are entirely outweighed by the attractions and the general overall amazing experience of island life.

• scuba diving and snorkeling
• carnival and jump-ups (endless dance parties in the streets)
• beaches, coastlines, and hilly countryside
• oceanside restaurants, beach bars and casinos
• watching salsa competitions!

I stayed for 15 months. It was an incredible time in my life, but ultimately it had to come to an end. The Washington D.C. Virginia region called me back with all its activity, commercial efficiency and job prospects. There is only so much relaxing in the sun that an ambitious northerner can take!

The pain that my current product CMarkup solves is the complexity and overhead of XML solutions for simple XML purposes. Experienced and practical C++ programmers take one look at those big things out there and say "Yuck! There's got to be something small and easy for such a trivial problem" and then they go online and find my product.

But CMarkup was just a nice accident as I explained in I See Markup, Part 3 - Transition To Commercial Product. The idea I introduced in my last post "Here Is My Idea" is the peer to peer spreadsheet idea that was in my mind when I started my company in 1999, though I have never brought it to market.

I made the mistake in "Here Is My Idea" of not powerfully invoking any real pain. Instead of describing pain I listed a few annoyances. Real pain is the number one means of communicating a good idea because everyone can understand real pain. As Joel Spolsky wrote:

Don't start a business if you can't explain what pain it solves, for whom, and why your product will eliminate this pain, and how the customer will pay to solve this pain. The other day I went to a presentation of six high tech startups and not one of them had a clear idea for what pain they were proposing to solve. I saw a startup that was building a way to set a time to meet your friends for coffee...

On one hand, my peer to peer spreadsheet idea is still somewhat of a solution in search of a problem. The are a number of ways to specialize and I plan to do that. On the other hand, my idea did come out of specific problem area for which I believe there is still no powerfully appropriate technology, the U.S. government budget coordination process. And that remains the area I would love to pursue again someday.

In The Right Idea?, Gavin Bowman lists a whole bunch of great points on selecting and proofing your idea. At the end he really nails what it all boils down to:

If that sounds like a lot of steps and a lot of work, it is, but it's a lot easier and less painful than building a product that nobody wants. Or, than building something and then trying to figure out who wants it and how you're going to get it in front of them.

My peer to peer spreadsheets idea is meant to address many of the pains in current distributed work-flow, approval, and publishing solutions. These are not new ideas, I just plan on executing a solution very specific to the needs of the domain. The challenges faced by systems that need to bring information managed by many people into one place often fail to get a good balance between management and data flow. There is the centralization bottleneck that works against the productive autonomy of a dispersed group of information managers. Peer to peer functionality supports managers in the field to control their own processes before dialing in to the central office. Centralized solutions have a bad dependency in times of movement and or turmoil when Internet access is unreliable, and being dependent on a server they make it difficult to move the center of operations.

I believe the pain amongst many existing customers of budget coordination systems is very real and that they, or the contractors providing their systems will benefit from a core product to address these pains. There are still a lot of questions to answer about this idea though, so stay tuned.

I started this blog six months ago to do just what Robert Scoble is saying: to explain why my idea is so good. I don't want venture capital, but I do want to capture the imaginations of colleagues and first adopters. I have some white papers and a prototype from about 5 years ago that I only showed under non-disclosure at the time, but now I intend to explain it all in this blog eventually. That old assumption that it needed to be kept secret has long since vanished.

My idea brings together spreadsheets, XML and peer to peer technology. Stay tuned to this blog as I expand on it.

I don't know how best to sell this idea yet, so instead this far I have been practicing by putting down some foundations of my philosophy of software. But Scoble's post catapulted me into posting this. Like the ebays and the Googles, my idea is not so much something new as it is an approach that recombines existing things. Since it is not likely to catch on with a quick description, it is up to me to carry it forward for the long haul.

One starting place is to talk about problems or pains with existing solutions such as Microsoft Excel and Access. But it is not to say the existing solutions are bad. They are powerful and widely used, so I would never try to compete directly. But any widely used software has to compromise and leave many of its users less than happy.

I have always found it very annoying that Excel and QuickBooks print that pesky extra column on a separate page when my table gets a little wide. There are ways to fix it, but it takes some mucking around, and its generally after I have already wasted paper. Another issue in Excel is centering the title on the whole page when it only occupies some of the columns, and doing headers and footers without editing them in the spreadsheet. Also, the VBA programming language is separate from the cell formula syntax, so when regular users reach the limit of a formula's capability they stop there.

What I am doing is identifying my constituency. My users are spread across several applications for which none of the established tools are quite right. Ideally I can carve out my place between those giants where I can grow without being stepped on.

Bookkeeping is often done with Excel and/or Access because of the difficulty of setting up and conforming to accounting applications. The idea is that you want to keep records in a way that is natural to your day to day business, and you would prefer to worry about doing reports and taxes later, when the time comes. But bookkeeping programs almost always require a lot of setup at the beginning and it is hard to know where and how to record unusual things.

Excel is great just to put stuff in columns and rows. Access takes you a step further to get it into columns and rows that are in database tables facilitating additional data processing. Word takes you one step back and allows regular text and pictures in addition to tables with columns and rows, but then programmatic access can be difficult.

The spreadsheet component of my idea has several defining aspects:

• When starting with a blank sheet, it focuses on rapid flexible freeform and tabular data entry on a page view where the number of columns can change between rows.
• Reports are simply new sheets which may be built from other sheets, and sheets are accessed like database tables.
• A sheet is like a document that prints the way it looks. This is a unifying principle that limits embedded components to ones which print as they appear in their non-focused state.
• Sheets are easily designed and programmed for end user friendly data entry and work flow.
• The formula shorthand and scripting language are unified with debugging and auditing, and wizards generate modifiable scripts.

The desktop application can act as a node in a peer to peer network for sharing a sheet or a bundle of sheets. In contrast to Groove, this P2P arrangement de-emphasizes infrastructure, and instead supports quick ad-hoc connections more akin to instant messaging.

While my spreadsheet prototype has never been released, you can try out my animated graphical demonstration of peer to peer messaging in Windows. The firstobject XML Messaging project is a simple executable you can run right out of the zip file and type "demo" to see the animated show of a buyer seller conversation. Run it from an additional machine on your LAN and without any setup you can hook it into the conversation occuring on the first machine by typing "loc sellerXYZ,ComputerA" and "buyer SellerXYZ". Full details are there on that page.

This post is only a first peek. There is already quite a bit of depth to this technology including written materials that I will be slowly going through and putting forth in this blog, and components under development in other (free) firstobject.com projects like the messaging one.

To make a large problem manageable you generally have to break it down into smaller problems and let separate teams handle the smaller problems. Of course you need to break it down along lines that allow the smaller problems to be solved individually, otherwise you have not really broken it down at all.

Big Design Up Front (BDUF)

The project was launched with a lot of excitement and enthusiasm about doing the job right. The goal was to modernize a whole bunch of legacy systems from various platforms and locations throught the world into a new design with a set of subsystems (roughly analogous to the original legacy systems but greatly enhanced and interconnected). They were aware of failures in the past, but they assembled a formidable team of domain experts and were well-versed in the latest software development best practices and technologies. Some of the key players were hot off of other successful (albeit smaller) projects and everyone felt the synergy was just right.

All of the initial goodwill may have gone to some peoples' heads because there seemed to be a sense that something really quite visionary was going on. They decided that as part of the requirements analysis and design phases, they would create a software framework that was meant to streamline the project by abstracting certain generalizable processes out of the overall application and data development effort. Though it was their pride and joy, I'm guessing that this framework concept may have hurt the design phase by allowing abstractions to dominate instead of tangibles.

The ideal of the BDUF (Big Design Up Front) approach used in virtually all large IT projects is that with good design and specifications to work from, the various teams can go off and implement their parts and later bring them all together and get them working with tweaks here and there. The principle is that it is much more efficient to resolve conflicts during requirements assessment and design than after such a large system is implemented.

Whether specifications can allow teams to go away and implement their parts independently is debatable. But even as a means of defining and integrating the efforts of teams on this large project BDUF had 3 major problems:

1. The design negotiation process got paralysed. Meetings never seemed to reach a resolution because of the sheer volume of legacy system details and experiences being considered. The more experts that got involved the worse it got. The more they earnestly wanted concensus the longer it took.
2. Specifications could not cover the enormity of details. Ultimately specifications are only as good as the people writing them can get their minds around the problem. Modeling tools and prototypes helped but the results were not inspiring.
3. No one read them. Well that's not true, but many times they were only read until a glitch or inconsistency was found, and then the author was contacted anyway.

The bottom line is that BDUF demands something of humans that we aren't any good at, at all.

After a year or two when the customers started to have misgivings about whether progress was being made, the design meetings became shouting matches. The project leaders tried to control the egos involved by planning a certain amount of time for each point to be addressed and opinions to be discussed. I went to a couple of these meetings a year apart and had intense déjà vu as I discovered some of the exact same issues were still being debated.

Common sense from trial and error should have told them that unifying the design of a large number of legacy systems on the scale they were attempting was simply not going to happen in meetings and spec-writing iterations. But ultimately the problem was not BDUF, it was a complete unwillingness to break the large problem down into smaller more manageable problems.

Special teams and new initiatives saddled

Giving true autonomy to teams tends to lead to non-uniform and duplicative solutions (which may nevertheless yield better results but I'll come back to that another time), and this prospect worried them so much that they ruled it out altogether. However, their stubbornness did not stop them from hedging their bet with some experiments. After insisting for years that the BDUF approach was paying off they began to consider other options behind closed doors.

They formed some small special teams to go and quietly start building software outside of the official process. They did this somewhat secretively because they didn't want to be seen as undercutting the process they had championed for so long, and also because they weren't exactly confident about it. But it was precisely because of their timidity that it did not bear much fruit.

The special teams weren't allowed to work freely with the other teams because of their unofficial status and yet were expected to utilize the progress of the other teams so as not to be duplicating effort. In addition they were forced to work within the same framework designed by the team leadership and so-called software architecture experts. The special players made a herculean effort and months of overtime turned into years, to no avail.

At one point a buzz circulated that the project leadership was embracing a new "iterative" approach, but it turned out to be in name only, since iterative development emphasizes building releasable software and they were still only producing a framework and system architecture. The deliverables still required the customers to use their imaginations; it was not a releasable end product. In fact it took a lot of smoke and mirrors to keep the customer from concluding that the emperor had no clothes.

The waterfall methodology (requirements analysis, design, implementation, testing, integration, and maintenance phases) used on this project (and most if not all large IT projects) has built in mechanisms to measure and predict progress. However, due to its reliance on volumes of specifications that the customers can never hope to digest, it is susceptible to being hijacked by misguided project management. In addition, it appeared that the customers had committed to the framework vision advanced by the project leaders. With all hopes pinned on a framework there was no room for truly introducing any new approaches to avert disaster.

Ultimately every well-meaning effort was undermined by subjugation to the over-architected, meddling, micro-managed whole.

Separating DBA, UI, reports into teams

The high level unified design being hammered out in meetings played into a manifestation of groupthink that had a dangerous effect on the division of problem areas. For example, in order to build the subsystems of the end product, there were 3 teams to manage different aspects of each of the subsystems:

• Database administration
• User interface development
• Report generation

The idea was that the report system would be written once and then each subsystem would be easily added after that, the UI team would utilize common UI components across all subsystems, and the database administrators would design and manage a unified data and application model to greatly reduce potential maintenance down the road. These were meant to be independent teams, not just tasks within a larger team, and were to be judged on their progress separately.

However, on any given subsystem used by a specific group of users there are many details affecting all of the data, UI and reports. Effectively, the 3 teams were expected to each learn the exact same things about every subsystem: the user roles, tables, indexes, and the ways of accessing, displaying and filtering the data, all of which effect each other a great deal (e.g. a DBA cannot design user roles and indexes without knowing about the reports and filters that will be needed for all the affected subsystems). The most sobering problem was that compromises made for the unified design often turned out to be showstoppers for individual subsystems and this would drive the issues back up into the design meetings.

A pragmatic approach would have been to give one team jurisdiction over all of the data, UI and reports of a particular subsystem and truly relegate the framework to optional services, but that idea was always dismissed as a maintenance nightmare.

Dividing each subsystem between the 3 teams ultimately led to an insurmountable effort, not to mention a dumbing down of the specific requirements that further hindered completion of any one subsystem. The cost of conformity and efficiencies across subsystems was too high a price to pay when it stopped progress in its tracks.

No matter what you do, you cannot get away from the reality that if a problem cannot be solved except in the context of the solutions to other problems, then you are greatly increasing the work by trying to assign it to a separate team.

Faced with increasing pressure and desperation from developers, the leadership eventually adopted a modified team structure based on subsystems with attached DBA and reports people. But they were too slow to ease the mandates on framework conformance to allow substantial productivity to set in. As a friend of mine likes to say, they were just "moving deck chairs around on the Titanic."

The consequences

After some 3 or 4 years, the customers started to make desperate demands to see something concrete. The project leaders chose a small sample subsystem and decided to accelerate it. This was going to be the showcase, the proof that the complex framework (the metaphorical "machine") they had put in place to generate the end product was going to yield vast efficiencies (see "Building the Machine That Will Build the Machine"). However, at every turn, it was the framework that was slowing them down. They took to spending whole days in the rooms with the developers, telling them to skip this and forget about that in a mad rush to get something concrete.

As the deadline approached they completely scrapped the application framework in favor of rapid application development, still maintaining that they would go back and retrofit the solution to their framework later. But they had already clung to framework goals too long to allow the team to adequately address the intricacies of the specific subsystem.

When the customers were presented with a demonstration of this sample program they were sorely disappointed. They recognized immediately that it was only a cursory treatment of the subsystem requirements it was meant to service. Many years and millions upon millions of dollars already invested were the only thing keeping them from bailing immediately. It would be some time still before the project was scaled back and effectively dissolved.

It is an unfortunate story that has affected me deeply. I observed this from a system services role (not any of the special, DBA, UI or report teams mentioned above) and wasn't in a position of influence. In a future article I'll talk about a "market" approach that I argue would have worked well because of this project's natural subsystem organization.

Well okay, may be it is not exactly an assembly line (although some have been known to use the term), they are just trying to define the architectural framework, i.e. the way that modules will work together and re-use common parts. But, to borrow Tim Bray's words, this attempt to design the framework beforehand fails in its "overreaching completism, trying to be, in Gall's terms, 'a complex system designed from scratch.'" He goes on to characterize one of the tenets of agile development this way:

Until you've implemented and deployed a feature, you don't really understand it. So don't try to build multiple layers of features on things you don't really understand.

I subcontracted on a large government contract that demonstrated how to go about it the wrong way. As I discussed in The Large Software Contract Problem, large IT projects are not like highway construction where a much clearer critical path can be drawn to schedule tasks that depend on each other. That certainly presents a challenge but it slowly crystallized for me that the reason this large government contract was on a death march was the fault of its leaders: they were stubbornly spinning their wheels trying to build the architectural framework that would eventually be used to build the end product.

So much was being implemented under the pretense that it would be part of generating and ensuring the consistency and efficiency of the end solution that they never even approached a satisfactory end solution. After 3 years they had absolutely nothing tangible to show the customer!

Of all the most preposterous absurdities, they chose Oracle Developer 2000 to "help" lay the foundation for this project. Developer 2000 was essentially a framework that never really arrived, even by 2002. But in 1999 they hadn't even started trying to use it before they were figuring out how to work-around certain fundamental characteristics such as some sort of repository that did not support their schema design. Rather than seeing the folly in this, they also had a subcontractor hire a lot of people to pursue the development of a new database framework as well.

When you commit to a framework, whether it is a database development framework or a web community portal or a distributed network component architecture, a significant effort is shifted to installation, configuration and training for that framework. It would be one thing if that framework was the end solution such as a web community that needs no alteration, but then that is not a large project. Actually the problem is often more than a matter of alteration, it involves an attempt to overcome certain shortfalls in the framework, sometimes via a horrid attempt to marry it with yet another framework!

This industry sees wave after wave of application frameworks that come and go. The desire to have a framework is so strong that developers constantly keep turning to the idea of a framework rather than getting on with it! Can you imagine what they might have accomplished in 3 years on the contract mentioned above if instead of chasing frameworks they put all that effort into working closely with selected customers to build a few small tight user interfaces that greatly improved upon existing legacy systems? The problem was that instead of just doing the thing they know how to do, they are hung up on creating this big complex thing that they feel they need.

Early on in the story "Why I Hate Frameworks" it sounds kind of plausible that rather than building the actual tool, they will build the factory that can easily produce the specialized tool.

"According to our research, what people really needed wasn't a Universal Hammer after all. It's always better to have the right kind of hammer for the job. So, we started selling hammer factories, capable of producing whatever kind of hammers you might be interested in using."

But the absurdity becomes evident as the same logic continues to find value in abstracting the task one level further out:

"As it turns out, people don't want to build an entire factory just to manufacture a couple of hammers. Leave the factory-building up to the factory-building experts, that's what I always say!! ... So we stopped selling those schematics and started selling hammer-factory-building factories."

I used to read books about software development. Now I look more to market dynamics and read stories in business magazines like Baseline about how inspired people strike the right balance of passion, action, and design.

In large projects you desperately need a strong technical leader to plot a pragmatic course keeping all eyes on the prize. But too often I find that people who focus on software development principles have an affliction that I call "building the machine that will build the machine."

Ben Bernanke, who succeeded Alan Greenspan as chairman of the Federal Reserve, explains the difficult nature of economics by comparing it to "learning how to repair a car while it is still running". That is one of the necessities of dealing with something as large and complex as the economy or (I would argue) IT solutions in major institutions. Starting over from scratch with a fresh new framework is simply not a viable option.