The Age of Innocence

A colleague just emailed me an article on whether open source can innovate, where:

• Larry McVoy says that open source can only "scrape together enough resources to reverse engineer stuff. That's easy... But if the world goes to 100% open source, innovation goes to zero."
• Linus Torvalds is quoted to retort, "Open source actually builds on a base that works even without any commercial interest [which] is almost always secondary."
  

In time, we'll probably look back on these quotes with a wistful "Ahh, those were the days..." comment, when the world was younger and so we were we.

We will probably realize that open source sometimes can succeed without commercial interests, but often requires it to be really relevant. (Just look at the history of Linux: It was created by Linus the college student but flourished under Linus the head of an international effort funded and supported by major hardware and software vendors.)

We will also realize that open source can innovate, sometimes more effectively than commercial software. But this will happen when:

1. Open source software itself catches up to commercial software in features. Most open source projects today are still playing catch up to their commercial counterparts simply because they are newer.
   
2. The business models and processes behind open source software become more developed, probably through trial and error.
3. Financial markets better understand open source software and provide funding access for open source software development.

If it seems like open source cannot compete with commercial software, it's probably because open source is just too new of a phenomenon. But try to imagine the early days of the software industry itself, before there were venture capitalists, startups, IPOs, software development processes, synch-and-stabilize methods, and all the software and libraries that exist today.

Would you have believed, in those days, that a bunch of hackers and geeks playing with toy computers could one day create software that pervades every part of our lives? Would you have believed that these people could innovate?

Fortunately for us, and himself, at least one person believed: Bill Gates.

The Clone Wars

A lot has been written about the IBM acquisition of GlueCode: that it's war on open source, that it validates open source, that it's just business as usual. One thing is for certain: we'll soon be seeing lots of GlueCode clones. There's even one group that's raised a fund specifically for Java middleware projects and companies using the Apache license.

Here's my guess at a plot for how it goes from here:

There are currently several frameworks, such as Spring, J2EE, .NET, and LAMP, for building (often web-based) applications. Underlying these frameworks are technologies such as PHP, Java, ASP .NET, Hibernate and products such as JBoss, WebSphere, WebLogic, Apache, etc.

These frameworks all try to streamline the process of developing web-database applications--namely, setting standards and making the process more efficient. This is happening because the underlying applications are maturing. (A website is no longer such a novelty.) Open source frameworks, including Java middleware, is a further step in this maturing process. It is essentially a step towards commoditizing the products that make up the frameworks. (This is a view that JBoss has espoused as well.)

If there are no new application needs, then the frameworks will battle themselves out in the usual manner. Debates about the various frameworks are already starting to sound like the "Pepsi Challenge."

But sooner or later a new technology will come along and create a whole new type of applications, and one particular framework (who knows which one?) will be particularly well suited for it. At that time, the demand for this new type of application will pull the appropriate framework far ahead of the rest.

Remember COBOL, FORTRAN, and Pascal? We used to have similar debates about those frameworks, until the Internet and web-based applications came along and pulled PHP, Perl, Java, and ASP .NET ahead.

The question is--what will that new technology be, and what will the next application look like? There are a few suspects: web services, utility computing, ubiquitous computing, and (a perennial contender) artificial intelligence. There could also be some hidden new technology, a phantom menance yet to be discovered.

I can't wait for the sequel...

The Developers' Utopia

Not long ago, the Internet was supposed to usher in a new Era, where a globally connected economy would do away with the inefficiencies of traditional distribution. Consumers and producers would connect directly with each other in this Utopia. Whole layers of distribution would disappear. Inventory and business cycles would be a thing of the past.

Nowadays, open source is spoken of in similar Utopian terms, of which this recent Groklaw article is a good example. Open source will sweep away the inefficiencies (indeed, the inhumanity) of commercial software companies and their managements. Developers will connect directly with businesses and users through open source projects. Merit will finally replace marketing.

In reality, the Internet revolutionized business not by eliminating distribution but rather by creating new forms of distribution. Similarly, open source will not do away with management and marketing but rather create new strategies for such. Just because open source projects do not have a formal CEO and a VP of Marketing does not mean that they do not involve tremendous management and marketing skills and effort.

Indeed, the large number of abandoned open source projects on SourceForge.net seems to prove this point. A successful open source project requires real management and marketing skills as well as a serious commitment of time and effort. Going forward, the bar will probably be higher, as new projects will have to compete with either "professional" open source companies like JBoss and MySQL or major software vendors like IBM, Novell, and Oracle, which are all now getting involved with open source.

But open source does offer a tremendous opportunity for developers to advance their careers based on merit. The right way to do this, though, is usually to join an existing project, rather than trying to start one's own. Successful open source projects already have the recongition required to draw in potential clients and users. They can give developers a highly visible stage to show off their skills.

At the same time, these projects also possess the subtle but definite structures for organizing the development process and promoting the project. These marketing and management processes may be invisible for someone used to commercial software, but they must be there for an open source project to succeed.

Remember, if it sounds fun to be a "Lead Developer," imagine yourself also being the CEO and VP of Marketing for a project. And imagine not necessarily getting paid to do those jobs.

For more information, see the academic articles on my research site.

It's Happening

It's gratifying and a bit scary to see your own predictions starting to come true. Here are a few headlines from the past week:

• Oracle will start certifying open source applications.
• IBM is buying GlueCode and getting involved with Apache's Geronimo project.
• Microsoft's Steve Ballmer met with Red Hat's Szulik.

The general convergence of open source and traditional software vendors is already happening, and it will continue. It's really just a matter of time before open source becomes a part of the mainstream.

The Least You Can Do

What do self-help books, get-rich-in-real-estate courses, and exercise videos have in common?

They all tell you that you can do great things, and all you have to do is just a little bit right now to get started. Usually, that little bit involves buying the book, course, or video.

Here's my version of the little bit that every software company can do with open source software. Today. Right now.

Find a few open source projects that are similar to your products and get on their mailing lists.

That's it. Easy, isn't it?

From those mailing lists, you can start seeing what questions people ask and what features they want. You can also learn a lot about users' experiences with a product similar to yours, in a way that would be difficult otherwise. Companies like Intuit and Microsoft spend millions to observe their customers in action, down to following them home and watching them install and use their software. Now all you have do to do is watch a mailing list.

In fact, an open source project's mailing list will give you a better view of users than any focus group or intense observation experiment. This is because whenever people are observed, they act differently. They tell you what they want you to hear, not what they really want. On a mailing list, though, people will actively describe their experiences, ask for help, and suggest new features because the mailing lists are the vehicles for doing just that.

Easy, isn't? And who knows? You might even decide to get involved.

A Boring Prediction

Someday I'd like to be a historian. I picture myself donning a tweed jacket, wearing a sweater vest, and delivering a stirring lecture about the Napoleonic Wars or the Ming dynasty in front of a blackboard (and hopefully some students, too.)

Unfortunately, people think historians are boring (doubly unfortunately, my wife is one of these people.) The reason is that historians have a dull, analytical perspective on events. They (we?) boil events down to cut and dry technological and demographic trends, devoid of any passion, idealism, intrigue, or heroism. Who wants to read about advances in metallurgy or seed planting techniques?

So, while I bide my time, I'll perform a boring historian's analysis about open source software:

Today, open source software is seen as an epic struggle of freedom versus property, empowerment versus establishment, even community versus capitalism. It is described as a fundamentally disruptive force, a transition to a gift culture, etc. Even manifestos are written about open source software.

When we look back a few years from now, though, we will realize that open source software was simply the result of two bigger trends:

1. The internet, which has created a globally connected software industry; and
2. The "blog generation," a more general demographic trend of greater user and consumer participation across all major industries.

The former makes a highly distributed model of production and distribution possible. The latter has made users activley participate in the development of their software.

That's it. Nothing more, nothing less. If this sounds boring to you, well it is. In time, open source will be a normal part of the software industry, indeed all technology industries. Having an open source strategy in, say 2010, will be no more groundbreaking than having a website today.

And here's my boring prediction:

In five years time (2010), there will no longer be a meaningful distinction between "open source" and "closed source" software vendors. All major software companies will:

1. Actively support some open source projects;
2. Actively adopt and use some open source projects as part of their products; and
3. Fight off some other open source projects' assaults on their products.
   

What about Microsoft, you ask? Well, in fact, they are already doing all three.