We'll first start by contrasting the "evolutionary" aspect of modern software architecture against the "predictable" aspect of modern hardware architecture. Moore's law best describes the predictability of modern hardware architecture: that almost every aspect of digital electronic devices (processing speed, memory capacity, etc.) are all improving exponentially, doubling approximately every two years. The trend was first observed by Intel co-founder Gordon E. Moore in a 1965 paper, has continued for almost a half of a century, and is not expected to stop anytime soon.
Modern software architecture has no such predictability. It is best described by the theory of evolution, well-known within the domain of biology. The primary reason for this is that software is used to model the knowledge and processes that form the foundation of a business. This intellectual property (IP) that gives a business its competitive differentiation comes from the coordinated effort of its people. In biology, evolution is change in the inherited traits of a population of organisms from one generation to the next. Evolution occurs naturally within any biological population, so it is natural to expect that that the knowledge and processes guarded by a group of people will evolve as well, and therefore that the software that models that IP must have the chief characteristic of "evolvability" in its design.
Whether you fall on the side of Evolution or Creation as it applies to the origins of humanity, no one can deny the fact that software is created. One could argue if all software was indeed created by an "intelligent designer", but that is the subject of another blog. Designing software to best support its own evolution is by far the hardest thing a software architect must attempt to do. While evolution is a relatively simple thing to comprehend, it is not even remotely simple to design for.
Service Oriented Architecture is the next step along the path our industry has taken since its inception to achieve "evolvability" in software design. At first we categorized and organized data "in-situ" with structured programming, then we began to add behavior and encapsulate both data and its behavior as objects with object-oriented design, and now we are making behavior a first-class design concern with SOA - pushing data to more of a "behind-the-scenes" supportive role. This is because the knowledge that forms the competitive advantage of a business is derived from the interpretation of data created and managed through the processes with which the business provides value to its customers. Data is only a means to an end, it is in the successful interpretation of data that a business lives or dies. That successful interpretation is inexorably linked to the evolving environment in which the business is competing. Safely and securely nurturing and advancing the IP of a business is indeed akin to "survival of the fittest" within the free market economy.
So why is Agile the best method to use when approaching the evolutionary characteristic that is the chief concern of SOA? Simple, because while those of us who architect software may be intelligent - we are not omniscient, and cannot predict the ways in which our software and its use within the business context will evolve over time. Whenever I discuss Agile, I like to go back to the 4 basic values that were recognized by Kent Beck in "Extreme Programming Explained: Embrace Change", Addison-Wesley, 1999. These values are Communication, Simplicity, Feedback, and Courage.
Communication. "The first value is communication. Problems with projects can invariably be traced back to somebody not talking to somebody else about something important."
An Agile approach to SOA is all about establishing, and continually improving, the communication between business and IT. Architecture is the way in which those of us involved with software design communicate to our business counterparts. This communication is meant to be "reflective", in that it should reflect the values, priorities, and requirements initially shared with us by the business. Perhaps the biggest way in which SOA improves the "reflective" property of IT to business communication around software design is that it de-couples the two concepts of business and application services. Business services are discovered in a top-down fashion from the activities that make up a business process. A business service delegates the automated work related to the activity it is modeling to application services. Application services are discovered in a bottom-up fashion from the existing systems and data that support the business. It is in the mapping of business services to application services that our business counter-parts can truly understand the value provided by their application portfolio, and that we in IT can rationalize the continuing investment from the business for maintaining it.
Simplicity. "The second value is simplicity. The [Agile practitioner] continually asks the question: What is the simplest thing that could possibly work?"
An Agile approach to SOA is all about deriving simplicity from the seemingly complex. Legacy application services are complex, mainly because those who designed them just didn't expect the internals of these applications ever to see the light of day in the business world. Legacy Enterprise Resource Planning (ERP) applications were a commodity and so long as they functioned properly as a whole within acceptable service levels, it didn't really matter how they were designed internally. However, in SOA, these legacy application monoliths must be broken down into the specific services they provide in order to effectively trace those application services to the business activities they support.
Simplicity in SOA comes down to reducing the "surface area" of the software we are integrating into our business. Legacy monolithic applications just have too much surface area to consider when attempting to integrate them within an evolving business process. The ERP days were also all about implementing these legacy monoliths "vanilla", with as few customizations as possible - because customizations were hard and cost money. This led to the business being forced to conduct their processes the way the software dictated they be conducted. This rigidity, introduced by IT, led to many businesses falling behind in their ability to compete by successfully adapting to their continually evolving industries. This has left a very bad taste in their mouths about IT (not to mention the debacle of Y2K that yielded absolutely no business value).
With SOA, it is an absolute imperative that we find the simplest way to keep our respective businesses competitive. SOA promotes simplicity in its design through modularity and flexibility. By flexibly composing more complex business services from simple, modular application service building blocks - we are better preparing that software to evolve along with the business processes it supports. Kent Beck says it best: "[Agile software development] is making a bet. It is betting that it is better to do a simple thing today and pay a little more tomorrow to change it if it needs it, than to do a more complicated thing today that may never be used anyway." Time is the enemy of complexity - only the simplest of solutions will endure.
Feedback. "The third value [of Agile] is feedback. Concrete feedback about the current state of the system is absolutely priceless. Optimism is an occupational hazard of [software architecture]. Feedback is the treatment."
An Agile approach to SOA is all about promoting feedback at both design-time and run-time, creating a continual feedback loop between the changes in the processes that support the business and the responsiveness of IT to those changes. At design-time it does this by improving the testability of software while at run-time it does this by allowing real business results to be tracked and monitored in real time.
In software, increased modularity leads to increased testability. Testability of software deals with the ability of software to hide errors from detection during testing. More complex software is more difficult to test because its complexity acts as a sort of camouflage for latent bugs, allowing them to survive longer without detection. Many studies have shown that the longer it takes to detect a bug, the more costly it is to correct that bug. Simple, modular software architecture allows for more effective testing of smaller, and therefore more manageable, chunks of an application. Since this testing can be focused on the module and performed independently of the rest of the application, it is far less likely to conceal bugs further into the software lifecycle.
SOA also promotes feedback at run-time by linking the work performed by application services to tangible business results. Business services represent activities that add value for the customer within the context of a business process. Run-time monitoring of these business services provides real-time feedback on how well the business is running. If a business service is not running within established service levels, it is also easy to "click into" a particular business service and see if the reliability, availability, scalability or performance of the supportive application services are the source of the problem. Finally, the real-time metrics collected by monitoring a business process can be used to simulate business process performance under load to better understand the impact of changes to that business process while those changes are still under test.
Courage. "Within the context of the first three values - communication, simplicity, and feedback - it's time to go like hell. If you aren't going at absolutely your top speed, somebody else will be, and they will eat your lunch."
An Agile approach to SOA is all about finding the courage to be wrong sometimes, while minimizing the impact of being wrong and optimizing the time to correct those mistakes as they are discovered. Due to the predictable nature of hardware architecture, it wasn't generally acceptable for the first generation of enterprise architects to be wrong. The new generation of architects must be capable of making educated decisions in the trade-off between "perfect" and "fast". If software is going to successfully evolve at the speed of business, we software architects must learn how to facilitate, not impede, that evolutionary progress. We must trust that if we follow the other tenants of the agile approach, that our mistakes will be fewer, of smaller scope, and much easier to detect and correct. The business does not really care if you can unequivocally point the finger at a software vendor as the source of a problem - they only care how long it will take you to diagnose and correct that problem. If you are able to keep pace with the business, then small, quickly recoverable problems with the software will be tolerated by the business. It is only when you cannot keep pace with the business that the business begins to expect the impossible. Your customer's expectations for the software you deliver are directly proportional to the time it takes for you to deliver it -- the longer it takes for you to deliver software to your customer, the higher your customer's expectations for that software will be. SOA gives you the modular and flexible design you need to embrace business change while an Agile approach gives you the most rapid approach to keep pace with that business change.
So if I've convinced you that an Agile approach to SOA is the right approach, you may come to see the monolithic SOA Suites the large vendors sell as overly-complex and the cumbersome development methods and processes they dictate to you as overly-slow. This will be the subject of my next blog - part III in this series.
