Agile SOA - Part II

After establishing the forces behind the paradigm shift of Service Oriented Architecture (SOA) in Part I of this series, I'd like to talk about how we should now be approaching enterprise architecture from the perspective of software architecture in this article. SOA is an architectural pattern for how to build software that better supports the evolutionary characteristics of your business. The emphasis within this discussion will be around the term "evolutionary", and how agile methods best support the evolutionary development of software.

We'll first start by contrasting the "evolutionary" aspect of modern software architecture against the "predictable" aspect of modern hardware architecture. Moore's law best describes the predictability of modern hardware architecture: that almost every aspect of digital electronic devices (processing speed, memory capacity, etc.) are all improving exponentially, doubling approximately every two years. The trend was first observed by Intel co-founder Gordon E. Moore in a 1965 paper, has continued for almost a half of a century, and is not expected to stop anytime soon.

Modern software architecture has no such predictability. It is best described by the theory of evolution, well-known within the domain of biology. The primary reason for this is that software is used to model the knowledge and processes that form the foundation of a business. This intellectual property (IP) that gives a business its competitive differentiation comes from the coordinated effort of its people. In biology, evolution is change in the inherited traits of a population of organisms from one generation to the next. Evolution occurs naturally within any biological population, so it is natural to expect that that the knowledge and processes guarded by a group of people will evolve as well, and therefore that the software that models that IP must have the chief characteristic of "evolvability" in its design.

Whether you fall on the side of Evolution or Creation as it applies to the origins of humanity, no one can deny the fact that software is created. One could argue if all software was indeed created by an "intelligent designer", but that is the subject of another blog. Designing software to best support its own evolution is by far the hardest thing a software architect must attempt to do. While evolution is a relatively simple thing to comprehend, it is not even remotely simple to design for.

Service Oriented Architecture is the next step along the path our industry has taken since its inception to achieve "evolvability" in software design. At first we categorized and organized data "in-situ" with structured programming, then we began to add behavior and encapsulate both data and its behavior as objects with object-oriented design, and now we are making behavior a first-class design concern with SOA - pushing data to more of a "behind-the-scenes" supportive role. This is because the knowledge that forms the competitive advantage of a business is derived from the interpretation of data created and managed through the processes with which the business provides value to its customers. Data is only a means to an end, it is in the successful interpretation of data that a business lives or dies. That successful interpretation is inexorably linked to the evolving environment in which the business is competing. Safely and securely nurturing and advancing the IP of a business is indeed akin to "survival of the fittest" within the free market economy.

So why is Agile the best method to use when approaching the evolutionary characteristic that is the chief concern of SOA? Simple, because while those of us who architect software may be intelligent - we are not omniscient, and cannot predict the ways in which our software and its use within the business context will evolve over time. Whenever I discuss Agile, I like to go back to the 4 basic values that were recognized by Kent Beck in "Extreme Programming Explained: Embrace Change", Addison-Wesley, 1999. These values are Communication, Simplicity, Feedback, and Courage.

Communication. "The first value is communication. Problems with projects can invariably be traced back to somebody not talking to somebody else about something important." 

An Agile approach to SOA is all about establishing, and continually improving, the communication between business and IT. Architecture is the way in which those of us involved with software design communicate to our business counterparts. This communication is meant to be "reflective", in that it should reflect the values, priorities, and requirements initially shared with us by the business. Perhaps the biggest way in which SOA improves the "reflective" property of IT to business communication around software design is that it de-couples the two concepts of business and application services. Business services are discovered in a top-down fashion from the activities that make up a business process. A business service delegates the automated work related to the activity it is modeling to application services. Application services are discovered in a bottom-up fashion from the existing systems and data that support the business. It is in the mapping of business services to application services that our business counter-parts can truly understand the value provided by their application portfolio, and that we in IT can rationalize the continuing investment from the business for maintaining it.

Simplicity. "The second value is simplicity. The [Agile practitioner] continually asks the question: What is the simplest thing that could possibly work?"

An Agile approach to SOA is all about deriving simplicity from the seemingly complex. Legacy application services are complex, mainly because those who designed them just didn't expect the internals of these applications ever to see the light of day in the business world. Legacy Enterprise Resource Planning (ERP)  applications were a commodity and so long as they functioned properly as a whole within acceptable service levels, it didn't really matter how they were designed internally. However, in SOA, these legacy application monoliths must be broken down into the specific services they provide in order to effectively trace those application services to the business activities they support. 

Simplicity in SOA comes down to reducing the "surface area" of the software we are integrating into our business. Legacy monolithic applications just have too much surface area to consider when attempting to integrate them within an evolving business process. The ERP days were also all about implementing these legacy monoliths "vanilla", with as few customizations as possible - because customizations were hard and cost money. This led to the business being forced to conduct their processes the way the software dictated they be conducted. This rigidity, introduced by IT, led to many businesses falling behind in their ability to compete by successfully adapting to their continually evolving industries. This has left a very bad taste in their mouths about IT (not to mention the debacle of Y2K that yielded absolutely no business value). 

With SOA, it is an absolute imperative that we find the simplest way to keep our respective businesses competitive. SOA promotes simplicity in its design through modularity and flexibility. By flexibly composing more complex business services from simple, modular application service building blocks - we are better preparing that software to evolve along with the business processes it supports. Kent Beck says it best: "[Agile software development] is making a bet. It is betting that it is better to do a simple thing today and pay a little more tomorrow to change it if it needs it, than to do a more complicated thing today that may never be used anyway." Time is the enemy of complexity - only the simplest of solutions will endure.

Feedback. "The third value [of Agile] is feedback. Concrete feedback about the current state of the system is absolutely priceless. Optimism is an occupational hazard of [software architecture]. Feedback is the treatment."

An Agile approach to SOA is all about promoting feedback at both design-time and run-time, creating a continual feedback loop between the changes in the processes that support the business and the responsiveness of IT to those changes. At design-time it does this by improving the testability of software while at run-time it does this by allowing real business results to be tracked and monitored in real time. 

In software, increased modularity leads to increased testability. Testability of software deals with the ability of software to hide errors from detection during testing. More complex software is more difficult to test because its complexity acts as a sort of  camouflage for latent bugs, allowing them to survive longer without detection. Many studies have shown that the longer it takes to detect a bug, the more costly it is to correct that bug. Simple, modular software architecture allows for more effective testing of smaller, and therefore more manageable, chunks of an  application. Since this testing can be focused on the module and performed independently of the rest of the application, it is far less likely to conceal bugs further into the software lifecycle.

SOA also promotes feedback at run-time by linking the work performed by application services to tangible business results. Business services represent activities that add value for the customer within the context of a business process. Run-time monitoring of these business services provides real-time feedback on how well the business is running. If a business service is not running within established service levels, it is also easy to "click into" a particular business service and see if the reliability, availability, scalability or performance of the supportive application services are the source of the problem. Finally, the real-time metrics collected by monitoring a business process can be used to simulate business process performance under load to better understand the impact of changes to that business process while those changes are still under test.

Courage. "Within the context of the first three values - communication, simplicity, and feedback - it's time to go like hell. If you aren't going at absolutely your top speed, somebody else will be, and they will eat your lunch."

An Agile approach to SOA is all about finding the courage to be wrong sometimes, while minimizing the impact of being wrong and optimizing the time to correct those mistakes as they are discovered. Due to the predictable nature of hardware architecture, it wasn't generally acceptable for the first generation of enterprise architects to be wrong. The new generation of architects must be capable of making educated decisions in the trade-off between "perfect" and "fast". If software is going to successfully evolve at the speed of business, we software architects must learn how to facilitate, not impede, that evolutionary progress. We must trust that if we follow the other tenants of the agile approach, that our mistakes will be fewer, of smaller scope, and much easier to detect and correct. The business does not really care if you can unequivocally point the finger at a software vendor as the source of a problem - they only care how long it will take you to diagnose and correct that problem. If you are able to keep pace with the business, then small, quickly recoverable problems with the software will be tolerated by the business. It is only when you cannot keep pace with the business that the business begins to expect the impossible. Your customer's expectations for the software you deliver are directly proportional to the time it takes for you to deliver it -- the longer it takes for you to deliver software to your customer, the higher your customer's expectations for that software will be. SOA gives you the modular and flexible design you need to embrace business change while an Agile approach gives you the most rapid approach to keep pace with that business change. 

So if I've convinced you that an Agile approach to SOA is the right approach, you may come to see the monolithic SOA Suites the large vendors sell as overly-complex and the cumbersome development methods and processes they dictate to you as overly-slow. This will be the subject of my next blog - part III in this series.

Agile SOA - Part I

Service Oriented Architecture is not an expensive suite of software products as the large software vendors would like to sell you. SOA is an architectural pattern. A pattern is a well-described, proven way of doing something that has evolved out of multiple attempts (successes and failures) across an entire community of professionals. SOA is a best-practice way of constructing an enterprise software architecture to better suit the evolving needs of your business.

The real challenge of adopting SOA is to change the way you and your organization think about enterprise architecture, not to change your information technology infrastructure yet again to continue the cold war-like arm's race against your competitors the software vendors have sold you on (at a great profit to them, I might add).

Why is SOA such a challenge? Simple, because it puts the focus of enterprise architecture on software architecture, not hardware and network architecture. The IT industry is in the midst of handoff between its first generation of infrastructure architects and a new generation of software architects. Businesses, who have come to rely heavily on their architects, are struggling to understand a new sofware-centric view of their technology portfolio. The old days were (somewhat) easy, let the IT guys handle the infrastructure, and we'll handle the business. Investments in technology were relatively simple and straightforward to both understand and manage - and the results were tangible assets that had an expected life and could be depreciated with comfortable precision.

The first generation of architects are extremely good at what they were asked to do - connect hardware through ever-growing and ever-speedier networks. Software, at least the software they intended to run the mission-critical elements of the business on, was considered to be a commodity - just like the hardware and network components they were used to implementing. Their view of software from an enterprise perspective was through the hardware nodes that the software was to be deployed on (this box is for General Ledger, that box is for Accounts Payables, this box is for email, that box is for our website, and so on ...). Rack it, stack it, connect it up, and turn it on.

The new generation of architects don't think of software only in terms of how it is deployed. Disrupting forces like the Internet, Mobile Computing, Virtualization, and Cloud Computing are making physical hardware and networks a ubiquitos (and somewhat abstract) concept. With the changes brought about by these forces, it is not likely that the IT group of the future will even continue to manage physical technology assets within business-owned data centers anymore.

So if IT departments are no longer managing hardware and networks for the business, what will that leave them with? Will IT cease to exist as a business-critical function within the enterprise? The answer, of course, is no (or else I'd be learning a new trade instead of blogging about this one). The answer is that IT departments will "move up the food-chain" within the business - becoming newly responsible for managing and securing its intellectual property (IP).

IP is the beating heart of today's business. It is composed of the knowledge and the processes that form the foundation of a business and give it its unique competitive differentiation within the marketplace. Knowledge and processes are modeled as software, not hardware - and those models can no longer be confined to the physical boundaries of hardware and networks. Look no further than Amazon, Google and other major internet-based companies that have survived the dot.com bubble to form the new guard of today's business for proof of this paradigm shift is real.

So how does this all tie back to SOA? SOA is simply a better way of managing the knowledge and processes that form the IP of your business. SOA is a better way because it is a software architecture that, like the knowledge and processes it manages, is not confined to the physical boundaries of hardware and networks.

So, if I shouldn't approach SOA the way I've approached technology investments in the past, by going out and buying some components from a vendor, connecting them up, and turning them on; how should I approach it? This will be the subject of my next blog - Part II in this series.

Revisiting Requirements Management

The number one job of the Architect in an IT project is to manage requirements. The Architect is in the only position within the enterprise to look deep enough into the business and deep enough into the technology to understand the translation of the requirements between the two. It is the Architect who is on the front-lines of business-IT alignment, and people should look no further when IT seems "out of touch" with the business.

A requirement, in its simplest form is the codification of a business need. In this regard, requirements should trace naturally to the goals of a business and the metrics that the business uses to track their performance against those goals. Business goals and performance metrics should trace directly to the business plan - which is the embodiment of the business strategy. 

Requirements management, the domain of an Architect, is about managing the full life-cycle of a requirement; ensuring that the business value (goals & metrics) traced to that requirement is both: 

1. effectively captured during requirements analysis and 
2. efficiently delivered by the technology traced to that requirement. 

In this way, all technology and associated investment should trace naturally to business value. 

Therefore, the first half of an effective requirements analysis process is for the Architect to ensure that this traceability to the business plan is well-defined, and that the business context associated with the requirement is clear to IT. Architects are on the front-line of IT strategy and planning, and it is their responsibility to challenge and guide their business counterparts in applying the same rigor to the business strategy and planning process as they apply to the technology strategy and planning process.

The second half of the requirements analysis process is for the Architect to ensure that the traceability to the technology plan is well defined, and that the IT context associated with the requirement is clear to the business. The IT context associated with a requirement can be broken down into three main areas: 

1. the expected impact of the requirement on the enterprise architecture, 
2. the planning input for the project required to deliver it, and 
3. the expected impact on IT operations to support it once it has been delivered.
   

The impact of a requirement (or set of requirements) on the enterprise architecture should be captured across three main enterprise technology views: the software architecture, the server(or hardware) architecture, and the network architecture. In order to properly trace a requirement from business to one of these technology views, the original requirement needs to be "refined". The process of requirements refinement is a meticulous one, that requires all of the skill of an experienced Architect. Refining requirements involves deriving new, more detailed requirements from the base requirement that are specific to the technology view being considered. This process should not add or remove from the scope of the original requirement nor otherwise alter the original requirement. The motivations, assumptions, and constraints considered while deriving a software, hardware, or network requirement from a business requirement should be well-documented by the Architect. This documentation serves as a feedback mechanism from the Architects to their business counterparts on their interpretation of the original requirement. Requirements refinement and associated documentation serves as the basis for involving the business in technology decisions by providing transparency to the IT strategy and planning process.

Planning input should include the basis of estimate that drives the variable costs, any required fixed costs, and any notable risks associated with delivering the requirement (or set of requirements).  The first half of the basis of estimate should be a quantifiable metric (other than hours) that drives effort. Some examples of quantifiable metrics include the number of expected users, a number of screens to be developed, or the number of entities in the data model. The second half of the basis of estimate should be an estimate of hours, by resource type, associated with delivering the requirement across that metric. For example, it may be estimated that it will take 6 hours of a DBA, 1 hour of a business analyst, and 1 hour of a project manager to implement the necessary physical database objects associated with a single entity. This information can then be easily used to generate the necessary staffing plan for a project to deliver the set of requirements being planned, which would smooth the consolidated effort estimate for each resource type across a set of requirements over the full timeline of the project being planned to deliver those requirements and associate a cost per hour for that resource type to determine the cost for that effort within the scope of that project. Fixed costs from subcontractors or vendors that will be providing specific services and / or products needed in the delivery of a requirement should also be included in the planning input for each requirement. Finally, any risks known at the time of planning associated with delivering the requirement should be well-documented. Included in the risk documentation should be a probability that the risk will be realized and its potential impact on project cost, time, and / or quality.

Lastly in the IT requirements analysis process should be the development or modification of a concept of operations for the requirement (or set of requirements) to be delivered. This concept of operations should include any on-going monitoring, management, or administrative activities required to operate the new software, hardware, and or network components delivered while fulfilling the requirement. Like project planning input, the concept of operations would document the basis of estimate for any ongoing variable costs, any regular fixed costs and their timing interval, and any risks associated with operations in support of the components being delivered to meet this requirement (or set of requirements).

Once proper analysis has been done for a requirement, an Architect's job is not over. It is the responsibility of the Architect to monitor the delivery and ongoing operations of each requirement for variances to plan. Variances in cost and time (efficiency) during delivery or operations, including the realization of risk, should be analyzed by the Architect to determine if they are one-time or systemic in nature. Systemic variances in IT efficiency discovered over time should be re-factored back into the requirements analysis output with proper change management controls in place to notify interested business and IT stakeholders. Finally, it is the Architect's responsibility to periodically review the business goals and performance metrics that each requirement is tied to to ensure that expected business value has been delivered by IT. Variances in delivering expected business value (effectiveness) should also be analyzed by the Architect to determine if they are one-time or systemic in nature. Systemic variances in IT effectiveness discovered over time should be re-factored back into the IT strategy and planning process with proper change management controls in place to notify interested business and IT stakeholders.

Architecture today often has the misconception by business and IT stakeholders as being a project-based activity. However; true requirements management, which is at the heart of all modern architecture, involves the regular monitoring and course-correcting (governance) of the business-IT alignment. Requirements management defines architecture as an on-going relationship activity between business and IT; one that should not be confined to the delivery of specific IT projects.

ReFactoring Enterprise Architecture

I was listening to David Linthicum's podcast from last week (http://akamai.infoworld.com/weblog/realworldsoa/archives/IFW_SOA_05-12-08.mp3) and got to thinking more about the problem of Re-Factoring Enterprise Architecture. Re-Factoring your Enterprise Architecture is a variant on the Enterprise Modernization angle we have been pursuing within the technology industry for legacy platform customers. The reason Enterprise Modernization is not enough in my mind is that it is only really targeted at customers (like the government) who are still on homogeneous (albeit legacy) platforms and due to personnel training and retention issues want to stay that way [homogenous]. Most commercial customers likely have a highly heterogenous environment or want one to better keep up with the heightened competition within their industry.

Most commercial enterprises would agree that their Enterprise Architecture has organically grown over the years in a similar way to how the Amish sew together a patch-work quilt. This is the problem of "shopping for your EA solution".

On the left-side of the "shopping for an EA" continuum, you have customers that consider themselves early adopters and are willing to try new things to "get an edge" on their competition. [insert your favorite market guru here] tells these customers that they should have a portal, and that Plumtree is one of the best point-solutions out there so they buy that. [insert your favorite market guru here] tells them they should have a framework, and that SilverStream is one of the best point-solutions out there so they buy that. [insert your favorite market guru here] tells them they need an ESB solution, and that Cape Clear is the one of the best point-solutions out there so they buy that. And so on. These folks may be fast out of the gate, but lose momentum over time, due to inefficiences caused by lack of integration along the way and the rapid turnover of products (and vendors) caused by the fickleness of the commodity software industry.

On the right-side of the "shopping for an EA" contiuum, you have customers that consider themselves more conservative and place a high value on staying with a single vendor. These folks are still "shopping for an EA", except that they wait to be told what to buy and when to buy it by IBM or Microsoft. These folks are always dealing with repressed feelings of frustration and doubt caused by their inability to keep up with software market innovation because their chosen vendor isn't getting them there fast enough.

In my opinion, customers at either end of this continuum can be refactored toward a planned Enterprise Architecture, highly customized to their specific needs, that can deal with change in a "systemic" or "repeatable" fashion through the combination of:

1. the strategic use of Open Source software to get at the core problems within the IT portfolio by either better glueing the pieces together or more rapidly extending the functionality of the monolith (with either approach based on open standards) - making the IT-side of these commercial customers more agile while also providing a forward-looking context for better supporting their engrained spending habits, and
2. the tactical use of Governance solutions to give the business-side of these commercial customers visibility into the measureable (metrics-driven) progress they are making towards the goals that define their reasons for investing in IT to begin with.

To summarize, I like the "ReFactoring your Enterprise Architecture" angle because it deals with the heterogeneity (or lack thereof) that likely exists in most commercial enterprises while still sending the "we aren't here to change you ... just make you better" message that Enterprise Modernization sends.