Business and Technology Alignment

Large-growth technology start-ups still predominantly originate in Silicon Valley and Boston, but there's no reason why it can't happen here in the DC / Baltimore Alley. The raw materials and opportunities are here to both innovate and impact business and government on a grand scale, but the culture of business and technology are not aligned in that mission the way they need to be. To succeed as a tech-driven or tech-reliant innovator, leadership must drive execution top down to align business opportunity with technical excellence and foster this constant balance of values as part of their culture. Every employee has got to be in a position to contribute on both ends of the equation and have the support and incentives to do so.

Having worked with and in multiple technology startups in all these areas, I've seen first-hand that the differences are stark. Successful tech startups in "The Alley" tend toward sales and business heaviness. While this drives revenue and validates opportunity early (which is excellent) serious issues arise at scale when sound, experienced technical leadership and decision making isn't proactively integrated into the culture and DNA. Great ideas and opportunities without technical execution are doomed to failure as reality strays more and more from the promises of sales and marketing. Worse yet, the repeating pattern of this misstep creates a false perception that the talent for technical execution and innovation is not to be found in the area at all. In this scenario, perception becomes self-fulfilling prophecy driven by poor selection and fostering of talent at the highest levels.

On the other side of the coin "The Alley" is brimming with brilliant and experienced technologists doing amazing things with what they know about both their trade and industry. Unfortunately, very few of them are looking at the alignment of that work with the business opportunity that could elevate them to innovation greatness. More than that, the syndrome above promises them little hope of reward or recognition for bridging this gap or for partnering with those with the business acumen they so desperately need. The "culture of misalignment" strikes again here, discouraging or preventing technologist from working at this balance in themselves and their organizations. The syndrome's final outcome is an "us and them" mentality that promotes technical mediocrity, which tends to be predictable and tactical - focused on low risk, low value "wins" and distance from largely orphaned, higher-risk, more innovative efforts that may or may not be overly ambitious.

Enterprise Architecture professionals and practices have been pushing this alignment top down in large organizations for decades and "The Alley" is a veritable gold mine of those professionals. That being said, especially in a young tech company, the continued ability to execute and get your hands dirty at the leadership level is still critical. In fact, Silicon Valley companies like Google and Amazon still look for that kind of background from their leaders and regularly recruit and relocate from here. Finding the right technical leaders, empowering them, and making sure that the whole organization is built to reward strategic innovation, measured risk taking, and technical excellence that maps to the bottom line and competitive advantage is the most critical commitment a company can make.

The commitment to business and technical alignment is the key missing piece in the alley's start-up culture. Where there has been success here, you'll see exceptions to the rule and people willing to truly push the envelope. Building a culture that evangelizes, fosters, and rewards this re-alignment could fundamentally transform the innovation landscape in the area and set the tone to make the alley a hub of new kinds of companies built around the unique value propositions available.

Pseudo Documents: Reconciling REST, Reporting, and RPC

When constructing APIs for complex systems, particularly web-based systems, REST is becoming the preferred protocol. While the trend is strong, many programmers are still used to designing and building standard Remote Procedure Call (RPC) style web services and SOAP and WSDL are still major players in integration. While REST is simple and straight-forward (once you get the hang of it), it seems at first blush to have difficulty with problems that standard function calls handle with ease.

Enter Pseudo Documents - conceptually sound business domain data critical to communication but not necessarily stored in your database.

Standard REST encourages exposure of the entire data model as an organized set of documents arranged in a logical folder structure. When considering directly exposing a database in this way, each table (or each major table) will have a path location on the web site, a simple defined document to represent its data, and support create, update, read list, read single item, and delete calls at that path using the POST, PUT, GET, GET with ID passed on the path, and DELETE HTTP methods respectively. Every major entity is arranged in a taxonomy by URL path as if each record were simply a document file and each table a directory on the web server. This structure makes it very easy to define the API for basic CRUD operations quickly with most of the thought going into organizing the taxonomy logically.

Where this simple strategy appears to fall down is around non-standard operations like searches, reporting, and batch operations. When operations don't map neatly to your data model, defining Pseudo Documents (you can think of them as data views or message) to represent specific actions allow the API to call out important concepts like report requests and search queries as first-order concepts. These special documents now declare data critical to specialty operations in the same way the data model entities themselves were before (actually these documents existed already in the procedural world - we just don't recognize them explicitly) and REST opens them up to being stored, cached, and even used in asynchronous services quite easily.

A great example of a Pseudo Document is a report request. Imagine the need to pull data from 3 joined, time-series data tables where the request takes in a date range (start and end dates). Let's assume the output data is also grouped by geographic region and includes total number of people, total spending, total revenue, and profit for each day and geographic area. In this case, the input is more complex than a simple identifier (although not much) and the output is not directly related to any one specific table. Moreover, the basic CRUD operations do not truly apply at all in the standard sense.

As the input of a report request becomes more complex, it often makes sense to think of the report request as a document in and of itself and to use the POST method as if you were creating a ticket to be fulfilled (i.e. CREATE Report Request). You can even merge the request and response into a single document so that requests may be created, queued, fulfilled, cached, and saved.

Here is an example request containing only the input data:

POST /data/report/geo/finance/day HTTP 1.0

content-type: application/json

{

    startDate: '10-10-2011',

    endDate: '11-12-2011'

}

In the simplest case the response comes back immediately along with the request data so that the document is still a single document as shown below.

content-type: application/json

{

    startDate: '10-10-2011',

    endDate: '11-12-2011',

    identifier: 12345678000,

    data: [

      {

        region: 'southeast',

        date: '10-10-2011',

        people: 12332,

        spending: 1233.22,

        revenue: 2344.22,

        profit: 1111.00

      }, 
      {

        region: 'southeast',

        date: '10-10-2011',

        people: 12332,

        spending: 1233.22,

        revenue: 2344.22,

        profit: 1111.00

      }

     ...

    ]

}

Because we consider the document a record of a specific report request and it's result for the daily geographic financial report we can represent the entire report as a single document cleanly. We can also consider the POST a creation activity in line with the REST specification. This strategy also opens up the possibility of moving a long-running report to be fulfilled asynchronous by returning just an identifier and a status at creation time as shown below.

content-type: application/json

{

    startDate: '10-10-2011',

    endDate: '11-12-2011',

    identifier: 12345678001,

    status: 'pending'

}

Since the response indicates that the data is not ready and returns a unique identifier for the report request, the client can now poll for the results until they come back. We can use the GET method with this ID to poll for the results as shown here.

GET /data/report/geo/finance/day/12345678001 HTTP 1.0

content-type: application/json

While the report is being run, the response will for the GET method will continue to be identical to the pending status response above. Once the asynchronous report has completed however, the response can return with with the complete data for the report as shown in the longer response above.

Pseudo Documents like these allow you to leverage REST to accomplish very interesting things as easily as in any standard procedure call. The added benefit is that every critical message in the business service becomes a declared and defined part of the information taxonomy which often improves communication. This strategy also allows for algorithmic translation between REST and standard RPC services like SOAP or RMI to ensure parity across different types of interfaces.

Domain Driven: The Consumer is King

One of the key reasons great software is successful is that it does a very good job of using metaphors and representations of information that everyone understands. Consumers of software systems must dictate the language used to deal with work or intuition works against them at every turn. This Domain Dictionary of terms, concepts, and metaphors are already in use wherever we build software and whether it's inherited from generally accepted concepts like finance or accounting, an established industry like publishing or banking, or are part of what differentiates the way a specific organization operates from another, years of evolution have ingrained the ideas (if not the entire lexicon) of the Domain Language in the minds of your customers.

The Domain Driven community refers to the result of capturing the domain language as a Ubiquitous Language stating that the terms of the language should proliferate all communication and code. This concept is among the most powerful concepts in Domain Driven Design and incredibly valuable in ensuring all members of a project team and the stakeholders communicate and share information properly. In the end, every API, class name, package name, URL pattern, e-mail must reference the domain language (or ubiquitous language) and a good architect will ensure that these definitions are up to date and spread across the team in some easily accessible and referable form (wikis make a great repository for the glossary).

What is key about this approach is that it alters the typical entity strategy (ground-up) construction of a system when done properly and focuses on the consumer of the system and its services first (top-down). This means leading with the language the consumers already use (and visualizations that make sense helps too) all the to the field level and making a commitment to every known view of information as they understand it. It also means abandoning the urge to force new metaphors or force-feed vocabulary (at least as much as possible) on the consumers (your customer) and an expectation that the architect and development team will handle translation of the conceptual model or domain model to the underlying Entity Model (that deals more directly with data as it's stored in a database).

If necessary, this abstraction can carry down to the data model in the form of views or materialized views particularly if your consumers feel strongly about reporting from the data model directly. Ideally, the chosen system architecture should supports a remote Services API layer, a business domain services layer, and a separate data access layer where domain data objects (DTOs or VOs) can be translated into more granular or less business friendly (although you should keep this minimal if possible) Entity Model representations. Still, the code at the domain data layer should be almost understandable to anyone in the domain because it should be completely reliant on actions and domain information definitions directly from the Domain Language Lexicon (DLL) or Dictionary (DLD). Actions, Entities (or Objects), Processes, and Events will make up the backbone of the language and those names should show up across the code base to ensure easy mapping between requirements, code, and sub-systems.

The more the underlying code base and user interface reinforce the DLD/DLL, the better chance of the development team delivering a system that is intuitive and clear to their consumers. The same goes for basic web services APIs and other integration points. The name of the game is consistent semantics across from top to bottom and clean, clear flow from one operation to the next. The great news is, great architecture strategies like MVC, OOD, N-Tier, Event-Driven, and Process Driven are all compatible with Domain Driven Analysis where the consumer is king. Done right, everyone wins and innovation will be well received because it will speak to everyone when it's delivered.