Business and Technology Alignment

Large-growth technology start-ups still predominantly originate in Silicon Valley and Boston, but there's no reason why it can't happen here in the DC / Baltimore Alley. The raw materials and opportunities are here to both innovate and impact business and government on a grand scale, but the culture of business and technology are not aligned in that mission the way they need to be. To succeed as a tech-driven or tech-reliant innovator, leadership must drive execution top down to align business opportunity with technical excellence and foster this constant balance of values as part of their culture. Every employee has got to be in a position to contribute on both ends of the equation and have the support and incentives to do so.

Having worked with and in multiple technology startups in all these areas, I've seen first-hand that the differences are stark. Successful tech startups in "The Alley" tend toward sales and business heaviness. While this drives revenue and validates opportunity early (which is excellent) serious issues arise at scale when sound, experienced technical leadership and decision making isn't proactively integrated into the culture and DNA. Great ideas and opportunities without technical execution are doomed to failure as reality strays more and more from the promises of sales and marketing. Worse yet, the repeating pattern of this misstep creates a false perception that the talent for technical execution and innovation is not to be found in the area at all. In this scenario, perception becomes self-fulfilling prophecy driven by poor selection and fostering of talent at the highest levels.

On the other side of the coin "The Alley" is brimming with brilliant and experienced technologists doing amazing things with what they know about both their trade and industry. Unfortunately, very few of them are looking at the alignment of that work with the business opportunity that could elevate them to innovation greatness. More than that, the syndrome above promises them little hope of reward or recognition for bridging this gap or for partnering with those with the business acumen they so desperately need. The "culture of misalignment" strikes again here, discouraging or preventing technologist from working at this balance in themselves and their organizations. The syndrome's final outcome is an "us and them" mentality that promotes technical mediocrity, which tends to be predictable and tactical - focused on low risk, low value "wins" and distance from largely orphaned, higher-risk, more innovative efforts that may or may not be overly ambitious.

Enterprise Architecture professionals and practices have been pushing this alignment top down in large organizations for decades and "The Alley" is a veritable gold mine of those professionals. That being said, especially in a young tech company, the continued ability to execute and get your hands dirty at the leadership level is still critical. In fact, Silicon Valley companies like Google and Amazon still look for that kind of background from their leaders and regularly recruit and relocate from here. Finding the right technical leaders, empowering them, and making sure that the whole organization is built to reward strategic innovation, measured risk taking, and technical excellence that maps to the bottom line and competitive advantage is the most critical commitment a company can make.

The commitment to business and technical alignment is the key missing piece in the alley's start-up culture. Where there has been success here, you'll see exceptions to the rule and people willing to truly push the envelope. Building a culture that evangelizes, fosters, and rewards this re-alignment could fundamentally transform the innovation landscape in the area and set the tone to make the alley a hub of new kinds of companies built around the unique value propositions available.

Pseudo Documents: Reconciling REST, Reporting, and RPC

When constructing APIs for complex systems, particularly web-based systems, REST is becoming the preferred protocol. While the trend is strong, many programmers are still used to designing and building standard Remote Procedure Call (RPC) style web services and SOAP and WSDL are still major players in integration. While REST is simple and straight-forward (once you get the hang of it), it seems at first blush to have difficulty with problems that standard function calls handle with ease.

Enter Pseudo Documents - conceptually sound business domain data critical to communication but not necessarily stored in your database.

Standard REST encourages exposure of the entire data model as an organized set of documents arranged in a logical folder structure. When considering directly exposing a database in this way, each table (or each major table) will have a path location on the web site, a simple defined document to represent its data, and support create, update, read list, read single item, and delete calls at that path using the POST, PUT, GET, GET with ID passed on the path, and DELETE HTTP methods respectively. Every major entity is arranged in a taxonomy by URL path as if each record were simply a document file and each table a directory on the web server. This structure makes it very easy to define the API for basic CRUD operations quickly with most of the thought going into organizing the taxonomy logically.

Where this simple strategy appears to fall down is around non-standard operations like searches, reporting, and batch operations. When operations don't map neatly to your data model, defining Pseudo Documents (you can think of them as data views or message) to represent specific actions allow the API to call out important concepts like report requests and search queries as first-order concepts. These special documents now declare data critical to specialty operations in the same way the data model entities themselves were before (actually these documents existed already in the procedural world - we just don't recognize them explicitly) and REST opens them up to being stored, cached, and even used in asynchronous services quite easily.

A great example of a Pseudo Document is a report request. Imagine the need to pull data from 3 joined, time-series data tables where the request takes in a date range (start and end dates). Let's assume the output data is also grouped by geographic region and includes total number of people, total spending, total revenue, and profit for each day and geographic area. In this case, the input is more complex than a simple identifier (although not much) and the output is not directly related to any one specific table. Moreover, the basic CRUD operations do not truly apply at all in the standard sense.

As the input of a report request becomes more complex, it often makes sense to think of the report request as a document in and of itself and to use the POST method as if you were creating a ticket to be fulfilled (i.e. CREATE Report Request). You can even merge the request and response into a single document so that requests may be created, queued, fulfilled, cached, and saved.

Here is an example request containing only the input data:

POST /data/report/geo/finance/day HTTP 1.0

content-type: application/json

{

    startDate: '10-10-2011',

    endDate: '11-12-2011'

}

In the simplest case the response comes back immediately along with the request data so that the document is still a single document as shown below.

content-type: application/json

{

    startDate: '10-10-2011',

    endDate: '11-12-2011',

    identifier: 12345678000,

    data: [

      {

        region: 'southeast',

        date: '10-10-2011',

        people: 12332,

        spending: 1233.22,

        revenue: 2344.22,

        profit: 1111.00

      }, 
      {

        region: 'southeast',

        date: '10-10-2011',

        people: 12332,

        spending: 1233.22,

        revenue: 2344.22,

        profit: 1111.00

      }

     ...

    ]

}

Because we consider the document a record of a specific report request and it's result for the daily geographic financial report we can represent the entire report as a single document cleanly. We can also consider the POST a creation activity in line with the REST specification. This strategy also opens up the possibility of moving a long-running report to be fulfilled asynchronous by returning just an identifier and a status at creation time as shown below.

content-type: application/json

{

    startDate: '10-10-2011',

    endDate: '11-12-2011',

    identifier: 12345678001,

    status: 'pending'

}

Since the response indicates that the data is not ready and returns a unique identifier for the report request, the client can now poll for the results until they come back. We can use the GET method with this ID to poll for the results as shown here.

GET /data/report/geo/finance/day/12345678001 HTTP 1.0

content-type: application/json

While the report is being run, the response will for the GET method will continue to be identical to the pending status response above. Once the asynchronous report has completed however, the response can return with with the complete data for the report as shown in the longer response above.

Pseudo Documents like these allow you to leverage REST to accomplish very interesting things as easily as in any standard procedure call. The added benefit is that every critical message in the business service becomes a declared and defined part of the information taxonomy which often improves communication. This strategy also allows for algorithmic translation between REST and standard RPC services like SOAP or RMI to ensure parity across different types of interfaces.