mca blog

I've been contemplating transactions quite a bit lately. The examples of "transactions over HTTP" I've seen appear to be strenuous attempts to force state-ful ACID and 2PC patterns originally designed for local area networks onto a loosely-coupled, stateless, representation-based application protocol running over an unreliable distributed network. I understand there are application models that specifically define a "final commit" that must be generated by the client application (e.g. bidding models that line up a series of partners and then require a final review and commit before proceeding). That case is not what I'm interested in here. Instead I am focusing on classic 2PC-type implementations over HTTP that ignore the realities of long-running operations over a distributed network.

The typical conversation thread, real or virtual, about transactions over HTTP goes something like this (elided for brevity):

"You don't want transactions over HTTP"

But I need to organize number of steps into a single unit I can deal with easily.

"OK, but you don't need transactions over HTTP"

But I need the ability to back out changes in multiple locations safely and consistently.

"OK, but you can't do transactions over HTTP!"

Really?

And here the topic usually dies or descends into a heated debate.

A better approach is to expose an HTTP-compliant transaction service interface (TSI) that takes advantage of the protocol's inherent architectural style. Transactions over HTTP should be optional, discoverable, negotiable, based on optimistic commits, and (in the case of failures) use compenstating requests as a way to reverse previous work.

Clearing things up

First, I think it's true that 2PC-style transactions over HTTP are not needed. Two-Phase Commit may be important on the server-side behind the HTTP interface between the client and server, but there is no reason to expose them to HTTP clients; they don't care what goes on at the server. Clients just want a 2xx response to their request.

Second, I agree there are times when it's important to have a way to safely cancel long-running operations that span multiple HTTP conversations. For example, a server that accepts a "buy" order from a client may need to interact with several other servers over HTTP in order to complete the required stock adjustments, arrange shipping, and confirm payment details. Note however, in my example, the "client" doing the "transacted work" is really the server accepting the initial request. Again, the original client doesn't give a hoot about any work going on behind the scenes no matter how many servers are involved. Lastly, while there is a need for handling failures in an encapsulated request chain (the "unit of work"), ACID/2PC is not appropriate over a distributed network. Instead, servers should implement and expose Saga-type compenstating requests.

A better approach

The way to handle transactions over HTTP is to think about them differently. The HTTP application protocol is than a uniform interface. Another important aspect of HTTP is reliance on control data that clients and servers use to share additional information about the associated message, implement runtime discovery, safely add optional service extensions, and negotiate preferences on how each party should treat the message body. One example of leveraging control data is implementing optimistic concurrency (also know as unreserved checkout) over a distributed network via the ETag header.

It is important to leverage the protocol's features and take advantage of it's architectural strengths in order to define and expose a transaction service interface that is "HTTP-compliant."

So, what would an "HTTP-compliant" transaction interface look like to clients and servers?

Optional, Discoverable, and Negotiable

The service should be optional. Requiring clients to know about and implement the service details ahead of time is a barrier to adoption and runs counter to the tenets of HTTP. Rather, servers should be able to advertise support for transactions and clients should be able to discover that support. This can be done using the OPTIONS method and two new headers: Accept-Trans for requests and Content-Trans for responses. These new headers should work like the existing Accept and Content-Type headers. Servers can use the headers to advertise support for one or more transaction implementations and clients can use them negotiate transaction implementation preferences. Extensions could be added and might include things like required=true|false, max-ttl={msec}, min-ttl={msec}, and other implementation-specific options.

Optimistic

Clients should not need to execute a second "commit" request in order to complete a unit of work. Instead, servers should always assume the client wants the work to complete unless explicitly told otherwise and should do so as soon as possible (or within specified time frames). Of course, servers are free to do whatever they wish in order to track and control the progress of work outside the HTTP request chain and return 4xx or 5xx response codes whenever appropriate.

Compensated

Failed or cancelled requests within a unit of work should be handled via compenstating requests. Enough research has been done on compensating models to show this to be viable for any operations that are commutative and can be interleaved with other similar operations. To enable compensation, servers should issue a Link header or Link element in responses with a rel="trans" (or some similar value) that clients can use to 1) GET the transaction's current state or; 2) DELETE an open transaction.

Transactions

There are several ways to support optional, discoverable, negotiable, optimistic, compenstated transactions. A trivial example might look like this (elided for brevity):

  *** REQUEST
  POST /orders/
  Host: www.example.org
  Accept-Trans: doctr
  
  *** RESPONSE
  HTTP/1.1 202 Accepted
  Location: http://www.example.org/orders/1
  Content-Trans:doctr
  Link <http://www.example.org/trans/abc>;rel=doctr;
  
  *** REQUEST  
  GET /trans/abc
  Host: www.example.org
  
  *** RESPONSE
  HTTP/1.1 200 OK
  
  ... body w/ details about the current transaction ...
  ... (open/closed, succeeded/failed, etc.) ...
  
  *** REQUEST 
  DELETE /trans/abc 
  Host: www.example.org
  
  *** RESPONSE
  HTTP/1.1 200 OK  

The advantages of this approach are 1) it's completely compatible w/ existing HTTP infrastructure; 2) it requires no additional request traffic forthe common case (2xx); 3) it does not lock any existing resources; 4) it can be rolled out on existing servers w/o breaking clients; 5) clients and servers are free to ignore, negotiate, and/or extend as desired; and 6) servers are free to continue to support any local transaction management they wish including ACID/2PC as long as it does not leak out past the HTTP boundary to clients.

There are details to work out, but they can be handled on the server where they belong. Clients only need to be able to negotiate for a transaction implementation and be prepared use the transaction LINK returned by the server when necessary. Servers, need to implement sagas locally and return the transaction LINK. They also need to implement support for cancel requests from the client and handle request failures from any "upstream" server enlisted in the unit of work.

A uniform interface

What is described here is a transaction service interface (TSI) that clients and servers can choose to implement in ways all parties can agree upon. This is the way media-types are currenly handled and TSIs could be defined, documented, and registered in much the same way. This includes the possibility of registering a media-type that matches the TSI implementation. True to the archetictural style of the HTTP protocol, this approach proposes a "uniform interface" for handling units of work over HTTP.

The good news is that clients don't need to know about or understand in advance any details of the TSI in order to interact with a server that supports it. Servers, however, can start engaging in compensated transactions among themselves at any time as long as they agree on the TSI implementation. Eventually, as clients catch up and TSI implementations proliferate, servers and clients can engage in negotations for various types or levels of transaction support before initiating the work.

Someone call for a DOCTR?

I'm currently working on a POC implementation I'm calling DOCTR (Discoverable, Optimistic, Compensated, TRansactions) [cute, eh?]. I'll post my results for others to review as soon as I can. If all goes well and feedback is positive, I'll continue to pursue a formal description that can stand up to further inspection.