mca blog

it's been a busy month or so for me. among other things, i was lucky enough to participate in GOTOCPH 2011 a couple weeks ago. while there, in between hanging w/ several rather smart people, i had the opportunity to present my talk " Beyond REST : An approach for crafting stable, evolve-able Web applications ." this talk is the first one i've delivered in quite a while aimed at "system-level thinking." the material is a bit "fuzzy" right now but, based on my experience over the last few years, there are some valuable bits here (even if i don't have much solid evidence or dispassionate rhetoric, or a cool graphical model to back up my talk[grin]).

you can check out the slides, but here is a tl;dnr version:

The Question

"How can we design and implement distributed network solutions that remain stable and flexible over time?"

"How can we create 'Evolvable systems?'"

frankly, it's not all that hard to put up an 'app' that can be reached 'on the Web.' the hard part is implementing a solution that can successfully evolve over time. this is especially challenging when done in a distributed network where there is no central control of all the participants; where some client apps or server apps are built and maintained by parties that have no direct physical or cultural ties (other than the 'app' itself).

my assertion is that one successful approach invovles a combintion of accepting and embracing some realites of dist-net architecture, isolating the key transient aspect of the environment, and focusing most of your creative eneregies on expressing the vital problem domain information in a flexible and evolvable way.

The Transfer Protocol

"HTTP is standardized, stable, and subject to careful review and control. Accept HTTP 'as-is' and stop trying to turn it into something else in order to solve your own private system problems."

today, dist-net apps are almost always built using HTTP as the Transfer Protocol. however, too many implementations treat HTTP as a 'burden' instead of a boon. many programming frameworks do thier best to hide HTTP from devs. most devs are actually encouraged to ignore key aspects of the protocol (caching, concurrency, state-lessness, etc.). finally, some environments treat HTTP as a 'malleable space' where adding new methods and meta-data (headers) in order to maintain 'the fiction' of a 'local application model' is perfectly acceptable instead of changing one's design paradigm to better fit HTTP's established communications model.

this leads to ineffciencies, frustration, and - in some cases - causes harm to both users and devs.

the better way to deal with HTTP is to embrace it. let HTTP lead system designers in the direction of lossy, chunky, state-less designs. accept HTTP as the stable rarely-changing foundation for your implementations. learn to 'think' and 'speak' HTTP so that all your dist-net designs reflect the power, stability, and reliability of HTTP itself. and don't fuss with it.

State Management

"If HTTP is the standard, stable foundation of the network, State Management is HTTP's transient, unreliable cousin."

time and again, i find devs trying to 'swim against the tide' of HTTP and figure out ways to persist state information over an extended set of interactions on the Web. while it usually works, it often takes way too much effort (from the dev or the framework) and results in brittle implementations that 'leak' personal data and/or screw up important data points somewhere along the way. some implementations go so far as to inject this transient information directly into representation responses in an attempt to get both client and server to 'share' huge chunks of volatile per-request data that should never be shipped repeatedly back and forth between the server and every single client that server is trying to support.

instead of constantly inventing new ways to ignore that state-less nature of dist-net interactions, designers and developers need to stop polluting the public internet with their transient personalization data bits. clients should keep track of the information users need (that's why they are often called user agents). servers should keep track of the information required to support the private components running behind public-facing URIs. and implementations should be designed to allow clients and servers to drop, and then later pick up, interaction threads at any time w/o complex work needed to 're-establish' session scaffolding all over again.

again, state is transient and volatile. let clients and servers deal with this information on their own, in their own ways, using their own local identification and storage mechanisms.

Domain Semantics

"While HTTP is the standardized, stable foundation protocol honored by all parties and state management is the private concern of each individual party, Domain Semantics is the shared knowledge between willing participants on the network. It's the Domain Semantics upon which implementors and designers need to focus."

leave HTTP alone, keep transient state a private concern of each party on the dist-net. the real place designers need to work is in the area of domain semantics; the shared understanding between consenting participants. and the way that works today is through the resource representations passed around the network.

not the URIs, the representations.

and the way domain semantics is shared is through a hypermedia type; one that supports link annotations (@rel), state-transfers via "form"-like constructions (@name), and decorations of individual elements in the response (@id and @class). it's the media type and its domain-specific details that provides the semantic knowledge that is shared. hard coding clients to "just know" what to ask for, how to construct URIs, and when to ask for it is the wrong way to implement dist-net solutions. these processes result in 'empty' messages that have no semantic details shipped between parties that maintain lots of 'hard-coding' details promoting brittle-ness, instability, and inflexibility.

it's the media type that provides the chance to customize messages between parties; to share understanding of what each data element and hyperlink 'means.' it's the media type that is specifically provided a registry where individuals who want to find shared understanding of a problem domain in order to implement mutually agreed appropriate domain-level solutions.

"In distributed networks, shared understanding is provided not through the standardized protocol and not through leaking private transient state bits, but through a well-designed and properly implemented hypermedia type. This is where the bulk of the effort should be spent."

and so...

over the last few years i've worked in a number of situations where the act of focusing on expressing the problem domain via media types has made it possible to build solutions that have the ability to evolve over time at very low expense (in both time and money). by making sure clients and servers get their shared understanding of the problem domain through the messages passed between parties and not through static, hard-coded rules, dist-net systems can successfully grow and evolve while still remaining stable and reliable for all participants.

in the end, building evolvable systems is not very complicated. it requires clear understanding of the transfer protocol, careful isolation of transient data, and dedicated work to properly express the problem domian via shared messages. it may not be a complex process, but it does take hard work.

hard work that pays off not only in the long run, but every day you use the system.