interoperability, not integration
it's a small diff, really - one of definitions - but a difference i think
is worth calling out. when working in large distributed systems, it's
better to aim for
interoperability
than
integration.
and here's why...
integration for a single system
The Merriam-Webster Online Dictionary has a definition for integration that speaks to my point:
...the combining and coordinating of separate parts or elements into a unified whole...
a common approach to working with large distributed systems (e.g. internal networked implementations that run at various locations within a single organization OR implementations that rely on some WWW-based service[s]) is to attempt to treat the entire operation as a single unit - a "whole system."
bad idea.
these "whole systems" can also be given the name "closed systems." IOW, people work to create a fully-controlled single unit that, even when separated by space (location) and time ("We built that part three years ago!"), the expectation is that things will work as if they are all physically local (on a single machine) and temporally local (there is no meaningful delay in the completion of requests). as you might expect, attempting this almost always goes badly - at least at any significant scale. IOW, "toy" systems will likely not exhibit any problems.
there are several reasons for attempting the approach. the most common is that treating everything as "your system" is mentally easy. another reason this "single system" view prevails is that most tooling acts this way. the legacy of edit and build tools is that all components and data are local and easily accessible. how else would we be able to do things like 'code completion' and data model validation?
anyway, the point here is that 'integration' is an anti-pattern on 'The Web.' it's not a good idea to use this as your mental model when designing, implementing, and deploying large-scale systems.
interoperability for working with other systems
as you might guess, i find Merriam-Webster's definition for interoperability much more valuable:
ability of a system ... to work with or use the parts or equipment of another system
the interoperability mindset takes a different approach. in this view, you want to treat as many things as possible as interchangeable; as things that can be 'swapped out' or re-purposed along the way. interestingly Merriam-Webster notes the first known use of this term was in 1977. so the idea of interoperability is relatively 'new' compared to 'integration' which Merriam's claims was first used in 1620.
an interoperability approach leads to systems that do not need to 'understand' each other, just ones that use 'interchangeable parts.' especially in widely-distributed systems, this interchangeability has a very high value. it's easier to replace existing items in a system (e.g. changing data-storage vendors), re-use existing parts for other needs (e.g. apply the same editing component built for a blogging service to a new print publishing service), and even re-purpose parts when needed (use the file-based document caching system to provide caching for logged-in user sessions).
the primary challenge to thinking like an inter-operator instead of an integrator is that there are no easy 'tools' for this kind of work. most all integration work is done by creative thinkers 'in the field' ("You know, we could just use our existing storage system for that..."). you usually need a rather specific knowledge of what's available on site and what the existing parts can do in order to execute on interoperability.
despite the extra cost of interoperability, there are important benefits for distributed systems that must operate over a long period of time. that's why so much of the WWW relies on interoperability. the standards we use for DNS, HTTP, HTML, etc. all assume that varying products and services are free to decide what they do and how they do it as long as they inter-operate with other products and services on 'The Web.'
treat the network space as a bunch of 'other systems'
if you take the approach that everything in your 'network space' (e.g. your local intranet or any system that relies on at least one WWW-based service) as a bunch of 'other systems' you'll be better of in the long term. you'll stop trying to get everyone to work the same way (e.g. use the same storage model, or object model, or resource model, etc.) and will be free to start working with other teams on how you can share information successfully across systems via interoperability.
even better, large organizations can get a big value out of creating an interoperability model for their implementations. this amounts to fostering an internal ethos where it's fine to be creative and solve problems in novel ways using whatever means are at your disposal as long as you make sure that you also support interoperability to the rest of the parts of the system. IOW, you have the freedom to build whatever is most effective locally as long as it does not threaten your interoperability with the other parts.
there are lots of other benefits to adopting interoperability as the long-term implementation goal, but i'll stop here for how. however, the upshot of all this rambling can be summed up here:
As a general rule, treat your implementations as exercises in interoperability, not integration.