De-mystifying the Enterprise Application Integration (EAI) landscape: Actors, terminology, cadence and protocols

Any form of Enterprise Application Integration (EAI) [1] work for data synchronization, digital transformation or customer self-service web implementation involves communication between the service providers and service consumers. A web of connections grows over time between systems, facilitated by tools specialising in “system-integration”; this article covers how the clients, services and integration tools communicate and nuances around this observed in the wild

EAI Actors

Depending on the context, a system becomes a data-consumer or data-provider. These consumers and providers can be internal to a business enterprise or external to them. External providers can be pure software-as-a-service or partners with platforms and build teams entrenched in the “client-site”

Screen Shot 2020-02-25 at 3.41.20 pm — 1. Services, Consumers within and outside a business enterprise

The provider and consumer systems are the key actors within an internal or external system-integration context. Cadences vary as some provider services are stable and mature, while others are developed with the client applications

Service/API Contract

Providers and Consumers communicate with each other using one of many standard protocols; also consumers have a direct dependency on service provider’s “service contract” to know about these protocols and service details

A service contract is a document describing one or more services offered by a service provider and covers details such as protocol, methods, data type and structure etc.

Screen Shot 2020-02-25 at 1.48.43 pm — 2. Consumer, Service provider and Service Contract

A good service contract contains well-documented details of the service as well as “examples” of the request, response and errors. RAML [3] and Swagger/OAS [4] are two of the popular tools used in documenting service contracts

For example, the “Address search” contract by a SaaS vendor below describes the method, URI, query parameters and provides the user with the ability to “try out” the service. This approach allows consumers to iterate faster when developing solutions that use address search without having to engage the SaaS vendor teams (self-service)

Screen Shot 2020-02-25 at 3.49.51 pm — 3. A Service Contract on an API Portal [2]

Service Contract Cadence: Contract Driven Development

Contract cadence is when a service contract is available compared to when the client wants to build their application. There are 2 cadences – mature and in-flight.

For a mature/pre-existing service, a good service contract allows the service consumer to develop a client application without having to engage a service provider person. Meanwhile, for a service being developed with the client application, there is an opportunity for both teams to author the service contract together during the elaboration phase of a project

Screen Shot 2020-02-25 at 4.29.52 pm — 4. Deliver Cadence based on Service Contract Availability

Service contracts are key to consuming a service; when consuming mature services look for descriptive, sandbox-enabled, self-service services to supercharge your delivery. For services being developed in a project (along with the client applications), by internal teams or external vendors, ask for contracts upfront and ask for consumers & service providers to co-author the contracts to remove ambiguities sooner

Avoid generating service contracts from developed components (Java Class to WSDL) as this technique leads to isolated, one-way, ambiguous specifications requiring considerable hand-holding and has the most defects during integration testing (from experience). Use Contract Driven Development [8] which facilitates the writing of a contract by the Service Provider and the Consumer together during Elaboration phase (sign in blood if adventurous)

API Styles: RPC, REST,

Now what we have services, contracts out of the way we can dig deeper into how messages get over the wire in the real-time services. We will ignore “B2B protocols” for this discussion and leave them for the future

The four common real-time service protocols we see all are built over HTTP using JSON or XML content-type and different in their implementation. Below is a short description of each

REST
- Is the most service protocol for front-end applications and modern enterprise APIs. Is stateless, cacheable, uniform, layered etc and came from this [6]
- A resource is a key abstraction in REST and a hard concept for integration-practitioners to master
- Mature REST APIs look like Hypertext, i.e. you can navigate them like you would the web
- Uses HTTP methods e.g. GET, PUT, POST, Patch for query and command
- Uses HTTP status codes for communicating the response e.g. 2xx, 4xx, 5xx
- Open to custom or standard request/response data-types. Standard hypermedia types include HAL, JSON API, JSON-LD, Siren etc. [4]
- Requires resource-centric thinking
GrapQL
Streaming APIs
- HTTP Streaming
- Websockets
- SSE
- HTTP2/Push
- gRPC
gRPC
- over HTTP/2
- Uses protobuf for data type
- https://developers.google.com/protocol-buffers/docs/proto3
SOAP
- Use to be popular before REST came along
- Uses HTTP POST with the body containing request details – method-name, request etc in XML
- Uses XML schemas for describing data types
Json-RPC
- Semi-popular with some legacy clients
- Uses HTTP POST with the body containing request details – method-name, request etc in JSON
- Uses JSON object to define the method and data
OData
- Used in CRM, ERP etc enterprise systems to reduce client-side development through config driven service consumptions
- Presents the end service as a “data source” to the consumer, allowing SQL like query
- Uses schema for describing data source objects and atom/XML for transmitting over the wire
- Requires custom parsers for parsing URL which contains the “query” to be mapped to the back-end service

There is plenty to cover in this space and in a future post, I will compare these protocols further and flavour them from experience. The key takeaway here though is that there are many many ways in which service provides and service consumers can communicate, most choose REST or SOAP over HTTP, there are passionate conversations over REST/SOAP, JSON/XML, HAL/JSON-API/Siren all the while OData remains a mystery to us -until we need to deal with it

EAI Patterns

There are heaps to learn about “how” these service providers and consumers communicate in a networked environment but below is a quick overview of these patterns. These patterns emerge because of the CAP Theorem [7] and the Network partition between these systems looking to exchange data

Screen Shot 2020-02-25 at 5.57.40 pm — 5. Integration Communication Patterns used by Service Providers and Consumers

Recap

Enterprise Integration involves internal, external, batch and real-time services
Key actors are service providers, consumers and mediators
Service contracts are key documents in integrating
The cadence between provider and consumer impacts delivery velocity
Service protocols vary but there are 4-main types: REST, SOAP, JSON-RPC and OData