System Integration 101: Patterns and Best-Practices

The Importance of Integration Patterns

In the intricate world of software development, the ability to efficiently connect and integrate different systems is vital. Integration patterns provide a roadmap for this process, offering standardized solutions to common integration challenges. This post delves into the essence of integration patterns, shedding light on their importance and how they facilitate seamless interactions between disparate systems in an ever-more connected digital landscape.

Decoding the Essentials of Integration Patterns

Integration patterns are architectural solutions that address specific problems in system integration. These patterns offer standardized methods to manage data exchange, consistency, and process orchestration across varied systems.

1. Message Broker Pattern: Centralizes message processing, allowing different systems to communicate through a reliable intermediary. This pattern is key in scenarios where direct system-to-system communication is impractical or creates tight coupling.

2. Publish-Subscribe Pattern: A publisher sends messages to a channel without knowing who the subscribers will be. Subscribers listen for messages of interest without knowing who the publisher is. This pattern is crucial for building scalable, event-driven architectures.

3. Request-Reply Pattern: Ensures synchronous communication where a requester sends a message and waits for a reply. This pattern is common in service-oriented architectures where a response from a service is essential for the continuation of a process.

Implementing Integration Patterns: Real-World Scenarios

The practical application of integration patterns is vast and diverse, addressing various integration needs across industries.

1. Message Broker in E-Commerce: An e-commerce platform uses a message broker to integrate its order management system with inventory and shipping services, ensuring smooth order processing and fulfillment.

2. Publish-Subscribe in IoT: In an IoT ecosystem, sensors publish data to a central system. Various services subscribe to this data for monitoring, analytics, or triggering other processes, thereby enabling a responsive and intelligent IoT network.

3. Request-Reply in Financial Services: A banking application uses the request-reply pattern to communicate with credit check services, where immediate responses are crucial for loan approval processes.

Conclusion: The Pivotal Role of Integration Patterns in System Design

Integration patterns, standing at the crossroads of system design and functionality, are more than just solutions—they are the cornerstone of modern system integration. Much like design patterns that address common challenges in software development, integration patterns are specialized design patterns tailored for the intricate realm of system integration. They are instrumental in enabling disparate systems to communicate effectively, ensuring data flows seamlessly across different platforms and applications.

These patterns are fundamental not only because they solve recurring problems but also because they encapsulate best practices refined over years of technological evolution. They represent distilled wisdom for connecting and orchestrating complex systems, making them an indispensable part of any system architect’s toolkit.

As we continue to forge ahead in an era marked by technological convergence, the importance of these patterns only magnifies. They are not just tools for integration but also catalysts for creating cohesive, efficient, and scalable systems. For developers and system architects, delving into integration patterns is not just an exploration of technical strategies; it’s an essential step towards mastering the art of building interconnected systems that are robust, flexible, and future-proof.

List of Patterns

IDNameFlowWhen to use
Synchronous reads
1.1Consistent Read (master)
Client wants to read the latest dataRequest volume is  low (throughput)Request latency can be high (slow response is tolerated) Request size is small (fine-grained items)
1.2Available Read
(operational data store)		Client is okay to read slightly old data or the data is fairly static within the refresh periodRequest volume is high (throughput)Request latency must be low (slow response is not tolerated) Request size is small (fine-grained items)
1.3Bulk-Read
Client is okay to read slightly old data or the data is fairly static within the refresh periodRequest volume is high (throughput)Request latency must be low (slow response is not tolerated) Request size is small (fine-grained items) but client wants to get all or a large subset of the data using their query
2. Asynchronous reads
2.1Bulk Read – Consumer Adapter Request
Consumer Adapter Polling		Scheduled bulk download is needed Consumer adapter cannot receive events and is okay to poll for new fileRequest size is Large and paginated reads are time-consuming or slow
2.2Bulk Read – Consumer Adapter Request
Provider Events		Scheduled bulk download is needed Consumer adapter can receive events (internal app or webhook)Request size is Large and paginated reads are time-consuming or slow
2.3Bulk Read – Consumer System Request
Consumer Adapter Polling  		On-demand client system user-driven request to get bulk downloadConsumer adapter can receive events (internal app or webhook)Request size is Large and paginated reads are time-consuming or slowThere is a staging area (file or database) is available to push to (e.g SFTP or Salesforce bulk publish)
2.4Bulk Read – Consumer System Request
Provider Events		On-demand client system user-driven request to get bulk downloadConsumer adapter cannot receive events (external app with no webhook)Request size is Large and paginated reads are time-consuming or slowThere is a staging area (file or database) is available to push to (e.g SFTP or Salesforce bulk publish)
3. Synchronous writes
3.1Consistent Writes to master Master system allows changing data directlyYou need to write one entity or a parent-child entity in a single transaction to the backendYou need to receive errors in create / update and can handle them You need an acknowledgement with ID from the backend to store and map
3.2Consistent Write Requests to master (202)Master system does not allow changing data directly – must be via a change request & workflow in backendYou need to receive errors in create / update and can handle them You need an acknowledgement with ID from the backend to store and map
4. Asynchronous writes
4.1Buffered WritesLarge volume of inbound write requests (e.g. Bulk uploads)Requests are one-way, i.e. client does not require a ID of the written or updated dataWrite contain Updates (data change) or Commands (jobs)Master system cannot handle high-volume of write requests or needs to be protected from internal or external clients
4.2Shopping-cart Writes to master can be processed in batch or async mode through existing processShopping-cart or Draft request must be captured near the consumer system and only when the full request is available we can publish to master Rules at master to publish a full-record not partial items
4.3Dual-copy SyncA shallow copy of the data needs to be captured in a highly available system initially This copy is okay to be later synchronised in small sets or bulk to the backend via batch synchronization processes 
5. Events
5.1External InboundExternal system publishes an event which needs to be published to internal topic for consumers
5.2External OutboundExternal system subscribes to internal events 
5.3Internal InboundInternal systems publishes an event which is broadcast on a topic

Reads

2.1. Real-time Consistent Read

DescriptionReads via synchronous endpoints that provide real-time consistent data from the master of the data
When to use?When the client wants the latest copy of the data (wants “C” in “C.A.P”) and can handle issues with API with partition ( i.e. gives up availability in C.A.P)
FlowPlantUML Expand source
Who can consume this?
What does it not provide?
API endpoints
UsageRequestcurl GET \
    --header "Content-Type: application/json" \
    --header "cache-control: no-cache" \
    http://my/api/urlResponse- Response Headers:
 
x-read-from-cache: false
last-updated-cache-dttm:  null
time-since-last-update: 0

2.2. Real-time Cached Read

DescriptionReads via synchronous endpoints that provide cached data from master of the data
Flow PlantUM Expand source
UsageRequestcurl GET \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

2.3. Real-time Cached Bulk-Read

DescriptionReads via synchronous endpoints that provide cached data from master of the data
Flow PlantUM Expand source
UsageRequestcurl GET \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

2.4.  Asynchronous Bulk Read

DescriptionReads via asynchronous mechanism 
Flow
Sub-patternsUseCases
1Asynchronous Bulk Read – Consumer Adapter Request, Consumer Adapter PollingPeriodic bulk file download using polling to locate file and send2Asynchronous Bulk Read – Consumer Adapter Request, Provider EventsPeriodic bulk file download using events for notification



3Asynchronous Bulk Read – Consumer System Request, Consumer Adapter PollingClient requested bulk download using events for notification.Client request is async and API consumer can publish to Client shared location4Asynchronous Bulk Read – Consumer System Request, Provider EventsClient requested bulk download using polling.Client request is async and API consumer can publish to Client shared location
 Expand source
What is this good for?When looking to get data via event-driven or bulk mechanisms 
Who can consume this?Clients that can invoke reads and handle asynchronous events related to a read  
What does it not provide?No bulk or large documents. For a large set the client needs to query the collection endpoint and iterateComplex queries only available in OData enabled endpoints with $expand directive  
API endpointsGet collectionSearch collectionGet by ID

 Writes 

3.1.   Real-time  Consistency Synchronous Write to master 

DescriptionWrites via synchronous endpoints that accept request, write to master system and acknowledge after database is consistent
FlowPattern SummaryHandling Parent-Child entitiesPlantUM Expand source
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

3.2. Real-time  Consistency Asynchronous Write to Master

DescriptionWrites via asynchronous endpoints that accept request, acknowledge before backend database is consistent and eventually process
Flow
UsageRequestcurl GET \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

3.3. Delayed Consistency Asynchronous Write to Master

3.3.1. Buffered write

DescriptionBuffered writes to enable high-volume to commands with delayed writes to master – queue or streams as write logs 
Flow
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

3.3.2. Shopping-cart / Draft copy

DescriptionWrites to a local database which returns temporary ID and submits to backend when client is done with changes
Flow
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

3.3.3. High Available Shallow Copy

DescriptionWrites to a local database which returns temporary ID and eventually synchronises with backend 
Flow
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://my/api/urlResponse
- Response Headers:x-read-from-cache: true
last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU
time-since-last-update: 12h

Events 

External Publish

DescriptionWhen External System events must be captured for internal systems
Flow
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://internal.webhookResponse
 

External Subscribe

DescriptionWhen External System events must be captured for internal systems
Flow
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://internal.webhookResponse
 

Internal Pub/Sub

DescriptionWhen External System events must be captured for internal systems
Flow
UsageRequestcurl POST \
    --header "Content-Type: application/json" \
    http://internal.webhookResponse
 
Unknown's avatar

Published by alokmishra

Alok brings experience in engineering and architecting distributed software systems from over 20 years across industry and consulting. His posts focus on Systems Integration, API design, Microservices and Event driven systems, Modern Enterprise Architecture and other related topics

Leave a Reply