System Integration 101: Patterns and Best-Practices

The Importance of Integration Patterns

In the intricate world of software development, the ability to efficiently connect and integrate different systems is vital. Integration patterns provide a roadmap for this process, offering standardized solutions to common integration challenges. This post delves into the essence of integration patterns, shedding light on their importance and how they facilitate seamless interactions between disparate systems in an ever-more connected digital landscape.

Decoding the Essentials of Integration Patterns

Integration patterns are architectural solutions that address specific problems in system integration. These patterns offer standardized methods to manage data exchange, consistency, and process orchestration across varied systems.

1. Message Broker Pattern: Centralizes message processing, allowing different systems to communicate through a reliable intermediary. This pattern is key in scenarios where direct system-to-system communication is impractical or creates tight coupling.

2. Publish-Subscribe Pattern: A publisher sends messages to a channel without knowing who the subscribers will be. Subscribers listen for messages of interest without knowing who the publisher is. This pattern is crucial for building scalable, event-driven architectures.

3. Request-Reply Pattern: Ensures synchronous communication where a requester sends a message and waits for a reply. This pattern is common in service-oriented architectures where a response from a service is essential for the continuation of a process.

Implementing Integration Patterns: Real-World Scenarios

The practical application of integration patterns is vast and diverse, addressing various integration needs across industries.

1. Message Broker in E-Commerce: An e-commerce platform uses a message broker to integrate its order management system with inventory and shipping services, ensuring smooth order processing and fulfillment.

2. Publish-Subscribe in IoT: In an IoT ecosystem, sensors publish data to a central system. Various services subscribe to this data for monitoring, analytics, or triggering other processes, thereby enabling a responsive and intelligent IoT network.

3. Request-Reply in Financial Services: A banking application uses the request-reply pattern to communicate with credit check services, where immediate responses are crucial for loan approval processes.

Conclusion: The Pivotal Role of Integration Patterns in System Design

Integration patterns, standing at the crossroads of system design and functionality, are more than just solutions—they are the cornerstone of modern system integration. Much like design patterns that address common challenges in software development, integration patterns are specialized design patterns tailored for the intricate realm of system integration. They are instrumental in enabling disparate systems to communicate effectively, ensuring data flows seamlessly across different platforms and applications.

These patterns are fundamental not only because they solve recurring problems but also because they encapsulate best practices refined over years of technological evolution. They represent distilled wisdom for connecting and orchestrating complex systems, making them an indispensable part of any system architect’s toolkit.

As we continue to forge ahead in an era marked by technological convergence, the importance of these patterns only magnifies. They are not just tools for integration but also catalysts for creating cohesive, efficient, and scalable systems. For developers and system architects, delving into integration patterns is not just an exploration of technical strategies; it’s an essential step towards mastering the art of building interconnected systems that are robust, flexible, and future-proof.

List of Patterns

ID	Name	Flow	When to use
Synchronous reads
1.1	Consistent Read (master)		Client wants to read the latest dataRequest volume is  low (throughput)Request latency can be high (slow response is tolerated) Request size is small (fine-grained items)
1.2	Available Read (operational data store)		Client is okay to read slightly old data or the data is fairly static within the refresh periodRequest volume is high (throughput)Request latency must be low (slow response is not tolerated) Request size is small (fine-grained items)
1.3	Bulk-Read		Client is okay to read slightly old data or the data is fairly static within the refresh periodRequest volume is high (throughput)Request latency must be low (slow response is not tolerated) Request size is small (fine-grained items) but client wants to get all or a large subset of the data using their query
2. Asynchronous reads
2.1	Bulk Read – Consumer Adapter Request Consumer Adapter Polling		Scheduled bulk download is needed Consumer adapter cannot receive events and is okay to poll for new fileRequest size is Large and paginated reads are time-consuming or slow
2.2	Bulk Read – Consumer Adapter Request Provider Events		Scheduled bulk download is needed Consumer adapter can receive events (internal app or webhook)Request size is Large and paginated reads are time-consuming or slow
2.3	Bulk Read – Consumer System Request Consumer Adapter Polling		On-demand client system user-driven request to get bulk downloadConsumer adapter can receive events (internal app or webhook)Request size is Large and paginated reads are time-consuming or slowThere is a staging area (file or database) is available to push to (e.g SFTP or Salesforce bulk publish)
2.4	Bulk Read – Consumer System Request Provider Events		On-demand client system user-driven request to get bulk downloadConsumer adapter cannot receive events (external app with no webhook)Request size is Large and paginated reads are time-consuming or slowThere is a staging area (file or database) is available to push to (e.g SFTP or Salesforce bulk publish)
3. Synchronous writes
3.1	Consistent Writes to master		Master system allows changing data directlyYou need to write one entity or a parent-child entity in a single transaction to the backendYou need to receive errors in create / update and can handle them You need an acknowledgement with ID from the backend to store and map
3.2	Consistent Write Requests to master (202)		Master system does not allow changing data directly – must be via a change request & workflow in backendYou need to receive errors in create / update and can handle them You need an acknowledgement with ID from the backend to store and map
4. Asynchronous writes
4.1	Buffered Writes		Large volume of inbound write requests (e.g. Bulk uploads)Requests are one-way, i.e. client does not require a ID of the written or updated dataWrite contain Updates (data change) or Commands (jobs)Master system cannot handle high-volume of write requests or needs to be protected from internal or external clients
4.2	Shopping-cart		Writes to master can be processed in batch or async mode through existing processShopping-cart or Draft request must be captured near the consumer system and only when the full request is available we can publish to master Rules at master to publish a full-record not partial items
4.3	Dual-copy Sync		A shallow copy of the data needs to be captured in a highly available system initially This copy is okay to be later synchronised in small sets or bulk to the backend via batch synchronization processes
5. Events
5.1	External Inbound		External system publishes an event which needs to be published to internal topic for consumers
5.2	External Outbound		External system subscribes to internal events
5.3	Internal Inbound		Internal systems publishes an event which is broadcast on a topic

Reads

2.1. Real-time Consistent Read

Description	Reads via synchronous endpoints that provide real-time consistent data from the master of the data
When to use?	When the client wants the latest copy of the data (wants “C” in “C.A.P”) and can handle issues with API with partition ( i.e. gives up availability in C.A.P)
Flow	PlantUML Expand source
Who can consume this?
What does it not provide?
API endpoints
Usage	Requestcurl GET \     --header "Content-Type: application/json" \     --header "cache-control: no-cache" \     http://my/api/urlResponse- Response Headers:   x-read-from-cache: false last-updated-cache-dttm:  null time-since-last-update: 0

2.2. Real-time Cached Read

Description	Reads via synchronous endpoints that provide cached data from master of the data
Flow	PlantUM Expand source
Usage	Requestcurl GET \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

2.3. Real-time Cached Bulk-Read

Description	Reads via synchronous endpoints that provide cached data from master of the data
Flow	PlantUM Expand source
Usage	Requestcurl GET \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

2.4.  Asynchronous Bulk Read

Description	Reads via asynchronous mechanism
Flow	Sub-patternsUseCases 1Asynchronous Bulk Read – Consumer Adapter Request, Consumer Adapter PollingPeriodic bulk file download using polling to locate file and send2Asynchronous Bulk Read – Consumer Adapter Request, Provider EventsPeriodic bulk file download using events for notification 3Asynchronous Bulk Read – Consumer System Request, Consumer Adapter PollingClient requested bulk download using events for notification.Client request is async and API consumer can publish to Client shared location4Asynchronous Bulk Read – Consumer System Request, Provider EventsClient requested bulk download using polling.Client request is async and API consumer can publish to Client shared location  Expand source
What is this good for?	When looking to get data via event-driven or bulk mechanisms
Who can consume this?	Clients that can invoke reads and handle asynchronous events related to a read
What does it not provide?	No bulk or large documents. For a large set the client needs to query the collection endpoint and iterateComplex queries only available in OData enabled endpoints with $expand directive
API endpoints	Get collectionSearch collectionGet by ID

 Writes 

3.1.   Real-time  Consistency Synchronous Write to master 

Description	Writes via synchronous endpoints that accept request, write to master system and acknowledge after database is consistent
Flow	Pattern SummaryHandling Parent-Child entitiesPlantUM Expand source
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

3.2. Real-time  Consistency Asynchronous Write to Master

Description	Writes via asynchronous endpoints that accept request, acknowledge before backend database is consistent and eventually process
Flow
Usage	Requestcurl GET \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

3.3. Delayed Consistency Asynchronous Write to Master

3.3.1. Buffered write

Description	Buffered writes to enable high-volume to commands with delayed writes to master – queue or streams as write logs
Flow
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

3.3.2. Shopping-cart / Draft copy

Description	Writes to a local database which returns temporary ID and submits to backend when client is done with changes
Flow
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

3.3.3. High Available Shallow Copy

Description	Writes to a local database which returns temporary ID and eventually synchronises with backend
Flow
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://my/api/urlResponse - Response Headers:x-read-from-cache: true last-updated-cache-dttm: mm-dd-yyyyThh:mm:ssU time-since-last-update: 12h

Events 

External Publish

Description	When External System events must be captured for internal systems
Flow
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://internal.webhookResponse

External Subscribe

Description	When External System events must be captured for internal systems
Flow
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://internal.webhookResponse

Internal Pub/Sub

Description	When External System events must be captured for internal systems
Flow
Usage	Requestcurl POST \     --header "Content-Type: application/json" \     http://internal.webhookResponse