Systems Integration is key to building good distributed solutions as software modules are spread across network within and outside the organisation. Knowing where we are building a solution, the key requirements for interaction, data, consistency etc can help pick the right integration patterns
Here is my list of what I look for when integrating systems
1. Integration Context
Systems integration used to happen a lot within the enterprise however increasingly we find this happen for digital channels, public data APIs and partner eco-system integrations. Knowing the context can help orient ourselves to the solution, the customers and lead to application of right integration and data security patterns
2. Interactions
Knowing the interaction desired is the next step after context. Is this data synchronisation? Is this real-time customer query? or a one-way form submission? Knowing what we need to do can help us pick the right operation – Command or Query, the right consistency model – real-time or delayed, the right volume – batch or real-time etc
Key interaction patterns
- What do we want to do: Command or Query?
- Do we need an answer now: One-way or Request-Response?
- Do we need the response now or can we tolerate later: Real-time, delayed or eventual consistency?
- How big is the data we want to move or read: Lightweight or Bulk?
3. Transactions
Often solution transactions are quite atomic and simple request/response interfaces for commands work. However for more complex distributed transactions, especially across multiple systems and bounded-contexts we may need to consider either orchestration or choreography to ensure the solution objectives are achieved
Consider for example, a business process that generates a set of documents from templates for a customer, attaches them, emails the customer and maintains a copy of the sent correspondence. Such solutions requires a bunch of steps that must complete before the next can happen and sometimes there is need to rollback all the previous steps if the one of the last steps fail – this is called compensation and the long-running process that orchestrates such a flow is called a SAGA. SAGAs are great for centrally orchestrating calls but they are tied to the endpoints they call and can tie a service directly to other downstream services
On the other hand, using events to coordinate an end-to-end solution through rules in the systems/contexts to handle a business or system events is choreography and is used to keep systems decoupled
4. Constraints
When integrating systems one key factor in building a robust solution is the systems at ends of the solution, what are their constraints? For example, these can range from having a stable, secure interface, to ability to handle large volume/throughput, to data security etc
5. Platform
The platform where we build our solutions has a huge impact the patterns we pick. For example, some clients subscribe to a platform license that is turnkey and comes out-of-box with MFT, database, monitoring, API gateway. In contrast there are other BYO platform options where there is freedom to roll-your-own platform and licensing constraints on what one can use
6. Security
We look at the context and use cases now to determine the level of system and user authentication and authorization we need to factor in. For example, internal context vs public facing API may require different or similar levels of security controls given the ask
7. Exchange and Data format
This where we start looking for protocol translations between systems (HTTP/REST or JSOn-RPC or SOAP), content negotiation (XML or CSV or JSON), model translation (data mapping in request/response)
8. Support
We also need to look at the support requirements which can include logging, message tracking etc. This is where we add tracking IDs, monitor message queue listeners, apply SLA monitoring, create health check, platform endpoints etc.
Some of these are engineering quality related and we are expected to always bake them into our solution, however certain solutions require greater support consideration to guarantee information is reliably and consistently provided
Alok Mishra 