OData vs. REST: The Integration Dance with DataHub

In our earlier post, we delved deep into how data hubs, especially when organized by business domains, are revolutionizing the way we handle data. We also touched upon the challenges faced when integrating with a data hub and how consumers ingest this data. If you missed it, give it a read here. Today, we’re back with another intriguing topic: OData vs. REST. But we’re not just looking at them as protocols; we’re diving into their paradigms and our hands-on experience with them. Buckle up!

Reference architecture for DataHub APIs

Before we look the ways to get the information, let us look at what our Data Hub architecture might look like. Below is a rough outline of how a consumer can query multiple domain APIs

Here is how the underlying implementation would look like

OData and REST

Let us first understand OData and REST

REST (Representational State Transfer) Demystified:

What is REST?
REST isn’t just an API; it’s an architectural style. It leverages the power of HTTP and focuses on the interaction between a client and server. The idea is to represent the system’s resources in a stateless manner, where every interaction is isolated and independent³.

Why REST?
REST is all about simplicity and scalability. It uses standard HTTP methods, making it easy to understand and implement. The stateless nature ensures that each request from a client to a server contains all the information needed to understand and process the request⁴.

Example of REST for a ‘Claim’ API:
GET /api/claims/1234
This RESTful endpoint retrieves the details of a claim with the ID 1234.

What is OData?
OData, short for Open Data Protocol, is a standardized protocol for creating and consuming data APIs. It builds on core protocols like HTTP and commonly accepted methodologies like REST. The magic of OData lies in its ability to simplify data sharing across disparate applications in enterprise environments using the semantics of REST¹.

OData for Data Hubs?
While REST is versatile, it can sometimes be verbose, especially when dealing with complex queries. OData brings a structured way to query and update data, allowing for more complex operations like filtering, sorting, and pagination. It’s like having a conversation with your data store but through APIs. Think of it as SQL, but tailored for large datasets and accessible over the web².

Example of OData for a ‘Claim’ API:
GET /odata/Claims?$filter=PolicyId eq 1234 and Injury ne null
This OData query retrieves all claims with a specific policy ID and where an injury is recorded.

Comparing REST and OData

HTTP/REST:

Protocol Overview:

• Resource-based model with explicit paths for command and query.
• Hyperlinks in relationships allow clients to navigate.
• Can be designed from a specification e.g. OAS 3.0 easily.

Strengths:

• Simple CLI commands can launch pre-templated environments in minutes vs months.
• Resource-based model with explicit paths for command and query.
• Hyperlinks in relationships allow clients to navigate.

Challenges:

• RESTful endpoints need to be implemented for parent and child entities.
• Attribute-based search can be complex to implement on a collection.
• Attribute params based query, filter, sort, etc. needs to be implemented.

Design Skill Set:

• Understanding HTTP/HTTPS protocols.
• Grasping REST concepts at an average level, including endpoints, actions (GET, PUT, POST), schema, response/requests.
• Familiarity with JSON and RAML formats used to design the API endpoints.
• Understanding the data model at a logical level (entities, associations, references, data types).
• Grasping RDBMS concepts like data-types, tables, views, primary-key, foreign keys.

ODATA:

Protocol Overview:

• Provides resources-based services.
• Comes with built-in features like $sort, $filter, $top, etc.

Strengths:

• Implementation is simpler compared to vanilla REST due to built-in features.
• Provides resources-based services.

Challenges:

• Design of the OData service is done by coding the interface and then generating the metadata XML.
• Not suitable for rpc-style endpoints.
• Requires implementing the EDM objects (parent-child relationship).
• If doing specification first, then you need to align the OData model to it.

Design Skill Set:

• Essential XML knowledge.
• Understanding HTTP/HTTPS protocols.
• Grasping ODATA concepts like data-types, references, entity, entity-set, roles, and associations.
• ODATA expressions are beneficial but mostly used in development.
• Understanding the data model at a logical level (entities, associations, references, data types).
• Grasping RDBMS concepts like data-types, tables, views, primary-key, foreign keys.

Comparing by key ilitis

Standardization:

REST:
REST operates on the principle of Representational State Transfer, offering an architectural style that provides developers with the flexibility to design their endpoints and interactions based on specific application needs. Unlike OData, REST does not adhere to a fixed set of conventions, allowing for a more tailored approach. For instance, while one application might use GET /api/claims/1234 to retrieve claim details, another might opt for GET /api/details?claim=1234.

OData:
OData, or the Open Data Protocol, adheres to a strict set of conventions. This standardization ensures a consistent approach to crafting and consuming data APIs, which can be beneficial for applications that prioritize uniformity and predictability. For a ‘Claim’ API, OData would have a standardized endpoint like GET /odata/Claims(1234) to retrieve a specific claim.

Query Capabilities:

REST:
While REST offers a flexible approach to data retrieval, it often requires developers to design and implement custom querying capabilities. This can sometimes feel like reinventing the wheel, especially when trying to implement complex querying logic. For example, filtering or sorting claims based on specific criteria might necessitate custom endpoints or query parameters.

OData:
OData shines in its querying capabilities. It allows developers to delve into metadata and execute intricate, almost SQL-like queries directly from the URL. This can expedite certain processes and reduce the need for custom endpoint design. For instance, with the ‘Claim’ API, OData allows for advanced querying like GET /odata/Claims?$filter=PolicyId eq 1234&$orderby=Date desc.

Interoperability:

REST:
REST’s flexibility means that interoperability can sometimes be a challenge, especially when trying to ensure consistent interactions across different systems. However, certain standards, like the “Relationships” structure in JSON API, can enhance REST’s interoperability, ensuring that different systems can communicate more seamlessly.

OData:
OData’s standardized nature enhances its interoperability, making it akin to a system that remembers and anticipates specific interactions. However, while OData provides a consistent approach, it’s not without its challenges. OData libraries, for instance, can sometimes introduce unexpected behaviors or constraints, which developers need to be aware of and navigate.

Data Accessibility:

OData:
When creating a data hub, a centralized repository where users can seamlessly access, explore, and retrieve data in a comprehensible format, irrespective of the volume of data. OData excels in this realm, offering a standardized protocol that facilitates such interactions. For instance, users can easily filter and sort data using OData’s query capabilities, making data retrieval more intuitive.

REST:
REST provides a flexible architectural style for designing APIs. However, there are moments when its simplicity, while advantageous, might not capture the full depth or complexity of certain data interactions. It’s akin to trying to encapsulate a complex narrative with limited descriptors. While REST is versatile, it might require additional layers or conventions to achieve the same depth of interaction as OData.

Scalability and Performance:

OData:
While OData offers a rich set of querying capabilities, it can become resource-intensive, especially when dealing with complex queries or large datasets. As the intricacy of the queries increases, there might be performance implications. For instance, when implementing features like pagination on vast datasets, OData might experience performance challenges.

REST:
REST, with its adaptable nature, often presents a more lightweight approach, especially when dealing with straightforward data interactions.

What about Flexibility and Security?

From a consumer perspective lets look at security and flexibility for both, from our experience

REST:
REST operates on the principle of Representational State Transfer, offering an architectural style that is akin to a blank canvas. This means developers have the autonomy to design their endpoints and interactions based on the specific needs of their application. For instance, in the context of a ‘Claim’ API, a RESTful design might have separate endpoints for each action:

• GET /api/claims/1234 to retrieve details of a claim.
• POST /api/claims to create a new claim.
• PUT /api/claims/1234 to update a specific claim.

OData:
OData, or the Open Data Protocol, offers a more standardized approach. It can be likened to a paint-by-numbers kit, providing a structured methodology for crafting and consuming data APIs. While this structure can expedite certain processes, it might also introduce constraints, especially when dealing with complex data structures. For instance, with the ‘Claim’ API, OData allows for more intricate querying directly from the URL:

• GET /odata/Claims?$filter=PolicyId eq 1234 to filter claims by a specific policy ID.
• GET /odata/Claims?$orderby=Date desc to order claims by date in descending order.

However, this structured approach might feel like trying to fit puzzle pieces from different sets when dealing with intricate data structures or custom requirements.

Security:

When discussing data transmission and accessibility, security remains paramount. It’s not merely about protecting data but also about ensuring that only authorized entities have access. We spent considerable amount of time struggling with custom OData libraries and poor documentation when working out how to implement row-level security – I will discuss this in a future post. Let us look at security implications for both

REST:
REST relies heavily on the underlying protocol, typically HTTP or HTTPS, for security. Standard methods like SSL/TLS encryption can be employed to secure data in transit. Additionally, authentication and authorization mechanisms, such as OAuth or JWT, can be integrated to determine who can access the ‘Claim’ API and what actions they can perform.

OData:
OData, being built upon REST principles, also leverages the security features of HTTP/HTTPS. However, due to its standardized nature, it might require additional considerations, especially when exposing complex queries. For instance, if not properly configured, an OData service might inadvertently expose sensitive data through its querying capabilities. As with the ‘Claim’ API, developers need to ensure that filters or other query options do not expose data that should remain confidential.

Conclusion

Our decision was to use OData for DataHub and in retrospect this choice was good as it allowed our consumers flexibility in requesting information. However with had to overcome some key technical challenges with data authorization, pagination and complex queries. For example, build a large CSV for data transfer was quite complex using OData Query via a single $expand call and perhaps we were better off providing collections & compound documents like JSON API to let the consumers iterate over a collection and travel deep into the graph.

Keen to hear your experience and in our subsequent post we will explore data authorization for data hubs, event subscription, command and real-time queries with direct routes to the core systems (like a no-cache option)

References

1. Microsoft Docs – OData Overview ↩
2. OData Official Website – Why OData? ↩
3. Microsoft Azure – REST API Design ↩
4. Mozilla Developer Network – REST ↩