When designing cloud-native data platforms and the APIs that interconnect them, software engineering architects face a wide range of challenges and considerations. From managing continual change to balancing standardisation with domain independence, ensuring data consistency and addressing scalability concerns, the task is daunting – but critical. In his role as CTO at metadata-driven API platform company Hasura, Ken Stott has more insights than most into building the security, governance and discoverability factors within these environments.
Looking at experiences drawn from his own customer base, Stott says that one of the most significant challenges architects face is managing change. Enterprise software and data systems typically last 5-10 years (okay yes, many legacy systems last longer, but there’s a wider point to be made here), during which everything around them evolves i.e. data sources, environments, client expectations and constraints. Architects must design platforms that adapt to these changes while maintaining functionality and performance.
“To manage this change, system architects should focus on predicting and prioritising areas of likely change. Moving [data] definitions into metadata where possible can provide flexibility and building utilities that operate against metadata will improve resiliency as features and requirements evolve. Regardless, professional judgment is paramount. The goal is to simplify, not complicate,” explained Stott.
Standardisation vs. domain independence
The Hasura tech leader says that architects must strike a delicate balance between standardising APIs for consistency and allowing domain teams the independence they need to optimise their data products. To achieve this balance, he reminds us that it’s important to establish core standards for interoperability while providing centralised services that domains can use.
“Allowing domains to optimise within defined guardrails ensures both consistency and flexibility. The focus should be on creating reusable ‘Lego-like’ API components that can be composited to meet different needs. This might include platform-agnostic data schemas (or contracts) and consistent predicates for filtering, sorting, pagination, selection, composition, aggregation and mutations,” clarified Stott. “Ensuring data consistency across multiple APIs and maintaining user trust is key to platform success. Architects should implement a unified semantic layer, expressed in business terms, that provides a common understanding of data across the organisation, which will help develop standardised data products and ensure consistency in data representation and interpretation.”
The argument here gravitates around the need to provide data composition and aggregation services that allow for flexible data manipulation while maintaining consistency. Such a service allows data consumers to generate bespoke data sets to address their narrow use cases while freeing data producers to focus on creating high-quality data products. This helps establish clear data contracts between domains and platforms and helps define expectations and responsibilities.
Scalability & performance
“Cloud-native platforms must be designed to handle varying loads and perform efficiently across different environments,” said Stott. “Scalability and performance concerns can be mitigated by designing for distributed data serving, allowing data to be processed and accessed across multiple nodes. Implementing efficient query planning and execution mechanisms helps optimise performance while pushing operations to the most efficient data platform, which can reduce latency and improve overall efficiency. It’s also important to consider cross-domain analytics requirements to ensure the platform can handle complex, multi-domain queries effectively.”
Protecting data and ensuring regulatory compliance is – obviously – crucial in API platform design (for any API, but especially when those APIs are accessing mission-critical data resources and repositories), so to achieve security and governance, architects must implement authentication and authorisation mechanisms that protect data at various levels. The Hasura team say that allowing domain owners to specify security requirements for their data products ensures that sensitive information is properly safeguarded, and providing auditing and monitoring capabilities helps track data access and usage patterns.
Again, clearly, conscientious work to design data platform access with regulatory compliance in mind from the outset ensures it will meet legal and industry-specific requirements.
Discoverability & self-service
“Enabling users to easily discover and utilise APIs is critical for platform adoption and efficiency. Architects can improve discoverability by developing a comprehensive data catalogue that provides detailed information about available APIs and data products,” clarified Stott.
This is where he says that a strong metadata strategy pays off – spoiler alert, the metadata-driven data specialist might well be expected to say this.
He goes further though and suggests that a metadata strategy can establish the data lingua franca for the organisation, eliminate layers of unnecessary transformation and translation and can power a data catalogue Implementing a user-friendly API marketplace allows consumers to browse and select the APIs they need.
“Adding the ability to include access requests in a data catalogue or data marketplace accelerates the adoption of the platform and propagates your data language. Clear documentation and examples help users understand how to effectively use the APIs. Offering tools for data exploration and composition enables users to combine and manipulate data to meet their specific needs,” said Stott.
“Establishing mechanisms to gather usage data and user feedback is essential for platform evolution, but architects struggle to truly facilitate continuous improvement. Adding analytics to track API usage and performance will provide insights into how the platform is used and where improvements can be made. Additionally, creating channels for user feedback and feature requests ensures that the platform evolves as needs change. Driving the common metadata into every log and trace simplifies support and root cause analysis. Establishing processes for regular platform reviews and updates helps maintain the platform’s relevance and efficacy. Finally, using metadata to drive operational insights can empower teams to optimise the platform’s performance and functionality,” he added.
Adaptability, consistency & user-centricity
As we navigate the complexities of cloud-native API platforms, it seems clear that success lies in striking a delicate balance between flexibility and standardisation, security and accessibility, performance and scalability.
By addressing the key concerns outlined above and implementing the suggested strategies, the advice from Hasura is that architects can create adaptable platforms that meet current needs and evolve with changing requirements. Stott’s closing sentiment is simple – the future of API design for data access is here… and it’s built on the pillars of adaptability, consistency and user-centricity.