Project Overview
The eHealth platform was developed in response to the urgent need for a centralized data system during the COVID-19 pandemic. Designed to manage data compliance with GDPR, the platform allowed stakeholders to access real-time data securely, enabling a coordinated response across the Flemish and Brussels regions.
As we architected the platform, we adopted principles similar to those found in data mesh architecture, focusing on decentralized data ownership and flexibility in managing various sources. By allowing each stakeholder—policymakers, virologists, researchers, and law enforcement—to access data relevant to their domain, the platform facilitated cross-functional collaboration without compromising data security or integrity.
Challenges and Objectives
We faced several key challenges:
- Data Complexity and Evolving Needs: The constantly changing understanding of COVID-19 demanded a system capable of handling numerous data sources and adapting to new requirements on-the-fly. This aligns with a data mesh approach, where distributed data sources must be integrated without bottlenecks.
- Outdated Technology: Traditional data architectures used within government systems weren’t built for the scale or flexibility needed to handle COVID-19. Moving to a cloud-first, data mesh-inspired model allowed us to manage multiple data domains efficiently, ensuring rapid integration of hospitals, vaccination centers, schools, and more.
The primary objective of the platform was to create a single, consistent source of up-to-date data that would drive decisions across all levels of governance, ensuring that policy formation and execution remained data-driven and coordinated.
Approach and Strategy
We chose to implement a cloud-first approach, leveraging Snowflake as the core data platform to scale and handle various sources securely. By drawing on the data mesh paradigm, we ensured that data was accessible at a domain level, allowing various stakeholders—healthcare providers, law enforcement, and researchers—to work with their data while maintaining consistency across the platform.
Using dbt (data build tool) to automate data transformation and governance, we ensured a 360° view of the pandemic. This mirrored the self-serve data infrastructure seen in data mesh strategies, where different domains (hospitals, vaccination centers, etc.) could be onboarded quickly, adapting to their unique data needs.
Solution and Implementation
Key features included:
- Domain-oriented data access: Similar to the data mesh model, strict access controls ensured that only authorized users could access data, based on their roles and legal requirements. This was crucial for GDPR compliance.
- Real-time Dashboards and the Control Tower: The Control Tower application operated as the central node for data access, presenting key stakeholders with the right information at the right time. This decentralized access model aligns with the data mesh philosophy, where domain-specific insights were available without overburdening the platform’s central infrastructure.
The platform was rolled out in phases, onboarding stakeholders as their roles became clear. This approach ensured that the system could grow in line with the crisis, thanks to its cloud-first, domain-oriented architecture.
Results and Impact
The platform achieved several important outcomes, addressing the challenges it was designed to solve:
- Consistent, reliable data: Daily public updates were made possible through the platform’s centralized yet flexible architecture.
- Uniform Policy Execution: By ensuring that the Flemish and Brussels regions operated under consistent data-driven policies, the platform helped eliminate disparities between regions, improving overall effectiveness.
- Rapid Regional Onboarding: A significant success was the rapid onboarding of the Brussels region, which was completed in just three days by leveraging the existing infrastructure built for Flanders. This showcased the platform’s scalability and adaptability to new regions and stakeholders.
These outcomes helped ensure that policy decisions were consistent, timely, and data-driven, a key requirement identified at the project’s outset.
Challenges Faced and How You Overcame Them
Despite its successes, the project faced several challenges:
- Legal and Privacy Concerns: The platform had to ensure GDPR compliance and proper handling of sensitive healthcare data. This was managed through continuous alignment with the DPO and the privacy regulator.
- Data Quality: With so many data sources involved, maintaining consistent data quality was a significant challenge. We addressed this through automated data quality controls and outsourced complex business rules to ensure accuracy.
- Information Security: As the platform expanded to more stakeholders, maintaining security was critical. We worked closely with vendors to ensure compliance with information security protocols.
- Scaling and Data Complexity: The cloud-first architecture, similar to a data mesh setup, allowed us to scale quickly, integrating new data domains seamlessly as the pandemic unfolded.
By overcoming these challenges, the platform remained secure, compliant, and scalable, directly addressing the initial problems identified.
Lessons Learned and Future Steps
Key lessons learned from the project include:
- Prioritize processes: Architecting the platform with a focus on key processes first ensured we automated the right elements and maintained flexibility in adapting to new demands.
- Enforce standards: Ensuring unified data and governance standards across disparate groups enabled smooth collaboration between different sectors.
- Buy-before-build: Leveraging existing technologies like Snowflake and dbt allowed for rapid deployment and scalability, avoiding the inefficiencies of building custom solutions from scratch.
The platform has been decommissioned at the operational level but remains in use for analytical purposes. It is flexible enough to be redeployed should a future health crisis arise, reflecting the modularity and flexibility found in data mesh strategies.
Outcome: This project left the government better prepared for future public health emergencies, with a scalable and adaptable data infrastructure in place.
