Case Study

About the Client

A leading Belgium-based company, known for its planning and license management software, is a global leader in providing essential solutions to the broadcasting industry. Their flagship product is integral to the operations of prominent media organizations, including global giants like Warner and Al Jazeera, enabling efficient broadcast planning across numerous channels and managing complex licensing agreements.

As the media industry evolved, driven by rapid technological advancements and changing consumer behaviors, the company identified the need to modernize its platform to maintain its leadership position and continue delivering top-tier service to its clients.

The Challenge

The company’s legacy deployment model, which involved creating dedicated end-to-end stacks for each customer, had become increasingly unsustainable as the business expanded. This approach, while effective in the past, led to high operational costs and inefficiencies.

The primary challenge was scalability. The PostgreSQL-based system that underpinned their analytics and reporting capabilities began to struggle under the strain of ever-growing data volumes. As the data demands of their largest customers intensified, the system encountered performance bottlenecks that threatened to delay the delivery of critical insights, a significant risk in the fast-paced broadcasting industry.

Additionally, there was a growing demand from customers for access to raw data, which they wanted to integrate with their own datasets to extract deeper insights. These customers expected data to be delivered via Snowflake data shares, a modern and secure data-sharing method that bypasses the traditional extract-and-load processes. However, the company’s existing infrastructure did not support this level of data accessibility and flexibility. These challenges highlighted the need for a scalable, flexible, and customer-centric solution.

This modernization wasn’t just a technical necessity; it was a strategic move to enhance service offerings, streamline operations, and reinforce the company’s market position.

Project Objectives

To address these challenges, the company and its partner, Tropos, set out a clear set of objectives for the platform modernization:

1. Cost Efficiency: The goal was to reduce the average cost per customer by aligning expenses with actual resource consumption. This meant adopting a pay-as-you-go model where the company would only incur costs for the resources that its customers actually used, making the business more financially sustainable and enabling cost savings to be passed on to customers.
2. Scalability and Flexibility: The modernization aimed to create a flexible architecture that could easily integrate new tools and technologies as they emerged. This would ensure the platform could scale seamlessly with customer needs, future-proofing it against the rapidly evolving media landscape.
3. Customer-Centric Customization: Empowering field teams to offer customized data analytics solutions tailored to each customer’s specific needs was a priority. By enabling this level of customization, the company could enhance customer satisfaction and loyalty, providing solutions perfectly aligned with each client’s unique operational requirements.
4. Service Reliability: Maintaining or improving high standards for service reliability, particularly regarding Recovery Point Objective (RPO) and Recovery Time Objective (RTO), was crucial. Even as the platform transitioned to a new architecture, it was essential that customers could continue to rely on uninterrupted access to their data and analytics.

The Approach

To meet these objectives, the company adopted a strategic and structured approach, partnering with Tropos, a consultancy with expertise in data platform modernization. Together, they developed a six-pillar strategy to guide the transformation:

1. Snowflake Deployment with Guardrails: The company made a strategic investment in Snowflake, a cloud data platform known for its scalability, security, and high performance. Snowflake was chosen for its ability to manage large volumes of data efficiently while providing the flexibility needed for future growth.
   
   To ensure a smooth transition, the company employed Tropos’ Asana framework, which provided the necessary governance and security measures. This framework enabled the company to deploy Snowflake confidently, with guardrails in place to maintain control over the deployment process.
2. Data Replication via AWS: With Snowflake as the new platform, the company leveraged its existing AWS contract to facilitate the migration of data from the legacy PostgreSQL system. AWS Database Migration Service (DMS) was utilized to replicate data accurately and efficiently.
   
   To expedite this process, the company used Tropos’ internal Data Loading Framework, which was designed to handle large-scale data migrations with minimal disruption. This ensured that the data transfer was completed swiftly, allowing the company to maintain business continuity during the transition.
3. Migration of Legacy Reporting Logic: A critical aspect of the modernization was the migration of existing reporting logic to the new platform. This involved transforming legacy reporting logic into dbt (data build tool) projects using Tropos’ Kali tool. This step was essential for preserving the robust analytics capabilities that the company’s customers had come to rely on while ensuring these capabilities could be supported on the new, more scalable Snowflake platform.
4. Data Transformation and Orchestration: To manage the complex data transformation processes required by the new platform, the company built a comprehensive orchestration and execution environment using dbt. This environment was designed to efficiently handle the transformation and processing of large datasets, ensuring that the new system could deliver insights rapidly and accurately. This capability was vital for supporting the increased data volumes expected as the customer base continued to grow.
5. Change Management: Recognizing the diverse needs of its customers, the company developed a dual approach to change management. This involved setting up rules and processes to manage changes across their entire customer base while still allowing for tailored updates for individual clients. This approach ensured that the company could implement broad platform updates efficiently while providing the flexibility to meet specific customer requirements, maintaining a high level of service customization.
6. Staff Training: Finally, the company invested heavily in training its internal teams to ensure they were fully equipped to manage and maintain the new platform. This training was critical for driving adoption of the new technologies and ensuring the long-term success of the modernization effort. By empowering their staff with the necessary skills and knowledge, the company ensured they could leverage the new platform to its full potential, delivering enhanced value to customers.

Project Results

The platform modernization project delivered substantial benefits across several key areas:

1. Dramatic Performance Improvements: The migration to Snowflake resulted in significant performance gains. Reports that had previously been sluggish on the PostgreSQL-based system now run tens of times faster on Snowflake, greatly improving the user experience. This performance boost has enabled the company’s customers to make faster, more informed decisions, enhancing their ability to respond to real-time events and trends in the broadcasting industry.
2. Reliable Uptime: One of the critical advantages of the new platform is its robust uptime guarantees. Snowflake’s architecture ensures that customers have consistent access to their data and analytics, reducing the risk of downtime that could disrupt operations. This reliability has strengthened customer trust, reinforcing the company’s reputation for delivering dependable, high-quality services.
3. Cost Control: The modernization effort was completed within the projected budget, demonstrating that the company could achieve high performance and scalability without incurring excessive costs. The adoption of a pay-as-you-go model also allowed the company to control costs more effectively, aligning expenses with actual usage and improving overall financial sustainability.
4. Reduced Data Latency: The new system achieved a latency of just 3 seconds between data registrations in operational tools and visibility in reporting. This near real-time access to data has provided the company’s customers with a crucial advantage, allowing them to act quickly on the most current information available. In the broadcasting industry, where timing is everything, this reduction in latency has been a game-changer.

Challenges Encountered during the project

Despite the overall success of the project, several challenges arose that required careful management:

1. Learning Curve for Legacy Tools: The engineering team encountered a steep learning curve due to the need to use a relatively outdated release management and code version management platform to manage and execute dbt code. To address this, the company implemented focused training sessions, which helped the team quickly become proficient in the necessary tools, ensuring that the dbt projects could be managed effectively.
2. Fluctuating Efficiency with AWS DMS: The efficiency of the AWS Database Migration Service varied due to the differing data volumes across the company’s customer base. Some customers had significantly higher data demands, leading to fluctuations in migration efficiency. To mitigate this issue, the company introduced a customer pooling strategy, grouping customers with similar data demands. This approach balanced the load more effectively, ensuring a smoother and more consistent data migration process.
3. Integration with Terraform Refactoring: During the modernization, the company was also undergoing a large-scale refactoring exercise to Terraform, a tool for building and versioning infrastructure. This provided an opportunity to align the new data platform with broader infrastructure changes. By integrating this refactoring into the data loading framework, the company enhanced the platform’s flexibility and ensured seamless integration with the overall system architecture.

Lessons Learned

The project provided several key insights that will inform future initiatives:

1. Early Alignment with AWS Experts: Engaging with AWS network engineers and solutions architects early in the process was crucial. Their expertise helped navigate the complexities of the platform setup and ensured that the infrastructure was optimized from the start.
2. Starting with Boilerplate Code: Using boilerplate code and kickstarters from the beginning allowed the team to establish a strong foundation quickly. As the project evolved and specifications became more complex, the code could be tailored accordingly. This approach provided the flexibility needed to adapt to changing requirements without compromising speed or quality.
3. Promoting Pilot Projects Early: Promoting pilot projects to external stakeholders as early as possible proved valuable. This strategy helped gather critical feedback, generate interest, and refine the product based on real-world input. Early promotion also strengthened relationships with stakeholders, enhancing the overall success of the project.
4. Documenting Datasets in dbt: Embedding documentation of datasets directly within dbt code streamlined internal processes and made it easier to share detailed, well-documented datasets with external customers via Snowflake data marketplaces. This practice not only improved usability but also increased transparency, making it easier for customers to understand and utilize the data provided.

Conclusion

The platform modernization effort was a significant achievement that addressed critical technical challenges while aligning the company’s operations with the latest industry standards. The project resulted in substantial performance improvements, enhanced service reliability, and controlled costs, all while maintaining a customer-centric approach.

These outcomes have not only strengthened the company’s market position but also laid the groundwork for future growth and innovation. The lessons learned from this initiative will guide future efforts, ensuring that the company continues to lead the way in providing cutting-edge solutions for the broadcasting industry.