Challenge
In-app, real-time analytics on large and complex data sets.
The client builds data analytics applications tailored to the energy management industry. For one of the business cases, end clients can upload internal corporate data to the application. The application applies proprietary algorithms and offers comprehensible insights to the analyst.
As the business case is rapidly evolving, the app builder has a requirement to iterate on new insights quickly. During the first phases of application development, custom visualizations have been created on top of a highly customized API interface. A cost-effective data visualization component is key to the platform to improve time-to-market cycles.
The application is commercialized as a consulting service, with paying customers being onboarded in large numbers. The client wants to offer a premium user experience. Thus, scalability and minimal data processing times are of primary importance. An intelligent but critical redesign of the architecture and upgrade of the application was requested.
Approach
Powerful data analytics platform combined with cost-effective embedded analytics
The assignment consists of multiple challenges to address at the same time:
- Ensuring new data processes within SLA limits;
- Ensuring that the app builder can develop new insights as fast as possible;
- Embedding data visualizations in a seamless way to business users;
The application was deployed on Amazon Web Services (AWS). After a short solutions architecture exercise, our team proposed to change the data backend from a row-based MySQL database to a column-based database.
Snowflake was chosen out of a shortlisted number of candidate technologies because of:
- Unexpected growth patterns at the clients’ application;
- Spikey usage patterns throughout the month, as analysts usually use the application at the beginning and in the middle of the month;
- The ability to quickly ingest new data and apply proprietary algorithms;
As for front-end technology, the client decided to step away from earlier efforts to develop a conventional VueJs-based data visualization dashboard, as the time to iterate over new ideas was too long. Rather instead, they chose to leverage their existing AWS contract and work with AWS Quicksight:
- Competent data visualization technology that easily competes with dashboarding brands such as PowerBI and Tableau;
- A serverless technology that bills based on activated user sessions rather than conventional core-based pricing;
- Graphs, charts, and tables are easily embeddable in existing applications to minimize the impact of new software rollouts on the user base.
AWS Quicksight was a convenient and pragmatic tool selection for the product owner, as its embedding capabilities made it a quick upgrade for the current, in-app vue.js visualizations.
The quick architecture summary above highlights critical components in the setup of the analytical application. It has been designed from the ground up to scale based on variable user demand.
Results
Improved user experience, the optimized total cost of ownership, and faster development of new insights
After 4 development sprints, our team re-engineered the application’s backend, integrated AWS Quicksight into the existing application. As most of the data ingestion framework had already been built on AWS, our team could easily leverage the wide array of data loading capabilities available in Snowflake.
Thanks to the authoring module of AWS Quicksight, business analysts, and salespeople, we’re able to quickly iterate over new insights and operationalize them without the involvement of application developers. This was a major breakthrough for our client as they wanted to improve time-to-market.
Furthermore, we occasionally coach the clients’ development team to make the best use of Snowflake and Quicksight.
