Nowadays, most organizations have programs to become more data-driven. Primarily, this is an organizational change effort in which many of the organizations we work with are either in the process, have gone through it, or are about to embark on it.  

One of the significant obstacles that need to be overcome is the time-to-market for new business insights. IT-centric business intelligence teams have often struggled to deliver insights in time. In contrast, business-centric data teams have been challenged to realize economies of scale and govern large data projects without IT.

Thanks to public clouds, organizations can now design an operating model between business and IT that allows for a healthy combination of both. Now, acceptable time-to-market for new insights, steady progress, and sustainable inflow of data product specifications based on actual business demand can be guaranteed.

Our key insight? An end-to-end vision on delivering a data platform with the capabilities to manage, analyze, and secure data at scale. An end-to-end vision should align collaboration processes, people’s roles, and responsibilities with data technology.

Business teams are in the driver’s seat

Looking at the previous trends of organizing business intelligence within organizations, we’ve noticed the trend of linking data-driven change with tool selection. We don’t feel this is the right way to go, based on our day-to-day jobs:

• Technology is an enabler, not a goal in itself. 
• Business and process knowledge are embedded in business users’ ways of working, not in data engineering practices.

On the other hand, we can’t cut the IT practices out of the equation because:

• One can’t expect businesses to both design and operate automation processes. They simply don’t have the time to govern such tasks.
• Getting secure access to data within service level agreements is closely linked to core IT operations, such as networking and identity management. 
• Working on the cloud and using data creates a particularly interesting field for T-shaped professionals who span teams and practices, resulting in efficiency boosts for business teams.

Build a co-creation model that works

Assuming that our previous assumptions regarding data product building are correct – which have proven accurate in the teams we’ve coached – we need to separate between information discovery tasks on the one hand and information distribution work on the other.

Businesses are typically better at information discovery

Want to improve data literacy and make your organization more data-driven? As a business-embedded data professional, you’ll be digging through data daily using technology. Typically, you’ll find many levels of appetite – working with data and the technology required to work with the data. Here lies a significant opportunity to get served in a better way. Ultimately, if you are in business, you should get an interface for your data that aligns with your current working methods. Treat it like a sketchbook –  a way to express your data-related thinking in a way that comes naturally to you. Although your data may export pretty well, talking about your business concepts might be challenging. 

• You want your data answers fast, without computer hiccups and without the need to negotiate over database time with, say,  Ana from Operations. 
• You want to be able to build new datasets by joining existing ones and analyze the summaries. 
• The interface that works best for you, with your background and use case, might not work for your peers on the other team.

Also, if you’re in business, you likely want to get your job done!

• You don’t want to see any tech lingo on your screen. 
• Nor do you want to be distracted by fancy industry buzzwords.
• Lastly, if you’re building an insight that you frequently look at – say your monthly sales numbers – you just want to view the output, not the governance and control processes that ensure that those numbers are correct.

Our key finding is the notion of building a collaboration model with IT. You can hand over work that has been prepared utilizing business thinking and expressed in a language or format that is convenient for data engineers to pick up, without any structural efforts from the business side. 

Data engineers are typically better at structuring processes

IT is all about providing reliable experiences. Data engineers love to build repeatable processes that deliver quality data sets in time. 

Although discovering the business value of data might not be their first focus, they do cater to some other essential factors that will enhance confidence in data:

• Making sure the data quality is up-to-standards.
• Making sure the data is tested (think of entry-related, out-of-range values that might distort your CAPEX report you’ve been pushing for). 

To ensure that you get the most value out of the typically overbooked agendas of your data engineering colleagues, you need to make sure you hand over your business requirements as clearly and as precisely as possible. 

Tip: the requirements.xlsx sheet that used to suffice in the past is not likely to work for much longer. Requirements have shifted – we are now demanding more output from our engineers, at a higher quality, and within a shortened time frame. Ouch!

The operating model that will likely work

The model that we’ve repeatedly put into practice is relatively straightforward. 

• Create a common data platform that can retain and transform all of your organisation’s data. It should be cost-efficient, require less maintenance and should be incredibly fast. One should create as few copies of data as possible on the platform, in order to simplify governance.
• Businesses need a convenient tool to combine and prepare data. We’re not building data processes that should stand the test of time. We just need to prepare datasets that will either be used once for ourselves or used as an example for data engineering. 
• IT wants businesses to use whitebox technology to do the preparatory work. By whitebox we mean that the preparatory work should be done using technology that allows for the specifications to be opened up. By doing so, qualified engineers can view the internals and copy over the data set reasoning to a platform that fits their engineering paradigms. 
• You’ll need a rockstar handover process in place.
  Technology is important, but requirements, expectations and other important factors that are not captured in a dataset, matter even more. Your product owner is a key player, as they steer the process and make sure that the business gets the most value out of their investment. 

Pick accessible technology that pushes businesses forward

We’ve coached several business teams to become more data-driven by deploying business-led data products and programs. In doing so, we’ve met many profiles, ranging from data analysts to citizen data scientists.

While we have met many individuals from different levels of experience and degrees of willingness to adapt to new technologies, we have found that Matillion is the product that best sticks with most user groups. 

Matillion positions itself as an enterprise-wide data transformation tool – something that your average data engineer would be happy to use. They also seem to have an untapped population of analysts as a future client base. 

We noticed that some of our end-users used “extended proof-of-concept-time” from other teams to do their data prep work. From an MS Access and Excel background, the analysts all of a sudden started a buzz about this new technology that was still under consideration. Interestingly enough, it seemed to solve many of the challenges that they had been facing for years:

• It was hard to come by new versions of their base data.
• Processing large amounts of data was near impossible.
• Their access to personal data (we’re in Europe, GDPR applies) was cut as they inevitably made offline copies of data sets.

Interface-wise, we had a solution that was both intuitive and low maintenance for the data engineering team, who suddenly saw this use case pop up unexpectedly.

The near-perfect ad-hoc integration and prototyping tool.

Let’s create a case for prototyping and ad-hoc data set building.  

Business users are already doing this with the tools to get the job done. We feel we have a great opportunity on our hands, one that will deploy the target operating model we’ve discussed before and put power in the hands of business users once again. 

To make sure prototypes are governable, we either:

• Hand them over to the data engineers who specialize in improving technology efficiency. They’ll take the prototypes and align them with common data pipeline practices.
• Or learn from the analysts’ logical reasoning put into the prototype, and rebuild it.

Culture, capabilities, and team composition will decide on which of the options is chosen. Perhaps both. Continuous business value delivery is what matters here; technology should enable it.

In Matillion, we found a technology partner proven to support both – with tangible results measured in skyrocketing user adoption rates and satisfaction surveys.

Thanks, @joswenters, for the collaboration on this article!