Harnessing Data Mesh for powerful Business Intelligence and Analytics at a Leading European Retailer

In retail, responding fast to the technology trends that are shaping the industry is vitally important to keep growing and remain competitive. This retail business had identified digital transformation as a key priority early on – specifically, the critical importance of data.

Data was flooding the business in the form of customer, product, and transaction insights gleaned across various business processes, both internal and client-facing. It offered the opportunity of becoming a truly data-driven organisation. But the team needed to create the right environment and culture for users to actually use the data successfully.

However, there was a problem: their existing analytics stack was holding them back. The company’s traditional business intelligence system, built on Microsoft SQL Server and Reporting Services, was showing its limitations. Plagued by scalability challenges, it consisted of multiple, distinct data and BI silos, each managed by separate data science teams.

This fragmented structure, coupled with the complex nature of its data and the extensive business expertise required for their data science teams to analyse it, severely limited the company’s ability to expand its reporting and analytical capabilities. This was only increasing as the volume and complexity of the data sets grew.

To revolutionise its analytical capabilities, the company collaborated with CID to develop and introduce a “modern data stack” to align with its existing decoupled, decentralised software architecture. This strategic shift was designed to minimise central data teams’ involvement in business-centric tasks, and place data ownership and responsibility directly within the business units where the data originated. The change promised to reduce friction and inefficiencies in communication between business and technical teams, as well as significantly increasing its analytics capacity.

A critical component of this transformation was establishing self-service features, enabling business users to autonomously create dashboards and ad-hoc reports. The aim was to increase acceptance, and adoption across the organisation by not relying on a single data unit.

Evolving Data Culture Demands Next-Gen BI

The demand for a more efficient, self-service data access solution came from this growing culture of team-owned data. Now, the retailer could better define key objectives and results for a next-gen analytics platform that would allow more informed and timely decisions by business users. Features included self-service functionalities through tools like Tableau, creating user-friendly data marts, and real-time analytics.

The envisioned solution revolutionised several important processes, including:

• HR: Enabling more timely and efficient assessment for sourcing needs
• Sales: Mitigating return rates across various geographies
• Logistics: Streamlining order and tailoring processes
• Online Sales: Offering personalised product recommendations to online customers

Creating a Foundation with State-of-the-Art Information Architecture

Transitioning to a “modern data stack” set off several pivotal decisions. Firstly, a commitment to using micro-architecture would allow decentralised processes and more specific data ownership, alleviating scalability issues. At the same time, a structured approach was adopted to model and describe data products, fostering a clearer, company-wide understanding of term definitions and data semantics.

Implementing a Data Mesh was a pivotal point in this transformation. As a sophisticated, decentralised system, the Data Mesh would offer scalable data products, integrating seamlessly with the existing decoupled software services architecture built by CID.

To align with this change in data usage the retailer migrated to a cloud data platform, opting for Snowflake on Microsoft Azure over the on-premises Microsoft SQL environment. Following a thorough evaluation, the company selected Tableau to enable business users to create dashboards and ad-hoc reports as part of the move to a self-service model. Finally, Kafka was chosen to stream data from the operational systems into the Data Mesh, allowing real-time analyses.

Conclusion

Building upon the existing software architecture and the foundational layer of the Data Mesh, the company successfully deployed the initial set of data products and data marts. This kick started a new era for the retailer, where business processes, analytics, and decision-making are continually enhanced through an evolving stable of data products and insights.

The organisation is now on an evolving, future-proofed journey, leveraging state-of-the-art analytics systems to enable a culture of informed decision-making, efficiency, and innovation.

Key benefits

• More comprehensive and timely self-service insights for business users
• Better scalability for analytics performance, supporting more data-driven processes and decision-making across the organisation
• Improving data culture and data literacy, fostering smarter, seamless data use while laying the foundation to unlock the true benefits of AI