How to Implement the Semantic Layer

The semantic layer allows your business teams to share a common language with the data team and provide consistent, reliable, and accurate business metrics for your entire organization. To find out more about how the semantic layer can help your organization check out our last blog post - Is the semantic layer suitable for you?

In this post, we’ll be going deeper into how you can actually go about implementing the semantic layer.

The implementation and adoption of the semantic layer have several components, let's discuss each.

Driving a Data Culture

Prior to implementing the semantic layer, you need to establish a data plan. Start from the use cases/needs of your team. This exercise should be done with the data and (relevant) business teams together, where you try to answer:

• What are the north metrics that we should be targeting?
• In case we don’t know them: why not?
• What are our business entities?
• Is there alignment on the clear mathematical definitions of our metrics?
• Why / why not?

Since such an organization-wide effort could be very time-consuming, we suggest starting out with a single use case within a single business function which is not being satisfied due to lack of (or too much of) data. Experimenting with and trying out the semantic layer in small pockets of business functions will help your organization get more experience on how to incrementally build your data culture.

Once the business teams are aligned on the metrics they want to track, and the data team can confirm that the required data points are actually being tracked in the warehouse - we can proceed to the next step.

Building the  Semantic Layer

Goals

The goals of a successfully implemented semantic layer can be outlined as follows:

• Provide reliable, consistent, and accurate metrics to your organization. We define these three terms as follows in our context:
• Reliable: the metric numbers are updated and available as per the internal SLAs decided among all stakeholders
• Consistent: Metric definition alignment throughout the organization
• Accurate: the metric numbers are correct as per the definition and calculation method expected by the consumers (business teams)

Metrics Management

One of the core functionalities of a semantic layer is to provide business users access to metrics they need while ensuring that these metrics are consistent and accurate.

For this, any semantic layer solution needs to make sure that there is a single source of truth for managing your business metric definitions, connected to the source of truth for your metrics: your data warehouse.

You can explore the open-source metrics framework released by dbt Labs as a starting point for setting up a semantic layer. It helps with standardizing metrics definitions in a single place, while also providing powerful abstractions on top of complicated calculations like period-over-period comparisons for metrics and target projections.

Our semantic layer also needs to be adaptive to the dynamic nature of business by allowing users the ability to modify definitions of metrics and create metrics for their use case whenever required (this is CRUD for the technical people). Traditionally, these use cases come up in the form of requests for:

• New metrics
• Updates to existing metrics
• New business datasets
• New columns in existing datasets

One of the ways in which you can solve this is by using platforms like Retool to create a simple interface for business users to raise their requests which can be reviewed by the data team. You can also integrate right into the workflows of your stakeholders (business users) by managing requests via Google Forms, Slack, or email!

Data Management

On the data side, you would need to ensure that these metrics are updated - that is, the data is available to the business teams whenever they need it, as per the internal SLAs defined among the stakeholders. This might involve setting up cron to manage the runs for updating these metrics.

Further, you would need to make sure that the queries being run to fetch this data are optimal and that they are able to query the required data points, in the minimum compute and time costs. It might be useful to explore some solutions being provided by data warehouses in this regard: specifically the Warehouse Native App Development framework by Snowflake.

Should you Build or Buy?

This question comes down to the resources and expertise you have available in-house in order to take up a project like this.

An off-the-shelf solution definitely cuts your implementation time from months to a few weeks, while also saving on precious engineering resources. However, a readymade product might not be able to solve specialized use cases (like dealing with non-standard schemas) that your organization might have.

We’ve come up with a simple framework to help you make your decision based on your organization’s data maturity.

• You can consider implementing the semantic layer in-house, tailored to your specific requirements if
• You’re an early-stage company with nascent data processes like:
• Custom ways of pulling data from your different SaaS sources (or just referring to it in your different tools’ dashboards)
• Limited use of data warehouses for storing business data
• You’re in a hyper-specialized domain that uses non-standard schemas for their data requirements
• You can consider buying an off-the-shelf solution for the semantic layer if
• You’re a company making a transition from an early stage to that of implementing a data warehouse, transitioning to the modern data stack, and setting up a standardized data culture across your organization
• An enterprise looking to increase the value you draw from data and eliminate the overheads involved with ensuring data quality and availability

You can check out the partners for the recently launched Semantic Layer by dbt Labs here (full disclosure: we’re one of the partners), and explore the solution that meets the requirements of your organization.

We hope that this guide gives you a good idea of how to implement the semantic layer for your organization. Do write to us at sidhant@houseware.io if you have any doubts, or would like to chat in general. We’d be happy to talk through your data woes. :)