You’ve probably heard of the data modeling technique referred to as ‘One Big Table’. The technique essentially revolves around preferring wide tables in order to make ingestion into BI tools easier for analysts. Less talked about is why analyst ease of use is important to data teams. Does the business care? I’m almost certain the answer to this question is a resounding “No”! The business tends to care about timely and accurate data products, not necessarily the developer experience.
Based on what I’ve seen professionally, data teams gravitate towards OBT because analysts generally do not enjoy writing SQL. If you are an analyst reading this and you DO enjoy writing SQL, congratulations! You’ll likely be a Data Engineer within a few years or less. But for the others, what tends to happen is they either (1) struggle with SQL, which leads to unnecessary errors, rework, and frustration, or (2) rely on the BI tool to perform their joins, which leads to performance issues. In both scenarios, the problem bubbles upstream until Data Engineering solves it with OBT. There’s nothing inherently wrong about this, however I do think companies would benefit from having their Data Analysts upskill. Repetitive joins don’t need to be addressed with engineering time if the analysts are sufficiently capable.
I think a side effect of abstracting away source system relationships is that the data product becomes a black box very quickly. When there is a data quality concern, the only team that can address it becomes Data Engineering since all of the logic is now in their OBT. If analysts wrote the queries themselves, triaging issues should be faster and involve fewer team members. We’re all about efficiency here 🙂 – Thanks for reading.
– DQC