The following question came up in a recent conversation:
“Why do you gravitate towards data-centric projects?”
I was surprised by my answer, which is why I’m sharing. Essentially, I believe that data projects are less emotional than traditional software projects. The expected outputs can be clearly defined and are less subjective. On a data project, although there are several paths to success, they all look and feel quite similar. For a software project, success has a much wider range of variability. I’ll give an analogy – Data Engineering is like a grocery store, while Software Engineering is like a restaurant. Let me explain…
With traditional software, companies often pay 6/7/8 figures for projects or SaaS tools to support mission critical business processes – think CRMs, financial systems, help desks, etc… These systems are typically customizable to fit company-specific preferences, much like a restaurant allows substitutions. And just like restaurants, user sentiment tends to be polarized. People love or hate systems like Salesforce – rarely anything in between. The same goes for McDonald’s. These industries require highly specific customer targeting. Grocery stores, on the other hand, cater more broadly and don’t rely as heavily on emotional appeal or fine-tuned experiences.
Another aspect worth considering is that Data Engineers typically work with data produced by software products. That means they’re often brought in after the fact – usually when querying data becomes slow, inconsistent, or expensive. To bring back the analogy: imagine someone who isn’t an experienced home chef but likes to eat good food. At first, they might rely heavily on restaurants or takeout. Eventually, health or financial concerns force a shift – maybe to frozen meals (think PF Chang’s in the grocery freezer), and eventually, to cooking from scratch with groceries.
Similarly, companies may start with SaaS dashboards or run ad-hoc SQL on siloed systems. But over time, data sprawl leads to confusion, inconsistencies, and frustration. That’s when data engineering becomes essential. Like cooking your own meals, it requires more effort up front, but it creates a healthier, more scalable data ecosystem in the long run.
Happy Querying
– DQC