There’s constant chatter amongst data professionals about what tools to use to get the job done. I could spend all day rattling them off: Snowflake, Spark, Databricks, Big Query, Kafka, dbt, SQLMesh, and the list goes on and on… But here’s the real question – When’s the last time you heard a discussion about getting back to the basics?
Personally, I think the fundamentals are what separates average performers and “10x” Data Engineers. It’s not generally the tooling that makes the big difference. I’ve seen more business problems solved with a relational database and SQL queries than the latest and greatest of the modern data stack. Better tools can actually mask poor decision making – until it can’t any longer. That’s the pivotal moment when you’ll be forced to get back to the basics anyway.
Do your stakeholders really care that you migrated to Databricks from a SQL Server database? Probably not. Adopting new tools can actually increase time to insights due to the resulting learning curve, which is largely unavoidable. So, while the data team is happily exploring new technologies and padding their resumés, the business isn’t making any real strides from a data perspective. I encourage my peers to focus on communication, effective requirements gathering, customer service, and actually understanding what the business needs so we can deliver and make an impact.
Even though we are usually in a support role, it’s amazing what a motivated data professional can accomplish when focusing on the fundamentals instead of shiny objects. Rant over. Thanks for reading.
– DQC