Data engineer roles vary but some core traits stand out for any data engineer. To be successful you don’t need to portray these traits 100% of the time, but they are ways of acting that you should be trying to incorporate regularly in how you work. If you missed it, check out my first posts in this series on What is a Data Engineer? and Data Engineer Skills for Success. Let’s finish off this series with the traits I see as most critical for success as a data engineer.
Key Traits
Problem Solver
Data engineers are not often handed detailed specs of what needs to be built so it’s key that they can understand the goals and find solutions. Understanding the problem and driving to solutions is expected from a data engineer while sitting back and asking how to implement the request is a burden to the team.
Analytical / Detail Oriented
Data pipelines process a lot of data with a wide variety of issues that need to be handled by the code. Small decisions need to be made during development. Strong analytical skills and attention to detail lead to the right decisions to ensure data is transformed correctly and small bugs do not lead to large issues downstream.
Self-starter
Talk is cheap, action is what matters. Business leaders are not able to tell you exactly what to do so data engineers need the drive to create value and high-quality systems.
Team Player
Consistent naming and standards are important. Maintainable code is critical. A data engineer who can work well with others, talk through ideas, and include others in the process will save the team the stress of relying on one person every time a failure happens.
Quality Focused
Data value is directly related to data quality. Data engineers may not have superpowers but they play a big role in the quality of the data used by data scientists and analysts. Testing, validation, and communication of quality issues are important responsibilities of a data engineering team.
Curious / Asks “Why?”
Who needs to use the data? What questions do they have? What is hard for them to do that I can make easier? Someone who asks these types of questions will make a bigger impact than a person who feels like they are just moving data from one system to another.
The data engineer I want to hire…
Takes initiative
In line with being a self-starter, I look for adding team members that take initiative and drive projects forward. The nuances of each problem faced can be time-consuming for someone else to fully grasp and solve, so if each person can take initiative in the areas they work on the team can deliver at a much higher velocity.
Takes responsibility
More so than other software developer roles, I want a data engineer on the team who feels ownership of the systems they build. Data pipeline code needs to be revisited often as data changes or new calculations need to be added to the process. The more pride they take in the work they do, the more willing they are to improve the system even when it’s not an exciting new feature.
Actively Learns
Modern data platforms involve many technologies that are rapidly changing. A data engineer will be asked to evaluate a new cloud service, learn a new API, and quickly pick up on business logic.
Diversifies Team
We don’t want a team that agrees on everything. We want team members that challenge one another’s ideas, review code critically, bring different cultural perspectives, and have different professional experiences. This will lead to more well-rounded thinking about the platforms and pipelines being built.
Writes Code
Arguably, some data engineer roles only require SQL programming and the use of low-code ETL tools to be successful. At some point, a project will require a custom data source or transformation that is not supported by the team’s tool of choice. In those cases, the ability to program in Python, Scala, or another language will keep the team from being stuck. Because of the eventual limits of ETL tools, I gravitate toward building pipelines with custom code or heavily customizable frameworks.
What if I don’t have these traits?
If you are feeling like these traits don’t describe you, please don’t be discouraged. Just like a job description, few people will check every box on the list. Two key points I will leave you with…
- This is a biased list from my perspective; not every team or role will want this exact set of traits.
- Acting out these traits on the job is something you can develop over time and start doing more and more as you grow in your career.