I'm too raw to be a supervisor/ senior data Engineer: I have no choice but to grow...

Miles MaghanoyMiles Maghanoy
2 min read

I have around 2.5 years of experience as a Data Engineer in a reputable company before getting the job here in my current company. I went through a series of interveiw and examinations which determined that I am the best candidate for the role. However, after ~2 months of working here, I think I am too raw to be a Data Engineer Supervisor yet (it might be converted to Sr. Data Engineer in some companies).

My previous company heavily utilized AWS for almost all of our pipelines. I thought this easily converts to on-prem DW system, but nah. This is my first time dealing with numerous datasources: eBTs and various data servers per business units. Company has no real data architecture, let alone metadata, and I spent the past 2 months just tring to understand company data stored in almost a thousand columnar tables and flat files... No documentation what so ever. Some data are manually managed by operations also and updated periodically which will always be a challenge.

Adding here as well that I am originally hired to lead the team to migrate data to cloud... But I proposed to organize data first and move only those that are useable to business units and needed live streaming for Tableau and such...

For the record I am the very first Data Engineer they ever hired. They have an amazing team of software engineers who manage their eBTs and data storage, but that it. It's now my job to manage the accumulated mess and "provide the source of truth"...

Though, I believe in my company. It's one of the biggest in my country, and I am amazed how they progressed this far without proper Data Management. I wanted to try and give my best here for the next 4 months: let them decide if they still want my services afterwards.

Sometimes though, I find myself getting overwhelmed. I now stutter during meetings and presentations: maybe my doubt is showing through. Maybe this is just part of the learning that I have to overcome with self-compassion.

I am now reading Kimball's WH Toolkit. I've seen this recommended in this sub frequently and I'm glad it's handy. There is also a plan to create a data lake in the future, can you suggest any book/courses to help me with this?

TLDR: Company has no Data Management, and I am hired to initiate it, but I feel I'm so raw for the role. Highly appreciate getting resource material recommendations for growth that is expected of me.

Link to original Reddit post: https://www.reddit.com/r/dataengineering/comments/1ibycre/im_too_raw_to_be_a_supervisor_senior_engineer_i/

0
Subscribe to my newsletter

Read articles from Miles Maghanoy directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Miles Maghanoy
Miles Maghanoy