I am a fresh graduate data engineer working at a small company in the oil and drilling industry.
I was hired 6 months ago as a freelance data engineer, and after proving myself through my work quality, I am now essentially functioning as a tech lead, with full responsibility and ownership of desig...
I was hired 6 months ago as a freelance data engineer, and after proving myself through my work quality, I am now essentially functioning as a tech lead, with full responsibility and ownership of designing, implementing, and hiring for the projects I'm assigned.
Our company is not a tech company, so I only have a couple of tech-oriented colleagues, and I barely interact with them. Now I directly report to the director of the company, who in all senses is awesome, with 40+ years of combined experience in some of the biggest oil and drilling companies globally.
However, I have some strong FOMO about not being able to learn much technical stuff from my peers or seniors. I am trying my best to learn and pick things up on my own, learning design principles, getting code reviews from chatGPT, etc. But even then, I'm afraid I am not producing the software to the highest standards of the industry since we don't have any rigorous cross-checking, and might be missing out on a lot of learning.
Can someone who has been in positions similar to these please guide me?
Comments
Couple of important lessons that will keep you in good stead for a long time:
1. Learn how to learn well, continuously, and sustainably. Tech changes rapidly. And you will want to hop from one domain to another, just for keeping things interesting and to move with markets. This is both a blessing and a curse. It is a blessing because you can start late and still be in the top percentile if you have the brains and work hard for it. It is a curse because you will be doing this no matter how many years of experience you have.
2. Hone your non-technical skills– caution: these are compounding over time (both good and bad habits) – being disciplined, thinking clearly, articulating clearly, being professional, being trustworthy, managing your physical and mental health, being dependable/reliable, having a growth mindset, thriving in ambiguity and uncertainty etc. then, honing your communication skills – effectively collaborating with people, give/receive effective feedback, do/get mentoring/coaching, working with cross-functional people, working with very seniors, very juniors, peers etc. read a lot, develop mental models, deeply craft your personal approach to first principles problem solving, to making tradeoffs/bets etc.
You can do the above all by yourself, through reading, and observing people from afar, and engaging with people (even strangers on forum like this one) in dialog.
I've been in a similar position of having to learn solo for 10+ years and lesson #2 above has been FAR more important than #1 in my experience. Clients and bosses care much more about communication, dependability, etc. than whether a product has been coded elegantly via best practices.
By Gokevin 2024-12-027:37 Agree;
By underdeserver 2024-12-029:062 reply I would argue that articulating clearly is not a non-technical skill, it's the essence of what software engineers do.
There is a big difference between articulating "code" clearly and articulating clearly when you speak / write to other humans.
I can see some differences, but none of them seem "big".
In code you don't have all the extra communication avenues that we have when speaking, like body language, intonation, sarcasm and so on.
On the other hand, when writing code we are not speaking in real time. We can think about a problem for a while, consider the best possible way to solve a problem and how to explain it.
What do you see as a big difference?
By Kabootit 2024-12-0215:45 1) Coding clearing in the moment vs (2) coding clearly for future selves is two different mindsets/contexts right there.
(3) Communicating clearly is an orthogonal skill to coding clearly. I think this skill is barely acknowledged in engineering cultures in comparison to the above.
I feel you have to have an engineering culture that values institutional knowledge retention, team education and growth — and not treating engineers as fungible — to get to level (2). Level (3) would be a great place to work.
By shafyy 2024-12-0210:04 Ideally, yes. And I agree that there's some correlation of how well clearly people can write prose and write code. But, articulating yourself with your teammates is not only about writing well. It's also about saying the right things. I have seen many engineers that write code well, can write a paragraph in English well, but just don't communicate enough and don't communicate the right things at the right time etc.
By heeton 2024-12-0210:27 I agree. Writing well is _highly_ technical.
For 2, I'll single out the skill of tailoring technical explanations to your counterparty's level of understanding and technical knowledge. The ability to explain to less technical people what your new project/feature does without going into too many unnecessary details and without being too high-level is invaluable. It builds confidence in your work for them (I know what this thing is doing - maybe not all nuts and bolts, but enough to operate with confidence) and in you as a professional (this guy clearly understands what he's working on and does not try to bury me in jargon or oversimplify things).
By _kb 2024-12-0511:18 This exact scenario is one of the best interview questions I’ve been asked and have repeatedly re-used when on panels myself.
Taking a complex domain and effectively communicating it (correctly) at different levels requires having not just rote knowledge but an actual understanding.
I tell myself to not adhd it and instead take time and read through a whole manual, experiment in a local sandbox, grasp the limits of knowledge and features and also where it gets deep.
Then in a regular work I explicitly detect where it pays off and feel “see I told you”. This creates a motivational loop to continue not-adhd-ing through tech.
Sometimes I still fly over the knowledge, but then may note that what I’ve been doing in a complex way could be solved with one parameter, if only I knew about it. This creates negative feedback against flying over.
This is ofc only one facet of learning, but I find this “see I told you” method very effective, cause my main issue with learning is unwillingness to learn for no clear reason.
My issue with this method of learning is… deadlines. During work time, I often feel that I need to solve something “quick”, which then leads me to usually learn and really deep dive outside of regular work hours instead.
Now, I mostly actually enjoy doing this and thus it has not really limited me. But I wish I could just spend some actual work time on more ‘non adhd-ing’ what I learn.
By mathgeek 2024-12-0214:53 Have you chatted with your manager about expectations and your personal growth goals?
By wruza 2024-12-0213:48 Yeah, deadlines suck. Historically I managed to "manage the management" into a reasonable rush/cook balance most of the time, which allows for healthy exploration, but that is absolutely "ymmw" thing.
By markus_zhang 2024-12-0222:09 I love the manual method and occasionally used it. Basically assuming there is no Internet, just a book and a repo of source code and try to figure out how to do something.
Sadly I'm now so easily burnout that even setting a dev env up can burn me out.
By christoff12 2024-12-0120:46 The key is to figure out what your learning process looks like.
For example, I discovered early on that I learn in three phases: 1. I get exposed to something (a concept, a process, etc); basically discover that something exists. 2. I then see how that thing is used whether through mentorship or tutorials or, increasingly, through trial and error. 3. I apply that thing to some novel problem.
Through this cycle of Discovery-Tutelage-Application, I can assess my level of comfort with new material and understand when my struggles are due to trying to short circuit the process.
It's likely that you have some form of learning process that is equally cyclical, yet undefined -- once you identify and codify those steps, you can evaluate your progress when it comes to acquiring new skills.
By theonething 2024-12-024:33 Dr. Barbara Oakly is pretty renowned in this area of study and has a free online class.
By emusan 2024-12-0120:30 This is a difficult question to prescribe an answer for that works for everyone, but the best I personally can think of is "practice".
To make that more actionable... My approach in life has generally been to find a project (even something seemingly incredibly dumb, as long as it is fun), then work through it, learning what I need to know as I go along. To learn "well", you must then also constantly question what you have done as you complete various stages of the project to see if you have done them as effectively as possible, and try to incorporate any lessons learned into future projects.
I have found that how individuals do the learning required for this differs significantly from person to person, so it is hard to recommend any particular approach.
By zer0x4d 2024-12-028:21 You learn to learn with time and imo it's different slightly from field to field. In tech field you just learn by doing, asking questions, making mistakes and gaining experience. Take a sample code in a new language, make it do something slightly different. Then add one more thing on top. Then something slightly more complex, then finally try to make it do what you want. Once ready, deploy, test, and iterate.
By tourmalinetaco 2024-12-024:57 At its simplest, rewrite and interface regularly with knowledge. There’s an entire hobby around personal knowledge management, and it‘s all up to you to find what works best, but taking meaningful notes and rewriting those notes, processing them further and further, will help form deeper mental connections regardless of method.
By disambiguation 2024-12-020:52 You're doing it right now, i.e ask questions and seek answers.
By j45 2024-12-026:51 There is a pretty good course called learning how to learn that seems to pop up every so often.
By chii 2024-12-026:17 i learnt that at university - by essentially not being taught, but being given curiosity on the subject and told to research it.
I am not sure there's any hand holding that can be given to someone to learn how to learn.
Any recommendations on learning to "think clearly"? I feel like this is something I struggle quite a bit with and haven't found any good resources on improving it other than "sleep" and "exercise", which I'm not lacking in.
By beezlebroxxxxxx 2024-12-0215:49 Unironically, that's what school is supposed to be for.
There's keeping the engine well maintained (the sleep and exercise part, for example) and there's driving the engine down new paths and honing you driving technique. You work on the latter by exposing yourself to novel and interesting arguments (interesting philosophy or argumentative non-fiction, for example) and then working through the argument again with counter-arguments in mind. I would not recommend pop-sci books for this, because their arguments and writing tends to be quite flabby.
I'd actually recommend something like RG Collingwood's "The Principles of Art" which is a relatively plain English example of well written philosophy:
https://archive.org/details/in.ernet.dli.2015.188470/page/n1...
You will disagree with him. The point is to understand how and why you disagree with him and to explain that clearly, not only to yourself but others as well. Thinking clearly is about responding and communicating clearly while displaying a sure and succinct understanding of the problem at hand.
You can apply that to everything you read or are confronted with, but the key thing to realize is that "thinking clearly" is something you practice. There is no one trick --- it's an approach.
I was in a similar position for the first half of my career.
1) Start contributing heavily to a popular open source project you're familiar with and use. Try to make your PRs as high quality as possible. You'll get free code reviews from some of the best engineers in the world doing this. That'll be a million times better than ChatGPT code reviews and you'll be able to learn a ton from it while also getting your code into production at thousands to millions of companies in the world that depend on the open source project/s you choose to contribute to.
2) Fill holes in your knowledge base. If you feel weak in a particular area like networking or DSA then study and practice with it until you no longer feel weak in that area. If you were on my team I would try to assign tasks to you to help you naturally build out knowledge and confidence in your weak areas but without someone to do that for you, you'll need to try to figure out your own weaknesses.
3) Always try to do your best when working professionally. This is all any of us can ever do and if you actively practice it then it'll become a habit that will guide you toward success in any environment.
By MichaelRazum 2024-12-0212:314 reply 1) Is this really the case? I mean some of the popular open source project are kind of mature. So you would get some feedback, regarding some specifics of the frameworks. For sure you would learn a bit. On the other hand they are kind of "done" software, so you would not really learn how to build something new from scratch.
Not sure about the other points.
By jrochkind1 2024-12-0215:372 reply You learn so much by looking at how mature succesful software looks under the hood, which you will have to do as a condition for writing a good PR.
But, sure, don't send PR's for things that don't matter just to send PR's, that's just annoying.
But if you send a 3-line PR to fix a bug, that required 15 hours of study of the code to diagnose and figure out the right 3-lines, that is HUGELY educational.
Working on immature or even poorly written software can be super educational too, nothing wrong with that too.
But if you only ever look at hacky not-yet-mature not-yet-proven software, how would you ever learn to write high-quality software, or even know the difference? You have to study known good code. The way programmers study something with a natural goal instead of just artificially, is in the course of trying to alter it "correctly", the study you must do to figure that out.
(I know some programmers who have literally never worked on anything but over-engineered architecture-switched-every-month-for-years barely-holding-together technical debt bankrupt software... and I think sometimes some of them literally don't know anything else is possible! Looking at mature respected code is crucial, if you don't even know what it looks like how could you possibly try to get there?)
By sha16 2024-12-0219:11 One of my best contributions was a 1 line fix in one of Microsoft's repos. It took me a day or two to understand the code and what it was doing but it was well worth it.
By maxmynter95 2024-12-0313:01 +1 on the "reading other high quality code" part to improve your craft.
Getting your own work critiqued is unmatched, but reading what others did is the next best thing.
By bluGill 2024-12-0215:09 Mature projects are probably the best to work on - they have built a good foundation to study.
As the other poster pointed out, heavily contribute implies the project still could use some major new features and there is plenty to do. Even porting something KDE to QT6 would be a major effort and teach a lot even if you don't write any new code. There are projects that are complete as well and so there isn't much to do.
By 27theo 2024-12-0213:35 'some of the popular open source projects are kind of mature', so find popular projects that aren't mature, or 'done'? There are countless.
I wouldn't say you'd only learn specifics of the frameworks. Maybe if you only submitted 2 or 3 PRs, but that's not what parent is suggesting: 'start contributing heavily'.
By pavel_lishin 2024-12-0215:15 Most PRs I've opened seem to languish forever, until I get tired of seeing them in /pulls and close them myself. It really depends on the project.
Not so sure about 3. Wouldn't it be more sensible to be flexible in terms if what you optimize? Some projects need best work, some need a friendly face, some should just die quickly.
By ensignavenger 2024-12-0215:21 Determining what a project needs and doing your best to deliver those needs is what "doing your best" means. If you spend months working on a solution that needs to be and can be deployed in a week, because you want to make it shiny and such, you aren't doing a great job on that work.
By sbarre 2024-12-0215:12 This is hindsight in my case, but I will say that a lot of the resilience and experience I've developed in my career has come from putting best effort into less-than-ideal or even failing projects (in my defense, in most cases I didn't know they were failing or in a bad state).
It wasn't fun, but I'm better for it.
By collinmcnulty 2024-12-0213:26 Suggestion 1 is a great idea. In particular, consider contributing to one of Airflow’s provider packages if you use Airflow and have any steps in your data pipeline that aren’t ultra-specific to your company not covered by existing providers.
By humanfromearth9 2024-12-0216:174 reply I'll speak for software development, but in principle it's the same for any other domain.
Read about your technologies. A lot.
When I started working (about 3 to 4 years) , I spent 20-30 minutes a day reading DZone articles about Java, software design, architecture, OOP. Persist and do it daily. Repetition and habit are key.
And be focused. Understand everything in every article, do not accept not understanding everything. For each article, try to find out whether it makes sense, try to understand what the author wants to tell you. Think about how you would have done it. When possible, apply what seems to be useful and overcome the limitations of what is written /proposed in the article.
Occasionally, do the same with IT books instead of articles.
Then explore further, for example, find out how certain popular OOP patterns may be replaced by other, equivalent, patterns in FP. Think about how OOP classes are equivalent or not to closures in FP.
Also, become an expert in fundamental practices, like transaction management.
Application of theory is key.
That's how you do it.
> Application of theory is key.
I often see people either get caught up in theory without practice, or proclaim that it's all about experience. Actual time spent writing code is probably the most important thing, but having some foundational ideas and getting to skip some things you've have to painfully learn yourself by reading and then getting to apply them directly is huge.
By bunderbunder 2024-12-0219:56 There has actually been some research on this. I didn't save the link, but the paper I found showed that time spent reading code is a much stronger predictor of skill development than time spent writing code.
I'm guessing in practice it's probably comparable to music and language learning, which seem to work similarly in this respect. You do need to spend time practicing by doing, but having that be most of your skill development time is sub-optimal for people who aren't already at an advanced level. Most your time should actually be spent carefully observing what the experts are doing. Because that's what builds the intuition you need to critique your own performance. Which, in turn, acts as an enormous force multiplier for your active practice.
That's arguably why contributing to open source projects is such a great method. Because, unlike drilling yourself on practice problems, the vast majority of time spent contributing to an existing project will be spent reading and understanding the code that's already there.
I believe it was Knuth who said practice informs theory and theory informs practice. You need both.
And there's an adage updated with modern understanding "Perfect practice makes perfect" . It's not just repetitions but ensuring you train the patterns you wish to do automatically next time.
Excellent code is slow to write, the first 10 times, but eventually it's just as fast as junky code.
You should keep in mind that code quality is mostly subjective.
You can create a PR that coworker 1 loves because it hides the complexity and then have coworker 2 come in and say that it's complete garbage, because while the complexity is hidden, it still has to be understood in this context, only increasing the difficulty of working with the project
I.e using a state-machine vs a switch/case to determine a status attribute.
Both viewpoints are valid. Some code is objectively terrible, but most discussions on code quality are massively overblown
By psunavy03 2024-12-0221:40 There are also those who view obscure and quasi-obfuscated ways of solving a problem as "elegant" as opposed to more straightforward and easy-to-read ones. Usually found near the ones ranting about superfluous code comments.
Or making/sharing LinkedIn posts showing the uber-obscure or complicated "Senior Developer Solution" next to the more straightforward and easy-to-read "Junior Developer Solution."
By sevensor 2024-12-0218:13 See the recent threads about Heaviside and Category Theory for interesting perspectives on practice informing theory (Heaviside) and theory informing practice (Category Theory naturally).
By intelVISA 2024-12-0220:11 Not just time spent writing code but variety; the 1YOE ten times trap of some 'seniors' is very real.
By zcw100 2024-12-0218:21 > Understand everything in every article, do not accept not understanding everything.
I’d be careful with that. It’s a great way to get lost down the rabbit hole and works against being focused. Eventually you have to get to a point where you say to yourself that it’s someone else’s problem beyond here and double back.
By DelightOne 2024-12-0219:22 > And be focused. Understand everything in every article, do not accept not understanding everything.
I read my first programming books that way. Always ask "Why is that word here" and "Why is that ! there". Made me see details.
By roeles 2024-12-0217:52 > Application of theory is key.
What works for me is deliberately testing something that I recently read in a project. Even if it is something that works, rewrite it from the perspective of what you just learnt.
This makes you look at problems from different perspectives. And if you get stuck, it's easy to ask for help.
