Data lineage is a problem because most companies have several tables and queries before humans consume it! This has well known challenges: changes do not propagate downstream from the source, and reliable (fresh, updated or complete) data is not always available. What does Minerva do? I was expecting Minerva to a database (collection of tables), but it turns out that Minerva is what I’ll call: Data Transformation Manager. It overlaps quite a bit with dbt but it’s not a pure execution layer.

This one’s gonna be brief and echoes 2 Less Obvious Ideas to the younger me. I am assuming that you already know the hygiene factors: Make few promises. Keep most of them and exceed few of them atleast. Get to like the top 5% in the skill of effort estimation for your own work at the very least. And so on. Contribute to Developer Ecosystem Improving any part of the developer ecosystem is useful and visible at the same time.

This is primarily for my future self. These are observations based on my own experience of 2 years at Verloop.io and helping a few companies hire for similar roles. Do Seniormost hire first: Start by hiring the senior most person you’re going to hire. E.g. start by hiring the ML Lead (assuming you already have a CTO) Have a means to tell that your investment in data science is working out or not Closest to User first: Hire the person who will consume the data to build for the user first Sourcing: Begin sourcing early and over-emphasise two channels: Referrals and Portfolios Typically, in India - expect: ~2 months to close a full time role at early career (0-3 years) and ~3 months to close a mid career (3-7 years) and 6+ months to close a senior hire If a developer has open source code contributions in the last 2-3 years, consider waving off the coding or algorithmic challenge to speed up the interview process Pay above market cash salaries In 12-18 months from now, when your ML Engineer will have internalised all the requirements, company culture and built a bunch of important tooling - she would get an offer which is 2-3x of today.

Question from a friend: I am interested in knowing how did you come to this decision of moving to SWE from DS/MLE. Since I’ve been asked a variant of this question quite a few times, I thought it would be good to share my answer. What kind of research did you do to get to this decision? I spoke to a lot of people who were both big companies and startups.

You learn a well-paying skill and years later - it comes back to hurt you in unexpected ways. That’s an Anti Skill. Consider this hypothetical: You start your software engineering career and build a reputation as someone who is good at iOS development. Each year, the money you make keeps improving as you keep getting better at it. The downside? You’ll find it hard to get job offers outside of iOS development [1]

Bulk of this is borrowed from notes made my teammate and friend at Verloop.io’s NLP/ML team of our conversations. I’ve taken the liberty to remove our internal slang and some boring stuff. I want to build a community around me on NLP. How can I get discovered by others? Broadly speaking, the aim in forming connections can be split into Long Term and Short term. A short term aim would be where you can receive something immediate out of the connections or a particular connection itself.

Data Science Org Design for Startups While there is plenty of good advice on making ML work and making a career as a Data Scientist - I think very little discussion happens on the organization design for Data Science itself. This blog will hopefully help folks not just build their team, but also understand the ecosystem from which they are hiring. Organization Design is determined by these 3 broad categories:

This is a 3 part essay. Part 1: Talent In my teenage years, I first understood that the kind of family you’re born into gives you access to certain kinds of wealth e.g. money, network, habits/knowledge and social capital to do things you want. Most people in my peer group and even among the adults I could speak to, used these resources to make their own life materially more comfortable.

This quick entry is partially inspired by a similar entry from Manas Saloi. Ofc, other folks have offered their opinions on Basecamp and earlier Coinbase. The woke mob is shitting on Jason and DHH, but I strongly agree with Basecamp’s new policies. IMHO a company is supposed to create an environment where people come and do their best work without having to align to one side of the political spectrum.