Skip to content

Writing

Diwali 2019: Annual Letter

I wanted to drop by and wish you a very Happy Diwali!

This is an annual email that I write to say thanks to mentors, friends, allies - everyone who has chipped in sculpt a better me.

So that is what I'll begin with: Thanks a ton for being there for me!

Also, checkout the 2018 edition of the annual email in attachment if you're curious :)

Moving onto the least comfortable piece, talking about myself:

Health

This has been a year of regression. For instance:

  • My BMI has slipped from 30 to 33 (higher is worse), gained about 7 kgs
  • Have worked out less than 2000 minutes (was 5000 minutes last year)
  • Have made tremendous unmeasured progress in managing my long term asthma, now I can frequently run up 2 floors without noticing that I did

Wealth

  • I was trying to invest in things other than mutual funds. Well, I've lost 25% of whatever I’ve invested in stocks (via smallcase) since then
  • I hired a financial advisor, which was a net loss making deal due to the fees, but helped me form better habits. Financial advisor in that sense are like gym trainers, more than the exercise - their main value addition is in following up
  • I’ve reduced my Fixed Deposit and Recurring Deposit investments to 0% of my savings
  • I’m diversifying my folio to a mix of index and bond funds now - this is still work in progress

Learning and Reading

Last year, I wrote how I was trying to think and dream better. Well, that has been a bust. The intent was to form a mental model, a scaffold of habits and information on which I could add new information and refine the scaffold itself.

This has been slower than I’d have guessed.

Intentionally, I read less books this year. I did read a lot more modern essays (blogs?) ranging from Peter Drucker, Marc Andressen, Naval Ravikant, and surprisingly boring, a former Coal India Chairman. I also re-read a couple of books like Courage to Be Disliked and Effective Engineer.

I also spent an unhealthy time reading beginner content on seemingly irrelevant topics like Policy making and Human Capital. Here is my favourite cross-topic discovery so far: Talent vs Luck: The role of randomness in success and failure.

Work

Since last Diwali, I quit Soroco and moved to Verloop.io - a really small, pre-Product Market Fit, loss-making business. I strongly considered a very enticing data science role at an investment bank. I finally leaned away from it, making a financially very expensive gut call.

The role here is challenging in two very important ways:

  • Autonomy: There is a much higher degree of autonomy on how I prioritize tasks and that has been scarier than I’d thought
  • Leading: I also hired 3 Machine Learning engineers who now work with me

The thing that could unlock 10x value? Technical and Learning-to-Manage Mentorship: Something I could really use a lot of help on, is critically missing.

The other welcome change is in how anxious I am about work: I am not! I am almost always excited to go work.

Work Adjacent

  • Public Speaking: I was invited to speak at PyCon India 2019 - India’s top conference on the Python programming language
  • Open Source: I’ve continued to maintain Awesome-NLP for years now. It now has over 9 thousand stars and 80+ contributors now. It’s officially recommended by Stanford to their Deep Learning students now.

What's Next?

  1. Writing and Tact

I suspect more knowledge workers in 20s should invest in learning how to communicate via either public speaking or writing. My chosen medium will probably be writing. I intend to write about a select list of topics: technology and how it has shaped our choices.

Owing partly to how I grew up, my default communication style has been confrontational. I'll incorporate a lot more tact, grace and consideration in the years ahead. Be more Kabira.

  1. Chase external validation

My professional and personal framework so far been focused internally. I did not value traditional credentials like degrees, grades, that exclusive college club and so on very highly. In hindsight, this was a bad career decision.

External validation is what sets apart sustainable, long term careers from short one-trick wonders. This mindset shift will be hard. And will probably be a long 5-15 year journey, and I’ll keep you posted - unless you want me to stop. Just let me know 🙂

Till then, Happy Diwali and a Happy New Year!

Best, Nirant

Strong Beliefs, Loosely Held

On Death

  • Is usually better the sooner it happens. Just like it’s best to retire from your cricket career at your peak for your own sake, it’s best to die when you are at your peak. You just might become Father of the Nation.

  • No one wishes that they’d accomplished less on their death beds. You might hear that no one wishes that they’d worked more. That’s right too. And there in lies the great human stupidity. You can’t hope to accomplish without working more. You might accomplish less despite working more.

This is also my answer for what I think is true, but almost no one agrees with me (from Peter Thiel’s Zero to One)

  • No one (with as many safety nets as me) would wish that they’d take less risks in their 20s. Cool people (like cool companies) are polite, go against less empathetic institutions and take risks (h/t Jeff Bezos)

On Purpose of Life

  • There is no higher purpose. Why would a microscopic life from a floating blue speck in the dark cold space have a higher purpose? It’s arrogant to assume otherwise

  • Serve the man next to you, then the one next to him and so on

  • Follow the Paradoxical Commandments, not because they help, but follow them anyway.

On Philosophy of Life

  • Options, choices are net bad. Decisions are good. Defaults are better. Burn your bridges. All-you-can eat buffets are scams. Having good looks or being visibly rich is a dating disadvantage because it creates options.

  • When in doubt, look for answers from the dead: Sufis, Stoics, Mystics, and Buddhists.

  • Eastern philosophy is more tolerable than Western philosophy. Here is how they contrast

  • Simple systems that fail at edges beat complicated systems that can handle edge cases. As in engineering, so in life. Regret minimization framework (h/t Jeff Bezos) seems like the best candidate and I am testing it as I write this uncomfortably honest piece

On Money

  • Time value of money is underrated

  • Money is like sex. It’s good to have a lot of it when you are young. That still won’t fill the hole left in your heart by not having a philosophy

  • If you’ve money, invest in your learning by paying for coffees with smarter people. Books, blogs, and those Coursera courses are a slow learning curve

On Work-Life Balance

An older version of this lives on Medium

ML Model Monitoring

Mayank asked on Twitter:

Some ideas/papers/tools on monitoring models in production. A use case would be say a classification task over large inputs. I want to visualise how are the predicted values or even confidence scores vary over time? (paraphrased)

Quick Hacks

pandas-profiling

If you are logging confidence scores, you can begin there. The quickest hack is to visualize with pandas-profiling: https://github.com/pandas-profiling/pandas-profiling/

Rolling means

Calculate rolling aggregates (e.g. mean, variance) of your confidence scores. pandas inbuilt. Quite quick. Add them to your set of monitoring and alerting product metrics.

A better version of this would be to do it on cohort level. Actually, doing all the following analysis on cohort level makes sense.

Confidence Scores and Thresholds

One of the most common mistakes is to use static threshold(s) on a confidence score(s).

If you hear someone saying that they do not use thresholds for a classification problem. Stop and think. They are using a threshold, usually 0.5 from within the ML library that you are using.

This is sub-optimal. The better option would be to use a holdout validation set and determine the threshold from that.

Tagging Data

It is obvious that you will tag the predictions for which the model is least confident -- so that the model can learn.

What you should also do is this:

  • Find out samples which have high confidence and tag them first, this is a form of negative sample mining

  • For multi-class classification: Figure out samples which did not clear your threshold, and the prediction is correct. Add these back to your new training+validation set

  • Tag samples which are too close to the threshold. This will help you understand your model and dataset's margin of separation better

Training-Serving

The most common causes of trouble in production ML models is training-serving skews or differences.

The differences can be on 3 levels: Data, Features, Predictions

Data Differences

Data differences can be of several types, the most frequest are these: Schema change - someone dropped a column!, Class Distribution Change - When did this 10% training class have 20% predictions, or Data Input Drift - users have started typing instead of copy-pasting!

Schema skew (from Google's ML Guide)

Training and serving input data do not conform to the same schema. The format of the serving data changes while your model continues to train on old data.

Solution? Use the same schema to validate training and serving data. Ensure you separately check for statistics not checked by your schema, such as the fraction of missing values

Class Distribution check with Great Expectations

Training and serving input data should conform to the same class frequency distribution. Confirm this. If not, update the model by training with updated class frequency distribution.

For monitoring these first two, check out: https://github.com/great-expectations/great_expectations

For understanding data drift, you need to visualize data itself. This is too data-domain specific (e.g. text, audio, image). And more often than not, it is just as better to visualize features or vectors.

Feature Viz for Monitoring

Almost all models for high dimensional data (images or text) vectorize data. I am using features and vectorized embedding as loosely synonymous here.

Let's take text as an example:

Class Level with umap

Use any dimensionality reduction like PCA or umap (https://github.com/lmcinnes/umap) for your feature space. Notice that these are on class level.

umap-tweet-plots

Plot similar plots for both training and test, and see if they have similar distributions.

Prediction Viz for Monitoring

Here you can get lazy, but I'd still recommend that you build data-domain specific explainers

Sample Level with LIME

Consider this for text:

lime-viz

Check out other black box ML explainers: https://lilianweng.github.io/lil-log/2017/08/01/how-to-explain-the-prediction-of-a-machine-learning-model.html by the amazing @lilianweng

Class Level

You can aggregate your predictions across multiple samples on a class level:

agg-lime-viz

Training Data Checks

Expanding on @aerinykim's tweet

Robustness

Adding in-domain noise or perturbations should not change the model training and inference both.

Citations and Resources

[1] Machine Learning Testing in Production: https://developers.google.com/machine-learning/testing-debugging/pipeline/production

[2] Recommended by DJ Patil as "Spot On, Excellent": http://www.unofficialgoogledatascience.com/2016/10/practical-advice-for-analysis-of-large.html

[3] Practical NLP by Ameisen: https://bit.ly/nlp-insight. The images for umap, LIME, and aggregated LIME are all from nlp-insight

[4] Machine Learning:The High-Interest Credit Card of Technical Debt: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43146.pdf