Marcel Krčah

Transition your career to big data

Published on , in #data-engineering

You have decided to do a career switch to THE BIG DATA, but you don't know how.

Here's what to do:

  1. Choose your role
  2. Build a sexy demo and job hunt guerilla-style

Step One: Choose your role #

There are four roles in this big-data world:

Role 1: Finding insights in the past #

This is a key role in jobs titled Data analyst and Marketing analyst. Also, it's a key prerequisite for Growth Hackers.

You answer questions about the past to help steer the business forward. For example:

You use BI tools (like Tableau, Looker) to interactively explore and visualize data. If you are more technical, you reach for SQL and/or R.

After a while, you start impacting metrics that you report. You use growth hacking and A/B testing. And you provably improve company's profit.

Learn the first steps by:

Role 2: Setting up & delivering a data warehouse #

This is a key role in jobs titled Data Engineer, ETL engineer and Hadoop developer.

Typically, your responsiblity is to combine multiple company data together from various sources.

For example, you combine customer data from Salesforce, support data from Intercom, product usage from a few production Postgres databases, real-time customer events from Kafka and you sprinkle it with historical weather information.

Then you create an abstraction on top of this data to provide a single source of truth. You operate between two departments:

Having all data together in this manner has HUGE benefits - all of a sudden, the time-to-answer drops from days/weeks to zero. Connect a self-service BI tool on top of it (e.g Looker, Metabase) and you are there.

Note that there are cloud solutions out there which provide the warehousing and data shoveling for you. Checkout Redshift and BigQuery for warehousing and Alooma, Stitch and Fivetran for shoveling.

You need to do the data merging yourself - that's the company domain. You do this either with SQL or a data processing tool:

Notes:

Role 3: Predicting future from past data #

This is a key role in jobs titled Data Scientist and Quants.

Your role is to predict behavior by learning from the past data:

You are the math guy in the company. You know machine-learning, simulations or numerical computing. Typically, machine-learning is required. This includes:

Daily bread is Pandas, Scikit-learn (my favourite), Matlab, R or SparkML. However, I know a guy in Booking.com who uses C++ for his machine-learning experiments.

Learn the first steps by:

Notes:

Role 4: Developing data-driven product features #

This is a common role in jobs titled Data Engineer.

You are putting into production features driven by prediction and machine-learning models:

If there is a dedicated data scientist, you work closely with him/her to translate their model to production code. If there is a growth-hacker, you work closely to implement their ideas.

Learn the first steps:

Choose your role by your current strengths: #

Don't worry too much with the initial choice. Especially in smaller companies, the roles overlap and you have freedom to move between the roles. I know a Data Scientist doing warehousing stuff for two years and a Data Engineer doing growth hacking.

Still not sure which role? [Drop me](mailto://m@marcel.is?subject=Which role to choose) an email.

Step 2: Build a sexy demo and go guerilla #

Do this:

For choosing the target company:

For the demo, here are a few examples:

Example 2 (for data analysts): Tailored growth-hacking recommendations #

Example 3: (for data engineers): Spark Streaming with real-time visualizations #

Example 4: (for data engineers): Click-stream processing with Kafka & Alooma #

Example 5: (for data scientists): Open-source your Kaggle solution #

Example 6: (for data scientists): Deliver tailored prediction #

FAQ: #

Shall I do pro-bono? #

Sure. If you don't feel confident about your current skills, do a pro-bono project. Reach to your network of friends, family and colleagues. Try to get a contact within your network that would tell you about their company problems and are willing to share their data. In exchange for your work (if done well), you'd ask for a referral that you can post on your LinkedIn/blog.


Enjoy the ride and [ping me](mailto:m@marcel.is?subject=Big data transition) with your transition story. I'm very much interested.

I'll leave the last words to the marketing guru Seth Godin:

Make something happen

If I had to pick one piece of marketing advice to give you, that would be it.

Now.

Make something happen today, before you go home, before the end of the week. Launch that idea, post that post, run that ad, call that customer. Go the edge, that edge you've been holding back from... and do it today. Without waiting for the committee or your boss or the market. Just go.

This blog is written by Marcel Krcah, an independent consultant for product-oriented software engineering. If you like what you read, sign up for my newsletter