
In the past month, two things happened to me that made me realize how glad I'm for using a password manager, namely [1Password](https://1password.com/).

First thing is about a login page. I was searching for a Github repo on DuckDuckGo, and I clicked on a link in the search results. As I was about to fill in the password with a 1Password shortcut, I noticed that 1Password wasn't filling in the login details. How strange, I thought. Puzzled, I looked at the url. And then I saw it—I almost fell for a phishing attack! The url was [github.com.cnpmjs.org](github.com.cnpmjs.org) instead of [github.com](https://github.com). I knew of phishing attacks, and thought I'd never fell for such a simple trick. Yet, I almost did.

Second thing is about email. As I'm migrating away from gmail, I thought I would check out the spam folder. I usually don't do that, but I thought I'd peek in. One email caught my eye: the email subject was my password. The site must have been compromised and the attacker got hold of my email and password. Fortunately, the damage radius was minimal, as I have generated unique passwords for each site.

![1password-email](/img/1password-email.png)


I'm glad for using 1Password


I have spent more time than expected trying to connect to AWS API Gateway with HTTP instead of HTTPS.

For the context, we have legacy IoT devices that make outbound connections via HTTP, not HTTPS. On the receiving side, we have a REST API service. The service, deployed via AWS API Gateway and AWS Lambdas,  supports only HTTPS, not HTTP.

So how to connect the two?

![aws-gateway-https-how](/img/aws-gateway-https-how.png)

It turns out you can use [AWS CloudFront Distribution][cf-guide] to receive HTTP and forward it to the API Gateway. In the most basic setup, it's [a few steps][cf-video]. [This StackOverflow question][so] contains more info.

![api-gateway-https-cloudfront](/img/api-gateway-https-cloudfront.png)

[cf-video]: https://www.youtube.com/watch?v=Dq2m2ZoZ234
[cf-guide]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-working-with.html
[so]: https://stackoverflow.com/questions/43236152/how-to-make-aws-api-gateway-accept-http-instead-of-https


I have spent more time than expected trying to connect to AWS API Gateway with HTTP instead of HTTPS.


Connecting to AWS API Gateway with HTTP


I've been facing the question of how to build a bidirectional near-real-time synchronization between two systems.  In the past years, I've seen multiple projects where data between two systems were synced bi-directionally. In these projects, the bi-directional sync was a cause of data loss, unresolved conflicts, and frustration. In one project, the company ended up with two employees tasked to manually copy data from one system to another.

So when tasked to build bidirectional sync, I'm concerned about these issues. To mitigate the concerns, here are the strategies I use: 

- Before building, I've found it worth investing time into [[conflict-resolution|understanding the underlying needs]] and organizational constraints and into communicating back concerns from the IT perspective. Then, to investigate different *business solutions* to the needs and concerns without bidirectional sync.
- If the sync is needed, we try to turn the bidirectional sync into two one-directional syncs. In one-directional sync, each table is assigned a system, and the system is the source of truth for that table. Any change in the source of truth for that table overwrites the table in the other system. We can do the same on a more granular level, that is, on columns or rows.
- If that's not possible, we look at the bidirectional sync as a problem of reaching *eventual consistency* in a *distributed system*. The eventual consistency can be resolved using [CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type), which are nicely explained [in this video](https://www.youtube.com/watch?v=vBU70EjwGfw). As far as I understand, CRDTs require tight control over the actors of the distributed system. We hit a dead-end if we cannot change the actor, such as if we cannot implement [the last-writer wins strategy](https://dzone.com/articles/conflict-resolution-using-last-write-wins-vs-crdts) within the actor. We encountered this dead-end when we needed to sync two SaaS products: we couldn't access the codebase, and their APIs didn't support the CRDT use-case. 
- If none of these are possible, the last resort I've found is to build the bidirectional sync as a standalone sync program. The program uses APIs, webhooks, or databases to fetch updates from the systems, resolve conflicts with business rules, and push updates back to the systems. However, the solution comes with a risk of data loss—if the data is updated *during* the time window when the sync runs. For such bi-sync to be delivered, the business needs to accept the risk of possible data loss. 



Building bi-directional sync between two system


Eliot Sober studies the role of altruism in evolutionary biology and psychology. In the paper [Kindness and Cruelty in Evolution](https://www.researchgate.net/publication/30528984_Kindness_and_Cruelty_in_Evolution) he observes that in evolution a group of altruists outperforms a group of self-oriented individuals.

Can we translate that to writing code? Let's compare two extremes:

- A team of altruistic devs: each dev writes code so that _other devs_ can understand the code easily. The focus is on clean code and helping others.
- A team of self-oriented devs: each dev writes code so that they, individually, deliver features as fast as possible. The focus is on immediate short-term gains.

Now, as years go by, which team, codebase and product has a higher chance of survival? And if so, what is it that incentives us to be self-oriented?


Clean code: can we learn from altruism & evolution?


I see more and more similarities between coding and writing. Here's one advice on writing:

> Concision isn't the number of words on the page; it's how long it takes readers to process what's on the page. You need to put readers in the equation all the time. Language is social.
>
> — Larry McEnerney (from [Writing Beyond the Academy](https://www.youtube.com/watch?v=aFwVf5a3pZM))

The same applies to writing code. Concision isn't about the lines of code or short variable names; it's about how long it takes fellow devs to process and understand code. You need to put fellow devs in the equation [[comments-with-domain-knowledge|all]] [[clean-code-can-we-learn-from-altruism-and-evolution|the]] [[simple-components|time]]. Code needs to be painfully obvious.

Code is social.


Coding and writing are alike


I started experimenting with adding domain knowledge to code comments. It turned out that such comments were super helpful to others. They even inspired others to write such comments themselves.  So if they are so useful, why is it that we don't write them more often?

## A bit of context: Say hello to hybrid inverters

To give a bit of context, I've been working on an application that calculates investment costs for photovoltaic (PV) systems. Specifically, I've been adding new investment logic for hybrid inverters. I knew what an inverter was: a device changing DC current to AC. However, I didn't know a thing about *hybrid* inverters & how they relate to investments.

So first, I needed to figure out what a hybrid inverter even was. I learned that if you install a PV system with a battery, you need a hybrid inverter. It turns out a hybrid inverter is a device that combines a solar power inverter & a battery power inverter into one. For the investment costs, a hybrid inverter relates to both the PV system & the battery system.

![hybrid-inverter](/img/hybrid-inverter.png)

## Updating the code & adding the comment

Once I had hybrid inverters figured out, I noticed it was easier to understand the functional requirement at hand. Also, I could better understand the existing code & I was quick to update it. I ended up with this Typescript code:

```typescript
const GetInvestmentCost = {
  forHybridInverter(/*...args...*/) { /*...body...*/ },
  forBatterySystem(/*...args...*/) {/*...body...*/},
  forPVSystem(/*...args...*/) {/*...body...*/},
}
```

Out of the total time I spent on this update, most of the time went to understanding the domain background: the hybrid inverters & their relation to investment. This knowledge, specific for this particular feature, hadn't been captured anywhere. I also noticed that the domain knowledge was loaded in my mind, readily available for usage. So I added what I knew to the code as a comment. As brief & to the point as possible. It took a few minutes.


```typescript
/*
 * If you install a PV system with a battery, 
 * you need a hybrid inverter. A hybrid inverter 
 * combines a solar inverter & battery inverter 
 * into one device.
 * 
 * Now, for investment cost, we put the hybrid 
 * inverter separately, as it doesn't exclusively 
 * belong to neither the PV system costs nor 
 * the battery system cost. The investment cost
 * for the battery & PV system excludes
 * the cost of the hybrid inverter.
 * */
const GetInvestmentCost = {
  forHybridInverter(/*...args...*/) { /*...body...*/ },
  forBatterySystem(/*...args...*/) {/*...body...*/},
  forPVSystem(/*...args...*/) {/*...body...*/},
}
```

It's not perfect, but it'll do. I aimed to bridge the gap between  the code & the domain. I assumed the next person looking at the code will understand the logic faster & thus make their change faster.

There's another thing I aimed for: the explanation & the code now sit right next to each other. As [Rich Hickey explains](https://www.infoq.com/presentations/Simple-Made-Easy/), when we put related things together, we understand the system more easily. And so we add features and fix bugs quicker. But if we put related things apart, such as the code and the domain knowledge driving the code, the system is harder for us to understand. This scatteredness of things, these *inconsistent* locations, is a form of complexity. It slows down our understanding of systems & speed of change.

## More examples

Encouraged, I added a bunch of other domain comments to the code I worked with:

```typescript
/*
 * With net metering, also called reverse-count metering, 
 * the meter rolls back when you inject energy back 
 * to the grid. The grid serves as a free & fully 
 * efficient virtual battery. 
 */
function NetMeteringCashFlow(/*...args...*/) { 
  /*...body...*/ 
} 
```

```typescript
/*  
 * With market valorization policy, you are reimbursed 
 * for the energy injected into the grid, according 
 * to the injection tariff. For the energy consumed 
 * off the grid, you pay the regular off-take tariff. 
 * Both the injection tariff & the off-take are specific 
 * to your grid operator.
 */
function MarketValorizationCashFlow(/*...args...*/) { 
  /*...body...*/ 
}
```


## Feedback from fellow developers

I was curious what other devs thought about this, so I sent them a few snippets and asked about their opinion. Their answers were so interesting that I asked them if it's ok to include them here. They agreed, so here they are:

> There are so many concepts and business rules to be changed relatively often, adding more context to the right place is incredibly helpful. However, I also think doing this is difficult and I'm very interested to see how well will the comments age with coming time.

When asked more about the difficulty, he replied:

> My personal hypothesis for why I haven't been able to write such comments more is that for me (and I suspect many other developers), writing is a different skill from coding. I'm much less experienced at it and it's difficult for me to switch back and forth between. Because of the "free form". It also feels much less forgiving and more permanent than code. <br/><br/>The remote work environment, like we have, with more asynchronous and written communication, may be very helpful in promoting this. In my previous jobs we preferred talking over writing, attempting to make all knowledge shared by implicit osmosis (extreme being pair/mob programming) rather than explicit knowledge base (extreme being almost complete asynchronicity like Gitlab culture).
> Like with anything interesting, there are constraints and trade-offs for both approaches, but the more I think about it, the more comments like these seem to worth pursuing and I'm excited to try it as well.

Another developer answered:

> I am all for it. The downsides are hardly there. There is a maintenance burden for the comments, but that is usually removing/editing here or there. It also has the upside of assisting code review.

And another one:

> I'm extremely in favor. I was looking for a value yesterday and was unsure of what it was from the ticket, and if it had a comment like the above, it may have been easier to decipher what that value actually did.

So it seems others find such comments also super helpful, but they see two possible issues: writing & maintenance.

## The difficulty of writing

The difficulty of writing resonates. I feel it too, especially since English is not my native language.  What has helped me is redirecting  attention to learning. It turns out that non-fiction writing is a skill I can try to improve. The best resource on non-fiction writing I have found so far has been [Steven Pinker's The Sense of Style](https://www.goodreads.com/book/show/20821371-the-sense-of-style) (thanks to Gergely [for this tip](https://blog.pragmaticengineer.com/on-writing-well/)).  The book is an eye-opener. It's an entertaining, practical & easy read that introduces basic concepts behind nonfictional writing. It puts logic & process & rules into the writing process. I like that.

Another strategy, thanks [Alexey](https://guzey.com/) for the tip, is to use bullet-points as they are easier to write.

## Comments maintenance

Now for the maintenance. Imagine I need to make a change to a complex code. So first, I need to understand the code. I try to get all the help I can get, such as looking into the code, the comments, reading docs & discussing with the team. Once I understand the code & the domain context, I can proceed and make the change. As I finish, I am about to move on to another problem. Now shall I invest a few minutes in writing (or updating) the comment or not?

No matter how I decide now, the cost of gathering  knowledge has already been paid. To assess if to comment or not, I need to consider only the cost of writing the comment, not the cost of obtaining the knowledge for the comment.

So let's say I decide to invest my time and write the comment as part of the change A.  With the comment, the next person updating the code with the change B will have most of the explanation they need already at hand. They wouldn't need to pay as much time as I did to understand the code. So for the two of us, the total return on my investment would be positive & high.

![time-saved-investment-yes](/img/time-saved-investment-yes.png)

What if I decide to save time & not write the comment? The upside is that I will deliver the change A faster since I am not writing the comment. The downside is that the next person would need to start from scratch for their change B. They would need to figure out all the heavy stuff again. So for the two of us, the total return on my investment would be negative.

![time-saved-investment-no](/img/time-saved-investment-no.png)

Now you may wonder: it's one change, does it matter? Perhaps not much for one change. But imagine more changes, more functions & more developers. The savings start to compound.

![investment-change-more](/img/investment-change-more.png)

A friend remarked:

> Taking the time to explain something for the next dev is usually even more of a good idea because the next dev to read the code will probably be yourself, so even when you feel selfish, you can think of it as a gift to your future self.

## Conclusion to this experiment

All in all, it seems to me that if we adopt domain comments throughout the codebase, we accelerate the engineering pace. Perhaps even significantly. So for now, I'll try to write more of them and see how it goes.


## Won't comments get out of sync?

It seems that domain comments don't change that often. A hybrid inverter's relation to a PV/battery system stays the same if we refactor imperative code into functional code or extract some code into a reusable component. Also, we can always try to encourage consistency, for example, via code reviews.

## Shouldn't good code document itself?

When discussing the domain comments with a friend of mine, he made this remark. I find it spot on:

> Yeah, if code is so clear that it needs no further comments, that is great. But honestly, if you look at code, how often does that encapsulate everything you've learned about the domain. If it doesn't, why not add your knowledge in the comments?

## Further reading

- Joel Spolsky, [Code Tells You How, Comments Tell You Why](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/)
- Donald Knuth, [Literate Programming](http://literateprogramming.com/index.html)


_Thanks a lot to Reinier, Peter, Andreas, and Julien for reading this & making it better._

I started experimenting with adding domain knowledge to code comments. It turned out that such comments were super helpful to others. They even inspired others to write such comments themselves.  So if they are so useful, why is it that we don't write them more often?


Adding domain knowledge to code comments


One of the hard things in building software, I realized, are decisions: Which tech debt to tackle? How to architect this feature? What to focus on?

> If you don't know where you are going, any road can take you there.<br/>
> — Lewis Carroll, from [Alice in Wonderland](https://www.goodreads.com/work/quotes/2933712)

What helps me to navigate the maze is an _architectural vision_. An architectural vision maps out what the applications should look like in the future from the architectural perspective. The vision should be reachable and reasonably challenging. For me, I'm excited when the vision is on the edge of my comfort zone—something to stretch my abilities. I've found that thinking in terms of one or two years from now is a reasonable timeframe: long enough to unlock substantial changes, short enough to be hands-on.

How does an architectural vision look like? Stripped to bare essence, it could be as short as a sentence. For example, a company has a CRM system causing them only headaches. They discover that customer relationship management is, in fact, their [[to-buy-or-to-build|core]]. So their architectural vision could be: _The CRM is built in-house_.

Another example: a company operates in two different lines of business. The IT architecture supports both lines. Because of the shared foundations, the team faces mounds of complexity and overhead. So the architectural vision could read: _IT systems are split into two separate buckets: systems serving retail customers and systems serving the infrastructure customers._

For me, such vision is a celestial compass, showing the path in everyday decisions. And with such a compass, [[kaizen|the days feel different]].

![architectural-vision](/img/architectural-vision.png)


Everyday decision making


I'm moved by the works of [Marshall Rosenberg](https://en.wikipedia.org/wiki/Marshall_Rosenberg). In the book [Living Nonviolent Communication](https://www.goodreads.com/book/show/14361179-living-nonviolent-communication),  Marshall tells a story of how he mediated peace negotiations between two African tribes at violent war.

In Chapter I, subtitled *Resolving Conflicts Peacefully and Powerfully*,  Marshall starts off by laying out  generic steps to resolve conflicts. He follows the principles laid out in his previous work, [NonViolent Communication](https://www.goodreads.com/book/show/71730.Nonviolent_Communication). We then peek into a counseling session, where Marshall uses the steps to mediate a married couple's life-long financial dispute. Within one session, the couple resolves the dispute to the benefit of both the wife and the husband. 

At this point, Marshall moves on to a different use-case—violent war. The move startled me: I didn't expect to see the steps from marriage counseling to be applied in peace negotiations. I'm wondering: *could all conflicts and tensions be resolved using  Marshall's approach?*

I include the peace negotiation story from the book:

<div class="blockquote">
	
>...As we were walking into the session, my colleague whispered to me, “Be prepared for a little bit of tension, Marshall. Three of the people in the room know that the person who killed their child is in that room.” It was very tense at first. There had been so much violence between these two groups, and it was the first time they had really sat down together.
>
> I started with the question with which I frequently start conflict resolution sessions in order to focus on people’s needs. I said to both sides, “I’d like whoever would like to speak first to say what your needs are in this situation. After everyone understands the needs of everyone else, then we’ll move to finding some ways of meeting the needs.”
>
> The chief from one side looked across the table and said, “You people are murderers,” and the other side responded, “You’ve been trying to dominate us. We’re not going to tolerate it anymore!”  ...
>
> I turned to the chief who had said “You people are murderers” and guessed, “Chief, do you have a need for safety and to be sure that whatever conflicts are going on will be resolved by some means other than violence?” The chief immediately said to me, “Of course, that’s what I’m saying!” Well, of course, he hadn’t said that. He’d said that the other person was a murderer and had made a judgment rather than expressing his needs. However, now we had his needs out on the table, so I turned to a chief from the other side and said, “Chief, would you please reflect back what he said his needs were?” ...
> 
> The chief responded to this man by asking in a very hostile way, “Then why did you kill my son? That started an uproar between the two groups. After things calmed down, I said, “Chief, we’ll deal with your reaction to his needs later, but at the moment, I suggest that you just hear his needs. Could you repeat back what he said his needs were?” He couldn’t do it. He was so emotionally involved in this situation and in his judgments of the other person that he didn’t hear what the other person’s needs were. I repeated the needs as I had heard them and said, “Chief, I heard the other chief saying that he has a need for safety. He has a need to feel secure—that no matter what conflicts are present, they’ll be resolved in some way other than by violence. Could you just reflect back what that need is, so that I’m sure everybody’s communicating?” He couldn’t do it. I had to repeat it two or three times before he could hear the other person’s needs. ...
>
>  After I spent this much time getting both sides to express their needs and to hear each other’s needs (this took close to two hours), another chief who hadn’t spoken jumped to his feet, looked at me, and said something very intensely in his own language. I was very curious about what he was trying to express to me with such intensity, and I eagerly awaited the translation. I was very touched when the translator said, “The chief says we cannot learn this way of communicating in one day. But he says that if we know how to communicate this way, we don’t have to kill each other.”
> 
> In fact, before I left that day, we had members from both tribes eager to learn this process that would allow everyone to hear needs behind whatever message was being expressed. I am happy to report that the war between the tribes ended that day.


</div>


Resolving conflicts peacefully and powerfully


In this post, we'll look at
- which aspects are required to develop a data-driven software product,
- how data scientists and data engineers fit these aspects,
- how to detect if your team needs a data engineer, and
- how to find a data-engineer for your team, if you need one.

## Three aspects for any data-driven product
Developing a data-driven software product is not only about analytics.
In fact, there are three aspects required for a product to succeed: Consulting, Analytics, and Automation.

Let’s dive into each of them individually.

#### Aspect #1: Consulting
Consulting is about
- translating the business problem into analytical terms,
- collaborating on a business-case, and
- validating outcomes with (internal) business clients.

It requires business-acumen and desire to understand the domain.

#### Aspect #2: Analytics

Analytics is about extracting insights from data to solve a given business problem. It requires in-depth knowledge in:

- preparing and cleaning the data/features,
- simple models like excel and data exploration/vizualization,
- complex models to reach out for if needed, like machine-learning, statistics and simulations.

The focus here is to demonstrate via a proof of concept that the analytical
solution adds value to the business.

#### Aspect #3: Automation

Automation is about optimising & preparing the product for long-term use.
The proof-of-concept is improved to production-level code, using  [software-engineering best-practices](/sebp).

Underestimating this step might result in:

- low confidence in the product from end-users, due to many unexpected bugs,
- an unexpected loss of production data,
- unexpected long-term service unavailability,
- inability to promptly add a feature/fix an issue.

Automation requires software-engineering skill set tailored for data-driven products.

## Data Scientists vs. Data Engineers
Data Scientists usually come from mathematical/statistics/operation-research/machine-learning background. Having business acumen, they are strong in both analytics and consulting. They live and breath for data insights and applying models to solve a business problem.

Data Engineers, on the other hand, come from software-engineering background. They live and breath for automation. They understand how to ship [high-quality production-level code](/sebp), including code readability, testability, architecture, DevOps, automated-deployment, robust ETL, etc.

Data engineers can also speed-up delivery of analytical parts by providing
technical support for data scientists. For example:

- create a discovery platform for data scientists (e.g. IPython notebooks automatically hooked to a cleaned data warehouse representing a single source of truth),
- provide infrastructure improvements for data scientists (e.g. a common Docker image with all data-science tooling installed),
- create shareable libraries, boilerplates, etc.

## Skills mapping: Summary

Here's a summary of expected expertize in the three discussed aspects
by role.

| Expected expertize      | Data Scientist | Data Engineer |
|------------------------|----------------|---------------|
| Consulting             |   :medal:  Strong     |      Basic      |
| Analytics              |   :medal:  Strong     |      Medium    |
| Automation/Engineering |      Basic     |     :medal: Strong    |

_NB: As applicable to any role, the more a data-engineer
knows about analytics and consulting, the better. And vice versea,
the more a data scientist knows about automation and engineering, the better._

## Do you need a Data Engineer?

This question should be simple simple to answer: Ask your team which activities they spend most of their time on.

If the answer includes mostly manual deployments, getting access to data, re-cleaning the data, code refactoring, application monitoring, dev-ops, fighting Spark/Hadoop/Kafka/Yarn issues, then you probably need an additional Data Engineer.

If the answer is modeling, feature-engineering, vizualisations, communicating
 with an internal customer, you are probably
not in need of additional Data Engineer.


## How to find Data Engineers

In the current job market, the demand for data engineers
exceeds supply. In this context, there seems to be two viable options
on how to get an additional data engineer to the team:

#### Option #1: Become an attractive workplace

Become an attractive workplace so that data-engineers come to you:
start open-source initiatives & analytical blogs,
strengthen conference presence, start organizing local meetups.

This option is the harder one but it pays off in long-term.

#### Option #2: Turn generalists into specialists

Alternatively, hire software-engineers who are generalists.
A strong generalist (e.g. a Python/Scala developer)
would grasp the required stack fast and would be a great
engineering complement to your existing team of data scientists.


## More resources

- [Infographics: Data Engineering vs. Data Science](https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic)
- [Video (8min): Data science and data engineering, the two sides of a dysfunctional marriage](https://www.youtube.com/watch?v=NFumbHzfk94&feature=youtu.be&lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BIi8eNkkySdSW2h8VZxMW7A%3D%3D)
- [The Data Science Hierarchy of Needs](https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007)
































Should you hire a data engineer instead of a data scientist?


If you are considering using Docker for data-science projects, in this post you'll find the conceptual advantages of migrating data-science/big-data/machine-learning projects onto Docker.

In the post, we start with the topic of team productivity.

### Exec Summary: Less technical and communication friction

Based on my experience, teams using Docker experience a decrease in technical and communication friction, namely due to the following advantages that Docker offers:

- New colleagues get onboard quickly
- System documentation is explicit and always up-to-date
- Docker eliminates most work-on-my-machine problems
- Docker enables transparent and peer-reviewed infrastructure updates
- Docker streamlines technical collaboration across teams
- Docker sharpens responsibility

These advantages come with a cost—the necessity to learn new tooling.

Let's dig into each of those individually.

### Advantage #1: New colleagues get onboarded quickly

With Docker, it is a matter of one command to get an application up and running. The ease of spinning up a complete application also holds for complex cases where an application requires custom binaries, tweaks in system configuration, a database, or a cluster-like setup.

With Docker, it is not required anymore to spend days getting an application running on a new laptop. I once worked on a micro-service-based application consisting of 20+ interconnected services. Docker allowed us to spin up the whole stack easily with just one command.

To learn more, check out `docker-compose`.

### Advantage #2: System documentation is explicit and always up-to-date

Docker uses files to define application system requirements and links between them. These files are human-readable, declarative, and define the desired end-state of an application. They are typically committed to a version control system. Thus, they serve as excellent documentation of all system and application dependencies.

With Docker, it is not necessary to manually document system requirements. The documentation is, by definition, always up-to-date and on-par with the corresponding application version because the definition of the system requirements is the documentation itself.

To learn more, check out `Dockerfile` and `docker-compose`.

### Advantage #3: Docker eliminates most work-on-my-machine problems

How many times have you encountered an issue that an application behaves correctly on one developer machine but misbehaves on another developer machine? Issues like different path separators, missing undocumented system configuration, missing libraries, and customer binaries (e.g., linear programming tools).

Since Docker packages the application together with all its system dependencies (operating system, binaries, configuration), all developers work with the same infrastructure.

Useful concepts: `Dockerfile`, `docker-compose.yaml`

### Advantage #4: Docker enables transparent & peer-reviewed infrastructure updates

Docker defines application infrastructure and dependencies in files. These files are typically committed to a version control system (for example, Git or Mercurial).

Thus, best-practices for team collaboration like peer reviews and pull-requests can be applied to Docker files as well. A whole team can thus transparently review any suggestions for any updates for system configuration, custom binaries, and libraries.

### Advantage #5: Docker streamlines technical collaboration across teams

Docker encapsulates everything needed to run an application into a single _Docker image_ and stores those images in a centrally accessible location. This mechanism enables a team to reuse an image developed by another team quickly.

Let's take a use-case where different teams concurrently develop a new set of services that depend on each other: a new version of a frontend coupled with a new REST API backend, for example. Each team would welcome having an option to quickly spin up the whole application stack (the frontend and the backend) locally for testing purposes. Docker enables such an option, for example, through docker-compose.

### Advantage #6: Docker sharpens responsibility

If you work for an older company where IT is not a core business, but rather, an optimization around a core business, chances are you will find yourself working in organizational silos. In this model, there's a lack of end-to-end responsibility for a product: one team develops a model, while another team deploys the model. This split leads, among other things, to high communication overhead, common technical misalignments (for example, different library versions between development machines and production environment), cumbersome application monitoring, and unclear support boundaries.

Ideally, these silos would be merged, and there would be a single team end-to-end responsible for a product. If that's not possible, Docker might partially help.

Docker allows shifting responsibility for application system requirements to the development team. At the same time, the responsibility of the deployment team would sharpen as well: Instead of supporting a wide range of applications half-way, the deployment team can be end-to-end responsible for deploying any application that is Dockerized.

### Disadvantage #7: Necessity to learn a new tool

Docker is an additional tool that team members need to have in their toolbelt. One option to learn Docker is to experiment on your own and invest time in exploring and learning the tool. However, team members need to find out what they don't know and learn it. 

### Bring Docker advantages to your team

An in-house Docker training specialized for data-science teams and geared towards your specific needs is an excellent option to learn Docker fundamentals effectively. ~~My Docker training~~ (Update from 2020: I no longer to these) offers high-level conceptual overview coupled with practical, focused, hands-on exercises. Also, your team would learn about hard-won lessons, pragmatic best-practices and conceptual high-level overviews which are all typically hard to find on the interwebs.

Why Docker for Data Scientists


For longer, I missed something in my communication, something between writing and talking. There's a whiteboard, that's not an option for me since I work remotely. Then there are online diagramming tools, but they felt cumbersome to use: it took me long to draw & I often couldn't find the shape I was looking for.

![drawing-writing-talking](/img/drawing-writing-talking.png)

Two years ago, I stumbled upon Doug's [course on sketch noting](https://www.verbaltovisual.com/our-courses/). I learned the basics of technical drawings, but it didn't occur to me I could use drawings for software-related concepts. Then a few weeks ago, I stumbled upon [Julia's blog](https://jvns.ca/) and her technical drawings. It turns out she wrote [a bunch of articles](https://jvns.ca/#on-writing-comics---zines) where she explains her drawing process. Inspired, I read them all. And then, the final drop: [Ryan's article on how he wrote ShapeUp](https://m.signalvnoise.com/how-i-wrote-shape-up/). I was hooked & determined to give drawings a try.

So I bought a used iPad 7 and an Apple Pencil 1st generation. I installed Notability: a simple note-taking app that both Julia and Ryan used in their beginnings. And I started experimenting.

A few days ago we finished a large refactoring project. I thought that would be a good time to experiment. So I decided to use drawings to explain the thinking process behind the big refactoring we did. Here are a few drawings from the talk:

![writing-first-steps-example](/img/writing-first-steps-example.png)

I didn't know what to expect. I was surprised to find out that my coworkers enjoyed the talk & they were quick to understand the driving principles behind the project. So it seems the drawings worked well. I also noticed I liked the process of making the talk: I could express myself better & it was fun.

It's hard to image now going back to the world without the drawings. Something clicked. I think I'm going to experiment with them more.

---

Some updates on the post after it was published:

- I stumbled upon [this interesting Twitter thread](https://twitter.com/GergelyOrosz/status/1337406214284652553) about various drawing apps for iPad. Also, if I ever need to make my own font, [here's a recommendation](https://twitter.com/b0rk/status/1337867237743595521).
- I was asked if I ever used pen & paper. I did when learning with Doug's video courses. However, I've discovered that I edit the drawings a lot, especially erasing & moving shapes around. For that reason, the digital version has been a huge time saver for me. On the other hand, one can [go pretty far](https://jvns.ca/blog/2019/09/01/ways-to-write-zines-without-fancy-tools/) with just a pen & paper. So to summarize: I started with pen & paper, the process resonated, so then I upgraded to digital.


For longer, I missed something in my communication, something between writing and talking.

Drawings for technical explanations: my first steps


I believe that one of the reasons why companies fail is a lack of focus and fear of simplicity. They focus on too many things at once. Have too many customer-specific features and processes. Want to squeeze a dollar from _every_ opportunity possible, not realizing the incurred debt.

Doing everything yet doing nothing properly.

I believe this is a road to disaster.

I have seen this multiple times.

Processes get complicated. IT development slows down. Products get harder to understand. More people are needed to handle the newly created complexity.

## Example 1: Building a generic solution instead of pushing for simplicity

Once, my team was given a task to build a generic pricing rule engine. The engine was supposed to support all the current and legacy business deals. Since the company had been lacking focus, the pricing had gotten out of hand. It got so complex that there was only one person in the whole company who understood the pricing intricacies completely. He tried to explain it to us many times, but the complexity was overwhelming. There were exceptions everywhere.

We should have paused and [not ignore the quenching](/quench). We should have pushed back to remove as much exceptions as possible. To simplify our pricing model.

This way, all parties would win:

- Pricing would be easy to explain and simple to understand.
- [Bus-factor](https://en.wikipedia.org/wiki/Bus_factor) would increase.
- Development would be quicker.
- Processes would streamline.

Instead, we have built the engine. It sure brought advantages. However, I believe that removing the root cause would bring benefits of another level.


## Example 2: Expanding B2B product with B2C capabilities

Another time, another team. We were given a task to expand our [B2B](https://en.wikipedia.org/wiki/Business-to-business) platform with [B2C](https://en.wikipedia.org/wiki/Retail) capabilities. The overlap between B2B and B2C features were minimal.  The B2B platform was hardly stable.

We [followed the thirst](/quench). Again. And things got very complex very quickly.

We were very slow. Discussions. Hacks.

It felt like fitting a circle in a square.

Instead, we should have focused on two things: polishing the B2B platform and start a green-field B2C product.

- The development would be faster
- Our end-users would understand the products easier.
- We would end up in two [loosely-coupled yet highly-cohesive](http://stackoverflow.com/questions/3085285/cohesion-coupling) products.
- Oh, and we would enjoy the hell out of it!


## Strive for focus and simplicity

Choose one thing and be the best at it. Stop marginal legacy business lines. Refuse niche requests from small customers. Simplify as much as possible.

Only when the product is saturated, expand. Do it with sharp focus. Create another product which will be the best at what it offers.

Simplify and keep focused!


Strive for focus & simplicity


Today, a colleague of mine recommended a [short video](https://www.youtube.com/watch?v=6XAu4EPQRmY)  by Jonathan Blow,  an acclaimed developer in the gaming industry. Jonathan mentions that sometimes it is better to make a hard decision later in the project rather than sooner. The later you decide: 

- the better you can explore and refine the idea, and 
- the better you know the technical things needed to build the idea.

I like that. I like to think of a project as having the start, the middle, and the end. For the start, I follow [[proof-of-concept|a process that works well for me]]. After the start, we move to the middle, where hard technical decisions often need to be made. Jonathan's approach gives me guidance on how to tackle such decisions with more control.

<div class="blockquote">

> Thing number one: If you get a rough draft of your program together, you can use that to figure out how you really want it to behave.  Some of your ideas about what you wanted to build in the beginning might not have been very good ideas. You can refine those ideas by having something approximating the thing that you were building.  And so the faster you get to that approximation the better. 
>
> Thing number two: The more time you spend working in that space of your approximation to the thing that you want, the more time you spend becoming an expert in the field of this specific application that you're making, the better you get at that subfield of programming, and the better your decision-making about technical issues in that field is going to be. 
>
> So if you make hard decisions later they will be made better, both by a more skilled person and with more contextual information than if you make those hard decisions early on.
>
>—Jonathan Blow, [On solving hard problems](https://www.youtube.com/watch?v=6XAu4EPQRmY)

</div>



Making hard decisions later


Let's talk about function inputs. Look at this Javascript function. What do you think it does?

```javascript
const signUp = (attrs) => {
  const user = saveUser(attrs)
  welcomeUser(user)
}
```

There are questions you might ask: How's the user saved? In the database? How's the user welcome? Do we send the email? Or do we, instead, send events to message queues?

Figuring out answers like this takes time. Sometimes surprisingly long. But it doesn't have to be. When writing code, we can [[clean-code-can-we-learn-from-altruism-and-evolution|help each other out]]: search for all global variables buried in the depths of nested functions and bring the variables up to the daylight for the reader to see.

```javascript
const signUp = (Db, Email, attrs) => () => {
  const user = saveUser(Db, attrs)
  welcomeUser(Email, user)
}
```

Now you made the function honest. All input is explicitly declared and [[coding-and-writing-are-alike|served on a silver platter]] to the reader. And you made the codebase [[simple-components|simpler]]. No more are the inputs scattered around, intertwined with different locations. Inputs now stand together in a single location, consistent. You also paved the way to simpler testing: no need for technical trickery and mocks of implementation details. All in all, the codebase is now [[sustain|easier to sustain]].

![honest-functions](/img/honest-functions.png)

To dive deeper into function inputs, checkout [Pure Happiness with Pure Functions](https://mostly-adequate.gitbooks.io/mostly-adequate-guide/content/ch03.html) in [Professor Frisby's Mostly Adequate Guide to Functional Programming](https://mostly-adequate.gitbooks.io/mostly-adequate-guide/content/).


Let's talk about function inputs. Look at this Javascript function. What do you think it does?


Honest function inputs


If you do not fully understand how Hive/Impala stores your data, it might cost you badly.

I've learnt the hard way.

## Symptom #1: Weird values in ingested Hive table

You double-checked with `select distinct(gender) from customers` that the `gender` column in your source RDBMS really contains only values `male`, `female` and `NULL`. However, when you ingest the table into Hive (maybe with [Apache Sqoop](http://sqoop.apache.org/) or [Cornet](https://github.com/datadudes/cornet)) and run the check there,  you see that weird values have creeped in:
```
> select distinct(gender) from customers;  // Run in Hive/Impala
+-----------------+
| gender          |
+-----------------+
| NULL            |
| CA 94304        |
| male            |
| Page Mill Road  |
| female          |
+-----------------+
```

## Symptom #2: Inconsistent size of ingested Hive table

You check with `select count(*) from customers` that the table in your RDBMS table has `156,010` rows.  You ingest the table into Hive and BAM! All of a sudden there are 14 more customers.

Maybe the business is doing great and you gained 14 customers before you started the ingestion? Wondering, you check the source table size again.

Still `156,010`.

## Symptom #3: Weird data when copied from another Hive table

This one is by far the most strange one.

You already have the `customers` table ingested in Hive. Values in the `gender` column look fine. The table has correct size of `156,010`. All is fine.

You do some data cleaning with SQL and copy the result into a new table as follows:

```
CREATE TABLE customers_superclean
AS SELECT name, coalesce(gender, 'unknown') FROM customers;
```

You check the size of the new table. BAM! 25 new customers in there. Impossible!


## No errors, no warnings

Neither Hive, Impala nor Sqoop gave you any error or warning. You have no idea what's going on. Somewhere at the back of your head, you start questioning the whole Hadoop infrastructure. You feel like the cool stuff you do all day with the data has been compromised.


## Cause: Hive delimiters present in the data

All these problems can occur if the ingested data contains characters that Hive uses to delimit fields and rows. Typically, these are newlines. For instance, let's assume that the `customers` table in the source RDBMS contains the following data (notice the newline in the first street name):

gender | street
---| --
male | Page Mill Road`\n`CA 94304
female | Great America Parkway

If one naively ingests this data into a Hive table using the default settings (a text file with rows delimited by `\01` and fields delimited by newlines), the data gets broken. Look at the corresponding delimited file in HDFS:

```
male\01Page Mill Road
CA 94304
female\01Great America Parkway
```

When Hive/Impala reads from this file, it finds three customers instead of two. The additional customer is of gender `CA 94304` and has no street specified. On top of that, the street field of the first customer misses the postal code.

## Particularly interesting case: Copying binary data to text file

Assume that the Hive table called `customers` uses Avro or Parquet for data storage and that the data contains newlines. Querying the `customers` table directly via Hive or Impala works as expected. However, let's create a new table as follows:

```
CREATE TABLE customers_superclean
AS SELECT gender, street FROM CUSTOMERS;
```

The problem with this command is that the new table is backed by a newline-delimited text file. An Avro/Parquet record which contains a newline will be split into two records in the new table. Symptom #3 is born.

## Solution 1: Use binary storage formats like Avro or Parquet

If possible, use binary storage for your Hive tables, for instance [Apache Avro](https://avro.apache.org/) or [Apache Parquet](http://parquet.apache.org/). Since these formats do not use dedicated characters to split a file into records and fields, Hive/Impala can read data with special characters properly.

Also, Avro and Parquet make it possible to safely copy records from one Hive table to another. For instance, checkout the following command which copies data to a Parquet table:

```
CREATE TABLE customers_superclean STORED AS parquet
AS SELECT gender, street FROM customers;
```

The Parquet storage will ensure that even if  data in the `customers` table contains newlines or another delimiters, the data will be properly copied and interpreted in the new table.

## Solution 2: Ensure the delimited text file does not contain Hive delimiters

When ingesting data into a delimited text file, you have to ensure that the file does not contain characters that Hive uses to split data into rows and fields. In general, there are two options to achieve this:

- Remove Hive delimiters from the data before ingestion. If you use Sqoop, there are handy parameters which can do that for you. Checkout the [Sqoop docs](https://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html#_importing_data_into_hive) and look for `--hive-drop-import-delims` and  `--hive-delims-replacement` parameters.
- Use custom Hive delimiters that are not present in the data. Unused [non-printable characters](http://en.wikipedia.org/wiki/ASCII#ASCII_control_characters) are good candidates, for instance `\01` or `\02`. You can instruct Hive [to use custom delimiters](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormat,StorageFormat,andSerDe) as follows:
```
CREATE TABLE customers
ROW FORMAT DELIMITED
  LINES TERMINATED BY '\002'
  FIELDS TERMINATED BY '\001';
```



Happy ingesting!

How a newline can ruin your Hive



What a great time it is nowadays for data geeks!

We have Pandas and Scikit-learn - fantastic Python stack for data analysis. On top of that we have IPython and IPython Notebook - powerful coding, documentation and visualization layer for experimenting.

Then we have the whole Hadoop stack with an amazingly fast Impala SQL query engine. We don't even have to build the Hadoop cluster in-house, we just choose the size and spin up a cluster via Amazon AWS and we're done.

**And now guys in Cloudera made [Impyla](https://github.com/cloudera/impyla) - Python connector to Impala.** And of course, they didn't forget to pack in an Impala connector for Pandas! How great is that?!

So, if you want to connect Pandas to Impala on Elastic MapReduce (EMR), here is how.

## 5 steps to connect Pandas to remote Impala


### Prerequisites

Install the awesome Pandas, Scikit-learn and IPython stack if you haven't done that already.

### Step 1: Install Impyla

```bash
$ pip install impyla
```

### Step 2: Create an SSH tunnel to Amazon EMR so you can access Impala from localhost

```bash
$ ssh -L 12345:localhost:21050 your_user_name@your_node.compute.amazonaws.com
```

The Impala query engine runs on port 21050 on your Hadoop master node. For security reasons, this port is not accessible from the outside.

This shell command will open up the port 12345 on your local machine and forward it to the port 21050 on the Hadoop master node where the Impala query engine listens. (Of course, you can choose whatever port you want, it doesn't have to be 12345.)


### Step 3: Connect Impyla to Impala via the tunnel
```python
>>> from impala.dbapi import connect
>>> conn = connect(host='localhost', port=12345)
>>> cur = conn.cursor()
```
Notice that we are connecting to `localhost:12345` which is (securely) forwarded to Impala on Amazon EMR.

### Step 4: Query Impala and convert the result into Pandas dataframe
```python
>>> from impala.util import as_pandas
>>> cur.execute('SELECT * FROM customers LIMIT 500')
>>> df = as_pandas(cur)
```
Note that the query result must be transported from remote cluster to your localhost. If the result is large, the download might take a while. You might want to check out the network traffic monitor on your system to see when the download is complete.

### Step 5. Enjoy Hadoop data in Pandas
```python
>>> from sklearn.ensemble import RandomForestClassifier
>>> clf = RandomForestClassifier(n_estimators=10)
>>> clf.fit(df[['age', 'gender']], df['lifetime_value'])
```

## Bonus: Impala with scikit-learn API

By the way, guys at Cloudera are now busy with implementing scikit-learn API in Impyla. They [already have](http://blog.cloudera.com/blog/2014/04/a-new-python-client-for-impala/) alpha implementation of linear regression, logistic regression and SVM ready. I'm quite excited where all this is going...


Analyzing Elastic MapReduce data with Python, Pandas and scikit-learn


Which version of the if-condition is easier for you to understand?

```javascript
// version A
if (!isValid(input) || input.command !== "foo") {
  ...
}
...

// version B
if (isValid(input) && input.command === "foo") {
  ...
}
...


```

Negation is a fascinating topic. It is explored by Steven Pinker in his book [The Sense of Style](https://www.goodreads.com/book/show/20821371-the-sense-of-style):

<div class="blockquote">

> Logically speaking, a sentence with a naysaying word like _not_, _no_, _neither_, _nor_, or _never_ is just the mirror image of an affirmative sentence. Saying that the integer 4 is not odd is logically the same as saying that it is even. If something is not alive, then it’s dead, and vice versa. But psychologically speaking, a negative statement and an affirmative statement are fundamentally different.
>
> A negative statement such as _The king is not dead_ is harder on the reader than an affirmative one, like _The king is alive_. Every negation requires mental homework, and when a sentence contains many of them, the reader can be overwhelmed.
>
> To hear or read a statement is to believe it, at least for a moment. For us to conclude that something is not the case, we must take the extra cognitive step of pinning the mental tag “false” on a proposition.

</div>

Can we apply that to the two `if` statements? To mentally comprehend version A of the if-statement, a fellow dev goes through five mental steps:

![if-vs-if-not-mental-steps-for-if-not](/img/if-vs-if-not-mental-steps-for-if-not.png)

To comprehend version B, they go through three steps:

![if-vs-if-not-mental-steps-for-if](/img/if-vs-if-not-mental-steps-for-if.png)

Because comprehension is shorter by two steps, version B is [[coding-and-writing-are-alike|easier on the mind]].

Here's another example, a config file of a static site generator:

```ini
# If true, the parser won't highlight inline code 
noInlineHighlight=false

# If true, the parser highlights inline code 
inlineHighlight=true
```


Which version of the if-condition is easier for you to understand?


The curious case of if-not



For those of you who haven't encountered it yet, [Azure DataMarket](https://datamarket.azure.com/browse/data)
is quite an exciting platform by Microsoft which provides standardized access
to plenty of interesting datasets. As of now, there are over 200 datasets  from broad range of topics, including weather, demographics, automotive, agriculture,  real-estate, etc. There might be a golden nugget for your bussiness hidden in there, so I would definitely recommend to go and [check the platform out](https://datamarket.azure.com/browse/data).

Recently, we have discovered such a golden nugget ourselves: details about every car sold in Netherlands in the past 15 years. A pretty exciting dataset, considering that the company I work for operates in the Dutch market of electric vehicles. What is more, the data is **free**, **updated daily** and comes shipped with a **REST API interface**. What more can we wish for?

To maximally leverage potential of the dataset, we ingest it into Hive, which allows us to:

- **run fast ad-hoc exploratory SQL queries** with [Impala](http://impala.io/),
- **explore the dataset** with BI tools,
- **enrich the data model** by combining car details with other data and
- **expand our dashboards** with insights about the car market.

## Avro-backed external Hive tables FTW!

Regarding the storage, we opted for an [Avro-backed](https://cwiki.apache.org/confluence/display/Hive/AvroSerDe) [external](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ExternalTables) Hive table. These types of tables rock, since:

- **Hive automatically infers table schema from the Avro schema**. There is no need to explicitly enumerate columns with their types, which is very useful if you deal with tables of many columns.
- **Changes in table schema are not an issue**. You just recreate the table with the new Avro schema and the [Avro schema evolution](http://blog.cloudera.com/blog/2011/05/three-reasons-why-apache-avro-data-serialization-is-a-good-choice-for-openrtb/) will automagically take care of properly reading the data stored with the old schema.
- **Data is easily processed by other tools** like Apache Spark or MapReduce. The data resides in a directory of your choice (rather than in a Hive directory) and the Avro format has first-class support in Hadoop tooling.
- **New data is easily added to the table**. Just add a new Avro file to the corresponding HDFS directory and Hive will automatically pick it up. With Impala, call `REFRESH <table-name>;` to notify Impala about arrival of new data.
- **[Hive data partitioning](http://blog.cloudera.com/blog/2014/08/improving-query-performance-using-partitioning-in-apache-hive/) is well supported** so you can optimize for query performance.
- **Dropping external Hive table does not delete data from HDFS**. Small thing, but very handy.


## Ingestion: From OData XML to an Avro Hive table

Here are detailed steps how we get that data from DataMarket into Hive.

### Step 1: Download the dataset as XML files, in a standardized OData format

DataMarket publishes datasets as XML. Each XML follows the same [OData schema](http://www.odata.org/documentation/odata-version-2-0/atom-format) (checkout the [example XML](https://api.datamarket.azure.com/opendata.rdw/VRTG.Open.Data/v1/KENT_VRTG_O_DAT) with cars data). DataMarket provides this data via a REST interface, which supports various [query parameters](https://msdn.microsoft.com/en-us/library/gg312156.aspx). Using the parameters, you control the subset of data you want to download.

Since the car dataset that we are interested in is about 15 GiB large, filtering proved very useful for our purposes. With the `$filter`, `$top` and `$inlinecount` parameters and a bit of `curl` and Bash, it is straightforward to download the whole dataset into files `cars_2000.xml` till  `cars_2015.xml`, where the number indicates the year where a car was registered in Netherlands.

### Step 2: Convert the dataset into Avro, using `odata2avro` utility

To convert an XML file in OData schema to Avro, we created a Python command-line tool called [odata2avro](https://github.com/datadudes/odata2avro) which does all the heavy lifting for you.

Just install the tool with `pip install odata2avro` and use it as follows:

```
$ odata2avro cars_2013.xml cars.avsc cars_2013.avro
```

This command reads an XML file `cars_2013.xml` and creates two files:

- `cars.avsc` - an Avro schema, in json, describing the dataset,
- `cars_2013.avro` - a binary Avro file containing the dataset from the XML.


### Step 3: Upload the Avro schema and Avro files to HDFS

In this case, let's create and upload the data to `/datamarket` HDFS directory:

```
$ hdfs dfs -mkdir -p /datamarket/cars
$ hdfs dfs -put cars_20*.xml /datamarket/cars
$ hdfs dfs -put cars.avsc /datamarket
```

### Step 4: Create an external Avro-backed Hive table


To create an external Hive table with a schema according to  `/datamarket/cars.avsc` and the data located in `/datamarket/cars`, use the following Hive command with the [AvroSerDe](https://cwiki.apache.org/confluence/display/Hive/AvroSerDe):
```
CREATE EXTERNAL TABLE cars
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/datamarket/cars'
TBLPROPERTIES ('avro.schema.url'='hdfs:///datamarket/cars.avsc');
```

### Step 5: Query & profit!

Congrats! As of now, the data is accessible in both Hive and Impala:

```
$ impala-shell --query 'select * from cars;'
```

## Bonus step: Keeping data updated

The *car* dataset is append-only, so for us it's pretty straightforward to keep the data in Hive updated. We run a daily job which:

- downloads XML data from DataMarket for the current year 2015,
- converts the XML data to an Avro file called `cars_2015.avro`,
- uploads the Avro file to HDFS and replaces the file uploaded on the previous day and
- calls `REFRESH cars;` so Impala registers the new data.

That's all!

How to ingest data from Azure DataMarket into Hadoop


Apache Kafka, Mesos, Hadoop/YARN, Neo4J, HBase, Solr. All of these services
(and [many others](https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy))
are built on top of Apache ZooKeeper. ZooKeeper is also a part of
all Hadoop distributions (e.g. [Cloudera CDH](http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html))
and is used in many companies, including LinkedIn, Twitter, Netflix or Yahoo.

So what is ZooKeeper all about and why is it that popular?

## In summary, ZooKeeper helps to build a distributed system

Following the Unix philosophy of small yet powerful tools,
Zookeeper is a service that allows distributed processes to coordinate with each other.
Instead of developing such a coordination system from scratch, you can re-use ZooKeeper
and benefit from its best-practice proven implementation.

Distributed systems use ZooKeeper for service discovery, cluster monitoring,
configuration management, leader election or naming service. Also, ZooKeeper
provides foundation for building higher-level distributed functions like locks, barriers,
queues, two-phase commits, etc.

Or, as the Apache Zookeeper PMC [puts it](https://www.hakkalabs.co/articles/apache-zookeeper-introduction/):
> If you are doing distributed locking and you are not using ZooKeeper, you are crazy - Camille Fournier, Apache ZooKeeper PMC

## ZooKeeper fundamentals: Z-nodes

ZooKeeper itself is a distributed highly-available service that manages
a shared hierarchical name space of data registers, called **z-nodes**.

**Z-nodes** very much resemble a filesystem. Each z-node has a path, e.g. `/services/product`, and data, given as `byte[]`. However, unlike directories in a filesystem, parent z-nodes can also carry data.

Here is an example of z-nodes hierarchy:
![Illustration of Zookeeper z-nodes]({filename}/images/zookeeper-znodes.jpg)

ZooKeeper provides a simple yet powerful [API](https://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/ZooKeeper.html)
for z-node management. You can *create*, *get*, *update* and *delete* a z-node, ask if it *exists*
and *get children*.

The interesting part, however, lies in the **consistency guarantees** that ZooKeeper provides:

- **Sequential consistency** -  Updates are applied in order they are received by ZooKeeper
- **Update atomicity** - Updates are either successful or failed. No partial results
- **Single system image** - A client sees the same view of the service regardless of the ZooKeeper server it connects to
- **Reliability** - Once an update has been applied, it will persist from that time forward until a client overwrites the update
- **Timeliness** - The clients view of the system is guaranteed to be up-to-date within a certain time bound

I hope that by now you start to feel the power of ZooKeeper. However, this is not all.

## Persistent, ephemeral and sequential z-nodes

Regarding persistence, ZooKeeper offers two types of z-nodes: **persistent** and **ephemeral**.

**Persistent z-nodes** are the default z-nodes, they exist until they are explicitly deleted.

**Ephemeral z-nodes**, on the other hand, are attached to a client session. If a
client dies, the session is terminated and the ephemeral z-done is deleted. How great is that!
You can very easily monitor clieant failures: If a client responsible for
performing an exclusive operation fails, the corresponding ephemeral z-node is deleted and other clients
are notified to take over the operation.

Both persistent and ephemeral z-nodes can be also marked as **sequential**.

For **sequential z-nodes**, ZooKeeper automatically appends a monotically increasing counter to the end of a path.
This is greatly helpful for synchronization: if more clients want to get a lock on a resource `A`,
they can each create a sequential z-node on path `resource/A`. The client getting the lowest number is entitled to the lock.

## Watches

ZooKeeper also offers a mechanism called **watches**.
A watch is just a one-time callback, which is triggered every time
a z-node changes. Watches are one-shot - if you need to continually monitor a z-node,
you need to reset the watch after each event.

Watches are very useful to if you don't want to periodically poll your z-nodes.


## Fault-tolerant, highly-available, read-performant

Zookeeper can run in both stand-alone and distributed fault-tolerant mode.
When running in a cluster, a group of ZooKeeper servers is called an **ensemble**.

Regarding node failures, ZooKeeper **tolerates loss of ensemble minority**
(there is [Paxos](http://en.wikipedia.org/wiki/Paxos_%28computer_science%29) under the hood).
So if the ensemble consists of three or four machines, ZooKeeper will tolerate failure
of only one machine in both of these cases. This is why it is recommended
to run the ZooKeeper ensemble on **an odd number of hosts**, since even number
of machines doesn't bring any benefit with respect to node failures.

ZooKeeper data is kept in memory and is backed up to a log for reliability.
It is optimized for read dominant workloads, handling up to 50k operations per second.

## Clients

ZooKeeper clients can connect to any of the ensemble members and maintain a connection.

ZooKeeper is build on Java, but there is a pretty [solid list](https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings) of client bindings,
including Java, Scala, Node.js, Erlang, Haskell, Python, C#, Go and Ruby.

The most popular client is [Apache Curator](http://curator.apache.org/) (former LinkedIn project).
Definitely checkout the [Curator support for high-level recipes](http://curator.apache.org/curator-recipes/index.html),
it's a very interesting read.

## Use-cases

Here is a couple of examples how ZooKeeper is used in practice:

Twitter uses ZooKeeper for **service discovery**. Each application instance
registers itself to ZooKeeper using an ephemeral z-node. In this way,
ZooKeeper can maintain an up-to-date list of running instances for each type of service.
Clients then query ZooKeeper to locate application instances.
In general, ZooKeeper might help you to build a
**micro-service infrastructure** or manage a network of **replicated
(REST) services**.

[Apache Solr](http://lucene.apache.org/solr/) uses ZooKeeper for **leader election, centralized configuration** and **cluster management**. ZooKeeper enables application servers to **bootstrap configuration**
from ZooKeeper as soon as they join the system and to keep the configuration up-to-date.


Cluster monitoring can be implemented as members registering to `/members/host-{i}` and
periodically updating the z-node with their status (load, memory, CPU etc). Each z-node update
then triggers an alert to z-node listeners using watches.

[Apache Kafka](https://www.google.cz/search?q=apache+kafka&rlz=1C5CHFA_enCZ546CZ546&oq=apache+kafka&aqs=chrome..69i57j69i59l2j0l3.1551j0j9&sourceid=chrome&es_sm=91&ie=UTF-8), a popular publish/subscribe system with persistent queues,
uses ZooKeeper to store client's last consumed
offset, to register Kafka brokers and to help load balance requests among live brokers.

[Apache Hadoop](http://hadoop.apache.org/) uses ZooKeeper for automatic failover of Hadoop HDFS Namenode and for high-availability of YARN resources.



Pretty amazing!

## Where to go from here

I was quite eager to experiment with the ensemble, so I made
this [automated script](https://github.com/mkrcah/virtual-zookeeper-cluster) which creates and provision a virtual 3-node ensemble on your local machine with one command using Vagrant and Ansible. [Check it out](https://github.com/mkrcah/virtual-zookeeper-cluster).

Also, if you want to dive deeper, I'd recommend to:

- Read the [Getting started guide ](http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html) on the official ZooKeeper website
- Checkout the [Zookeeper recipies](http://zookeeper.apache.org/doc/trunk/recipes.html)
- Bookmark the [curated list of Zookeeper presentations](https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeperPresentations)
- Read a use-case of [building a distributed concurrent queue in Zookeeper](http://blog.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/)
- Checkout [Grid Computing with Fault-Tolerant Actors and ZooKeeper](http://www.ebaytechblog.com/2012/09/13/grid-computing-with-fault-tolerant-actors-and-zookeeper/#.VKLmpsAAE) - eBay tech blog

**Happy ZooKeeping!**


Introduction to Apache Zookeeper, backbone of big-data systems


There is a section in the interview with [John Danaher](<https://en.wikipedia.org/wiki/John_Danaher_(martial_artist)>) that I've been thinking about a lot.

<div class="blockquote">

> I'm a huge believer in the idea of small progressive movements towards goals. The vast majority of our days are unexceptional mundane days. There's a chance you can drift through your life when only 1-5% of your days have any real meaning and that 95% of your life was a waste of time. And that's a tragedy.
>
> If we make a concerted effort to build one day upon another, even if it's only a very small thing (and in most cases, it will be a very small thing, it's rare when we have a day where something monumental happens, most days are not monumental, they are mundane), and we do this over a period of ten years, something remarkable can happen. You get this compounding interest effect where at the end of five or ten years you may literally have reinvented yourself.
>
> — John Danaher, from an [interview](https://londonreal.tv/john-danaher-the-philosophy-of-martial-arts-the-man-who-inspired-me-to-learn-brazilian-jiu-jitsu/)

</div>


The spirit of kaizen


I stumbled upon an interesting way to visualize things. In the book [An Invitation to Applied Category Theory](https://www.researchgate.net/publication/334292752_An_Invitation_to_Applied_Category_Theory_Seven_Sketches_in_Compositionality), they have these fascinating diagrams. There's something I really like about them.

- Basic schema (page X.) ![invitation-to-applied-theory-lemon-pie](/img/invitation-to-applied-theory-lemon-pie.png)

- Different Styles of wiring (page 43) ![invitation-to-applied-theory-different-styles.png](/img/invitation-to-applied-theory-different-styles.png)
- Keeping track of waste (page 49) ![invitation-to-applied-theory-waste](/img/invitation-to-applied-theory-waste.png)
- Copying (page 50) ![invitation-to-applied-theory-copying](/img/invitation-to-applied-theory-copying.png)
- Collaborative Design (page 118) ![invitation-to-applied-theory-collaborative-design](/img/invitation-to-applied-theory-collaborative-design.png)
- Databases (page 79): ![invitation-to-applied-theory-databases](/img/invitation-to-applied-theory-databases.png)


Preparing a lemon pie: visualization


I use [dotfiles](http://dotfiles.github.io/) to configure my environment.
I had the dotfiles stored in a git repo and used helper scripts to create symlinks in my `$HOME` directory. However, adding & updating the dotfiles felt cumbersome, like a pebble in a shoe.

Recently, I stumbled upon [this article](https://www.atlassian.com/git/tutorials/dotfiles), which was inspired by [this HN comment](https://news.ycombinator.com/item?id=11070797). They mention something called a _bare_ git repo to manage dotfiles. I have never heard of it & it took me a while to figure it out. So I thought I will share it here.

## How a typical git repo looks like

Typically, when you initialize or clone a git repo, it consists of two parts: _the working tree_ and _the [ `.git` directory](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain#ch10-git-internals)_. The working tree contains things like source code and the README. The `.git` directory contains [git internals](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain), such as commit objects, local git config, and hooks. For example, when you checkout a different branch, git updates the _working tree_ based on data stored in the `.git` directory.

![bare-repo-typical](/img/bare-repo-typical.png)




## Cumbersome symlinking

Now, when symlinking dotfiles, the working tree for dotfiles is located in the dotfiles git directory, not in `$HOME`. So to make dotfiles work, one needs to create symlinks. And the symlinks need to be managed. That's the pebble.

![bare-repo-symlinks](/img/bare-repo-symlinks.png)

You may wonder why not initialize a git repo directly in  `$HOME`? If git doesn't find the `.git` directory in the current directory, it continues the search in parent directories. So if a directory is not versioned, git would still assume we are in the dotfile git repo. That could be confusing & prone to errors.

If only there would be a way to put the  dotfiles working tree directly in `$HOME`, while keeping the `.git` directory separated.

## Say hello to bare repositories

It turns out there is a way: a _bare_ git repository. A bare git repo is a repo that contains only the content of the `.git` directory, not the working tree. When interacting with git, you specify the working tree directory as a command-line parameter. That's pretty nice.

![bare-repository](/img/bare-repository.png)

Technically, it's done like this.

```bash
$ DOTFILES=$HOME/projects/dotfiles.git

# init the repository with the --bare flag
$ git init --bare $DOTFILES

# setup a git alias which sets 
# the .git directory to $DOTFILES and
# the working tree to $HOME 
$ alias gitdf='git --git-dir=$DOTFILES --work-tree=$HOME'

# for gitdf status don't show untracked files
$ gitdf config status.showUntrackedFiles no

# making a change & commiting a dotfile
$ echo "syntax on" >> .vimrc
$ gitdf add .vimrc
$ gitdf commit 
```

I've been using this setup for a few days now, and it feels smoother. I like it.

## Further reading

Here I focus on the high-level concept. If you want to read more about the technical workings, checkout [this](http://www.saintsjd.com/2011/01/what-is-a-bare-git-repository/) and [this](https://www.ackama.com/blog/posts/the-best-way-to-store-your-dotfiles-a-bare-git-repository-explained) post.

Additionally, it turns out that git supports [multiple working trees](https://git-scm.com/docs/git-worktree) in one git repository. See the `git-worktree`. Interesting, I haven't used it yet.


## Side note on complexity

I now try to see everything through the eyes of [Rich Hickey's Simple Made Easy](https://www.infoq.com/presentations/Simple-Made-Easy/) & to spot complexity. The dotfiles use case is, perhaps, more of a theoretical exercise, but I find it valuable nevertheless. By doing thought experiments like this, I aim to solidify my understanding of complexity.

So, in this case, it seems that the symlinking approach complects two things: the dotfiles (the `.gitconfig`, the `.zshrc`) with the dotfile management (symlinks). These are two different concepts, intertwined in one repository. With a bare repository, we remove the complexity: the dotfiles are taken care of by the working tree, while dotfiles management is taken care of by git.

![bare-repo-complexity](/img/bare-repo-complexity.png)

Another form of complexity is the imperative code for symlinking: first symlink that, then symlink that. At least that's how I had it coded. Since imperative programming complects _how_ with _what_, it's complex. With a bare repository, we remove this form of complexity: the dotfiles become declarative; it's only the _what_.

There is one tradeoff, though. We introduce complexity in the form of inconsistent interaction with git. For dotfiles, we use an alias, such as `gitdf`; while for all other repos, we use `git`. For me, this complexity is negligible compared to the benefit of simplifying the symlinking process.


I use dotfiles to configure my environment. I had the dotfiles stored in a git repo and used helper scripts to create symlinks in my `$HOME` directory. However, adding & updating the dotfiles felt cumbersome, like a pebble in a shoe.

Managing dotfiles with a bare git repo


"We are going to implement Hadoop in the company."<br/>
"OK. Just to understand better, which problem are we trying to solve?"<br/>
"We want to have a single source of truth for all our data."<br/>
"OK, I understand. Which problem exactly is Hadoop going to solve?"<br/>
"The CTO went to the conference and heard that Hadoop was the way we should do things now."<br/>
"OK. Do we know which exact problems would Hadoop help us solve?"<br/>
"Hm... that's a good question."<br/>

Another example: <br/>
"We are going to put parameters of this machine learning model in the configuration file."<br/>
"OK. To understand, which problem would this solve?"<br/>
"It will make the model easily modifiable by business-users."<br/>
"I understand. Are there any business users who would use it? Which problem would we solve by that?"<br/>
"Currently, it's just us developers touching the code, but it might happen in future."<br/>
"Hm, OK. So is there an actual problem we are solving by doing this?"<br/>
"Hm... good question."<br/>

<br/>

The purpose of teams is to solve problems. Each problem has a _root cause_ which triggers
complications to a _customer_. A customer is an actual person or a group. A customer can be internal
within the company or external.

Without a clear understanding of the root cause and the customer,
a problem cannot be solved effectively. Without a root cause, we might
be solving a non-existent or a wrong problem. Without a customer, it's not possible
to validate if the solution resolves the problem.

Do you know the root causes and the customers of projects you currently work on?


Which problem would that solve?


The start of a project, I've noticed, often determines the pace for the whole project. I discovered that what works well is to start with a proof of concept that is _embarrassingly simple_. "Embarrassingly simple" clarifies what to expect: a half-baked barely-working solution. "Embarrassingly simple" is also kind of funny when said out loud, and it sets a good mood within the team. To me, a recovering perfectionist, "embarrassingly simple" also crystalizes what to build and what to exclude.

Here's an example: We needed to integrate system A with system B by periodically calling the APIs. To make the proof of concept embarrassingly simple, we excluded tests, devops and monitoring, we ignored edge cases and unhappy paths, and we were pretty relaxed about code quality.

We ended up with a manually-triggered script that reached out to system A, extracted one field from one object, and stored the field in system B. We could have run API calls manually, but a bit of automation is a nice touch for demos. We also aimed for high impact, so we asked which object and which field have high priority.

![vertical-slice-of-functionality](/img/vertical-slice-of-functionality.png)

Cutting corners where possible, the concept was delivered quickly. Clients like that, it turns out. We also learned a ton about the systems:

- we discovered the need for a sandbox environment,
- we figured out API access and ironed out surprising API errors,
- we learned the foundations of the system's data models,
- we learned how to map the models for one specific use case,
- we discovered gaps in requirements that we didn't know before.

We reached the first deliverable, we set the pace, and the client is satisfied. Now, it's about keeping this up through the middle to the end of the project.


Embarrassingly simple proof of concepts


We, software engineers, love to build.

The thrilling sensation of adding a feature or creating something new from scratch. The excitement of using the latest technology that we've been reading about on Hacker News for so long.

And thus, we build.

We dive into coding, into architecture, into design. Quenching another dose of the thrill.

I am guilty of this.

Yet, I came to believe that blindly following the thirst is by far the largest productivity killer. I believe that coding should be considered the very last resort, used only when all other options are exhausted.

Let me explain.

Development is very expensive. It takes a long time to finish. It requires future time investment. It requires maintenance and bug fixes. There is an opportunity cost, and there is a probability that our work won't be used in production, because the priorities have shifted.

No matter the team, the agile process or the technology stack, the development will always be this way - resource intensive. Coding slows down the company, drains the IT budget and creates technical debt for future.

So before blindly jumping into coding, take a step back. Understand arguments why the problem or the feature at hand needs to be built in the first place. Find if there is a way to simplify the problem. Find if there is an existing solution in-house or off-the-shelf that can be reused to solve it.

Fight that internal thirst as much as possible.

Only as the very last resort, only when all other options fail, start thinking about the very minimal coding solution to the problem you have and very cautiously take that first sip.


Resist the quench for coding.



I came across an interesting problem a couple of days ago. A friend of mine runs quite a large website with a lot of users registering on their web. The problem is that there are users which are lazy to enter proper information so they just enter random text, as this for example:

![Example of randomly typed words]({filename}/images/random-text.png)

Since the correct registration information is crucial to their business, he would like to have a solution which would detect such random text automatically.

It would be very useful for him to have a tool which would check all data gathered from registration forms and mark rows which look like  random text. He calls this *false user registrations*. Also, it would be interesting to have an online detector which would notify the customer-care team if such a false registration is submitted.

Although there are many approaches to this problem, I was curious if such problem can be solved algorithmically. In particular, **I was curious what would be the most simple solution that would be useful in 80% of the cases.** The reason is that when I am presented with a (data) problem, I always find it very effective to start with the most simple solution first. And often these solutions turn out to be satisfactory enough.

Anyway, I was first tempted to go with supervised algorithms, like Naive Bayes classifier for example. However, I realized that it would be much more interesting to come up with an unsupervised algorithm, which would identify random words automatically without any prior knowledge. **Just give the algorithm a list of words and it gives you back words from the list which look random.** Such solution would also be practical from the business perspective since you can immediately identify suspicious registrations in a sea of production data.

So here is a description of a very simple - yet surprisingly effective -  unsupervised solution to this challenge. Based on frequency-analysis of character bigrams.

## Before we dive in, let's the see results

For the sake of simplicity, I focused on the field of the registration form that contains a customer name. To mock up production data, I took the 2000 most common names in US and added 20 random texts on top of the list. I got [this list of words](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-input.txt), which I gave as input to the algorithm. (The order of words doesn't matter.)

What the algorithm does is to compute a *suspicious score* for each word. The higher the score, the more suspicious (i.e. random) the word looks when compared to the rest of the words.

If you run the algorithm, you get [this list of words sorted by a suspicious score](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-output-adjusted-idf.txt). I marked the 20 random words with an asterix, so I can quickly evaluate quality of the result.

We see that the algorithm is pretty effective: **the top 10 words with the highest suspicious score (above 25.0) consist of random words only.** Actually, the top 16 words consist of 15 random words and only one regular name - *guadalupe*.

The five random words which didn't make it to the top 20 include *tyrwereh* (#23), *qwerpo* (#43) and *sfert* (#422). However, I'm ok with the low scores here, because these words don't look very random to me anyway :)

I have made also an interactive version of the algorithm, it is [available here](http://random-text-detector.herokuapp.com). Go ahead and give it a try. *(Note that it might take up to 10 seconds to load the page because the Heroku node might need to wake up.)*


## How it works

If you look at a random word, e.g. *sdfjtwd*, you see a lot of rare combinations of characters like  *sd* or *ft* or *jt*. If you look at a  word *dustin*, there is no combination of characters which strikes you are rare. Maybe *st*, but that's about it.

A combination of two adjacent characters in a word is called a character bigram. So *sd*, *ft* and *jt* are all bigrams of the word *sdfjtwd*. (By the way, there are also *word-bigrams* which are adjacent words in a sentence.)

What you want to do to detect random text is the following:

1. Compute a score for each bigram. This score should be high if the bigram is rare among all given words. The score should be low if the bigram is very common among given words.
2. Compute suspicious score of a given word as a sum of all bigram scores that are contained in the word.

## Step 1 - Compute bigram scores

Consider the words *aabaa*, *abbb* and *ababa* for example. Let's compute for each bigram how many times it occurs in each word and also how many words in total contain this bigram:

Word / Bigram | aa  | ab  | ba  | bb
---   | --- | --- | --- | ---
aabaa | 2   | 1   | 1   | -   
abb   | -   | 1   | -   | 1   
ababa | -   | 2   | 2   | -   
**Total words containing the bigram** | **1** | **3** | **2** | **1**

We see that the bigram *aa* occurs twice in the word *aabaa*, while bigram *bb* doesn't occur at all in *aabaa*. Also, we see that the bigram *ab* is the most common bigram (used in all 3 words) and the bigrams *aa* and *bb* are the most rare (used only in one word).

Now we know which bigrams are rare and which are common in the dataset. We just need to translate it into a score.

The most popular metric for this purpose is the [inverse document frequency (IDF)](http://en.wikipedia.org/wiki/Tf–idf). For a given bigram $b$, the IDF is computed as

$$ \{IDF}(b) = \log\frac{N}{n_b}, $$

where $N$ is the total number of words (in our case $N=3$), and $n_b$ is the number of words which contain the bigram $b$.

In our example of words *aabaa*, *abbb* and *ababa*, the IDFs are as follows:

&nbsp; | aa  | ab  | ba  | bb
---   | --- | --- | --- | ---
$n_b$ | 1   | 3   | 2   | 1   
$\{IDF}(b)$  | 1.1 | 0.0 | 0.4 | 1.1

This is exactly what we need: The most common bigram (*ab*) has zero IDF, while the most rare bigrams (*aa* and *bb*) have highest IDF.

By the way, to see the IDFs of all bigrams from our data of 2020 words, click [here](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-idf-standard.txt).

## Step 2 - Sum up bigram scores

To get the suspicious score for a given word, we just need to sum up IDFs of all bigrams occurring in a word:

Word / Bigram | aa  | ab  | ba  | bb | Suspicious score (sum of IDFs)
---   | --- | --- | --- | --- | ---
aabaa | 2   | 1   | 1   | -   | **2.6** = 2 * 1.1 + 1 * 0.0 + 1 * 0.4
abb   | -   | 1   | -   | 1   | **1.1** = 1 * 0.0 + 1 * 1.1
ababa | -   | 2   | 2   | -  |  **0.8** = 2 * 0.0 + 2 * 0.4
**IDF**  | **1.1** | **0.0** | **0.4** | **1.1**

We see that the word *aabaa* with a score of 2.6 is the most suspicious word out of our fabricated example.

If you use this algorithm to compute suspicious score on our 2020 words, you get [this result](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-output-standard-idf.txt). Looks ok, but take a look at positions #9, #11 and #13. *Christopher* and *Jefferson* are regular names but they have very high suspicious scores (e.g. higher than *lkjwqer*, *erwtrepo* and others).

Can we improve the algorithm?

## Improve the algorithm with adjusted-IDF

The problem with *Christpoher* or *Jefferson* is that they are long and contain a lot of common bigrams. The sum of IDFs of these common bigrams is higher that the sum of IDFs of fewer but more rare bigrams.

To overcome this, we can adjust the IDF formula to return lower score for common bigrams. One of many possible options is to use the following formula:

$$ \text{IDF}_\{adj}(b) = \log\frac{\max_c(n_c)}{n_b}, $$

where the $\max_c(n_c)$ term is just the number of occurrences of the most common bigram. Using this formula, the bigram score should be zero for the most common bigram and almost zero for very common bigrams.

To see the adjusted-IDFs for the bigram in our sample of 2020 words, click [here](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-idf-adjusted.txt). You can see that scores of very common bigrams decreased from 2.5 to 0.5, while the score of most rare bigram remain above 5. *(You can experiment and tweak this formula more in order to fit your data. Apart from logarithm, you might also consider [different  functions](http://en.wikipedia.org/wiki/Exponential_distribution))*

The adjusted-IDF is the score that I used to arrive at the [results](https://github.com/mkrcah/random-text-detector/raw/master/data/names-output-adjusted-idf.txt) presented in the beginning of the article.

## Grab the source code here

I put the source code (Scala) on [github](https://github.com/mkrcah/random-text-detector). It tried to make it simple to get the tool up and running on your local machine (either in batch-mode or in server-mode with API). Also, it should be possible to deploy code to [Heroku](http://random-text-detector.herokuapp.com) without any code changes.


Detecting random text in web registration forms


That’s the one you need to do first. That’s where the highest impact and peak productivity lies, at least based on my experience.

Here are my three observations.

First, my willpower is strongest in the morning. [Research](https://www.amazon.com/Willpower-Instinct-Self-Control-Works-Matters/dp/1583335080) suggests the same. Focusing on the most resented task first thing in the morning allows me to utilize my willpower at its peak. Other tasks in the morning would deplete my willpower and thus increase the risk of postponing the important task to the next day. (This is the same reason why I exercise in the morning. Have you ever skipped a planned evening workout because there was no willpower, energy, or time left?)

Second, there is a feeling of satisfaction from the beginning of the day. This satisfaction remains present throughout the whole day, no matter how the rest of the day evolves. (How do you feel during the day if you workout in the morning?)

Third, it leads to tremendous productivity boost via a compound effect. I have observed that resented tasks usually fall into the [important not-due-soon quadrant](https://www.youtube.com/watch?v=oTugjssqOT0&feature=youtu.be&t=20m56s), known as the holy grail of peak long-term productivity. Since resented tasks often relate to each other, results get built upon each other.

<br/>

What would change if, for the upcoming month, the first task you do at work is the one you resent the most?


Which task do you resent the most today?


At [PyData Warsaw 2017](http://pydata.org/warsaw2017) and [Prague Hadoop Meetup 2017](ttps://www.meetup.com/CS-HUG/events/244755865) I talked about best-practices for putting a prototype of a machine-learning model or a data application into production. I tried my best to cover principles agnostic to a language or a technology. I see now I put perhaps too much detail in the slides & I was also more opinionated back then.

Slide deck: [http://talks.marcel.is/hugcz2017-sebp](http://talks.marcel.is/hugcz2017-sebp)

Video from the PyData Warsaw:

<iframe width="420" height="315"
src="https://www.youtube.com/embed/MKrPXfvIWoc">
</iframe>




From a model to production: Software-engineering best-practices


When building software, we introduce changes to the system. To change the system, we first need to understand the system. Each system consists of components. So to change the system, we need to understand its components. What makes components hard to understand?

Imagine component A intertwines with component B. And B intertwines with C. And C intertwines with D. To change component A, we need to dive into B and C and D. But we are limited in our cognitive capacity; in our heads we can consider [at most a few things](https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two) at a time. So to change a component, we need to understand the whole braid of intertwined components. That takes time.

> Everytime you combine two things, you twist two more things. So every intertwining adds this burden, and the burden is combinatorial. Fundamentally, this complexity, this twisting, limits our capability to understand our systems.
>
> — Rich Hickey, [Simple Made Easy](https://www.infoq.com/presentations/Simple-Made-Easy/)

To gain speed, we decomplect the _complex_ braid and turn it into a set of _simple_ components. A simple component consists of one fold, of one braid, of one ingredient. A simple component doesn't intertwine with other components. To change a simple component, we don't spend time understanding the whole braid. And so we are faster.

![simplicity](/img/simple-components.png)


To change a system, we first need to understand the system. Each system consists of components. So to change the system, we need to understand the components. What makes components hard to understand?


What slows us down when building software?


On the road to improve my writing, I stumbled upon this [super interesting video by Larry McEnerney](https://www.youtube.com/watch?v=aFwVf5a3pZM). The techniques covered in the video bend my mind. One such technique is about the subjects of sentences. Consider the two sentences below, which one do you think is better?

1. The dog chased the cat.
2. The cat was chased by the dog.

The first sentence is shorter and has an active verb. So, according to conventional wisdom, the first sentence is better. However, that might not be optimal. When we write, as Larry explains, we should aim for concision. But not in the conventional sense, that is, the number of words. Instead, we should target the _time_ it takes for the reader to understand the sentence.

We can improve concision by multiple routes. One route, it turns out, is to focus on the sentence's subject. It is because the subject defines what the sentence is _about_. So the first sentence, "The _dog_ chased the cat," is about the dog; while the second sentence, "The _cat_ was chased by the dog," is about the cat. The two sentences are about different things.
Now, what if we serve the reader the sentence about the dog, but they care about the cat? That's inconsistent. So the reader needs to cognitively reprocess the sentence to make it about the cat. This reprocessing is taxing, and it harms concision. The reader becomes disinterested or bored, and you can lose him.

![subjects](/img/subjects.png)

We should serve the reader subjects that are aligned with what they care about. If the reader cares about the dog, the subject should be the dog. If they care about the cat, the subject should be the cat.

So I started experimenting with the technique. When drafting a product update to the rest of the company, I wrote: "We delivered the feature \[X\]". However, the readers care about the product and the customers. So I can write the sentence as "Customers can now do \[X\]," and bring the product update closer to the readers' interest.

There's this story from Larry that I remind myself in hope to lubricate my rusty subjects:

> Sometimes people pay me stupid amounts of money to fly to often wonderful cities to sit down with them. They are saying to me: "My readers don't find this valuable" or "I can't get this published" or "I can't get promoted." So I make them take their pen out and underline the subjects of their sentences. And I ask: "Do your readers care about this?" And they say: "No."
>
> — Larry McEnerney, from the lecture [Writing Beyond the Academy](https://www.youtube.com/watch?v=aFwVf5a3pZM),


Which sentence is more concise: "We fixed the bug in production" or "The bug in production has been fixed" ?


Exploring my writing: The subjects of sentences


James Clear is the author of Atomic Habits, who popularized the idea of habits over goals. Last month he shared [the rules he uses for running his business](https://twitter.com/JamesClear/status/1344776679315689480), based on his 10-year experience. Among the rules was this one:

> Build assets that compound. Eliminate problems, responsibilities, and obligations that compound. Your business should feel easier to sustain each year.
>
> — James Clear (from [How I run my business](https://twitter.com/JamesClear/status/1344776679315689480))

"Your business should feel easier to sustain each year." What if, instead of business, we focus on building software? And what if we turn the advice into a question? Then, we get: _does our codebase feel easier to sustain each year?_

I love that question:
- It reveals the long-term vision of how to approach software engineering.
- It encourages challenging missions to [remove complexity](https://www.infoq.com/presentations/Simple-Made-Easy/) and [target the important non-urgent quadrant](https://youtu.be/oTugjssqOT0?t=1256).
- It exposes the murky alternative, that is, descending further into the complexity tar pit.

All in all, I've found that the question brings fresh air and whirls settled dust.



James Clear is an author of Atomic Habits, who popularized the idea of habits over goals. Last month he shared the rules he uses for running his business, based on his 10-year experience.

Does your codebase feel easier to sustain each year?


_Update: See [HN comments](https://news.ycombinator.com/item?id=17868066) for this article._

A company is not data-driven *by using a BI tool.*

A data-science team is not agile *by using a ticketing system.*

A study is not useful *by presenting results in a fancy tool.*

<br />

Using modern software does not imply intrinsic value.

<br />

A company can be data-driven just with a pen and paper.

A data-science team can be highly-performant just with a whiteboard.

A study can be impactful if presented just in a notepad.

<br />

What counts is the underlying hard-work, the focus on stuff that matters, and courage to make a decision.



On software tooling


It's an option. Pushing a proposal because it’s _your proposal._

Such an approach, however, often leads to discussions that are unpleasant for attendees. People get irritated or angry. Relationships and a team spirit get harmed.

There is another way.

Striving to be objective.

Proactively considering limitations of your proposal. Being aware of and acknowledge the advantages of other proposals. Choosing a solution based on mutually understandable pros and cons.

Becoming a curious observer, a detached messenger of facts and opinions. Including your own.

It’s an appealing state of mind, worth the higher mental investment.

Perhaps an occasional subjective push from a team’s authority would be eventually needed. However, it would come from a different angle, based on mutual understanding of each other.


Taking things personally


What do _you_ want to do?

Find this out for each team member.

Then align the roles accordingly.

It's dead simple.

### Why?

At least among software engineers (but I believe it's among all of us), there is this hidden voice inside. It silently and always reminds us what we want to do, what the forbidden fruit is.

Java engineers seeking transition to Scala.

Scala programmers wanting to do Big Data.

Big Data folks wanting the transition to product management.

<br/>

When this calling is enabled, magic happens.

Energy goes up. Productivity goes up. Satisfaction goes up.

Flow happens.

Cannot-wait-for-Monday-morning happens.

And when enabled within a team, synergies occur. The team becomes a warp-engine rocket, delivering like crazy while having tons of fun.

<br/>

Of course, a team might not need a new member; some roles might be lacking, etc.

But it's the best baseline. Once defined, it's just a matter of time until it's reached.


Supercharge your team


_By a data analytics warehouse we mean a solution that combines and cleans data from various internal and external sources and offers reporting and querying capabilities to business users. Sometimes also referred to as a big data platform._

Based on failures and hard-won lessons, here's 21 tips for teams and individuals who are responsible for making a company more data-driven.

### Solve the right problem

1. Understand that building an analytical warehouse is [not about the technical tooling](/sw-tooling).
2. Get an in-depth understanding of [the root problem](/problem) to minimize the risk of delivering a solution that doesn't solve a tangible business problem.


### Choose the right tool for the job
3. Consider the most straightforward solutions to resolve the root problem. Perhaps, an excel would suffice.
5. Do not choose a solution according to [personal preference](/quench).
6. Be aware of maintenance costs. Consider off-the-shelf solutions before building your own. If a cloud platform is an option, consider no-ops and SaaS offerings. There is a plethora of solutions out there for ETL, storage and BI tools. Be [pragmatic about Hadoop](https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html).
3. Start with delivering a narrow [vertical slice of functionality](https://en.wikipedia.org/wiki/Vertical_slice) to discover possible problems early and to validate the chosen approach.
4. Validate new deliverables as frequently as possible through an intimate collaboration with business.



### Provide correct and consistent data

8. Check for data correctness: integrity violations (foreign keys, unique values), invalid value types, errorneus ranges (values out of bound, text too long or too short), missing data, etc.
8. Check for suspicious data: values out of expected bounds, the number of new records too large/low, etc.
9. Treat incorrect and suspicious data as exceptions and warnings, respectively. Report on them automatically and periodically.
13. Use consistent prefixes and suffices per field type to limit guesswork. For example, use `is_` for booleans and  `_at` or `_time` for time.
10. Ensure consistency in values. For example, use `null` for missing data and [ISO codes](https://en.wikipedia.org/wiki/ISO_3166-1) for countries among all tables.
17. Convert values to native types as early as possible. For example, convert date strings to dates, and binary integers to booleans.

### Bring data closer to business
14. Add as much semantics to field names as possible. Where applicable, include units, timeframes, and aggregation functions. For example, use `avg_kwh_consumed_per_year`  instead of `energy`.
18. Establish and use common terminology with business. Rely on [ubiquitous language](https://martinfowler.com/bliki/UbiquitousLanguage.html) for table names, field names, and values. Reconsider [using abbreviations](https://www.ibtimes.com/spacex-boss-elon-musk-threatened-drastic-action-against-employees-who-use-unnecessary-1972972).
9. If possible, encode domain knowledge and undocumented business rules into data to increase value for data users.
15. If relevant, teach SQL to data users to enable advanced self-service querying.
20. Understand that data is [an abstract model of reality](https://fs.blog/2015/11/map-and-territory/). Identify and communicate data limitations.

### Develop efficiently
19. Learn [window functions](https://community.modeanalytics.com/sql/tutorial/sql-window-functions/) and [`with` queries](https://www.postgresql.org/docs/current/static/queries-with.html) to simplify complex SQL queries.
16. Create views for frequently reused SQL queries to ensure users [don’t repeat themselves](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
23. Follow software engineering best practices, incl. version control, clean code, testing, peer reviews and [boring and predictable deployments](https://www.youtube.com/watch?v=aoMfbgF2D_4).



Tips on building a data analytics warehouse


Should we buy the IT system from a vendor, or should we build it in-house? For some folks out there, myself included, this is a crossroad clouded with uncertainty.
To scatter the clouds and shed light into the paths ahead, let's hear what the elderlies of software engineering craftsmanship say:

<div class="blockquote">

> If it’s a core business function — do it yourself, no matter what.
>
> Pick your core business competencies and goals, and do those in house. If you’re a software company, writing excellent code is how you’re going to succeed. Go ahead and outsource the company cafeteria and the CD-ROM duplication. If you’re a pharmaceutical company, write software for drug research, but don’t write your own accounting package. If you’re a web accounting service, write your own accounting package, but don’t try to create your own magazine ads. If you have customers, never outsource customer service.
>
> — Joel Spolsky, from [In Defense of Not-Invented-Here Syndrome](https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/)

</div>
<div class="blockquote">

> Cores are the central competencies of the organization. These are things that customers are willing to pay for and what investors reward. Context is everything else. \[...\] As we fund core features and applications, we need to make sure that context doesn’t kill core.
>
> Of the applications and services that you manage, which of them are customers willing to pay you for? Which ones truly enhance competitive advantage? And which can you rely on vendors for?
>
> — Kim Gene, from [The Unicorn Project](https://www.goodreads.com/book/show/44333183-the-unicorn-project), chapter 17

</div>

<div class="blockquote">

> When it comes to build versus buy, the frequently repeated but rarely followed wisdom is good advice: if you're a technology company, vendors usually generate significant value if they're outside your company's core competency; within your core competency, they generally slow you down.
>
> — Will Larson, from [Build vs Buy](https://lethain.com/build-vs-buy/)

</div>

So according to the advice, the crossroad sign can read:

![buy-or-build](/img/buy-or-build.png)


Should we buy the IT system from a vendor, or should we build it in-house? For some folks out there, myself included, this is a crossroad clouded with uncertainty.

To buy or to build?


For the current project, I need to track hours in [Clockify](https://clockify.me/). But as I was opening the webpage and filling in the hours, I felt dispassionate. So I decided to turn this activity into a joyful one.

It turns out that Clockify has an easy to use [API](https://clockify.me/developers-api). So I decided to build a script that 1)
gets a list of all projects via an API call, 2) lets me interactively select a project, 3) lets me select duration I'd spent on the project that day, and 4) adds hours to the projects via an API call. To do the job, I summoned my favorite command-line tools:

- [HTTPie](https://httpie.io), a command-line HTTP client, out of the box ready for JSON REST APIs,
- [jq](https://stedolan.github.io/jq/), a swiss-knife for parsing and transforming JSON,
- [awk](https://ferd.ca/awk-in-20-minutes.html), a gnu stronghold for text processing,
- [fzf](https://github.com/junegunn/fzf), an interactive fuzzy search for the command-line,
- `cut` and `tr`, text processing unix commands.

For example, to interactively select a project id I piped the tools together:

![track-hours-select-a-project](/img/track-hours-select-a-project.png)

Using the script almost every day, I'm now playfully enjoying the tracking routine.

Here's the full script:

```bash
#!/usr/bin/env bash

# fail on error, unused var and pipe failure
set -euo pipefail

# clockify api config
CLOCKIFY_BASE_URL='https://api.clockify.me/api/v1'
CLOCKIFY_DATE_FORMAT="%FT%T"

# let me select a project
PROJECT_ID=$(http $CLOCKIFY_BASE_URL/workspaces/"$CLOCKIFY_WORKSPACE_ID"/projects \
  'X-Api-Key':"$CLOCKIFY_API_KEY" | \
  jq -r ".[] | .id, .name" | \
  awk 'NR%2{printf "(%s) ", $0;next}1' | \
  fzf --nth=2.. | \
  cut -f1 -d' ' | \
  tr -d '()'
)

# let me pick how many hours I worked today
HOURS=$(seq 1 12 | awk '{print $0 " hours"}' | fzf | awk '{print $1}')

# start the day at 9:00am
TASK_STARTED_AT=$(date -v9H -v0M -v0S +$CLOCKIFY_DATE_FORMAT)

# and stop after $HOURS have passed
TASK_STOPPED_AT=$(date -j -v +"${HOURS}H" -f $CLOCKIFY_DATE_FORMAT "$TASK_STARTED_AT" +$DATE_FORMAT)

http -q $CLOCKIFY_BASE_URL/workspaces/"$CLOCKIFY_WORKSPACE_ID"/time-entries \
  'X-Api-Key':"$CLOCKIFY_API_KEY" \
  start="${TASK_STARTED_AT}Z" \
  end="${TASK_STOPPED_AT}Z" \
  projectId="$PROJECT_ID" \

echo "Done 🚀"
```


Tracking hours with Clockify, fzf, jq and awk


You have decided to do a career switch to THE BIG DATA, but you don't know how.

Here's what to do:

1. Choose your role
2. Build a sexy demo and job hunt guerilla-style

## Step One: Choose your role

There are four roles in this big-data world:

- Finding insights in the past
- Setting up & delivering a data warehouse
- Predicting future from past data
- Coding data-driven product features

### Role 1: Finding insights in the past

This is a key role in jobs titled Data analyst and Marketing analyst. Also, it's a key prerequisite for Growth Hackers.

You answer questions about the past to help steer the business forward. For example:

- How does the [cohorts](http://cohortanalysis.com/) look like in our customer base?
- Which airport gates lead to the most late passangers?
- Which customer micro-segment is driving the most revenue?

You use BI tools (like Tableau, Looker) to interactively explore and visualize data. If you are more technical, you reach for SQL and/or R.

After a while, you start impacting metrics that you report. You use [growth hacking](https://www.quicksprout.com/the-definitive-guide-to-growth-hacking ) and [A/B testing](https://www.optimizely.com/ab-testing/). And you provably improve company's profit.

Learn the first steps by:

- Read the first 50-pages of the [Pyramid principle](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516) to get yourself acquainted with top-down reasoning
- Search for online-sources for Data Analysis (example: [edX](https://www.edx.org/course/subject/data-analysis-statistics), [Coursera](https://www.coursera.org/courses?languages=en&query=data+analysis))

### Role 2: Setting up & delivering a data warehouse

This is a key role in jobs titled Data Engineer, ETL engineer and Hadoop developer.

Typically, your responsiblity is to combine multiple company data  together from various sources.

For example, you combine customer data from Salesforce, support data from Intercom, product usage from a few production Postgres databases, real-time customer events from Kafka and you sprinkle it with historical weather information.

Then you create an abstraction on top of this data to provide a single source of truth. You operate between two departments:

- the business:  to align on metric definitions and
- the IT: to get the integrations right.

Having all data together in this manner has HUGE benefits - all of a sudden, the time-to-answer drops from days/weeks to zero. Connect a self-service BI tool on top of it (e.g [Looker](https://looker.com/), [Metabase](http://www.metabase.com/)) and you are there.

Note that there are cloud solutions out there which provide the warehousing and data shoveling for you. Checkout [Redshift](https://aws.amazon.com/redshift/?hp=tile&so-exp=below) and [BigQuery](https://cloud.google.com/bigquery/) for warehousing and [Alooma](https://www.alooma.com/), [Stitch](https://www.stitchdata.com/) and [Fivetran](https://www.fivetran.com/) for shoveling.

You need to do the data merging yourself - that's the company domain. You do this either with SQL or a data processing tool:

- Learn SQL [here](https://community.modeanalytics.com/sql/tutorial/introduction-to-sql/), especially the [cool](http://stackoverflow.com/questions/12552288/sql-with-clause-example) [features](https://community.modeanalytics.com/sql/tutorial/sql-window-functions/)
- Learn [Spark](http://spark.apache.org/), it's sexy these days. Learn it [here](https://youtu.be/fn3WeMZZcCk?t=878) and [here](https://databricks.com/training)


Notes:

- Software engineering skills are expected. Learn principles of [clean-coding](https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670/ref=pd_sbs_14_t_1?_encoding=UTF8&psc=1&refRID=3RJ1DN7QVHCWJEKCBDGT) and learn Python or Scala.
- Scala is hot. Get the certificate from [this legendary course](https://www.coursera.org/learn/progfun1) and put it on your LinkedIn.
- You *don't* need to learn Hadoop. Most companies [don't need Hadoop](https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html).


### Role 3: Predicting future from past data

This is a key role in jobs titled Data Scientist and Quants.

Your role is to predict behavior by learning from the past data:

- What are the products that customer X is most interested in?
- How much is product X going to be traded on the exchange tomorrow?
- How to optimally coordinate airplane routes to minimize fuel costs?
- How utilized will the Tesla super-chargers be tomorrow at location X?

You are the math guy in the company. You know machine-learning, simulations or numerical computing. Typically, machine-learning is required. This includes:

- supervised/unsupervised machine-learning models
- ability to prevent over-fitting
- ability to engineer and select the right features

Daily bread is Pandas, Scikit-learn (my favourite), Matlab, R or SparkML. However, I know a guy in Booking.com who uses C++ for his machine-learning experiments.  

Learn the first steps by:

- Follow the legendary [Andrew Ng's lecture](https://www.coursera.org/learn/machine-learning) on machine-learning
- Go to [Kaggle](https://www.kaggle.com/competitions) to practice your skills.
- The Kaggle [forums](https://www.kaggle.com/forums) are legendary, you will learn a LOT of practical info there

Notes:

- Focus is on understanding the business and the underlying data so you can come up with the right data [features](http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/).
- Once the features are there, a simple model would do.
- Less software engineering skills is expected. Clean-code is not a priority. There is another role for translation the model into production.


### Role 4: Developing data-driven product features

This is a common role in jobs titled Data Engineer.

You are putting into production features driven by prediction and machine-learning models:

- compute product recommendations and expose them via API, which is consumed by team-members building the mobile app
- implement a stock prediction algorithm within a production trading platform

If there is a dedicated data scientist, you work closely with him/her to translate their model to production code. If there is a growth-hacker, you work closely to implement their ideas.

- You are a software engineer by heart: you know a few languages, [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)/[YAGNI](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it), clean-code, continuous delivery, unit/functional tests, monitoring - the whole shebang.
- On top of it, you understand the big-data ecosystem used by the team (e.g. Kafka, Spark, [ELK Stack](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04))
- On top of it, you understand machine-learning and can translate data scientists model into production.
- Sometimes, data scientists are also engineers who put their models into production

Learn the first steps:

- If you are not a coder, get an engineering job first. I recommend Scala. Try to grasp the functional approach of it. Scala is used heavily in big-data community. Another option would be Python. Once you are good in engineering, go to the next step:
- If you are a coder, learn Spark and a few other technologies. It's the tech-stack you know that will get you this role.
- Although companies think they need Hadoop, they [most probably don't]((https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html)). But it's good to learn it, so you understand situations when Hadoop is useful. Checkout the major Hadoop distributions (e.g. [Cloudera](www.cloudera.com), [Hortonworks](http://hortonworks.com/), [MapR](http://www.mapr.com)), install it locally and try to get the basic ETL up and running. Or checkout cloud solutions like [AWS EMR](https://aws.amazon.com/emr/) and [Google Dataproc](https://cloud.google.com/dataproc/).





### Choose your role by your current strengths:

- If you have an economic background and no engineering skills yet, go for Data Analyst.
- If you have system administration background, go for setting up a warehouse.
- If you have software engineering background, go for data engineering.
- If you have strong mathematical, model, statistics background, go for Data Scientist.
- If you are a student, go learn machine-learning, [clean coding](https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670/ref=pd_sbs_14_t_1?_encoding=UTF8&psc=1&refRID=3RJ1DN7QVHCWJEKCBDGT), Scala and Clojure.

Don't worry too much with the initial choice. Especially in smaller companies, the roles overlap and you have freedom to move between the roles. I know a Data Scientist doing warehousing stuff for two years and a Data Engineer doing growth hacking.

Still not sure which role? [Drop me](mailto://m@marcel.is?subject=Which role to choose) an email.



## Step 2: Build a sexy demo and go guerilla

Do this:

- Read [this article](https://medium.com/the-mission/how-to-get-any-job-you-want-even-if-you-re-unqualified-6f49a65f5491#.c1cuqxawh) and [this article](http://brohrer.github.io/get_data_science_job.html) and choose one of these two approaches.
- Choose your target company.
- Build a (targeted) demo


For choosing the target company:

- aim for a smaller one, ideally a startup. There is much more freedom to experiment and switch roles.
- aim for a team where only 1-2 people do what you do. This gives you a lot of space to experiment. Plus the opportunity of taking up the full responsibility once they decide to move to something else.


For the demo, here are a few examples:

#### Example 1 (for data analysts): Data insights and recommended next-steps

- Find interesting data about the company you are hunting, e.g. [in this collection on GitHub](https://github.com/caesar0301/awesome-public-datasets).
- Get Excel, [Tableau Public](https://public.tableau.com/s/) or a similar tool and dive deep into data
- Deliver a [top-down](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516/ref=pd_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=KBBSY0YBQYA53CZEREKJ) structured research story with recommendations on next steps

#### Example 2 (for data analysts): Tailored growth-hacking recommendations

- Read more about [growth-hacking](https://github.com/btomashvili/awesome-growth-hacking) and [lean startup](http://theleanstartup.com/book)
- Guess problems the target company is facing by studying their products
- Deliver a list of 5-10 actions you would advise them and metrics on how to evaluate progress

#### Example 3: (for data engineers): Spark Streaming with real-time visualizations

- Spark is hot, [Spark Streaming](http://spark.apache.org/streaming/) especially.
- Play with combination of Twitter and Spark Streaming, like [here](https://www.youtube.com/watch?v=fn3WeMZZcCk&feature=youtu.be&t=1038)
- Or play locally on your computer, example [here](https://github.com/mkrcah/scala-kafka-twitter).

#### Example 4: (for data engineers): Click-stream processing with Kafka & Alooma

- Build a [simple web app](http://flask.pocoo.org), collect clicks via [Kafka](https://kafka.apache.org/)
- From Kafka, send the clicks to [Alooma](https://www.alooma.com/). Sessionize the clicks in Alooma and pipe it to [BigQuery](https://bigquery.cloud.google.com).
- Connect [Metabase](metabase.com) to BigQuery to show reports on basic usage
- You have just build a simple version of Google Analytics!
- Bonus: Find which SaaS product your target company is using (e.g. for customer support) and merge that data with the clickstream.

#### Example 5: (for data scientists): Open-source your Kaggle solution

- Compete on [Kaggle](https://www.kaggle.com/competitions)
- Once you have a decent model, put it on Github and LinkedIn and blog about it.

#### Example 6: (for data scientists): Deliver tailored prediction

- Get hold of data which might be interesting for your target company, for example [here](https://github.com/caesar0301/awesome-public-datasets)
- Find a problem which can be framed as a machine-learning problem.
- Solve it, package the result in a [top-down story](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516/ref=pd_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=KBBSY0YBQYA53CZEREKJ) in a [Jupyter](http://jupyter.org/) notebook and [share it only](http://nbviewer.jupyter.org/). Put it on Linked and blog and it.


## FAQ:

#### Shall I do pro-bono?

Sure. If you don't feel confident about your current skills, do a pro-bono project.
Reach to your network of friends, family and colleagues. Try to get a contact
within your network that would tell you about their company problems and are willing
to share their data. In exchange for your work (if done well), you'd ask for a referral that you can post on your LinkedIn/blog.



<br/>
Enjoy the ride and [ping me](mailto:m@marcel.is?subject=Big data transition) with your transition story. I'm very much interested.

I'll leave the last words to the marketing guru [Seth Godin](http://sethgodin.typepad.com/seths_blog/2006/10/make_something_.html):

> **Make something happen**
>
> If I had to pick one piece of marketing advice to give you, that would be it.
>
> Now.
>
> Make something happen today, before you go home, before the end of the week. Launch that idea, post that post, run that ad, call that customer. Go the edge, that edge you've been holding back from... and do it today. Without waiting for the committee or your boss or the market. Just go.


Transition your career to big data


In the book [The Metaphors to Live By](https://www.goodreads.com/book/show/34459.Metaphors_We_Live_By), the authors observe that we think and write and communicate in conceptual metaphors. Here are a few examples:

- **Conscious is up, unconscious is down**: Get *up*. She *rises* early in the morning. He *fell* asleep. He's *under* hypnosis. He *sank into* a coma.
- **Good is up, bad is down**: We hit a *peak* last year, but it's been *downhill* ever since. Things are at an all-time *low*. He does *high*-quality work.
- **Argument is a building**: That is the *foundation* of our theory. The theory needs more *support*. The argument is *shaky*, it *falls apart*. We need to *construct* a strong argument.
- **Argument is a battle**: Your claims are *indefensible*. He *attacked* every *weak* point. His criticisms were on *target*. We need to *aim* for concision. They have a *strategy*.
- **Argument is a container**: Your argument doesn't have much *content*. That argument *has holes in it*. His objections have *less substance*. Your arguments are *empty*.
- **Argument is a journey**: We have *set out* to prove that. *So far*, we've seen that theories will not work. We will *proceed* in a *step-by-step* fashion. We have *arrived* at a conclusion. We are at a  *crossroad*.
- **A problem is a body of water**: He *dived into* the problem. He  *immersed* himself in the problem. The problem is *murky*. Finally the answer *surfaced*.
- **A problem is a region in a landscape**: We've got to *explore* this problem. Let's *map out* the problem before doing anything else. We're *heading* in the right direction. The solution *lies far ahead*.
- **Ideas are food**: That's *food for thought.* We don't need to *spoon-feed* our students. This is the *meaty* part of the paper.  There are too many facts to *digest*.
- **Ideas are cutting instruments**: That's an *incisive* idea. That *cuts right* to the heart of the matter. He's *sharp*. He has a *razor* wit.
- **Ideas are plants**: The *seeds* of his ideas were *planted* in his youth. She has a *fertile* imagination. Mathematics has many *branches*. Here's an idea I'd like to *plant* in your mind.

It turns out [there are](http://www.lang.osaka-u.ac.jp/~sugimoto/MasterMetaphorList/metaphors/index.html) [catalogs](https://metaphor.icsi.berkeley.edu/pub/en/index.php/Category:Metaphor) of conceptual metaphors. We use them all the time. For example, also in the [man pages](https://jvns.ca/blog/2020/05/08/metaphors-in-man-pages/).








We think in conceptual metaphors


Today at work, we discussed when to write a script in shell and when in a language such as Python or Javascript. I realized I enjoy writing a shell script when:

-   I integrate heavily with the system, such as creating SSH tunnels, mounting a device, or copying files around. I could do so in Python, but I would need to add boilerplate code around system commands. Or I would need to find an equivalent Python/Javascript library. In most cases, I've found vanilla shell scripts easier to write.
-   I do simple text processing over a mound of files, such as searching log files or replacing text. I feel like Usain Bolt when I pipe together commands from the powerful family of `awk`, `sed`, `grep`, `cut`, `wc` and the like.
-   I run a script in an environment where I have no access to another language. Unix commands are available on any unix machine, even if Python is not. 

However, the more the script grows in complexity, the more cumbersome I feel when writing it in shell. I think it takes an extraordinary level of discipline and experience to make sure that a large shell script is easy to read and maintain. For that reason, I feel relieved when I can switch to Python or Javascript.

Another decision axis I find important is the team preference. Imagine the team doesn't need and doesn't want to learn shell. If I introduce shell scripts, I would be concerned that I would be the only one left to maintain the script.

Shell scripts vs. Python/Javascript scripts


As I've been doubling down on my writing, I started to observe situations when I hedge. Hedging should be avoided, the writing advice goes, but I've found that it depends.


> Many writers cushion their prose with wads of fluff that imply that they are not willing to stand behind what they are saying.
>
> — Steven Pinker (from [The Sense of Style](https://www.goodreads.com/book/show/20821371-the-sense-of-style), p. 43)

In one situation, a colleague asked about our website's unexpected behavior. I quickly investigated the issue and discovered that the most probable cause was a bug in the backend. So I replied: "It seems this is a bug in the backend." Such hedging, I realized, allows me to quickly communicate uncertainty in the case of the unknown. It's a shortcut I find acceptable.

![hedging-unknown](/img/hedging-unknown.png)

However, there are other situations. Once, I wrote: "It seems that the data connector is working." But this situation was different from the one above: I wrote the sentence _after_ investigating the data connector, not before. I knew the connector, I knew the facts, yet I presented the outcome with hedging.

I can do better. Instead of cushion the uncertain generic conclusion with a safety net, I can organize the facts in a logical flow. I can write: "The  connector filled in the missing data for October. To declare the connector working, we need to check the connector's real-time ingestion." In this situation, the lack of hedging forces me to clarify my thoughts, which, in turn, helps others to understand the situation, the known, better.

![hedging-after](/img/hedging-after.png)

> To write well, you need first to know what you are talking about. Thus, you have to do your research. ... Then you have to be able to organize your argument, at the level of the word, the sentence, the paragraph and the essay itself. If you can do that, you can organize your thoughts and, in consequence, your brain. Then you can help organize other brains, and other structures.
>
> Jordan Peterson (from [Quora](https://www.quora.com/What-are-the-necessary-skills-for-21st-century-students-and-future-leaders/answer/Jordan-B-Peterson))




As I've been doubling down on my writing, I started to observe situations when I hedge. Hedging should be avoided, the writing advice goes, but I've found that it depends.


Feb 24, 2021	Building bi-directional sync between two system #data-engineering
Feb 10, 2021	Resolving conflicts peacefully and powerfully #process
Feb 08, 2021	Making hard decisions later #engineering
Feb 04, 2021	Shell scripts vs. Python/Javascript scripts #tools
Feb 02, 2021	I'm glad for using 1Password #tools
Feb 01, 2021	Tracking hours with Clockify, fzf, jq and awk #tools
Jan 27, 2021	Preparing a lemon pie: visualization #writing
Jan 26, 2021	Embarrassingly simple proof of concepts #engineering
Jan 25, 2021	Everyday decision making #architecture
Jan 22, 2021	The spirit of kaizen #process

Jan, 2021	Preparing a lemon pie: visualization
Jan, 2021	We think in conceptual metaphors
Jan, 2021	Exploring my writing: The subjects of sentences
Dec, 2020	Exploring my writing: Hedging
Dec, 2020	Drawings for technical explanations: my first steps

Marcel Krčah

Most recent 10 posts

engineering

architecture

tools

writing

process

aws

data-engineering

talks

Feb, 2021	Making hard decisions later
Jan, 2021	Embarrassingly simple proof of concepts
Jan, 2021	The curious case of if-not
Jan, 2021	Honest function inputs
Jan, 2021	Coding and writing are alike
Jan, 2021	What slows us down when building software?
Jan, 2021	Clean code: can we learn from altruism & evolution?
Jan, 2021	Does your codebase feel easier to sustain each year?
Dec, 2020	Adding domain knowledge to code comments
Aug, 2018	On software tooling
Sep, 2016	Resist the quench for coding.

Feb, 2021	Building bi-directional sync between two system
Sep, 2018	Tips on building a data analytics warehouse
Aug, 2018	Why Docker for Data Scientists
Nov, 2017	Should you hire a data engineer instead of a data scientist?
Oct, 2016	Transition your career to big data
May, 2015	How a newline can ruin your Hive
Apr, 2015	How to ingest data from Azure DataMarket into Hadoop
Jan, 2015	Introduction to Apache Zookeeper, backbone of big-data systems
Aug, 2014	Analyzing Elastic MapReduce data with Python, Pandas and scikit-learn
Jul, 2014	Detecting random text in web registration forms