
In the past month, two things happened to me that made me realize how glad I'm for using a password manager, namely [1Password](https://1password.com/).

First thing is about a login page. I was searching for a Github repo on DuckDuckGo, and I clicked on a link in the search results. As I was about to fill in the password with a 1Password shortcut, I noticed that 1Password wasn't filling in the login details. How strange, I thought. Puzzled, I looked at the url. And then I saw it—I almost fell for a phishing attack! The url was [github.com.cnpmjs.org](https://github.com.cnpmjs.org) instead of [github.com](https://github.com). I knew of phishing attacks, and thought I'd never fell for such a simple trick. Yet, I almost did.

Second thing is about email. As I'm migrating away from gmail, I thought I would check out the spam folder. I usually don't do that, but I thought I'd peek in. One email caught my eye: the email subject was my password. The site must have been compromised and the attacker got hold of my email and password. Fortunately, the damage radius was minimal, as I have generated unique passwords for each site.

![1password-email](/img/1password-email.png)


I'm glad for using 1Password


I have spent more time than expected trying to connect to AWS API Gateway with HTTP instead of HTTPS.

For the context, we have legacy IoT devices that make outbound connections via HTTP, not HTTPS. On the receiving side, we have a REST API service. The service, deployed via AWS API Gateway and AWS Lambdas,  supports only HTTPS, not HTTP.

So how to connect the two?

![aws-gateway-https-how](/img/aws-gateway-https-how.png)

It turns out you can use [AWS CloudFront Distribution][cf-guide] to receive HTTP and forward it to the API Gateway. In the most basic setup, it's [a few steps][cf-video]. [This StackOverflow question][so] contains more info.

![api-gateway-https-cloudfront](/img/api-gateway-https-cloudfront.png)

[cf-video]: https://www.youtube.com/watch?v=Dq2m2ZoZ234
[cf-guide]: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-working-with.html
[so]: https://stackoverflow.com/questions/43236152/how-to-make-aws-api-gateway-accept-http-instead-of-https


Connecting to AWS API Gateway with HTTP


I've been facing the question of how to build a bidirectional near-real-time synchronization between two systems.  In the past years, I've seen multiple projects where data between two systems were synced bi-directionally. In these projects, the bi-directional sync was a cause of data loss, unresolved conflicts, and frustration. In one project, the company ended up with two employees tasked to manually copy data from one system to another.

So when tasked to build bidirectional sync, I'm concerned about these issues. To mitigate the concerns, here are the strategies I use: 

- Before building, I've found it worth investing time into [[conflict-resolution|understanding the underlying needs]] and organizational constraints and into communicating back concerns from the IT perspective. Then, to investigate different *business solutions* to the needs and concerns without bidirectional sync.
- If the sync is needed, we try to turn the bidirectional sync into two one-directional syncs. In one-directional sync, each table is assigned a system, and the system is the source of truth for that table. Any change in the source of truth for that table overwrites the table in the other system. We can do the same on a more granular level, that is, on columns or rows.
- If that's not possible, we look at the bidirectional sync as a problem of reaching *eventual consistency* in a *distributed system*. The eventual consistency can be resolved using [CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type), which are nicely explained [in this video](https://www.youtube.com/watch?v=vBU70EjwGfw). As far as I understand, CRDTs require tight control over the actors of the distributed system. We hit a dead-end if we cannot change the actor, such as if we cannot implement [the last-writer wins strategy](https://dzone.com/articles/conflict-resolution-using-last-write-wins-vs-crdts) within the actor. We encountered this dead-end when we needed to sync two SaaS products: we couldn't access the codebase, and their APIs didn't support the CRDT use-case. 
- If none of these are possible, the last resort I've found is to build the bidirectional sync as a standalone sync program. The program uses APIs, webhooks, or databases to fetch updates from the systems, resolve conflicts with business rules, and push updates back to the systems. However, the solution comes with a risk of data loss—if the data is updated *during* the time window when the sync runs. For such bi-sync to be delivered, the business needs to accept the risk of possible data loss. 



Building bi-directional sync between two system

In the talk [There is no talent shortage](https://www.youtube.com/watch?v=P_sWGl7MzhU&t=1544), Andrew Clay Shafer referenced this parable:



<div class="blockquote">
	
> Three stone cutters were asked about their jobs. The first said he was paid to cut stones. The second replied that he used special techniques to shape stones in an exceptional way, and proceeded to demonstrate his skills. The third stone cutter just smiled and said: I build cathedrals.
> 
> — Richardo Semler, from [Maverick](https://www.goodreads.com/book/show/32994.Maverick),

</div>

There's also a similar creed:

<div class="blockquote">

> We who cut mere stones must always be envisioning cathedrals.
>
> — Medieval quarry worker’s creed

</div>

I _care_ about building cathedrals. Personally, cathedrals allow me to contemplate on _which_ cathedrals to build. Through contemplation, I can explore my motives and [[kaizen|goals]]. Professionally, in software engineering, cathedrals can guide [[compass-for-everyday-decision-making|everyday decision-making and create a shared vision for a team.]]

Also, a cathedral is more long-term so that I can experience both positive and negative consequences of my actions. I have more [skin in the game](<https://en.wikipedia.org/wiki/Skin_in_the_Game_(book)>) that I can learn from.

I've also noticed that focusing on a cathedral eliminates a whole bunch of burdening questions. A cathedral gives me clarity and focus.

Building cathedrals

I thought I will experiment and start a changelog of technical improvements for this website, excl. the posts themselves.



The website is written in [NextJS](https://nextjs.org/) and Typescript,
and relies on tools from [the Unified ecosystem](https://unifiedjs.com/).

## 2021-04

- added the [newsletter generator](/ng/) that automates most part of preparing the [Buttondown newsletter](https://buttondown.email/emails). It's not perfect, it's ugly, but it gets 90% done. Excerpts are extracted with [graymatter](https://github.com/jonschlinkert/gray-matter) (still used for reading Markdown files, could be replaced with Unified and [react-excerpt](https://github.com/manovotny/remark-excerpt)). Date picker by [react-datepicker](https://reactdatepicker.com/).
- added stars to posts overview indicating the favorite posts of mine
- reorganized blog categories, inspired by the structure of [Julia](https://jvns.ca/blog/2019/02/17/organizing-this-blog-into-categories/) and [Will](https://lethain.com)
- added table of content with [rehype-toc](https://jstools.dev/rehype-toc/) if a post requests so in the frontmatter
- added external link symbol: basic css [here](https://davidwalsh.name/external-links-css), symbol [copied from Quora as SVG](https://www.quora.com/Is-the-symbol-for-external-link-available-in-Unicode-If-so-how-do-I-get-in-on-my-Mac?share=1), SVG then serialized to CSS using [this tool](https://yoksel.github.io/url-encoder/)

## 2021-03

- switched to [dark code theme](https://prismjs.com/) that I find personally more readable
- added [changelogs](/post-changelog.md) to posts, using data from Markdown frontmatter
- added links to headers with [rehype-autolink-headings](https://github.com/rehypejs/rehype-autolink-headings) and [rehype-slug](https://github.com/rehypejs/rehype-slug)
- added this changelog

Changelog of this blog (meta)


Eliot Sober studies the role of altruism in evolutionary biology and psychology. In the paper [Kindness and Cruelty in Evolution](https://www.researchgate.net/publication/30528984_Kindness_and_Cruelty_in_Evolution) he observes that in evolution a group of altruists outperforms a group of self-oriented individuals.

Can we translate that to writing code? Let's compare two extremes:

- A team of altruistic devs: each dev writes code so that _other devs_ can understand the code easily. The focus is on clean code and helping others.
- A team of self-oriented devs: each dev writes code so that they, individually, deliver features as fast as possible. The focus is on immediate short-term gains.

Now, as years go by, which team, codebase and product has a higher chance of survival? And if so, what is it that incentives us to be self-oriented?


Clean code: can we learn from altruism & evolution?


I see more and more similarities between coding and writing. Here's one advice on writing:

> Concision isn't the number of words on the page; it's how long it takes readers to process what's on the page. You need to put readers in the equation all the time. Language is social.
>
> — Larry McEnerney (from [Writing Beyond the Academy](https://www.youtube.com/watch?v=aFwVf5a3pZM))

The same applies to writing code. Concision isn't about the lines of code or short variable names; it's about how long it takes fellow devs to process and understand code. You need to put fellow devs in the equation [[comments-with-domain-knowledge|all]] [[clean-code-can-we-learn-from-altruism-and-evolution|the]] [[simple-components|time]]. Code needs to be painfully obvious.

Code is social.


Coding and writing are alike


I started experimenting with adding domain knowledge to code comments. It turned out that such comments were super helpful to others. They even inspired others to write such comments themselves. So if they are so useful, why is it that we don't write them more often?

## A bit of context: Say hello to hybrid inverters

To give a bit of context, I've been working on an application that calculates investment costs for photovoltaic (PV) systems. Specifically, I've been adding new investment logic for hybrid inverters. I knew what an inverter was: a device changing DC current to AC. However, I didn't know a thing about _hybrid_ inverters & how they relate to investments.

So first, I needed to figure out what a hybrid inverter even was. I learned that if you install a PV system with a battery, you need a hybrid inverter. It turns out a hybrid inverter is a device that combines a solar power inverter & a battery power inverter into one. For the investment costs, a hybrid inverter relates to both the PV system & the battery system.

![hybrid-inverter](/img/hybrid-inverter.png)

## Updating the code & adding the comment

Once I had hybrid inverters figured out, I noticed it was easier to understand the functional requirement at hand. Also, I could better understand the existing code & I was quick to update it. I ended up with this Typescript code:

```typescript
const GetInvestmentCost = {
  forHybridInverter(/*...args...*/) {
    /*...body...*/
  },
  forBatterySystem(/*...args...*/) {
    /*...body...*/
  },
  forPVSystem(/*...args...*/) {
    /*...body...*/
  },
}
```

Out of the total time I spent on this update, most of the time went to understanding the domain background: the hybrid inverters & their relation to investment. This knowledge, specific for this particular feature, hadn't been captured anywhere. I also noticed that the domain knowledge was loaded in my mind, readily available for usage. So I added what I knew to the code as a comment. As brief & to the point as possible. It took a few minutes.

```typescript
/*
 * If you install a PV system with a battery,
 * you need a hybrid inverter. A hybrid inverter
 * combines a solar inverter & battery inverter
 * into one device.
 *
 * Now, for investment cost, we put the hybrid
 * inverter separately, as it doesn't exclusively
 * belong to neither the PV system costs nor
 * the battery system cost. The investment cost
 * for the battery & PV system excludes
 * the cost of the hybrid inverter.
 * */
const GetInvestmentCost = {
  forHybridInverter(/*...args...*/) {
    /*...body...*/
  },
  forBatterySystem(/*...args...*/) {
    /*...body...*/
  },
  forPVSystem(/*...args...*/) {
    /*...body...*/
  },
}
```

It's not perfect, but it'll do. I aimed to bridge the gap between the code & the domain. I assumed the next person looking at the code will understand the logic faster & thus make their change faster.

There's another thing I aimed for: the explanation & the code now sit right next to each other. As [Rich Hickey explains](https://www.infoq.com/presentations/Simple-Made-Easy/), when we put related things together, we understand the system more easily. And so we add features and fix bugs quicker. But if we put related things apart, such as the code and the domain knowledge driving the code, the system is harder for us to understand. This scatteredness of things, these _inconsistent_ locations, is a form of complexity. It slows down our understanding of systems & speed of change.

## More examples

Encouraged, I added a bunch of other domain comments to the code I worked with:

```typescript
/*
 * With net metering, also called reverse-count metering,
 * the meter rolls back when you inject energy back
 * to the grid. The grid serves as a free & fully
 * efficient virtual battery.
 */
function NetMeteringCashFlow(/*...args...*/) {
  /*...body...*/
}
```

```typescript
/*
 * With market valorization policy, you are reimbursed
 * for the energy injected into the grid, according
 * to the injection tariff. For the energy consumed
 * off the grid, you pay the regular off-take tariff.
 * Both the injection tariff & the off-take are specific
 * to your grid operator.
 */
function MarketValorizationCashFlow(/*...args...*/) {
  /*...body...*/
}
```

## Feedback from fellow developers

I was curious what other devs thought about this, so I sent them a few snippets and asked about their opinion. Their answers were so interesting that I asked them if it's ok to include them here. They agreed, so here they are:

> There are so many concepts and business rules to be changed relatively often, adding more context to the right place is incredibly helpful. However, I also think doing this is difficult and I'm very interested to see how well will the comments age with coming time.

When asked more about the difficulty, he replied:

> My personal hypothesis for why I haven't been able to write such comments more is that for me (and I suspect many other developers), writing is a different skill from coding. I'm much less experienced at it and it's difficult for me to switch back and forth between. Because of the "free form". It also feels much less forgiving and more permanent than code. <br/><br/>The remote work environment, like we have, with more asynchronous and written communication, may be very helpful in promoting this. In my previous jobs we preferred talking over writing, attempting to make all knowledge shared by implicit osmosis (extreme being pair/mob programming) rather than explicit knowledge base (extreme being almost complete asynchronicity like Gitlab culture).
> Like with anything interesting, there are constraints and trade-offs for both approaches, but the more I think about it, the more comments like these seem to worth pursuing and I'm excited to try it as well.

Another developer answered:

> I am all for it. The downsides are hardly there. There is a maintenance burden for the comments, but that is usually removing/editing here or there. It also has the upside of assisting code review.

And another one:

> I'm extremely in favor. I was looking for a value yesterday and was unsure of what it was from the ticket, and if it had a comment like the above, it may have been easier to decipher what that value actually did.

So it seems others find such comments also super helpful, but they see two possible issues: writing & maintenance.

## The difficulty of writing

The difficulty of writing resonates. I feel it too, especially since English is not my native language. What has helped me is redirecting attention to learning. It turns out that non-fiction writing is a skill I can try to improve. The best resource on non-fiction writing I have found so far has been [Steven Pinker's The Sense of Style](https://www.goodreads.com/book/show/20821371-the-sense-of-style) (thanks to Gergely [for this tip](https://blog.pragmaticengineer.com/on-writing-well/)). The book is an eye-opener. It's an entertaining, practical & easy read that introduces basic concepts behind nonfictional writing. It puts logic & process & rules into the writing process. I like that.

Another strategy, thanks [Alexey](https://guzey.com/) for the tip, is to use bullet-points as they are easier to write.

## Comments maintenance

Now for the maintenance. Imagine I need to make a change to a complex code. So first, I need to understand the code. I try to get all the help I can get, such as looking into the code, the comments, reading docs & discussing with the team. Once I understand the code & the domain context, I can proceed and make the change. As I finish, I am about to move on to another problem. Now shall I invest a few minutes in writing (or updating) the comment or not?

No matter how I decide now, the cost of gathering knowledge has already been paid. To assess if to comment or not, I need to consider only the cost of writing the comment, not the cost of obtaining the knowledge for the comment.

So let's say I decide to invest my time and write the comment as part of the change A. With the comment, the next person updating the code with the change B will have most of the explanation they need already at hand. They wouldn't need to pay as much time as I did to understand the code. So for the two of us, the total return on my investment would be positive & high.

![time-saved-investment-yes](/img/time-saved-investment-yes.png)

What if I decide to save time & not write the comment? The upside is that I will deliver the change A faster since I am not writing the comment. The downside is that the next person would need to start from scratch for their change B. They would need to figure out all the heavy stuff again. So for the two of us, the total return on my investment would be negative.

![time-saved-investment-no](/img/time-saved-investment-no.png)

Now you may wonder: it's one change, does it matter? Perhaps not much for one change. But imagine more changes, more functions & more developers. The savings start to compound.

![investment-change-more](/img/investment-change-more.png)

A friend remarked:

> Taking the time to explain something for the next dev is usually even more of a good idea because the next dev to read the code will probably be yourself, so even when you feel selfish, you can think of it as a gift to your future self.

## Conclusion to this experiment

All in all, it seems to me that if we adopt domain comments throughout the codebase, we accelerate the engineering pace. Perhaps even significantly. So for now, I'll try to write more of them and see how it goes.

## Won't comments get out of sync?

It seems that domain comments don't change that often. A hybrid inverter's relation to a PV/battery system stays the same if we refactor imperative code into functional code or extract some code into a reusable component. Also, we can always try to encourage consistency, for example, via code reviews.

## Shouldn't good code document itself?

When discussing the domain comments with a friend of mine, he made this remark. I find it spot on:

> Yeah, if code is so clear that it needs no further comments, that is great. But honestly, if you look at code, how often does that encapsulate everything you've learned about the domain. If it doesn't, why not add your knowledge in the comments?

## Further reading

- Joel Spolsky, [Code Tells You How, Comments Tell You Why](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/)
- Donald Knuth, [Literate Programming](http://literateprogramming.com/index.html)

_Thanks a lot to Reinier, Peter, Andreas, and Julien for reading this & making it better._


Adding domain knowledge to code comments


One of the hard things in building software, I realized, are decisions: Which tech debt to tackle? How to architect this feature? What to focus on?

> If you don't know where you are going, any road can take you there.<br/>
> — Lewis Carroll, from [Alice in Wonderland](https://www.goodreads.com/work/quotes/2933712)

What helps me to navigate the maze is an _architectural vision_. An architectural vision maps out what the applications should look like in the future from the architectural perspective. The vision should be reachable and reasonably challenging. For me, I'm excited when the vision is on the edge of my comfort zone—something to stretch my abilities. I've found that thinking in terms of one or two years from now is a reasonable timeframe: long enough to unlock substantial changes, short enough to be hands-on.

How does an architectural vision look like? Stripped to bare essence, it could be as short as a sentence. For example, a company has a CRM system causing them only headaches. They discover that customer relationship management is, in fact, their [[to-buy-or-to-build|core]]. So their architectural vision could be: _The CRM is built in-house_.

Another example: a company operates in two different lines of business. The IT architecture supports both lines. Because of the shared foundations, the team faces mounds of complexity and overhead. So the architectural vision could read: _IT systems are split into two separate buckets: systems serving retail customers and systems serving the infrastructure customers._

For me, such vision is a celestial compass, showing the path in everyday decisions. And with such a compass, [[kaizen|the days feel different]].

![architectural-vision](/img/architectural-vision.png)


Everyday decision making & architectural vision


I'm moved by the works of [Marshall Rosenberg](https://en.wikipedia.org/wiki/Marshall_Rosenberg). In the book [Living Nonviolent Communication](https://www.goodreads.com/book/show/14361179-living-nonviolent-communication), Marshall tells a story of how he mediated peace negotiations between two African tribes at violent war.

In Chapter I, subtitled _Resolving Conflicts Peacefully and Powerfully_, Marshall starts off by laying out generic steps to resolve conflicts. He follows the principles laid out in his previous work, [NonViolent Communication](https://www.goodreads.com/book/show/71730.Nonviolent_Communication). We then peek into a counseling session, where Marshall uses the steps to mediate a married couple's life-long financial dispute. Within one session, the couple resolves the dispute to the benefit of both the wife and the husband.

At this point, Marshall moves on to a different use-case—violent war. The move startled me: I didn't expect to see the steps from marriage counseling to be applied in peace negotiations. I'm wondering: _could all conflicts and tensions be resolved using Marshall's approach?_

I include the peace negotiation story from the book:

<div class="blockquote">
	
>...As we were walking into the session, my colleague whispered to me, “Be prepared for a little bit of tension, Marshall. Three of the people in the room know that the person who killed their child is in that room.” It was very tense at first. There had been so much violence between these two groups, and it was the first time they had really sat down together.
>
> I started with the question with which I frequently start conflict resolution sessions in order to focus on people’s needs. I said to both sides, “I’d like whoever would like to speak first to say what your needs are in this situation. After everyone understands the needs of everyone else, then we’ll move to finding some ways of meeting the needs.”
>
> The chief from one side looked across the table and said, “You people are murderers,” and the other side responded, “You’ve been trying to dominate us. We’re not going to tolerate it anymore!”  ...
>
> I turned to the chief who had said “You people are murderers” and guessed, “Chief, do you have a need for safety and to be sure that whatever conflicts are going on will be resolved by some means other than violence?” The chief immediately said to me, “Of course, that’s what I’m saying!” Well, of course, he hadn’t said that. He’d said that the other person was a murderer and had made a judgment rather than expressing his needs. However, now we had his needs out on the table, so I turned to a chief from the other side and said, “Chief, would you please reflect back what he said his needs were?” ...
> 
> The chief responded to this man by asking in a very hostile way, “Then why did you kill my son? That started an uproar between the two groups. After things calmed down, I said, “Chief, we’ll deal with your reaction to his needs later, but at the moment, I suggest that you just hear his needs. Could you repeat back what he said his needs were?” He couldn’t do it. He was so emotionally involved in this situation and in his judgments of the other person that he didn’t hear what the other person’s needs were. I repeated the needs as I had heard them and said, “Chief, I heard the other chief saying that he has a need for safety. He has a need to feel secure—that no matter what conflicts are present, they’ll be resolved in some way other than by violence. Could you just reflect back what that need is, so that I’m sure everybody’s communicating?” He couldn’t do it. I had to repeat it two or three times before he could hear the other person’s needs. ...
>
>  After I spent this much time getting both sides to express their needs and to hear each other’s needs (this took close to two hours), another chief who hadn’t spoken jumped to his feet, looked at me, and said something very intensely in his own language. I was very curious about what he was trying to express to me with such intensity, and I eagerly awaited the translation. I was very touched when the translator said, “The chief says we cannot learn this way of communicating in one day. But he says that if we know how to communicate this way, we don’t have to kill each other.”
> 
> In fact, before I left that day, we had members from both tribes eager to learn this process that would allow everyone to hear needs behind whatever message was being expressed. I am happy to report that the war between the tribes ended that day.

</div>


Resolving conflicts peacefully and powerfully


A client recently asked for my input in the following situation: the company works with an external contractor, and the contractor worked for months on a project. When the contractor delivered the project, the output wasn't what had been expected. The misalignment resulted in tension between the company and the contractor, as well as in tension _within_ the company. Now, how to resolve this?

To understand my client's needs, I first asked the client about their exact concern: "Are you interested in resolving the _current_ conflict, or is it that you are concerned about the _future_ conflicts from happening?" I asked because resolutions differ for the two. Plus, my background in conflict mediation is mostly theoretical, so I was hoping for the latter. Thankfully, that's what the client was interested in.

To prevent the conflict from happening, I suggested three strategies.

![contractor-didnt-deliver](/img/contractor-didnt-deliver.png)

The first strategy is changing the frequency of the contractor's deliverables. The contractor worked for months in isolation. The longer the isolation, the higher the risk of a discrepancy between expectations and deliverables. An alternative way is to deliver the project in smaller, more frequent pieces. The contractor can start from an [[proof-of-concept|embarrassingly simple proof of concept]] and iterate over it. This applies to both software engineering and [design](https://basecamp.com/shapeup/3.2-chapter-11#case-study-clients-in-projects).

Another strategy, which can be combined with the first one, is to flash out the exact description of what needs to be done in a _functional spec_, also known as a _design doc_. I learnt about design docs first from [this hilarious blog series by Joel Spolsky](https://www.joelonsoftware.com/2000/10/02/painless-functional-specifications-part-1-why-bother/), as well as from [this pragmatic post by Julia Evans](https://jvns.ca/blog/2016/06/03/learning-to-like-design-documents/).

The third strategy is to focus on the communication aspect. As I [[conflict-resolution|wrote earlier]], I see the works of Marshall Rosenberg on [Nonviolent Communication](https://www.goodreads.com/book/show/71730.Nonviolent_Communication?from_search=true&from_srp=true&qid=6aJYMcNkFS&rank=1) to be a powerful approach for conflict resolution and conflict prevention. When I follow the _essence_ of the principles–as opposed to shallowly repeating the suggested phrases–I see a notable change in my professional relationships. I believe the same change can be achieved in a company-contractor relationship.


Contractor didn't deliver: how to resolve?


In this post, we'll look at
- which aspects are required to develop a data-driven software product,
- how data scientists and data engineers fit these aspects,
- how to detect if your team needs a data engineer, and
- how to find a data-engineer for your team, if you need one.

## Three aspects for any data-driven product
Developing a data-driven software product is not only about analytics.
In fact, there are three aspects required for a product to succeed: Consulting, Analytics, and Automation.

Let’s dive into each of them individually.

#### Aspect #1: Consulting
Consulting is about
- translating the business problem into analytical terms,
- collaborating on a business-case, and
- validating outcomes with (internal) business clients.

It requires business-acumen and desire to understand the domain.

#### Aspect #2: Analytics

Analytics is about extracting insights from data to solve a given business problem. It requires in-depth knowledge in:

- preparing and cleaning the data/features,
- simple models like excel and data exploration/vizualization,
- complex models to reach out for if needed, like machine-learning, statistics and simulations.

The focus here is to demonstrate via a proof of concept that the analytical
solution adds value to the business.

#### Aspect #3: Automation

Automation is about optimising & preparing the product for long-term use.
The proof-of-concept is improved to production-level code, using  [software-engineering best-practices](/sebp).

Underestimating this step might result in:

- low confidence in the product from end-users, due to many unexpected bugs,
- an unexpected loss of production data,
- unexpected long-term service unavailability,
- inability to promptly add a feature/fix an issue.

Automation requires software-engineering skill set tailored for data-driven products.

## Data Scientists vs. Data Engineers
Data Scientists usually come from mathematical/statistics/operation-research/machine-learning background. Having business acumen, they are strong in both analytics and consulting. They live and breath for data insights and applying models to solve a business problem.

Data Engineers, on the other hand, come from software-engineering background. They live and breath for automation. They understand how to ship [high-quality production-level code](/sebp), including code readability, testability, architecture, DevOps, automated-deployment, robust ETL, etc.

Data engineers can also speed-up delivery of analytical parts by providing
technical support for data scientists. For example:

- create a discovery platform for data scientists (e.g. IPython notebooks automatically hooked to a cleaned data warehouse representing a single source of truth),
- provide infrastructure improvements for data scientists (e.g. a common Docker image with all data-science tooling installed),
- create shareable libraries, boilerplates, etc.

## Skills mapping: Summary

Here's a summary of expected expertize in the three discussed aspects
by role.

| Expected expertize      | Data Scientist | Data Engineer |
|------------------------|----------------|---------------|
| Consulting             |   :medal:  Strong     |      Basic      |
| Analytics              |   :medal:  Strong     |      Medium    |
| Automation/Engineering |      Basic     |     :medal: Strong    |

_NB: As applicable to any role, the more a data-engineer
knows about analytics and consulting, the better. And vice versea,
the more a data scientist knows about automation and engineering, the better._

## Do you need a Data Engineer?

This question should be simple simple to answer: Ask your team which activities they spend most of their time on.

If the answer includes mostly manual deployments, getting access to data, re-cleaning the data, code refactoring, application monitoring, dev-ops, fighting Spark/Hadoop/Kafka/Yarn issues, then you probably need an additional Data Engineer.

If the answer is modeling, feature-engineering, vizualisations, communicating
 with an internal customer, you are probably
not in need of additional Data Engineer.


## How to find Data Engineers

In the current job market, the demand for data engineers
exceeds supply. In this context, there seems to be two viable options
on how to get an additional data engineer to the team:

#### Option #1: Become an attractive workplace

Become an attractive workplace so that data-engineers come to you:
start open-source initiatives & analytical blogs,
strengthen conference presence, start organizing local meetups.

This option is the harder one but it pays off in long-term.

#### Option #2: Turn generalists into specialists

Alternatively, hire software-engineers who are generalists.
A strong generalist (e.g. a Python/Scala developer)
would grasp the required stack fast and would be a great
engineering complement to your existing team of data scientists.


## More resources

- [Infographics: Data Engineering vs. Data Science](https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic)
- [Video (8min): Data science and data engineering, the two sides of a dysfunctional marriage](https://www.youtube.com/watch?v=NFumbHzfk94&feature=youtu.be&lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BIi8eNkkySdSW2h8VZxMW7A%3D%3D)
- [The Data Science Hierarchy of Needs](https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007)
































Should you hire a data engineer instead of a data scientist?


If you are considering using Docker for data-science projects, in this post you'll find the conceptual advantages of migrating data-science/big-data/machine-learning projects onto Docker.

In the post, we start with the topic of team productivity.

### Exec Summary: Less technical and communication friction

Based on my experience, teams using Docker experience a decrease in technical and communication friction, namely due to the following advantages that Docker offers:

- New colleagues get onboard quickly
- System documentation is explicit and always up-to-date
- Docker eliminates most work-on-my-machine problems
- Docker enables transparent and peer-reviewed infrastructure updates
- Docker streamlines technical collaboration across teams
- Docker sharpens responsibility

These advantages come with a cost—the necessity to learn new tooling.

Let's dig into each of those individually.

### Advantage #1: New colleagues get onboarded quickly

With Docker, it is a matter of one command to get an application up and running. The ease of spinning up a complete application also holds for complex cases where an application requires custom binaries, tweaks in system configuration, a database, or a cluster-like setup.

With Docker, it is not required anymore to spend days getting an application running on a new laptop. I once worked on a micro-service-based application consisting of 20+ interconnected services. Docker allowed us to spin up the whole stack easily with just one command.

To learn more, check out `docker-compose`.

### Advantage #2: System documentation is explicit and always up-to-date

Docker uses files to define application system requirements and links between them. These files are human-readable, declarative, and define the desired end-state of an application. They are typically committed to a version control system. Thus, they serve as excellent documentation of all system and application dependencies.

With Docker, it is not necessary to manually document system requirements. The documentation is, by definition, always up-to-date and on-par with the corresponding application version because the definition of the system requirements is the documentation itself.

To learn more, check out `Dockerfile` and `docker-compose`.

### Advantage #3: Docker eliminates most work-on-my-machine problems

How many times have you encountered an issue that an application behaves correctly on one developer machine but misbehaves on another developer machine? Issues like different path separators, missing undocumented system configuration, missing libraries, and customer binaries (e.g., linear programming tools).

Since Docker packages the application together with all its system dependencies (operating system, binaries, configuration), all developers work with the same infrastructure.

Useful concepts: `Dockerfile`, `docker-compose.yaml`

### Advantage #4: Docker enables transparent & peer-reviewed infrastructure updates

Docker defines application infrastructure and dependencies in files. These files are typically committed to a version control system (for example, Git or Mercurial).

Thus, best-practices for team collaboration like peer reviews and pull-requests can be applied to Docker files as well. A whole team can thus transparently review any suggestions for any updates for system configuration, custom binaries, and libraries.

### Advantage #5: Docker streamlines technical collaboration across teams

Docker encapsulates everything needed to run an application into a single _Docker image_ and stores those images in a centrally accessible location. This mechanism enables a team to reuse an image developed by another team quickly.

Let's take a use-case where different teams concurrently develop a new set of services that depend on each other: a new version of a frontend coupled with a new REST API backend, for example. Each team would welcome having an option to quickly spin up the whole application stack (the frontend and the backend) locally for testing purposes. Docker enables such an option, for example, through docker-compose.

### Advantage #6: Docker sharpens responsibility

If you work for an older company where IT is not a core business, but rather, an optimization around a core business, chances are you will find yourself working in organizational silos. In this model, there's a lack of end-to-end responsibility for a product: one team develops a model, while another team deploys the model. This split leads, among other things, to high communication overhead, common technical misalignments (for example, different library versions between development machines and production environment), cumbersome application monitoring, and unclear support boundaries.

Ideally, these silos would be merged, and there would be a single team end-to-end responsible for a product. If that's not possible, Docker might partially help.

Docker allows shifting responsibility for application system requirements to the development team. At the same time, the responsibility of the deployment team would sharpen as well: Instead of supporting a wide range of applications half-way, the deployment team can be end-to-end responsible for deploying any application that is Dockerized.

### Disadvantage #7: Necessity to learn a new tool

Docker is an additional tool that team members need to have in their toolbelt. One option to learn Docker is to experiment on your own and invest time in exploring and learning the tool. However, team members need to find out what they don't know and learn it.

### Bring Docker advantages to your team

An in-house Docker training specialized for data-science teams and geared towards your specific needs is an excellent option to learn Docker fundamentals effectively. ~~My Docker training~~ (Update from 2020: I no longer to these) offers high-level conceptual overview coupled with practical, focused, hands-on exercises. Also, your team would learn about hard-won lessons, pragmatic best-practices and conceptual high-level overviews which are all typically hard to find on the interwebs.


Why Docker for Data Scientists


For longer, I missed something in my communication, something between writing and talking. There's a whiteboard, that's not an option for me since I work remotely. Then there are online diagramming tools, but they felt cumbersome to use: it took me long to draw & I often couldn't find the shape I was looking for.

![drawing-writing-talking](/img/drawing-writing-talking.png)

Two years ago, I stumbled upon Doug's [course on sketch noting](https://www.verbaltovisual.com/our-courses/). I learned the basics of technical drawings, but it didn't occur to me I could use drawings for software-related concepts. Then a few weeks ago, I stumbled upon [Julia's blog](https://jvns.ca/) and her technical drawings. It turns out she wrote [a bunch of articles](https://jvns.ca/#on-writing-comics---zines) where she explains her drawing process. Inspired, I read them all. And then, the final drop: [Ryan's article on how he wrote ShapeUp](https://m.signalvnoise.com/how-i-wrote-shape-up/). I was hooked & determined to give drawings a try.

So I bought a used iPad 7 and an Apple Pencil 1st generation. I installed Notability: a simple note-taking app that both Julia and Ryan used in their beginnings. And I started experimenting.

A few days ago we finished a large refactoring project. I thought that would be a good time to experiment. So I decided to use drawings to explain the thinking process behind the big refactoring we did. Here are a few drawings from the talk:

![writing-first-steps-example](/img/writing-first-steps-example.png)

I didn't know what to expect. I was surprised to find out that my coworkers enjoyed the talk & they were quick to understand the driving principles behind the project. So it seems the drawings worked well. I also noticed I liked the process of making the talk: I could express myself better & it was fun.

It's hard to image now going back to the world without the drawings. Something clicked. I think I'm going to experiment with them more.

---

Some updates on the post after it was published:

- I stumbled upon [this interesting Twitter thread](https://twitter.com/GergelyOrosz/status/1337406214284652553) about various drawing apps for iPad. Also, if I ever need to make my own font, [here's a recommendation](https://twitter.com/b0rk/status/1337867237743595521).
- I was asked if I ever used pen & paper. I did when learning with Doug's video courses. However, I've discovered that I edit the drawings a lot, especially erasing & moving shapes around. For that reason, the digital version has been a huge time saver for me. On the other hand, one can [go pretty far](https://jvns.ca/blog/2019/09/01/ways-to-write-zines-without-fancy-tools/) with just a pen & paper. So to summarize: I started with pen & paper, the process resonated, so then I upgraded to digital.


Drawings for technical explanations: my first steps

This week I attended a knowledge sharing session about the DynamoDB. I knew Dynamo's basics concepts—I used different key-value NoSQL db before, such as Redis, MongoDb or HBase. But I was curious about the technical details of DynamoDb because this has been new territory for me. So here's what I've learned:



- Dynamo emerged to the public in 2007 when Amazon engineers published **the [Dynamo Paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)**, where they presented a solution to their ever-increasing demands for Amazon's key-value store. The paper was first mentioned in [Werner Vogel's blog](https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html), the CTO of Amazon.
- As it's typical for NoSQL dbs, Dynamo is a key-value store optimized for high read/write throughput and low latency. Dynamo is also fully-managed by AWS. That means no need to care about servers, plus there's high-resiliency and auto-scaling out-of-the-box. There's no need to provision for the peak load. Dynamo is **public**, that is, it's not hosted within VPC.
- Dynamo organizes data into **tables**. A table consists of **items** (rows) and **attributes** (columns). Dynamo supports three attribute types: `Binary`, `Number`, and `String`. When creating a table, we don't specify the schema; we specify the primary key (see the next point).
- Each table has **a primary key** that uniquely identifies each item in the table. It consists of a mandatory **partition key** and an optional **sort key**. The partition key determines data distribution (see the next point), and is identified as `HASH` in code. The sort-key enables richer querying and is identified as `RANGE` in code. A primary key consisting only of a partition key is called a **simple primary key**; a primary key consisting of both a partition key and a sort key is called a **composite primary key**. (Phew, this took me a while to grasp and explain)
- Tables are internally stored in **partitions**. The split into partitions is determined by the primary key, specifically by the partition key. Once a table is created, the primary key and partitions cannot be changed.
- Data are read with two types of operations: **Query** and **Scan**. A query searches within a partition, a scan searches all partitions. If speed matters, use queries.
- When **minimizing latency or costs, consider query patterns**, that is, how do you read data. Then, align primary keys with your query patterns. For newly emerged query patterns, consider [secondary indices](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-general.html).
- Dynamo charges by **read capacity units (RCU)** and **write capacity units (WCU)**. Capacity units attach price for reading/writing a given size of data within one second. Capacity units are allocated in two modes: **provisioned capacity** or **on-demand capacity**. With provisioned capacity, units are fixed: if exceeded, the data is lost, and Dynamo raises an exception. There is no queue for the lost data or anything like that. With on-demand capacity, units are scaled on demand. So on-demand is handy, but it comes with a risk of high cost in case of an unexpected event (such as a DDOS attack).
- Dynamo supports **TTLs**. When the time to live is exceeded, the given item is automatically deleted. Handy for temporary data, such as sessions or event logs.
- Similar to RDS, Dynamo features **backups, point-in-time recovery**, and **data encryption at rest**.
- Dynamo supports **in-memory cache for improved performance**, called [DAX](https://aws.amazon.com/dynamodb/dax/). Useful for read-intensive apps that require extremely low latency, such as trading or gaming.
- By default, Dynamo offers **[eventually consistent reads](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html)**. It's possible to request **a strongly consistent read**, at the trade-off of slower reads and higher price.
- Dynamo also supports event-driven integration via **streams**, where a change to an item invokes AWS Lambda or Kinesis Firehose. Streams are like triggers or stored-procedures in RDBMS dbs, but disconnected from the table space. Here are [posts](https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/) [with use-cases](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/streamsmain.html). Streams support inserts, updates, and deletes.
- When interacting with Dynamo from Javascript/Typescript, consider **[marshalling/unmarshalling utilities](https://stackoverflow.com/questions/44535445/unmarshall-dynamodb-json)** to simplify code and prevent issues with [reserved words](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ReservedWords.html)

Thanks a lot to Evgeny for sharing most of these tips.

### Side note on knowledge sharing sessions

I'm discovering more and more that I enjoy knowledge-sharing sessions. I learned a lot in a very short time (it's effective), I can ask questions (it's interactive), I can discuss the topic with others (it's collaborative). The same applies when I host a session: I can solidify my knowledge, discover gaps, and discuss the topic with others.

### CloudFormation example

Here's an example of a Dynamo table provisioned via [CloudFormation Serverless template](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification.html). The template provisions a table called `charging_sessions` with an attribute `start_date` set as a simple primary key. The table is configured with provisioned capacity units. The template also contains an example Lambda function that accesses the Dynamo table.

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

ChargeSessionsTable:
	Type: AWS::DynamoDB::Table
	Properties:
		TableName: 'charge_sessions'
		AttributeDefinitions:
		  - AttributeName: 'start_date'
		AttributeType: 'N'   # type Number
		KeySchema:
		  - AttributeName: 'start_date'
			KeyType: 'HASH'  # simple primary key
		ProvisionedThroughput:
		  ReadCapacityUnits: 10
		  WriteCapacityUnits: 10

AddChargeSessionLambda:
	Type: AWS::Serverless::Function
	Environment:
		Variables:
			CHARGE_SESSION_DDB_TABLE: !Ref ChargeSessionsTable
	Policies:
		- DynamoDBCrudPolicy:
			TableName: !Ref ChargeSessionsTable
	# Properties, CodeUri, etc...

```

DynamoDB: key technical concepts & features


I believe that one of the reasons why companies fail is a lack of focus and fear of simplicity. They focus on too many things at once. Have too many customer-specific features and processes. Want to squeeze a dollar from _every_ opportunity possible, not realizing the incurred debt.

Doing everything yet doing nothing properly.

I believe this is a road to disaster.

I have seen this multiple times.

Processes get complicated. IT development slows down. Products get harder to understand. More people are needed to handle the newly created complexity.

## Example 1: Building a generic solution instead of pushing for simplicity

Once, my team was given a task to build a generic pricing rule engine. The engine was supposed to support all the current and legacy business deals. Since the company had been lacking focus, the pricing had gotten out of hand. It got so complex that there was only one person in the whole company who understood the pricing intricacies completely. He tried to explain it to us many times, but the complexity was overwhelming. There were exceptions everywhere.

We should have paused and [not ignore the quenching](/quench). We should have pushed back to remove as much exceptions as possible. To simplify our pricing model.

This way, all parties would win:

- Pricing would be easy to explain and simple to understand.
- [Bus-factor](https://en.wikipedia.org/wiki/Bus_factor) would increase.
- Development would be quicker.
- Processes would streamline.

Instead, we have built the engine. It sure brought advantages. However, I believe that removing the root cause would bring benefits of another level.

## Example 2: Expanding B2B product with B2C capabilities

Another time, another team. We were given a task to expand our [B2B](https://en.wikipedia.org/wiki/Business-to-business) platform with [B2C](https://en.wikipedia.org/wiki/Retail) capabilities. The overlap between B2B and B2C features were minimal. The B2B platform was hardly stable.

We [followed the thirst](/quench). Again. And things got very complex very quickly.

We were very slow. Discussions. Hacks.

It felt like fitting a circle in a square.

Instead, we should have focused on two things: polishing the B2B platform and start a green-field B2C product.

- The development would be faster
- Our end-users would understand the products easier.
- We would end up in two [loosely-coupled yet highly-cohesive](http://stackoverflow.com/questions/3085285/cohesion-coupling) products.
- Oh, and we would enjoy the hell out of it!

## Strive for focus and simplicity

Choose one thing and be the best at it. Stop marginal legacy business lines. Refuse niche requests from small customers. Simplify as much as possible.

Only when the product is saturated, expand. Do it with sharp focus. Create another product which will be the best at what it offers.

Simplify and keep focused!


Strive for focus & simplicity

I'm building a Typescript app, and I have reached a level of [accidental complexity](/simple-components) that I'm not able to remove. I believe a paradigm change is needed to cross the chasm. Namely, to introduce parts of functional programming (FP). Now, to start with functional programming in Typescript, I need two things: a library to use as a backbone and resources to learn from. Here's what I've found so far.



## Libraries

Based on what I've seen, the most popular libraries are [Lodash](https://lodash.com/), [Ramda](https://github.com/ramda/ramda) ([Rambda](https://github.com/selfrefactor/rambda), [Rambdax](https://github.com/ramda/ramda)) and [fp-ts](https://github.com/gcanti/fp-ts). It seems that Lodash and Ramda and Rambda were built for JS first, with Typescript types added later, while fp-ts was built for Typescript first. One can see that in the interfaces:

```typescript
# rambdax
import { pipeAsync } from 'rambdax'
await pipeAsync(foo)('hello')
// rambdax's pipeAsync is curried,
// the compiler has no way to check if foo is compatible with 'hello'

# fp-ts
import { pipe } from 'fp-ts/function'
pipe('hello', foo)
// fp-ts's pipe is not curried,
// the compiler can check the compatibility of foo with 'hello'
```

Interestingly, fp-ts also contains `Task` that can replace `Promise` for better type safety. See [Should I use fp-ts Task?](https://dev.to/anthonyjoeseph/should-i-use-fp-ts-task-h52) post.

I'm intrigued by fp-ts.

## Resources

I've found these posts to be excellent resources for starting with FP:

- [Practical Guide to Fp‑ts](https://rlee.dev/writing/practical-guide-to-fp-ts-part-1)
- [Mostly adequate guide to FP](https://mostly-adequate.gitbook.io/mostly-adequate-guide/#play-around-with-code)
- [The State monad](https://paulgray.net/the-state-monad/)
- [fp-ts Learning Resources](https://gcanti.github.io/fp-ts/learning-resources/) and [fp-ts recipes](https://grossbart.github.io/fp-ts-recipes/#/)
- [Functors, Applicatives, And Monads In Pictures](https://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html) (in Haskell, but with pictures)

And here are some more interesting articles/posts about fp-ts:

- [fp-ts and Even More Beautiful API Calls (w/ sum types!)](https://dev.to/anthonyjoeseph/fp-ts-and-even-more-beautiful-api-calls-w-sum-types-53j0)
- [Getting started with fp-ts: Reader](https://dev.to/gcanti/getting-started-with-fp-ts-reader-1ie5), for dependency injection
- [https://samhh.github.io/fp-ts-std/](https://samhh.github.io/fp-ts-std/), dubbed "the missing pseudo-standard library"

## fp-ts example

Here's a non-trivial example ([source](https://dev.to/gnomff_65/fp-ts-and-beautiful-api-calls-1f55)) of fp-ts that includes async execution against a REST API, validation, and error handling. Many things here [decomplected](./simple-components), code is concise, extensible, and with static type checking. The tradeoff is the
cost of learning a whole new functional API.

```typescript
import axios, { AxiosResponse } from "axios"
import { flatten, map } from "fp-ts/lib/Array"
import * as TE from "fp-ts/lib/TaskEither"
import * as E from "fp-ts/lib/Either"
import * as T from "fp-ts/lib/Task"
import { sequenceT } from "fp-ts/lib/Apply"
import { pipe } from "fp-ts/lib/pipeable"
import { flow } from "fp-ts/lib/function"
import { failure } from "io-ts/lib/PathReporter"
import * as t from "io-ts"

//create a schema to load our user data into
const users = t.type({
  data: t.array(
    t.type({
      first_name: t.string,
    })
  ),
})
type Users = t.TypeOf<typeof users>

//schema to hold the deepest of answers
const answer = t.type({
  ans: t.number,
})

//Convert our api call to a TaskEither
const httpGet = (url: string) =>
  TE.tryCatch<Error, AxiosResponse>(
    () => axios.get(url),
    (reason) => new Error(String(reason))
  )

//function to decode an unknown into an A
const decodeWith = <A>(decoder: t.Decoder<unknown, A>) =>
  flow(
    decoder.decode,
    E.mapLeft((errors) => new Error(failure(errors).join("\n"))),
    TE.fromEither
  )

//takes a url and a decoder and gives you back an Either<Error, A>
const getFromUrl = <A>(url: string, codec: t.Decoder<unknown, A>) =>
  pipe(
    httpGet(url),
    TE.map((x) => x.data),
    TE.chain(decodeWith(codec))
  )

const getAnswer = pipe(TE.right({ ans: 42 }), TE.chain(decodeWith(answer)))

const apiUrl = (page: number) => `https://reqres.in/api/users?page=${page}`

const smashUsersTogether = (users1: Users, users2: Users) =>
  pipe(
    flatten([users1.data, users2.data]),
    map((item) => item.first_name)
  )

const runProgram = pipe(
  sequenceT(TE.taskEither)(
    getAnswer,
    getFromUrl(apiUrl(1), users),
    getFromUrl(apiUrl(2), users)
  ),
  TE.fold(
    (errors) => T.of(errors.message),
    ([ans, users1, users2]) =>
      T.of(
        smashUsersTogether(users1, users2).join(",") +
          `\nThe answer was ${ans.ans} for all of you`
      )
  )
)()

runProgram.then(console.log)
```

Getting started with functional programming in Typescript


Today, a colleague of mine recommended a [short video](https://www.youtube.com/watch?v=6XAu4EPQRmY)  by Jonathan Blow,  an acclaimed developer in the gaming industry. Jonathan mentions that sometimes it is better to make a hard decision later in the project rather than sooner. The later you decide: 

- the better you can explore and refine the idea, and 
- the better you know the technical things needed to build the idea.

I like that. I like to think of a project as having the start, the middle, and the end. For the start, I follow [[proof-of-concept|a process that works well for me]]. After the start, we move to the middle, where hard technical decisions often need to be made. Jonathan's approach gives me guidance on how to tackle such decisions with more control.

<div class="blockquote">

> Thing number one: If you get a rough draft of your program together, you can use that to figure out how you really want it to behave.  Some of your ideas about what you wanted to build in the beginning might not have been very good ideas. You can refine those ideas by having something approximating the thing that you were building.  And so the faster you get to that approximation the better. 
>
> Thing number two: The more time you spend working in that space of your approximation to the thing that you want, the more time you spend becoming an expert in the field of this specific application that you're making, the better you get at that subfield of programming, and the better your decision-making about technical issues in that field is going to be. 
>
> So if you make hard decisions later they will be made better, both by a more skilled person and with more contextual information than if you make those hard decisions early on.
>
>—Jonathan Blow, [On solving hard problems](https://www.youtube.com/watch?v=6XAu4EPQRmY)

</div>



Making hard decisions later


Let's talk about function inputs. Look at this Javascript function. What do you think it does?

```javascript
const signUp = (attrs) => {
  const user = saveUser(attrs)
  welcomeUser(user)
}
```

There are questions you might ask: How's the user saved? In the database? How's the user welcome? Do we send the email? Or do we, instead, send events to message queues?

Figuring out answers like this takes time. Sometimes surprisingly long. But it doesn't have to be. When writing code, we can [[clean-code-can-we-learn-from-altruism-and-evolution|help each other out]]: search for all global variables buried in the depths of nested functions and bring the variables up to the daylight for the reader to see.

```javascript
const signUp = (Db, Email, attrs) => () => {
  const user = saveUser(Db, attrs)
  welcomeUser(Email, user)
}
```

Now you made the function honest. All input is explicitly declared and [[coding-and-writing-are-alike|served on a silver platter]] to the reader. And you made the codebase [[simple-components|simpler]]. No more are the inputs scattered around, intertwined with different locations. Inputs now stand together in a single location, consistent. You also paved the way to simpler testing: no need for technical trickery and mocks of implementation details. All in all, the codebase is now [[sustain|easier to sustain]].

![honest-functions](/img/honest-functions.png)

To dive deeper into function inputs, checkout [Pure Happiness with Pure Functions](https://mostly-adequate.gitbooks.io/mostly-adequate-guide/content/ch03.html) in [Professor Frisby's Mostly Adequate Guide to Functional Programming](https://mostly-adequate.gitbooks.io/mostly-adequate-guide/content/).


Honest function inputs


If you do not fully understand how Hive/Impala stores your data, it might cost you badly.

I've learnt the hard way.

## Symptom #1: Weird values in ingested Hive table

You double-checked with `select distinct(gender) from customers` that the `gender` column in your source RDBMS really contains only values `male`, `female` and `NULL`. However, when you ingest the table into Hive (maybe with [Apache Sqoop](http://sqoop.apache.org/) or [Cornet](https://github.com/datadudes/cornet)) and run the check there,  you see that weird values have creeped in:
```
> select distinct(gender) from customers;  // Run in Hive/Impala
+-----------------+
| gender          |
+-----------------+
| NULL            |
| CA 94304        |
| male            |
| Page Mill Road  |
| female          |
+-----------------+
```

## Symptom #2: Inconsistent size of ingested Hive table

You check with `select count(*) from customers` that the table in your RDBMS table has `156,010` rows.  You ingest the table into Hive and BAM! All of a sudden there are 14 more customers.

Maybe the business is doing great and you gained 14 customers before you started the ingestion? Wondering, you check the source table size again.

Still `156,010`.

## Symptom #3: Weird data when copied from another Hive table

This one is by far the most strange one.

You already have the `customers` table ingested in Hive. Values in the `gender` column look fine. The table has correct size of `156,010`. All is fine.

You do some data cleaning with SQL and copy the result into a new table as follows:

```
CREATE TABLE customers_superclean
AS SELECT name, coalesce(gender, 'unknown') FROM customers;
```

You check the size of the new table. BAM! 25 new customers in there. Impossible!


## No errors, no warnings

Neither Hive, Impala nor Sqoop gave you any error or warning. You have no idea what's going on. Somewhere at the back of your head, you start questioning the whole Hadoop infrastructure. You feel like the cool stuff you do all day with the data has been compromised.


## Cause: Hive delimiters present in the data

All these problems can occur if the ingested data contains characters that Hive uses to delimit fields and rows. Typically, these are newlines. For instance, let's assume that the `customers` table in the source RDBMS contains the following data (notice the newline in the first street name):

gender | street
---| --
male | Page Mill Road`\n`CA 94304
female | Great America Parkway

If one naively ingests this data into a Hive table using the default settings (a text file with rows delimited by `\01` and fields delimited by newlines), the data gets broken. Look at the corresponding delimited file in HDFS:

```
male\01Page Mill Road
CA 94304
female\01Great America Parkway
```

When Hive/Impala reads from this file, it finds three customers instead of two. The additional customer is of gender `CA 94304` and has no street specified. On top of that, the street field of the first customer misses the postal code.

## Particularly interesting case: Copying binary data to text file

Assume that the Hive table called `customers` uses Avro or Parquet for data storage and that the data contains newlines. Querying the `customers` table directly via Hive or Impala works as expected. However, let's create a new table as follows:

```
CREATE TABLE customers_superclean
AS SELECT gender, street FROM CUSTOMERS;
```

The problem with this command is that the new table is backed by a newline-delimited text file. An Avro/Parquet record which contains a newline will be split into two records in the new table. Symptom #3 is born.

## Solution 1: Use binary storage formats like Avro or Parquet

If possible, use binary storage for your Hive tables, for instance [Apache Avro](https://avro.apache.org/) or [Apache Parquet](http://parquet.apache.org/). Since these formats do not use dedicated characters to split a file into records and fields, Hive/Impala can read data with special characters properly.

Also, Avro and Parquet make it possible to safely copy records from one Hive table to another. For instance, checkout the following command which copies data to a Parquet table:

```
CREATE TABLE customers_superclean STORED AS parquet
AS SELECT gender, street FROM customers;
```

The Parquet storage will ensure that even if  data in the `customers` table contains newlines or another delimiters, the data will be properly copied and interpreted in the new table.

## Solution 2: Ensure the delimited text file does not contain Hive delimiters

When ingesting data into a delimited text file, you have to ensure that the file does not contain characters that Hive uses to split data into rows and fields. In general, there are two options to achieve this:

- Remove Hive delimiters from the data before ingestion. If you use Sqoop, there are handy parameters which can do that for you. Checkout the [Sqoop docs](https://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html#_importing_data_into_hive) and look for `--hive-drop-import-delims` and  `--hive-delims-replacement` parameters.
- Use custom Hive delimiters that are not present in the data. Unused [non-printable characters](http://en.wikipedia.org/wiki/ASCII#ASCII_control_characters) are good candidates, for instance `\01` or `\02`. You can instruct Hive [to use custom delimiters](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormat,StorageFormat,andSerDe) as follows:
```
CREATE TABLE customers
ROW FORMAT DELIMITED
  LINES TERMINATED BY '\002'
  FIELDS TERMINATED BY '\001';
```



Happy ingesting!


How a newline can ruin your Hive



What a great time it is nowadays for data geeks!

We have Pandas and Scikit-learn - fantastic Python stack for data analysis. On top of that we have IPython and IPython Notebook - powerful coding, documentation and visualization layer for experimenting.

Then we have the whole Hadoop stack with an amazingly fast Impala SQL query engine. We don't even have to build the Hadoop cluster in-house, we just choose the size and spin up a cluster via Amazon AWS and we're done.

**And now guys in Cloudera made [Impyla](https://github.com/cloudera/impyla) - Python connector to Impala.** And of course, they didn't forget to pack in an Impala connector for Pandas! How great is that?!

So, if you want to connect Pandas to Impala on Elastic MapReduce (EMR), here is how.

## 5 steps to connect Pandas to remote Impala


### Prerequisites

Install the awesome Pandas, Scikit-learn and IPython stack if you haven't done that already.

### Step 1: Install Impyla

```bash
$ pip install impyla
```

### Step 2: Create an SSH tunnel to Amazon EMR so you can access Impala from localhost

```bash
$ ssh -L 12345:localhost:21050 your_user_name@your_node.compute.amazonaws.com
```

The Impala query engine runs on port 21050 on your Hadoop master node. For security reasons, this port is not accessible from the outside.

This shell command will open up the port 12345 on your local machine and forward it to the port 21050 on the Hadoop master node where the Impala query engine listens. (Of course, you can choose whatever port you want, it doesn't have to be 12345.)


### Step 3: Connect Impyla to Impala via the tunnel
```python
>>> from impala.dbapi import connect
>>> conn = connect(host='localhost', port=12345)
>>> cur = conn.cursor()
```
Notice that we are connecting to `localhost:12345` which is (securely) forwarded to Impala on Amazon EMR.

### Step 4: Query Impala and convert the result into Pandas dataframe
```python
>>> from impala.util import as_pandas
>>> cur.execute('SELECT * FROM customers LIMIT 500')
>>> df = as_pandas(cur)
```
Note that the query result must be transported from remote cluster to your localhost. If the result is large, the download might take a while. You might want to check out the network traffic monitor on your system to see when the download is complete.

### Step 5. Enjoy Hadoop data in Pandas
```python
>>> from sklearn.ensemble import RandomForestClassifier
>>> clf = RandomForestClassifier(n_estimators=10)
>>> clf.fit(df[['age', 'gender']], df['lifetime_value'])
```

## Bonus: Impala with scikit-learn API

By the way, guys at Cloudera are now busy with implementing scikit-learn API in Impyla. They [already have](http://blog.cloudera.com/blog/2014/04/a-new-python-client-for-impala/) alpha implementation of linear regression, logistic regression and SVM ready. I'm quite excited where all this is going...


Analyzing Elastic MapReduce data with Python, Pandas and scikit-learn


Which version of the if-condition is easier for you to understand?

```javascript
// version A
if (!isValid(input) || input.command !== "foo") {
  ...
}
...

// version B
if (isValid(input) && input.command === "foo") {
  ...
}
...


```

Negation is a fascinating topic. It is explored by Steven Pinker in his book [The Sense of Style](https://www.goodreads.com/book/show/20821371-the-sense-of-style):

<div class="blockquote">

> Logically speaking, a sentence with a naysaying word like _not_, _no_, _neither_, _nor_, or _never_ is just the mirror image of an affirmative sentence. Saying that the integer 4 is not odd is logically the same as saying that it is even. If something is not alive, then it’s dead, and vice versa. But psychologically speaking, a negative statement and an affirmative statement are fundamentally different.
>
> A negative statement such as _The king is not dead_ is harder on the reader than an affirmative one, like _The king is alive_. Every negation requires mental homework, and when a sentence contains many of them, the reader can be overwhelmed.
>
> To hear or read a statement is to believe it, at least for a moment. For us to conclude that something is not the case, we must take the extra cognitive step of pinning the mental tag “false” on a proposition.

</div>

Can we apply that to the two `if` statements? To mentally comprehend version A of the if-statement, a fellow dev goes through five mental steps:

![if-vs-if-not-mental-steps-for-if-not](/img/if-vs-if-not-mental-steps-for-if-not.png)

To comprehend version B, they go through three steps:

![if-vs-if-not-mental-steps-for-if](/img/if-vs-if-not-mental-steps-for-if.png)

Because comprehension is shorter by two steps, version B is [[coding-and-writing-are-alike|easier on the mind]].

Here's another example, a config file of a static site generator:

```ini
# If true, the parser won't highlight inline code
noInlineHighlight=false

# If true, the parser highlights inline code
inlineHighlight=true
```


The curious case of if-not



For those of you who haven't encountered it yet, [Azure DataMarket](https://datamarket.azure.com/browse/data)
is quite an exciting platform by Microsoft which provides standardized access
to plenty of interesting datasets. As of now, there are over 200 datasets  from broad range of topics, including weather, demographics, automotive, agriculture,  real-estate, etc. There might be a golden nugget for your bussiness hidden in there, so I would definitely recommend to go and [check the platform out](https://datamarket.azure.com/browse/data).

Recently, we have discovered such a golden nugget ourselves: details about every car sold in Netherlands in the past 15 years. A pretty exciting dataset, considering that the company I work for operates in the Dutch market of electric vehicles. What is more, the data is **free**, **updated daily** and comes shipped with a **REST API interface**. What more can we wish for?

To maximally leverage potential of the dataset, we ingest it into Hive, which allows us to:

- **run fast ad-hoc exploratory SQL queries** with [Impala](http://impala.io/),
- **explore the dataset** with BI tools,
- **enrich the data model** by combining car details with other data and
- **expand our dashboards** with insights about the car market.

## Avro-backed external Hive tables FTW!

Regarding the storage, we opted for an [Avro-backed](https://cwiki.apache.org/confluence/display/Hive/AvroSerDe) [external](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ExternalTables) Hive table. These types of tables rock, since:

- **Hive automatically infers table schema from the Avro schema**. There is no need to explicitly enumerate columns with their types, which is very useful if you deal with tables of many columns.
- **Changes in table schema are not an issue**. You just recreate the table with the new Avro schema and the [Avro schema evolution](http://blog.cloudera.com/blog/2011/05/three-reasons-why-apache-avro-data-serialization-is-a-good-choice-for-openrtb/) will automagically take care of properly reading the data stored with the old schema.
- **Data is easily processed by other tools** like Apache Spark or MapReduce. The data resides in a directory of your choice (rather than in a Hive directory) and the Avro format has first-class support in Hadoop tooling.
- **New data is easily added to the table**. Just add a new Avro file to the corresponding HDFS directory and Hive will automatically pick it up. With Impala, call `REFRESH <table-name>;` to notify Impala about arrival of new data.
- **[Hive data partitioning](http://blog.cloudera.com/blog/2014/08/improving-query-performance-using-partitioning-in-apache-hive/) is well supported** so you can optimize for query performance.
- **Dropping external Hive table does not delete data from HDFS**. Small thing, but very handy.


## Ingestion: From OData XML to an Avro Hive table

Here are detailed steps how we get that data from DataMarket into Hive.

### Step 1: Download the dataset as XML files, in a standardized OData format

DataMarket publishes datasets as XML. Each XML follows the same [OData schema](http://www.odata.org/documentation/odata-version-2-0/atom-format) (checkout the [example XML](https://api.datamarket.azure.com/opendata.rdw/VRTG.Open.Data/v1/KENT_VRTG_O_DAT) with cars data). DataMarket provides this data via a REST interface, which supports various [query parameters](https://msdn.microsoft.com/en-us/library/gg312156.aspx). Using the parameters, you control the subset of data you want to download.

Since the car dataset that we are interested in is about 15 GiB large, filtering proved very useful for our purposes. With the `$filter`, `$top` and `$inlinecount` parameters and a bit of `curl` and Bash, it is straightforward to download the whole dataset into files `cars_2000.xml` till  `cars_2015.xml`, where the number indicates the year where a car was registered in Netherlands.

### Step 2: Convert the dataset into Avro, using `odata2avro` utility

To convert an XML file in OData schema to Avro, we created a Python command-line tool called [odata2avro](https://github.com/datadudes/odata2avro) which does all the heavy lifting for you.

Just install the tool with `pip install odata2avro` and use it as follows:

```
$ odata2avro cars_2013.xml cars.avsc cars_2013.avro
```

This command reads an XML file `cars_2013.xml` and creates two files:

- `cars.avsc` - an Avro schema, in json, describing the dataset,
- `cars_2013.avro` - a binary Avro file containing the dataset from the XML.


### Step 3: Upload the Avro schema and Avro files to HDFS

In this case, let's create and upload the data to `/datamarket` HDFS directory:

```
$ hdfs dfs -mkdir -p /datamarket/cars
$ hdfs dfs -put cars_20*.xml /datamarket/cars
$ hdfs dfs -put cars.avsc /datamarket
```

### Step 4: Create an external Avro-backed Hive table


To create an external Hive table with a schema according to  `/datamarket/cars.avsc` and the data located in `/datamarket/cars`, use the following Hive command with the [AvroSerDe](https://cwiki.apache.org/confluence/display/Hive/AvroSerDe):
```
CREATE EXTERNAL TABLE cars
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/datamarket/cars'
TBLPROPERTIES ('avro.schema.url'='hdfs:///datamarket/cars.avsc');
```

### Step 5: Query & profit!

Congrats! As of now, the data is accessible in both Hive and Impala:

```
$ impala-shell --query 'select * from cars;'
```

## Bonus step: Keeping data updated

The *car* dataset is append-only, so for us it's pretty straightforward to keep the data in Hive updated. We run a daily job which:

- downloads XML data from DataMarket for the current year 2015,
- converts the XML data to an Avro file called `cars_2015.avro`,
- uploads the Avro file to HDFS and replaces the file uploaded on the previous day and
- calls `REFRESH cars;` so Impala registers the new data.

That's all!


How to ingest data from Azure DataMarket into Hadoop


Apache Kafka, Mesos, Hadoop/YARN, Neo4J, HBase, Solr. All of these services
(and [many others](https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy))
are built on top of Apache ZooKeeper. ZooKeeper is also a part of
all Hadoop distributions (e.g. [Cloudera CDH](http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html))
and is used in many companies, including LinkedIn, Twitter, Netflix or Yahoo.

So what is ZooKeeper all about and why is it that popular?

## In summary, ZooKeeper helps to build a distributed system

Following the Unix philosophy of small yet powerful tools,
Zookeeper is a service that allows distributed processes to coordinate with each other.
Instead of developing such a coordination system from scratch, you can re-use ZooKeeper
and benefit from its best-practice proven implementation.

Distributed systems use ZooKeeper for service discovery, cluster monitoring,
configuration management, leader election or naming service. Also, ZooKeeper
provides foundation for building higher-level distributed functions like locks, barriers,
queues, two-phase commits, etc.

Or, as the Apache Zookeeper PMC [puts it](https://www.hakkalabs.co/articles/apache-zookeeper-introduction/):
> If you are doing distributed locking and you are not using ZooKeeper, you are crazy - Camille Fournier, Apache ZooKeeper PMC

## ZooKeeper fundamentals: Z-nodes

ZooKeeper itself is a distributed highly-available service that manages
a shared hierarchical name space of data registers, called **z-nodes**.

**Z-nodes** very much resemble a filesystem. Each z-node has a path, e.g. `/services/product`, and data, given as `byte[]`. However, unlike directories in a filesystem, parent z-nodes can also carry data.

Here is an example of z-nodes hierarchy:
![Illustration of Zookeeper z-nodes]({filename}/images/zookeeper-znodes.jpg)

ZooKeeper provides a simple yet powerful [API](https://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/ZooKeeper.html)
for z-node management. You can *create*, *get*, *update* and *delete* a z-node, ask if it *exists*
and *get children*.

The interesting part, however, lies in the **consistency guarantees** that ZooKeeper provides:

- **Sequential consistency** -  Updates are applied in order they are received by ZooKeeper
- **Update atomicity** - Updates are either successful or failed. No partial results
- **Single system image** - A client sees the same view of the service regardless of the ZooKeeper server it connects to
- **Reliability** - Once an update has been applied, it will persist from that time forward until a client overwrites the update
- **Timeliness** - The clients view of the system is guaranteed to be up-to-date within a certain time bound

I hope that by now you start to feel the power of ZooKeeper. However, this is not all.

## Persistent, ephemeral and sequential z-nodes

Regarding persistence, ZooKeeper offers two types of z-nodes: **persistent** and **ephemeral**.

**Persistent z-nodes** are the default z-nodes, they exist until they are explicitly deleted.

**Ephemeral z-nodes**, on the other hand, are attached to a client session. If a
client dies, the session is terminated and the ephemeral z-done is deleted. How great is that!
You can very easily monitor clieant failures: If a client responsible for
performing an exclusive operation fails, the corresponding ephemeral z-node is deleted and other clients
are notified to take over the operation.

Both persistent and ephemeral z-nodes can be also marked as **sequential**.

For **sequential z-nodes**, ZooKeeper automatically appends a monotically increasing counter to the end of a path.
This is greatly helpful for synchronization: if more clients want to get a lock on a resource `A`,
they can each create a sequential z-node on path `resource/A`. The client getting the lowest number is entitled to the lock.

## Watches

ZooKeeper also offers a mechanism called **watches**.
A watch is just a one-time callback, which is triggered every time
a z-node changes. Watches are one-shot - if you need to continually monitor a z-node,
you need to reset the watch after each event.

Watches are very useful to if you don't want to periodically poll your z-nodes.


## Fault-tolerant, highly-available, read-performant

Zookeeper can run in both stand-alone and distributed fault-tolerant mode.
When running in a cluster, a group of ZooKeeper servers is called an **ensemble**.

Regarding node failures, ZooKeeper **tolerates loss of ensemble minority**
(there is [Paxos](http://en.wikipedia.org/wiki/Paxos_%28computer_science%29) under the hood).
So if the ensemble consists of three or four machines, ZooKeeper will tolerate failure
of only one machine in both of these cases. This is why it is recommended
to run the ZooKeeper ensemble on **an odd number of hosts**, since even number
of machines doesn't bring any benefit with respect to node failures.

ZooKeeper data is kept in memory and is backed up to a log for reliability.
It is optimized for read dominant workloads, handling up to 50k operations per second.

## Clients

ZooKeeper clients can connect to any of the ensemble members and maintain a connection.

ZooKeeper is build on Java, but there is a pretty [solid list](https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings) of client bindings,
including Java, Scala, Node.js, Erlang, Haskell, Python, C#, Go and Ruby.

The most popular client is [Apache Curator](http://curator.apache.org/) (former LinkedIn project).
Definitely checkout the [Curator support for high-level recipes](http://curator.apache.org/curator-recipes/index.html),
it's a very interesting read.

## Use-cases

Here is a couple of examples how ZooKeeper is used in practice:

Twitter uses ZooKeeper for **service discovery**. Each application instance
registers itself to ZooKeeper using an ephemeral z-node. In this way,
ZooKeeper can maintain an up-to-date list of running instances for each type of service.
Clients then query ZooKeeper to locate application instances.
In general, ZooKeeper might help you to build a
**micro-service infrastructure** or manage a network of **replicated
(REST) services**.

[Apache Solr](http://lucene.apache.org/solr/) uses ZooKeeper for **leader election, centralized configuration** and **cluster management**. ZooKeeper enables application servers to **bootstrap configuration**
from ZooKeeper as soon as they join the system and to keep the configuration up-to-date.


Cluster monitoring can be implemented as members registering to `/members/host-{i}` and
periodically updating the z-node with their status (load, memory, CPU etc). Each z-node update
then triggers an alert to z-node listeners using watches.

[Apache Kafka](https://www.google.cz/search?q=apache+kafka&rlz=1C5CHFA_enCZ546CZ546&oq=apache+kafka&aqs=chrome..69i57j69i59l2j0l3.1551j0j9&sourceid=chrome&es_sm=91&ie=UTF-8), a popular publish/subscribe system with persistent queues,
uses ZooKeeper to store client's last consumed
offset, to register Kafka brokers and to help load balance requests among live brokers.

[Apache Hadoop](http://hadoop.apache.org/) uses ZooKeeper for automatic failover of Hadoop HDFS Namenode and for high-availability of YARN resources.



Pretty amazing!

## Where to go from here

I was quite eager to experiment with the ensemble, so I made
this [automated script](https://github.com/mkrcah/virtual-zookeeper-cluster) which creates and provision a virtual 3-node ensemble on your local machine with one command using Vagrant and Ansible. [Check it out](https://github.com/mkrcah/virtual-zookeeper-cluster).

Also, if you want to dive deeper, I'd recommend to:

- Read the [Getting started guide ](http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html) on the official ZooKeeper website
- Checkout the [Zookeeper recipies](http://zookeeper.apache.org/doc/trunk/recipes.html)
- Bookmark the [curated list of Zookeeper presentations](https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeperPresentations)
- Read a use-case of [building a distributed concurrent queue in Zookeeper](http://blog.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/)
- Checkout [Grid Computing with Fault-Tolerant Actors and ZooKeeper](http://www.ebaytechblog.com/2012/09/13/grid-computing-with-fault-tolerant-actors-and-zookeeper/#.VKLmpsAAE) - eBay tech blog

**Happy ZooKeeping!**


Introduction to Apache Zookeeper, backbone of big-data systems


There is a section in the interview with [John Danaher](<https://en.wikipedia.org/wiki/John_Danaher_(martial_artist)>) that I've been thinking about a lot.

<div class="blockquote">

> I'm a huge believer in the idea of small progressive movements towards goals. The vast majority of our days are unexceptional mundane days. There's a chance you can drift through your life when only 1-5% of your days have any real meaning and that 95% of your life was a waste of time. And that's a tragedy.
>
> If we make a concerted effort to build one day upon another, even if it's only a very small thing (and in most cases, it will be a very small thing, it's rare when we have a day where something monumental happens, most days are not monumental, they are mundane), and we do this over a period of ten years, something remarkable can happen. You get this compounding interest effect where at the end of five or ten years you may literally have reinvented yourself.
>
> — John Danaher, from an [interview](https://londonreal.tv/john-danaher-the-philosophy-of-martial-arts-the-man-who-inspired-me-to-learn-brazilian-jiu-jitsu/)

</div>

Relates to _[Gradatim Ferociter](https://www.geekwire.com/2016/jeff-bezos-blue-origin-motto-logo-boots/)_ ("Step by Step, ferociously") and the [important non-urgent quadrant](https://youtu.be/oTugjssqOT0?t=1256).


The spirit of kaizen

I enjoy working with [Sentry](https://sentry.io/organizations/sungevity/issues/) for error tracking: I have a real-time overview of errors, I set up various alerts, and I use it in both backend and frontend systems. Now, how to setup Sentry for AWS Lambda that is provisioned by AWS CloudFormation?



There are currently two ways to do it: wrapping up the Lambda handler or using AWS Lambda layers. I focus on the Lambda handler here.

## Step 1: Wrapping up handler

Following the Sentry docs on integrating with AWS Lambda, this step was straightforward. Here's an example of a wrapped Node handler:

1. Install Sentry lib:

   ```bash
   npm install --save @sentry/serverless
   ```

1. Wrap the Lambda handler:

   ```typescript
   const Sentry = require("@sentry/serverless")

   Sentry.AWSLambda.init({
     dsn: "<Your Sentry DSN>",
     // ...
   })

   exports.handler = Sentry.AWSLambda.wrapHandler(async (event, context) => {
     // Your handler code
   })
   ```

In the current version of the library (6.2.3), I had to [tweak the setup a bit](https://github.com/getsentry/sentry-javascript/issues/2984#issuecomment-805857077) to make it work with WebPack and Typescript.

## Step 2: Setting up CloudFormation

Now, Sentry supports monitoring errors per _environment_, such as dev, acceptance, or production. I like that because I have more control over errors in different environments. The environment is set up in the Lambda itself:

```typescript
Sentry.AWSLambda.init({
  dsn: "<Your Sentry DSN>",
  environemnt: "acceptnace", // <--- here we supply the lambda environment
})
```

Now, how to supply the lambda environment using CloudFormation? It turns out, this can be achieved with [parameters](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html).

(In the steps below, I also use AWS Parameter Store to fetch the Sentry DSN instead of having the DSN hardcoded in the Sentry init):

1. Setup the Cloudformation template: add an environment parameter and Sentry DSN parameter

   ```yaml
   # content of template.yml
   AWSTemplateFormatVersion: '2010-09-09'
   Transform: AWS::Serverless-2016-10-31

   Parameters:
     # here we define that the CloudFormation stack accepts environment parameter
     LambdaEnvironment:
        Type: String
        AllowedValues:
          - dev
          - acc
          - prod

     # here we fetch the sentry URL from SSM Parameter store
     SentryDSN:
        Type: AWS::SSM::Parameter::Value<String>
        Default: '/<your-path-to-sentry-dsn-ssm-parameter>'

   Resources:
   	ExampleLambda:
            Type: AWS::Serverless::Function
            Properties:
              Environment:
                # here we supply sentry dsn & lambda environment to lambda
                Variables:
                  SENTRY_DSN: !Ref SentryDSN
                  LAMBDA_ENVIRONMENT: !Ref LambdaEnvironment
            Policies:
                # we need to allow reads from SSM parameter store
               - SSMParameterReadPolicy:
                 ParameterName: '/<your-path-to-sentry-dsn-ssm-parameter>'
            #CodeUri, Events, etc...
   ```

2. In Lambda, read from the env vars instead:

   ```typescript
   Sentry.AWSLambda.init({
     dsn: process.env.SENTRY_DSN,
     environemnt: process.env.LAMBDA_ENVIRONMENT,
   })
   ```

3. When deploying the CloudFormation stack in a CI pipeline, supply `--parameter-overrides` argument to `aws cloudformation deploy`:

   ```bash
   aws cloudformation deploy --template-file template.yml  \
   	--parameter-overrides "LAMBDA_ENVIRONMENT=acc" \
   	# ... other args & options ...
   ```

This should be it 🎉.

## Sidenote on AWS Lambda logs & CLI

Speaking of errors, it turns out one can tail aws logs via a CLI using `aws logs`. Plus, there's no need to specify the log stream, which is handy:

```bash
#  print all log groups
aws logs describe-log-groups

# tail logs for lambda example-lambda since 1 day ago
aws logs tail /aws/lambda/example-lambda --since 1d --follow |
```

I like to combine `aws logs` with [grep context lines](https://www.gnu.org/software/grep/manual/html_node/Context-Line-Control.html) to search for errors with the surrounding context:

```bash
# show all ERROR log messages. For each ERROR msg, show 10 preceding messages
aws logs tail /aws/lambda/example-name --since 1d | grep -B 10 'ERROR'
```

Tracking AWS Lambda errors with Sentry (and CloudFormation)


I stumbled upon an interesting way to visualize things. In the book [An Invitation to Applied Category Theory](https://www.researchgate.net/publication/334292752_An_Invitation_to_Applied_Category_Theory_Seven_Sketches_in_Compositionality), they have these fascinating diagrams. There's something I really like about them.

- Basic schema (page X.) ![invitation-to-applied-theory-lemon-pie](/img/invitation-to-applied-theory-lemon-pie.png)

- Different Styles of wiring (page 43) ![invitation-to-applied-theory-different-styles.png](/img/invitation-to-applied-theory-different-styles.png)
- Keeping track of waste (page 49) ![invitation-to-applied-theory-waste](/img/invitation-to-applied-theory-waste.png)
- Copying (page 50) ![invitation-to-applied-theory-copying](/img/invitation-to-applied-theory-copying.png)
- Collaborative Design (page 118) ![invitation-to-applied-theory-collaborative-design](/img/invitation-to-applied-theory-collaborative-design.png)
- Databases (page 79): ![invitation-to-applied-theory-databases](/img/invitation-to-applied-theory-databases.png)


Preparing a lemon pie: visualization

I finished a 6-month project where we delivered a greenfield IT system. Here are my lessons learned, focusing on the technical side.



For the context, the system was written in Typescript, on top of AWS Lambda. Functionally, the system integrated two other systems via their REST APIs. The team consisted of two engineers, one project manager, and one person from the business. I was the engineer.

## What went well: Tests

Automatic testing is such a delicate topic:

- too many implementation-specific tests and code gets tedious to refactor and to structurally improve;
- too few tests and one is afraid to refactor, afraid to break code that works.

So how to solve this? In this project, we used two batteries of tests. First, we added component tests. The tests were implementation-agnostic and used [HTTP server mocking](https://github.com/nock/nock). Such tests were a great decision:

- majority of code got covered by a few high-level component tests,
- we gained the confidence to structurally refactor code,
- we also enjoyed the development experience: tests with HTTP mocks were easy to work with, especially with [Nock's automatic test recorder](https://github.com/nock/nock#recording)

The second battery of tests contained unit tests. We unit tested functions with nontrivial calculations, such as complex text parsing. With such tests, we gained confidence that the calculations were correct and all edge cases were covered.

## What went well: Code structure

Speaking of refactoring: the closer we were to completion, the clearer it was how code should be organized. So towards the end of the project and thanks to the component tests, I heavily refactored the overall code structure. [[simple-components|Complex things got decomplected]]. Afterward, when unexpected business requests arrived, we added them as a simple unit; instead of adding to a hairball of complexity.

What I take from this: it's OK to start adding functionality without worrying about high-level code structure. The structure [[hard-decisions|will reveal itself]]. I wish I'd have added component tests even sooner, to enable continuous structural refactoring.

## What went well: Idempotency

Where possible, I made components [idempotent](https://ericlathrop.com/2021/04/idempotence-now-prevents-pain-later/) ([wiki](https://en.wikipedia.org/wiki/Idempotence)). What a relief it was to later perform data migration based on such an idempotent execution. We felt in control, even during complex data migration.

Admittedly, some parts were not idempotent. This was due to business constraints. For the next project, I'll try to advocate for idempotency more.

(By the way, as I'm writing this post, another team is facing an issue in the billing system. They discovered that the issue is related to AWS Lambda services not being idempotent. What a day for idempotency.)

## What went well: Logging

I'm deepening my appreciation for verbose logging, especially with AWS Lambdas: function inputs, calls to external systems, currently executed steps. When things went unexpected, logging turned out to be half the success.

## What went well: Validating AWS lambda input

What a time saver! Handler input validation turned out to be super helpful as we were switching systems that were responsible for invoking the AWS Lambda. From now on, I'm validating Lambda's handler input every time I write an AWS Lambda function. It's a low-hanging fruit that can save hours.

## What went well: Error reporting

We ended up distinguishing between two types of errors: _application errors_ and _business warnings_.

- _Application errors_ were unhandled exceptions or AWS Lambda timeouts. The responsibility to fix these is on our side, the engineers. We handled such errors with [[lambda-sentry|Sentry]], to our great satisfaction.
- _Business warnings_ were data errors, out of our responsibility. When a data error was detected, we sent a Slack notification to the business.

The combination of application errors and business warnings worked well: we had full and immediate visibility into exceptions, and the business had immediate visibility into data issues. They were even eager to fix it..

All in all, the team was very happy with this immediate and proactive approach.

## To improve: Using unfinished open-source library

We used an unfinished open-source library that wrapped API calls around one system. We wanted to gain speed. It turned out this was a bad investment: the library didn't cover edge-cases, missed functionality, behaved unpredictably, missed Typescript types. We ended up rewriting most of the functionality from scratch to fit our needs.

## To improve: Function inputs

For some functions, I supplied more inputs than the function needed:

```typescript
interface Foo {
  id: number
  name: string
}

// supplying the whole Foo object
const func1 = (foo: Foo) => {
  console.log(foo.name) // but only name is needed
}

// supplying only what's needed, Foo's name
const func2 = (name: Foo["name"]) => {
  console.log(name)
  // the function doesn't need to know about the whole Foo object
}
```

During heavier refactoring, I needed the form of `func2` instead of `func1`. In retrospect, I'm not surprised: `func1` [[simple-components|complects]] an operation executed on a `name` with the whole `Foo` object. This complexity prevented me to have simpler components elsewhere.

Also, testing `func2` got easier, we didn't need to supply unnecessary test data.

## Open questions: Dependency injection in Typescript

Where possible, I aim to write functions that [[honest-function-inputs|are honest about their inputs]]. However, passing all the context around soon becomes cumbersome. I'm wondering about possible solutions in Typescript that [don't feel like magic](https://inversify.io/). One approach I'd like to experiment with is the [fp-ts Reader monad](https://dev.to/gcanti/getting-started-with-fp-ts-reader-1ie5).

## Open question: Code organization

I thought that organizing code by domains or features (feature A, feature B, etc.) is preferred over organizing by layers (db, api, etc.).

```
# organizing by features
src/
  featureA/
  featureB/
  ...

# organizing by layers
src/
  db/
  api/
  ...
```

My reasoning was that if code related to _one feature_ sits together, it is easier to reason about it. However, I discovered that if I put all code related to _one layer_ together, I spot more opportunities for simplification and decomplecting. So now I face this conundrum: organize code by domain or organize code by layer?

## Open question: Duplication for the win?

For many years, I was a strong advocate of the [DRY principle](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself). However, I'm now discovering benefits of duplication:

- duplicated code is easier to modify
- duplicated code can be made more domain-specific and use-case specific
- duplication can simplify code, such as functions that contain many if-else branches, where each branch denotes a different business use-case.

So now I face another delicate conundrum: when to duplicate code and when to make code DRY?

## Open question: Risk of missed business outcome

This question is not technical, but to me, it is the most important one. We were asked to deliver a technical solution that didn't immediately bring business value. Only _after_ the project had been delivered, another team started to monetize the system we had built. But what if the monetization won't go as planned? All our efforts will be wasted.

The client was determined to go ahead with the implementation, despite the risk of wasted effort. As an engineer, I face a conundrum: build the system knowing the system could be a waste; or push for the business impact. There are [differing views](https://news.ycombinator.com/item?id=27771219) on the topic. What I'm contemplating is to start from myself by [reflecting on the outcome of everything we release](https://www.jpattonassociates.com/mindset-that-kills-product-thinking/).

Lessons-learned from a 6-month Typescript/AWS Lambda project


I use [dotfiles](http://dotfiles.github.io/) to configure my environment.
I had the dotfiles stored in a git repo and used helper scripts to create symlinks in my `$HOME` directory. However, adding & updating the dotfiles felt cumbersome, like a pebble in a shoe.

Recently, I stumbled upon [this article](https://www.atlassian.com/git/tutorials/dotfiles), which was inspired by [this HN comment](https://news.ycombinator.com/item?id=11070797). They mention something called a _bare_ git repo to manage dotfiles. I have never heard of it & it took me a while to figure it out. So I thought I will share it here.

## How a typical git repo looks like

Typically, when you initialize or clone a git repo, it consists of two parts: _the working tree_ and _the [ `.git` directory](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain#ch10-git-internals)_. The working tree contains things like source code and the README. The `.git` directory contains [git internals](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain), such as commit objects, local git config, and hooks. For example, when you checkout a different branch, git updates the _working tree_ based on data stored in the `.git` directory.

![bare-repo-typical](/img/bare-repo-typical.png)




## Cumbersome symlinking

Now, when symlinking dotfiles, the working tree for dotfiles is located in the dotfiles git directory, not in `$HOME`. So to make dotfiles work, one needs to create symlinks. And the symlinks need to be managed. That's the pebble.

![bare-repo-symlinks](/img/bare-repo-symlinks.png)

You may wonder why not initialize a git repo directly in  `$HOME`? If git doesn't find the `.git` directory in the current directory, it continues the search in parent directories. So if a directory is not versioned, git would still assume we are in the dotfile git repo. That could be confusing & prone to errors.

If only there would be a way to put the  dotfiles working tree directly in `$HOME`, while keeping the `.git` directory separated.

## Say hello to bare repositories

It turns out there is a way: a _bare_ git repository. A bare git repo is a repo that contains only the content of the `.git` directory, not the working tree. When interacting with git, you specify the working tree directory as a command-line parameter. That's pretty nice.

![bare-repository](/img/bare-repository.png)

Technically, it's done like this.

```bash
$ DOTFILES=$HOME/projects/dotfiles.git

# init the repository with the --bare flag
$ git init --bare $DOTFILES

# setup a git alias which sets 
# the .git directory to $DOTFILES and
# the working tree to $HOME 
$ alias gitdf='git --git-dir=$DOTFILES --work-tree=$HOME'

# for gitdf status don't show untracked files
$ gitdf config status.showUntrackedFiles no

# making a change & commiting a dotfile
$ echo "syntax on" >> .vimrc
$ gitdf add .vimrc
$ gitdf commit 
```

I've been using this setup for a few days now, and it feels smoother. I like it.

## Further reading

Here I focus on the high-level concept. If you want to read more about the technical workings, checkout [this](http://www.saintsjd.com/2011/01/what-is-a-bare-git-repository/) and [this](https://www.ackama.com/blog/posts/the-best-way-to-store-your-dotfiles-a-bare-git-repository-explained) post.

Additionally, it turns out that git supports [multiple working trees](https://git-scm.com/docs/git-worktree) in one git repository. See the `git-worktree`. Interesting, I haven't used it yet.


## Side note on complexity

I now try to see everything through the eyes of [Rich Hickey's Simple Made Easy](https://www.infoq.com/presentations/Simple-Made-Easy/) & to spot complexity. The dotfiles use case is, perhaps, more of a theoretical exercise, but I find it valuable nevertheless. By doing thought experiments like this, I aim to solidify my understanding of complexity.

So, in this case, it seems that the symlinking approach complects two things: the dotfiles (the `.gitconfig`, the `.zshrc`) with the dotfile management (symlinks). These are two different concepts, intertwined in one repository. With a bare repository, we remove the complexity: the dotfiles are taken care of by the working tree, while dotfiles management is taken care of by git.

![bare-repo-complexity](/img/bare-repo-complexity.png)

Another form of complexity is the imperative code for symlinking: first symlink that, then symlink that. At least that's how I had it coded. Since imperative programming complects _how_ with _what_, it's complex. With a bare repository, we remove this form of complexity: the dotfiles become declarative; it's only the _what_.

There is one tradeoff, though. We introduce complexity in the form of inconsistent interaction with git. For dotfiles, we use an alias, such as `gitdf`; while for all other repos, we use `git`. For me, this complexity is negligible compared to the benefit of simplifying the symlinking process.


Managing dotfiles with a bare git repo

This week I published a post (a [[til-typescript|collection of today-I-learn nuggets]]) that I might further update in the future. For the updates I'd love to add a changelog to the post bottom, and also add the _last updated_ to the post top. However: I store posts in Markdown with a frontmatter; and the frontmatter contains the `date` field; and the one field is not sufficient to capture the changes. So here's how I changed the frontmatter and how I migrated existing posts.



![blog-post-changelog](/img/blog-post-changelog.png)

## Storing the changelog

First, I discovered that Markdown frontmatters are, in fact, full-fledged YAML. So I can store a complex data structure, such as a list of maps in there. I think this is a really cool feature of frontmatters. Here's a new frontmatter shape:

```yaml
---
title: "Typescript & NodeJS: Collection of TILs"
changelog: 	# 🎉
  - date: 2021-03-30
    change: postUpdated
    detail: >
		Added 3 TILs: absolute imports in Webstorm,
		const assertions and
		checking string literal types at runtime
  - date: 2021-03-29
    change: postPublished
# ... rest of the frontmatter ...
---

# ... content of the post...
```

## Updating the static-site generator

I use [NextJS](https://nextjs.org/) for this blog. NextJS is a React-based static site generator. I use NextJS together with [[til-typescript|Typescript]]. I have an interface representing a post, so I expanded it as follows:

```typescript
interface ChangelogItem {
  date: string // string because of NextJS serializion
  change: "postPublished" | "postUpdated"
  detail?: string
}

interface Post {
  changelog: ChangelogItem[]
  publishedOn?: string
  lastUpdatedOn?: string
  // ...
}
```

I updated the code that parses the frontmatter and also updated the React component that generates the site.

## Migrating existing posts

Ok, so that should do for new posts. However, I have around 50 existing posts with a, now obsolete, `date` field:

```yaml
---
title: "A post with an obsolete date"
date: 2021-03-30 # <----- this needs to be migrated to a new shape
tags: engineering
---
Lorem ipsum dolor
```

What I needed is to migrate the old frontmatters into a new shape. So I summed my favorite `awk` to do the heavy lifting:

```bash
# content of migrate-to-change-log.awk
BEGIN { delims = 0 }
# count the delimiters to determine if we are inside the frontmatter
/---/ { delims++; }

{
  if (delims == 1 && /^date:/) {
    printf "changeLog:\\n  - date: %s\\n    change: postPublished\\n",$2
  } else {
    print
  }
}
```

Once I had the `awk` script down, I migrated all posts with:

```bash
echo *.md | xargs -n1 gawk -i inplace -f migrate-to-change-log.awk
```

Here's an outcome on the sample post:

```yaml
---
title: "some title"
changeLog: # 🎉🎉🎉
  - date: 2021-03-30
    change: postPublished
tags: engineering
---
Lorem ipsum dolor
```

## Using git instead?

When writing this post, it occurred to me that instead of the frontmatter, git can also be used as a data source for the changelog. For this to work, I would have to exclude refactoring commits, which change the post structure. For example, a commit that replaces the `date` field with the `changeLog` field. Or a commit that moves the file around. I'll keep this idea in [the background mind](https://www.youtube.com/watch?v=f84n5oFoZBc) for now.

Adding changelogs to blog posts


"We are going to implement Hadoop in the company."<br/>
"OK. Just to understand better, which problem are we trying to solve?"<br/>
"We want to have a single source of truth for all our data."<br/>
"OK, I understand. Which problem exactly is Hadoop going to solve?"<br/>
"The CTO went to the conference and heard that Hadoop was the way we should do things now."<br/>
"OK. Do we know which exact problems would Hadoop help us solve?"<br/>
"Hm... that's a good question."<br/>

Another example: <br/>
"We are going to put parameters of this machine learning model in the configuration file."<br/>
"OK. To understand, which problem would this solve?"<br/>
"It will make the model easily modifiable by business-users."<br/>
"I understand. Are there any business users who would use it? Which problem would we solve by that?"<br/>
"Currently, it's just us developers touching the code, but it might happen in future."<br/>
"Hm, OK. So is there an actual problem we are solving by doing this?"<br/>
"Hm... good question."<br/>

<br/>

The purpose of teams is to solve problems. Each problem has a _root cause_ which triggers
complications to a _customer_. A customer is an actual person or a group. A customer can be internal
within the company or external.

Without a clear understanding of the root cause and the customer,
a problem cannot be solved effectively. Without a root cause, we might
be solving a non-existent or a wrong problem. Without a customer, it's not possible
to validate if the solution resolves the problem.

Do you know the root causes and the customers of projects you currently work on?


Which problem would that solve?


The start of a project, I've noticed, often determines the pace for the whole project. I discovered that what works well is to start with a proof of concept that is _embarrassingly simple_. "Embarrassingly simple" clarifies what to expect: a half-baked barely-working solution. "Embarrassingly simple" is also kind of funny when said out loud, and it sets a good mood within the team. To me, a recovering perfectionist, "embarrassingly simple" also crystalizes what to build and what to exclude.

Here's an example: We needed to integrate system A with system B by periodically calling the APIs. To make the proof of concept embarrassingly simple, we excluded tests, devops and monitoring, we ignored edge cases and unhappy paths, and we were pretty relaxed about code quality.

We ended up with a manually-triggered script that reached out to system A, extracted one field from one object, and stored the field in system B. We could have run API calls manually, but a bit of automation is a nice touch for demos. We also aimed for high impact, so we asked which object and which field have high priority.

![vertical-slice-of-functionality](/img/vertical-slice-of-functionality.png)

Cutting corners where possible, the concept was delivered quickly. Clients like that, it turns out. We also learned a ton about the systems:

- we discovered the need for a sandbox environment,
- we figured out API access and ironed out surprising API errors,
- we learned the foundations of the system's data models,
- we learned how to map the models for one specific use case,
- we discovered gaps in requirements that we didn't know before.

We reached the first deliverable, we set the pace, and the client is satisfied. Now, it's about keeping this up through the middle to the end of the project.


Embarrassingly simple proof of concepts


We, software engineers, love to build.

The thrilling sensation of adding a feature or creating something new from scratch. The excitement of using the latest technology that we've been reading about on Hacker News for so long.

And thus, we build.

We dive into coding, into architecture, into design. Quenching another dose of the thrill.

I am guilty of this.

Yet, I came to believe that blindly following the thirst is by far the largest productivity killer. I believe that coding should be considered the very last resort, used only when all other options are exhausted.

Let me explain.

Development is very expensive. It takes a long time to finish. It requires future time investment. It requires maintenance and bug fixes. There is an opportunity cost, and there is a probability that our work won't be used in production, because the priorities have shifted.

No matter the team, the agile process or the technology stack, the development will always be this way - resource intensive. Coding slows down the company, drains the IT budget and creates technical debt for future.

So before blindly jumping into coding, take a step back. Understand arguments why the problem or the feature at hand needs to be built in the first place. Find if there is a way to simplify the problem. Find if there is an existing solution in-house or off-the-shelf that can be reused to solve it.

Fight that internal thirst as much as possible.

Only as the very last resort, only when all other options fail, start thinking about the very minimal coding solution to the problem you have and very cautiously take that first sip.


Resist the quench for coding



I came across an interesting problem a couple of days ago. A friend of mine runs quite a large website with a lot of users registering on their web. The problem is that there are users which are lazy to enter proper information so they just enter random text, as this for example:

![Example of randomly typed words]({filename}/images/random-text.png)

Since the correct registration information is crucial to their business, he would like to have a solution which would detect such random text automatically.

It would be very useful for him to have a tool which would check all data gathered from registration forms and mark rows which look like  random text. He calls this *false user registrations*. Also, it would be interesting to have an online detector which would notify the customer-care team if such a false registration is submitted.

Although there are many approaches to this problem, I was curious if such problem can be solved algorithmically. In particular, **I was curious what would be the most simple solution that would be useful in 80% of the cases.** The reason is that when I am presented with a (data) problem, I always find it very effective to start with the most simple solution first. And often these solutions turn out to be satisfactory enough.

Anyway, I was first tempted to go with supervised algorithms, like Naive Bayes classifier for example. However, I realized that it would be much more interesting to come up with an unsupervised algorithm, which would identify random words automatically without any prior knowledge. **Just give the algorithm a list of words and it gives you back words from the list which look random.** Such solution would also be practical from the business perspective since you can immediately identify suspicious registrations in a sea of production data.

So here is a description of a very simple - yet surprisingly effective -  unsupervised solution to this challenge. Based on frequency-analysis of character bigrams.

## Before we dive in, let's the see results

For the sake of simplicity, I focused on the field of the registration form that contains a customer name. To mock up production data, I took the 2000 most common names in US and added 20 random texts on top of the list. I got [this list of words](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-input.txt), which I gave as input to the algorithm. (The order of words doesn't matter.)

What the algorithm does is to compute a *suspicious score* for each word. The higher the score, the more suspicious (i.e. random) the word looks when compared to the rest of the words.

If you run the algorithm, you get [this list of words sorted by a suspicious score](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-output-adjusted-idf.txt). I marked the 20 random words with an asterix, so I can quickly evaluate quality of the result.

We see that the algorithm is pretty effective: **the top 10 words with the highest suspicious score (above 25.0) consist of random words only.** Actually, the top 16 words consist of 15 random words and only one regular name - *guadalupe*.

The five random words which didn't make it to the top 20 include *tyrwereh* (#23), *qwerpo* (#43) and *sfert* (#422). However, I'm ok with the low scores here, because these words don't look very random to me anyway :)

I have made also an interactive version of the algorithm, it is [available here](http://random-text-detector.herokuapp.com). Go ahead and give it a try. *(Note that it might take up to 10 seconds to load the page because the Heroku node might need to wake up.)*


## How it works

If you look at a random word, e.g. *sdfjtwd*, you see a lot of rare combinations of characters like  *sd* or *ft* or *jt*. If you look at a  word *dustin*, there is no combination of characters which strikes you are rare. Maybe *st*, but that's about it.

A combination of two adjacent characters in a word is called a character bigram. So *sd*, *ft* and *jt* are all bigrams of the word *sdfjtwd*. (By the way, there are also *word-bigrams* which are adjacent words in a sentence.)

What you want to do to detect random text is the following:

1. Compute a score for each bigram. This score should be high if the bigram is rare among all given words. The score should be low if the bigram is very common among given words.
2. Compute suspicious score of a given word as a sum of all bigram scores that are contained in the word.

## Step 1 - Compute bigram scores

Consider the words *aabaa*, *abbb* and *ababa* for example. Let's compute for each bigram how many times it occurs in each word and also how many words in total contain this bigram:

Word / Bigram | aa  | ab  | ba  | bb
---   | --- | --- | --- | ---
aabaa | 2   | 1   | 1   | -   
abb   | -   | 1   | -   | 1   
ababa | -   | 2   | 2   | -   
**Total words containing the bigram** | **1** | **3** | **2** | **1**

We see that the bigram *aa* occurs twice in the word *aabaa*, while bigram *bb* doesn't occur at all in *aabaa*. Also, we see that the bigram *ab* is the most common bigram (used in all 3 words) and the bigrams *aa* and *bb* are the most rare (used only in one word).

Now we know which bigrams are rare and which are common in the dataset. We just need to translate it into a score.

The most popular metric for this purpose is the [inverse document frequency (IDF)](http://en.wikipedia.org/wiki/Tf–idf). For a given bigram $b$, the IDF is computed as

$$ \{IDF}(b) = \log\frac{N}{n_b}, $$

where $N$ is the total number of words (in our case $N=3$), and $n_b$ is the number of words which contain the bigram $b$.

In our example of words *aabaa*, *abbb* and *ababa*, the IDFs are as follows:

&nbsp; | aa  | ab  | ba  | bb
---   | --- | --- | --- | ---
$n_b$ | 1   | 3   | 2   | 1   
$\{IDF}(b)$  | 1.1 | 0.0 | 0.4 | 1.1

This is exactly what we need: The most common bigram (*ab*) has zero IDF, while the most rare bigrams (*aa* and *bb*) have highest IDF.

By the way, to see the IDFs of all bigrams from our data of 2020 words, click [here](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-idf-standard.txt).

## Step 2 - Sum up bigram scores

To get the suspicious score for a given word, we just need to sum up IDFs of all bigrams occurring in a word:

Word / Bigram | aa  | ab  | ba  | bb | Suspicious score (sum of IDFs)
---   | --- | --- | --- | --- | ---
aabaa | 2   | 1   | 1   | -   | **2.6** = 2 * 1.1 + 1 * 0.0 + 1 * 0.4
abb   | -   | 1   | -   | 1   | **1.1** = 1 * 0.0 + 1 * 1.1
ababa | -   | 2   | 2   | -  |  **0.8** = 2 * 0.0 + 2 * 0.4
**IDF**  | **1.1** | **0.0** | **0.4** | **1.1**

We see that the word *aabaa* with a score of 2.6 is the most suspicious word out of our fabricated example.

If you use this algorithm to compute suspicious score on our 2020 words, you get [this result](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-output-standard-idf.txt). Looks ok, but take a look at positions #9, #11 and #13. *Christopher* and *Jefferson* are regular names but they have very high suspicious scores (e.g. higher than *lkjwqer*, *erwtrepo* and others).

Can we improve the algorithm?

## Improve the algorithm with adjusted-IDF

The problem with *Christpoher* or *Jefferson* is that they are long and contain a lot of common bigrams. The sum of IDFs of these common bigrams is higher that the sum of IDFs of fewer but more rare bigrams.

To overcome this, we can adjust the IDF formula to return lower score for common bigrams. One of many possible options is to use the following formula:

$$ \text{IDF}_\{adj}(b) = \log\frac{\max_c(n_c)}{n_b}, $$

where the $\max_c(n_c)$ term is just the number of occurrences of the most common bigram. Using this formula, the bigram score should be zero for the most common bigram and almost zero for very common bigrams.

To see the adjusted-IDFs for the bigram in our sample of 2020 words, click [here](https://raw.githubusercontent.com/mkrcah/random-text-detector/master/data/names-idf-adjusted.txt). You can see that scores of very common bigrams decreased from 2.5 to 0.5, while the score of most rare bigram remain above 5. *(You can experiment and tweak this formula more in order to fit your data. Apart from logarithm, you might also consider [different  functions](http://en.wikipedia.org/wiki/Exponential_distribution))*

The adjusted-IDF is the score that I used to arrive at the [results](https://github.com/mkrcah/random-text-detector/raw/master/data/names-output-adjusted-idf.txt) presented in the beginning of the article.

## Grab the source code here

I put the source code (Scala) on [github](https://github.com/mkrcah/random-text-detector). It tried to make it simple to get the tool up and running on your local machine (either in batch-mode or in server-mode with API). Also, it should be possible to deploy code to [Heroku](http://random-text-detector.herokuapp.com) without any code changes.


Detecting random text in web registration forms


That’s the one you need to do first. That’s where the highest impact and peak productivity lies, at least based on my experience.

Here are my three observations.

First, my willpower is strongest in the morning. [Research](https://www.amazon.com/Willpower-Instinct-Self-Control-Works-Matters/dp/1583335080) suggests the same. Focusing on the most resented task first thing in the morning allows me to utilize my willpower at its peak. Other tasks in the morning would deplete my willpower and thus increase the risk of postponing the important task to the next day. (This is the same reason why I exercise in the morning. Have you ever skipped a planned evening workout because there was no willpower, energy, or time left?)

Second, there is a feeling of satisfaction from the beginning of the day. This satisfaction remains present throughout the whole day, no matter how the rest of the day evolves. (How do you feel during the day if you workout in the morning?)

Third, it leads to tremendous productivity boost via a compound effect. I have observed that resented tasks usually fall into the [important not-due-soon quadrant](https://www.youtube.com/watch?v=oTugjssqOT0&feature=youtu.be&t=20m56s), known as the holy grail of peak long-term productivity. Since resented tasks often relate to each other, results get built upon each other.

<br/>

What would change if, for the upcoming month, the first task you do at work is the one you resent the most?


Which task do you resent the most today?


At [PyData Warsaw 2017](http://pydata.org/warsaw2017) and [Prague Hadoop Meetup 2017](ttps://www.meetup.com/CS-HUG/events/244755865) I talked about best-practices for putting a prototype of a machine-learning model or a data application into production. I tried my best to cover principles agnostic to a language or a technology. I see now I put perhaps too much detail in the slides & I was also more opinionated back then.

Slide deck: [http://talks.marcel.is/hugcz2017-sebp](http://talks.marcel.is/hugcz2017-sebp)

Video from the PyData Warsaw:

<iframe width="420" height="315"
src="https://www.youtube.com/embed/MKrPXfvIWoc">
</iframe>




From a model to production: Software-engineering best-practices


When building software, we introduce changes to the system. To change the system, we first need to understand the system. Each system consists of components. So to change the system, we need to understand its components. What makes components hard to understand?

Imagine component A intertwines with component B. And B intertwines with C. And C intertwines with D. To change component A, we need to dive into B and C and D. But we are limited in our cognitive capacity; in our heads we can consider [at most a few things](https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two) at a time. So to change a component, we need to understand the whole braid of intertwined components. That takes time.

> Everytime you combine two things, you twist two more things. So every intertwining adds this burden, and the burden is combinatorial. Fundamentally, this complexity, this twisting, limits our capability to understand our systems.
>
> — Rich Hickey, [Simple Made Easy](https://www.infoq.com/presentations/Simple-Made-Easy/)

To gain speed, we decomplect the _complex_ braid and turn it into a set of _simple_ components. A simple component consists of one fold, of one braid, of one ingredient. A simple component doesn't intertwine with other components. To change a simple component, we don't spend time understanding the whole braid. And so we are faster.

![simplicity](/img/simple-components.png)


What is complexity?


On the road to improve my writing, I stumbled upon this [super interesting video by Larry McEnerney](https://www.youtube.com/watch?v=aFwVf5a3pZM). The techniques covered in the video bend my mind. One such technique is about the subjects of sentences. Consider the two sentences below, which one do you think is better?

1. The dog chased the cat.
2. The cat was chased by the dog.

The first sentence is shorter and has an active verb. So, according to conventional wisdom, the first sentence is better. However, that might not be optimal. When we write, as Larry explains, we should aim for concision. But not in the conventional sense, that is, the number of words. Instead, we should target the _time_ it takes for the reader to understand the sentence.

We can improve concision by multiple routes. One route, it turns out, is to focus on the sentence's subject. It is because the subject defines what the sentence is _about_. So the first sentence, "The _dog_ chased the cat," is about the dog; while the second sentence, "The _cat_ was chased by the dog," is about the cat. The two sentences are about different things.
Now, what if we serve the reader the sentence about the dog, but they care about the cat? That's inconsistent. So the reader needs to cognitively reprocess the sentence to make it about the cat. This reprocessing is taxing, and it harms concision. The reader becomes disinterested or bored, and you can lose him.

![subjects](/img/subjects.png)

We should serve the reader subjects that are aligned with what they care about. If the reader cares about the dog, the subject should be the dog. If they care about the cat, the subject should be the cat.

So I started experimenting with the technique. When drafting a product update to the rest of the company, I wrote: "We delivered the feature \[X\]". However, the readers care about the product and the customers. So I can write the sentence as "Customers can now do \[X\]," and bring the product update closer to the readers' interest.

There's this story from Larry that I remind myself in hope to lubricate my rusty subjects:

> Sometimes people pay me stupid amounts of money to fly to often wonderful cities to sit down with them. They are saying to me: "My readers don't find this valuable" or "I can't get this published" or "I can't get promoted." So I make them take their pen out and underline the subjects of their sentences. And I ask: "Do your readers care about this?" And they say: "No."
>
> — Larry McEnerney, from the lecture [Writing Beyond the Academy](https://www.youtube.com/watch?v=aFwVf5a3pZM),


Exploring my writing: The subjects of sentences


James Clear is the author of Atomic Habits, who popularized the idea of habits over goals. Last month he shared [the rules he uses for running his business](https://twitter.com/JamesClear/status/1344776679315689480), based on his 10-year experience. Among the rules was this one:

> Build assets that compound. Eliminate problems, responsibilities, and obligations that compound. Your business should feel easier to sustain each year.
>
> — James Clear (from [How I run my business](https://twitter.com/JamesClear/status/1344776679315689480))

"Your business should feel easier to sustain each year." What if, instead of business, we focus on building software? And what if we turn the advice into a question? Then, we get: _does our codebase feel easier to sustain each year?_

I love that question:

- It reveals the long-term vision of how to approach software engineering.
- It encourages challenging missions to [remove complexity](https://www.infoq.com/presentations/Simple-Made-Easy/) and [target the important non-urgent quadrant](https://youtu.be/oTugjssqOT0?t=1256).
- It exposes the murky alternative, that is, descending further into the complexity tar pit.

All in all, I've found that the question brings fresh air and whirls settled dust.


Does your codebase feel easier to sustain each year?


_Update: See [HN comments](https://news.ycombinator.com/item?id=17868066) for this article._

A company is not data-driven _by using a BI tool._

A data-science team is not agile _by using a ticketing system._

A study is not useful _by presenting results in a fancy tool._

<br />

Using modern software does not imply intrinsic value.

<br />

A company can be data-driven just with a pen and paper.

A data-science team can be highly-performant just with a whiteboard.

A study can be impactful if presented just in a notepad.

<br />

What counts is the underlying hard-work, the focus on stuff that matters, and courage to make a decision.


On software tooling


It's an option. Pushing a proposal because it’s _your proposal._

Such an approach, however, often leads to discussions that are unpleasant for attendees. People get irritated or angry. Relationships and a team spirit get harmed.

There is another way.

Striving to be objective.

Proactively considering limitations of your proposal. Being aware of and acknowledge the advantages of other proposals. Choosing a solution based on mutually understandable pros and cons.

Becoming a curious observer, a detached messenger of facts and opinions. Including your own.

It’s an appealing state of mind, worth the higher mental investment.

Perhaps an occasional subjective push from a team’s authority would be eventually needed. However, it would come from a different angle, based on mutual understanding of each other.


Taking things personally


Every month I need to create a tax report using this ancient government website. Facing the choir, I found myself procrastinating every month, and this procrastination had been draining my energy. So I decided to face my daemons.

Encouraged by [[tracking-hours|my last automation project]], I discovered it is possible to semi-automate the choir. It turned out the website supports uploading the reports in XML format. It also turned out that my invoicing system provides REST API. I was onto something. 

So I summoned my favorite tools for file generation—Python and [Jinja ](https://jinja2docs.readthedocs.io/en/stable/)—and added [Click](https://click.palletsprojects.com/en/7.x/) to the mix. The result is [a CLI tool](https://github.com/mkrcah/souhrnne-hlaseni) that extracts invoices from the API and generates the XML report. 

While coding, I discovered new delightful things about Jinja and Click:

- Jinja can raise an error during template generation if a variable is not set. This behavior is configured with 
	```python
	Environment(...,undefined=jinja2.StrictUndefined)
	```
- Jinja can automatically escape content injected into XML files. So if my name would be "Marcel<", Jinja would generate  `<person name="{{ name }} />` as  `<person name="Marcel&lt;" />`.   Escaping is configured with
	```python
	Environment(...,autoescape=select_autoescape(['xml'])
	```
- Click can [read options from environment variables](https://click.palletsprojects.com/en/7.x/options/?highlight=auto_envvar_prefix#dynamic-defaults-for-prompts). This is great: I added invoicing auth credentials to my shell startup script—in my case `.zshrc`—and now I don't have to think about them anymore. 

I spent two evenings coding. Strictly from the time perspective, it was probably [not an optimal investment](https://xkcd.com/1205/). But when I generated my first report, I noticed a stark difference in my mood: I *enjoyed* the choir. 











Generating tax reports with Python, Jinja, and Click


What do _you_ want to do?

Find this out for each team member.

Then align the roles accordingly.

It's dead simple.

### Why?

At least among software engineers (but I believe it's among all of us), there is this hidden voice inside. It silently and always reminds us what we want to do, what the forbidden fruit is.

Java engineers seeking transition to Scala.

Scala programmers wanting to do Big Data.

Big Data folks wanting the transition to product management.

<br/>

When this calling is enabled, magic happens.

Energy goes up. Productivity goes up. Satisfaction goes up.

Flow happens.

Cannot-wait-for-Monday-morning happens.

And when enabled within a team, synergies occur. The team becomes a warp-engine rocket, delivering like crazy while having tons of fun.

<br/>

Of course, a team might not need a new member; some roles might be lacking, etc.

But it's the best baseline. Once defined, it's just a matter of time until it's reached.


Supercharge your team


I'm spending these days kicking off end-to-end tests. By end-to-end tests, I mean tests that interact with a set of IT systems as if the set was a big black box. For example, one test sends a REST API request to system A, waits until the data propagates through systems B and C to system D, and checks the result in system D via another REST API call. Now, how to implement the _waiting_ part for this asynchronous test?

![testing-async-systems](/img/testing-async-systems.png)

The first option is to wait for a fixed period of time, such as 10 seconds or 1 minute. But with the period being too long, it's cumbersome to develop such tests; I want fast feedback loops when developing. On the other hand, with the period being too short, the tests can fail; and I need tests to be stable to trust them.

Another option is to set up a service that provides visibility into the asynchronous request. So we can periodically poll the status of the request from tests. However, building such a service can be a mound of engineering work across multiple systems. Such a service could not be even feasible to build.

The third option is to write tests that check periodically that the expectation is met until a given timeout is reached. For example, we can check every second that the result in system D is present; and we can keep checking up until one minute. I find the third option to be the sweet spot: minimal engineering work, minimal waiting time, and fast feedback loops.

It turns out there are libraries built for such periodic checks (thanks Peter for introducing me to these). For my current project based on Javascript/Typescript and [jest](https://jestjs.io/), there's [wait-for-expect](https://github.com/TheBrainFamily/wait-for-expect). For JVM-based languages, such as Scala, there's [awaitility](https://github.com/awaitility/awaitility).


Testing asynchronous systems


_By a data analytics warehouse we mean a solution that combines and cleans data from various internal and external sources and offers reporting and querying capabilities to business users. Sometimes also referred to as a big data platform._

Based on failures and hard-won lessons, here's 21 tips for teams and individuals who are responsible for making a company more data-driven.

### Solve the right problem

1. Understand that building an analytical warehouse is [not about the technical tooling](/sw-tooling).
2. Get an in-depth understanding of [the root problem](/problem) to minimize the risk of delivering a solution that doesn't solve a tangible business problem.


### Choose the right tool for the job
3. Consider the most straightforward solutions to resolve the root problem. Perhaps, an excel would suffice.
5. Do not choose a solution according to [personal preference](/quench).
6. Be aware of maintenance costs. Consider off-the-shelf solutions before building your own. If a cloud platform is an option, consider no-ops and SaaS offerings. There is a plethora of solutions out there for ETL, storage and BI tools. Be [pragmatic about Hadoop](https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html).
3. Start with delivering a narrow [vertical slice of functionality](https://en.wikipedia.org/wiki/Vertical_slice) to discover possible problems early and to validate the chosen approach.
4. Validate new deliverables as frequently as possible through an intimate collaboration with business.



### Provide correct and consistent data

8. Check for data correctness: integrity violations (foreign keys, unique values), invalid value types, errorneus ranges (values out of bound, text too long or too short), missing data, etc.
8. Check for suspicious data: values out of expected bounds, the number of new records too large/low, etc.
9. Treat incorrect and suspicious data as exceptions and warnings, respectively. Report on them automatically and periodically.
13. Use consistent prefixes and suffices per field type to limit guesswork. For example, use `is_` for booleans and  `_at` or `_time` for time.
10. Ensure consistency in values. For example, use `null` for missing data and [ISO codes](https://en.wikipedia.org/wiki/ISO_3166-1) for countries among all tables.
17. Convert values to native types as early as possible. For example, convert date strings to dates, and binary integers to booleans.

### Bring data closer to business
14. Add as much semantics to field names as possible. Where applicable, include units, timeframes, and aggregation functions. For example, use `avg_kwh_consumed_per_year`  instead of `energy`.
18. Establish and use common terminology with business. Rely on [ubiquitous language](https://martinfowler.com/bliki/UbiquitousLanguage.html) for table names, field names, and values. Reconsider [using abbreviations](https://www.ibtimes.com/spacex-boss-elon-musk-threatened-drastic-action-against-employees-who-use-unnecessary-1972972).
9. If possible, encode domain knowledge and undocumented business rules into data to increase value for data users.
15. If relevant, teach SQL to data users to enable advanced self-service querying.
20. Understand that data is [an abstract model of reality](https://fs.blog/2015/11/map-and-territory/). Identify and communicate data limitations.

### Develop efficiently
19. Learn [window functions](https://community.modeanalytics.com/sql/tutorial/sql-window-functions/) and [`with` queries](https://www.postgresql.org/docs/current/static/queries-with.html) to simplify complex SQL queries.
16. Create views for frequently reused SQL queries to ensure users [don’t repeat themselves](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
23. Follow software engineering best practices, incl. version control, clean code, testing, peer reviews and [boring and predictable deployments](https://www.youtube.com/watch?v=aoMfbgF2D_4).



Tips on building a data analytics warehouse

Should we buy the IT system from a vendor, or should we build it in-house? For some folks out there, myself included, this is a crossroad clouded with uncertainty.



To scatter the clouds and shed light into the paths ahead, let's hear what the elderlies of software engineering craftsmanship say:

<div class="blockquote">

> If it’s a core business function — do it yourself, no matter what.
>
> Pick your core business competencies and goals, and do those in house. If you’re a software company, writing excellent code is how you’re going to succeed. Go ahead and outsource the company cafeteria and the CD-ROM duplication. If you’re a pharmaceutical company, write software for drug research, but don’t write your own accounting package. If you’re a web accounting service, write your own accounting package, but don’t try to create your own magazine ads. If you have customers, never outsource customer service.
>
> — Joel Spolsky, from [In Defense of Not-Invented-Here Syndrome](https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/)

</div>
<div class="blockquote">

> Cores are the central competencies of the organization. These are things that customers are willing to pay for and what investors reward. Context is everything else. \[...\] As we fund core features and applications, we need to make sure that context doesn’t kill core.
>
> Of the applications and services that you manage, which of them are customers willing to pay you for? Which ones truly enhance competitive advantage? And which can you rely on vendors for?
>
> — Kim Gene, from [The Unicorn Project](https://www.goodreads.com/book/show/44333183-the-unicorn-project), chapter 17

</div>

<div class="blockquote">

> When it comes to build versus buy, the frequently repeated but rarely followed wisdom is good advice: if you're a technology company, vendors usually generate significant value if they're outside your company's core competency; within your core competency, they generally slow you down.
>
> — Will Larson, from [Build vs Buy](https://lethain.com/build-vs-buy/)

</div>

So according to the advice, the crossroad sign can read:

![buy-or-build](/img/buy-or-build.png)

Update 08/2021: Another good post: [Why is it so hard to decide to buy?](https://skamille.medium.com/why-is-it-so-hard-to-decide-to-buy-d86fee98e88e) by Camille Fournier

To buy or to build?


For the current project, I need to track hours in [Clockify](https://clockify.me/). But as I was opening the webpage and filling in the hours, I felt dispassionate. So I decided to turn this activity into a joyful one.

It turns out that Clockify has an easy to use [API](https://clockify.me/developers-api). So I decided to build a script that 1)
gets a list of all projects via an API call, 2) lets me interactively select a project, 3) lets me select duration I'd spent on the project that day, and 4) adds hours to the projects via an API call. To do the job, I summoned my favorite command-line tools:

- [HTTPie](https://httpie.io), a command-line HTTP client, out of the box ready for JSON REST APIs,
- [jq](https://stedolan.github.io/jq/), a swiss-knife for parsing and transforming JSON,
- [awk](https://ferd.ca/awk-in-20-minutes.html), a gnu stronghold for text processing,
- [fzf](https://github.com/junegunn/fzf), an interactive fuzzy search for the command-line,
- `cut` and `tr`, text processing unix commands.

For example, to interactively select a project id I piped the tools together:

![track-hours-select-a-project](/img/track-hours-select-a-project.png)

Using the script almost every day, I'm now playfully enjoying the tracking routine.

Here's the full script:

```bash
#!/usr/bin/env bash

# fail on error, unused var and pipe failure
set -euo pipefail

# clockify api config
CLOCKIFY_BASE_URL='https://api.clockify.me/api/v1'
CLOCKIFY_DATE_FORMAT="%FT%T"

# let me select a project
PROJECT_ID=$(http $CLOCKIFY_BASE_URL/workspaces/"$CLOCKIFY_WORKSPACE_ID"/projects \
  'X-Api-Key':"$CLOCKIFY_API_KEY" | \
  jq -r ".[] | .id, .name" | \
  awk 'NR%2{printf "(%s) ", $0;next}1' | \
  fzf --nth=2.. | \
  cut -f1 -d' ' | \
  tr -d '()'
)

# let me pick how many hours I worked today
HOURS=$(seq 1 12 | awk '{print $0 " hours"}' | fzf | awk '{print $1}')

# start the day at 9:00am
TASK_STARTED_AT=$(date -v9H -v0M -v0S +$CLOCKIFY_DATE_FORMAT)

# and stop after $HOURS have passed
TASK_STOPPED_AT=$(date -j -v +"${HOURS}H" -f $CLOCKIFY_DATE_FORMAT "$TASK_STARTED_AT" +$DATE_FORMAT)

http -q $CLOCKIFY_BASE_URL/workspaces/"$CLOCKIFY_WORKSPACE_ID"/time-entries \
  'X-Api-Key':"$CLOCKIFY_API_KEY" \
  start="${TASK_STARTED_AT}Z" \
  end="${TASK_STOPPED_AT}Z" \
  projectId="$PROJECT_ID" \

echo "Done 🚀"
```


Tracking hours with Clockify, fzf, jq and awk


You have decided to do a career switch to THE BIG DATA, but you don't know how.

Here's what to do:

1. Choose your role
2. Build a sexy demo and job hunt guerilla-style

## Step One: Choose your role

There are four roles in this big-data world:

- Finding insights in the past
- Setting up & delivering a data warehouse
- Predicting future from past data
- Coding data-driven product features

### Role 1: Finding insights in the past

This is a key role in jobs titled Data analyst and Marketing analyst. Also, it's a key prerequisite for Growth Hackers.

You answer questions about the past to help steer the business forward. For example:

- How does the [cohorts](http://cohortanalysis.com/) look like in our customer base?
- Which airport gates lead to the most late passangers?
- Which customer micro-segment is driving the most revenue?

You use BI tools (like Tableau, Looker) to interactively explore and visualize data. If you are more technical, you reach for SQL and/or R.

After a while, you start impacting metrics that you report. You use [growth hacking](https://www.quicksprout.com/the-definitive-guide-to-growth-hacking) and [A/B testing](https://www.optimizely.com/ab-testing/). And you provably improve company's profit.

Learn the first steps by:

- Read the first 50-pages of the [Pyramid principle](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516) to get yourself acquainted with top-down reasoning
- Search for online-sources for Data Analysis (example: [edX](https://www.edx.org/course/subject/data-analysis-statistics), [Coursera](https://www.coursera.org/courses?languages=en&query=data+analysis))

### Role 2: Setting up & delivering a data warehouse

This is a key role in jobs titled Data Engineer, ETL engineer and Hadoop developer.

Typically, your responsiblity is to combine multiple company data together from various sources.

For example, you combine customer data from Salesforce, support data from Intercom, product usage from a few production Postgres databases, real-time customer events from Kafka and you sprinkle it with historical weather information.

Then you create an abstraction on top of this data to provide a single source of truth. You operate between two departments:

- the business: to align on metric definitions and
- the IT: to get the integrations right.

Having all data together in this manner has HUGE benefits - all of a sudden, the time-to-answer drops from days/weeks to zero. Connect a self-service BI tool on top of it (e.g [Looker](https://looker.com/), [Metabase](http://www.metabase.com/)) and you are there.

Note that there are cloud solutions out there which provide the warehousing and data shoveling for you. Checkout [Redshift](https://aws.amazon.com/redshift/?hp=tile&so-exp=below) and [BigQuery](https://cloud.google.com/bigquery/) for warehousing and [Alooma](https://www.alooma.com/), [Stitch](https://www.stitchdata.com/) and [Fivetran](https://www.fivetran.com/) for shoveling.

You need to do the data merging yourself - that's the company domain. You do this either with SQL or a data processing tool:

- Learn SQL [here](https://community.modeanalytics.com/sql/tutorial/introduction-to-sql/), especially the [cool](http://stackoverflow.com/questions/12552288/sql-with-clause-example) [features](https://community.modeanalytics.com/sql/tutorial/sql-window-functions/)
- Learn [Spark](http://spark.apache.org/), it's sexy these days. Learn it [here](https://youtu.be/fn3WeMZZcCk?t=878) and [here](https://databricks.com/training)

Notes:

- Software engineering skills are expected. Learn principles of [clean-coding](https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670/ref=pd_sbs_14_t_1?_encoding=UTF8&psc=1&refRID=3RJ1DN7QVHCWJEKCBDGT) and learn Python or Scala.
- Scala is hot. Get the certificate from [this legendary course](https://www.coursera.org/learn/progfun1) and put it on your LinkedIn.
- You _don't_ need to learn Hadoop. Most companies [don't need Hadoop](https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html).

### Role 3: Predicting future from past data

This is a key role in jobs titled Data Scientist and Quants.

Your role is to predict behavior by learning from the past data:

- What are the products that customer X is most interested in?
- How much is product X going to be traded on the exchange tomorrow?
- How to optimally coordinate airplane routes to minimize fuel costs?
- How utilized will the Tesla super-chargers be tomorrow at location X?

You are the math guy in the company. You know machine-learning, simulations or numerical computing. Typically, machine-learning is required. This includes:

- supervised/unsupervised machine-learning models
- ability to prevent over-fitting
- ability to engineer and select the right features

Daily bread is Pandas, Scikit-learn (my favourite), Matlab, R or SparkML. However, I know a guy in Booking.com who uses C++ for his machine-learning experiments.

Learn the first steps by:

- Follow the legendary [Andrew Ng's lecture](https://www.coursera.org/learn/machine-learning) on machine-learning
- Go to [Kaggle](https://www.kaggle.com/competitions) to practice your skills.
- The Kaggle [forums](https://www.kaggle.com/forums) are legendary, you will learn a LOT of practical info there

Notes:

- Focus is on understanding the business and the underlying data so you can come up with the right data [features](http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/).
- Once the features are there, a simple model would do.
- Less software engineering skills is expected. Clean-code is not a priority. There is another role for translation the model into production.

### Role 4: Developing data-driven product features

This is a common role in jobs titled Data Engineer.

You are putting into production features driven by prediction and machine-learning models:

- compute product recommendations and expose them via API, which is consumed by team-members building the mobile app
- implement a stock prediction algorithm within a production trading platform

If there is a dedicated data scientist, you work closely with him/her to translate their model to production code. If there is a growth-hacker, you work closely to implement their ideas.

- You are a software engineer by heart: you know a few languages, [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)/[YAGNI](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it), clean-code, continuous delivery, unit/functional tests, monitoring - the whole shebang.
- On top of it, you understand the big-data ecosystem used by the team (e.g. Kafka, Spark, [ELK Stack](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04))
- On top of it, you understand machine-learning and can translate data scientists model into production.
- Sometimes, data scientists are also engineers who put their models into production

Learn the first steps:

- If you are not a coder, get an engineering job first. I recommend Scala. Try to grasp the functional approach of it. Scala is used heavily in big-data community. Another option would be Python. Once you are good in engineering, go to the next step:
- If you are a coder, learn Spark and a few other technologies. It's the tech-stack you know that will get you this role.
- Although companies think they need Hadoop, they [most probably don't](<(https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html)>). But it's good to learn it, so you understand situations when Hadoop is useful. Checkout the major Hadoop distributions (e.g. [Cloudera](www.cloudera.com), [Hortonworks](http://hortonworks.com/), [MapR](http://www.mapr.com)), install it locally and try to get the basic ETL up and running. Or checkout cloud solutions like [AWS EMR](https://aws.amazon.com/emr/) and [Google Dataproc](https://cloud.google.com/dataproc/).

### Choose your role by your current strengths:

- If you have an economic background and no engineering skills yet, go for Data Analyst.
- If you have system administration background, go for setting up a warehouse.
- If you have software engineering background, go for data engineering.
- If you have strong mathematical, model, statistics background, go for Data Scientist.
- If you are a student, go learn machine-learning, [clean coding](https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670/ref=pd_sbs_14_t_1?_encoding=UTF8&psc=1&refRID=3RJ1DN7QVHCWJEKCBDGT), Scala and Clojure.

Don't worry too much with the initial choice. Especially in smaller companies, the roles overlap and you have freedom to move between the roles. I know a Data Scientist doing warehousing stuff for two years and a Data Engineer doing growth hacking.

Still not sure which role? [Drop me](mailto://m@marcel.is?subject=Which role to choose) an email.

## Step 2: Build a sexy demo and go guerilla

Do this:

- Read [this article](https://medium.com/the-mission/how-to-get-any-job-you-want-even-if-you-re-unqualified-6f49a65f5491#.c1cuqxawh) and [this article](http://brohrer.github.io/get_data_science_job.html) and choose one of these two approaches.
- Choose your target company.
- Build a (targeted) demo

For choosing the target company:

- aim for a smaller one, ideally a startup. There is much more freedom to experiment and switch roles.
- aim for a team where only 1-2 people do what you do. This gives you a lot of space to experiment. Plus the opportunity of taking up the full responsibility once they decide to move to something else.

For the demo, here are a few examples:

#### Example 1 (for data analysts): Data insights and recommended next-steps

- Find interesting data about the company you are hunting, e.g. [in this collection on GitHub](https://github.com/caesar0301/awesome-public-datasets).
- Get Excel, [Tableau Public](https://public.tableau.com/s/) or a similar tool and dive deep into data
- Deliver a [top-down](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516/ref=pd_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=KBBSY0YBQYA53CZEREKJ) structured research story with recommendations on next steps

#### Example 2 (for data analysts): Tailored growth-hacking recommendations

- Read more about [growth-hacking](https://github.com/btomashvili/awesome-growth-hacking) and [lean startup](http://theleanstartup.com/book)
- Guess problems the target company is facing by studying their products
- Deliver a list of 5-10 actions you would advise them and metrics on how to evaluate progress

#### Example 3: (for data engineers): Spark Streaming with real-time visualizations

- Spark is hot, [Spark Streaming](http://spark.apache.org/streaming/) especially.
- Play with combination of Twitter and Spark Streaming, like [here](https://www.youtube.com/watch?v=fn3WeMZZcCk&feature=youtu.be&t=1038)
- Or play locally on your computer, example [here](https://github.com/mkrcah/scala-kafka-twitter).

#### Example 4: (for data engineers): Click-stream processing with Kafka & Alooma

- Build a [simple web app](http://flask.pocoo.org), collect clicks via [Kafka](https://kafka.apache.org/)
- From Kafka, send the clicks to [Alooma](https://www.alooma.com/). Sessionize the clicks in Alooma and pipe it to [BigQuery](https://bigquery.cloud.google.com).
- Connect [Metabase](metabase.com) to BigQuery to show reports on basic usage
- You have just build a simple version of Google Analytics!
- Bonus: Find which SaaS product your target company is using (e.g. for customer support) and merge that data with the clickstream.

#### Example 5: (for data scientists): Open-source your Kaggle solution

- Compete on [Kaggle](https://www.kaggle.com/competitions)
- Once you have a decent model, put it on Github and LinkedIn and blog about it.

#### Example 6: (for data scientists): Deliver tailored prediction

- Get hold of data which might be interesting for your target company, for example [here](https://github.com/caesar0301/awesome-public-datasets)
- Find a problem which can be framed as a machine-learning problem.
- Solve it, package the result in a [top-down story](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516/ref=pd_sbs_14_img_0?_encoding=UTF8&psc=1&refRID=KBBSY0YBQYA53CZEREKJ) in a [Jupyter](http://jupyter.org/) notebook and [share it only](http://nbviewer.jupyter.org/). Put it on Linked and blog and it.

## FAQ:

#### Shall I do pro-bono?

Sure. If you don't feel confident about your current skills, do a pro-bono project.
Reach to your network of friends, family and colleagues. Try to get a contact
within your network that would tell you about their company problems and are willing
to share their data. In exchange for your work (if done well), you'd ask for a referral that you can post on your LinkedIn/blog.

<br/>
Enjoy the ride and [ping me](mailto:m@marcel.is?subject=Big data transition) with your transition story. I'm very much interested.

I'll leave the last words to the marketing guru [Seth Godin](http://sethgodin.typepad.com/seths_blog/2006/10/make_something_.html):

> **Make something happen**
>
> If I had to pick one piece of marketing advice to give you, that would be it.
>
> Now.
>
> Make something happen today, before you go home, before the end of the week. Launch that idea, post that post, run that ad, call that customer. Go the edge, that edge you've been holding back from... and do it today. Without waiting for the committee or your boss or the market. Just go.


Transition your career to big data


In the book [The Metaphors to Live By](https://www.goodreads.com/book/show/34459.Metaphors_We_Live_By), the authors observe that we think and write and communicate in conceptual metaphors. Here are a few examples:

- **Conscious is up, unconscious is down**: Get _up_. She _rises_ early in the morning. He _fell_ asleep. He's _under_ hypnosis. He _sank into_ a coma.
- **Good is up, bad is down**: We hit a _peak_ last year, but it's been _downhill_ ever since. Things are at an all-time _low_. He does _high_-quality work.
- **Argument is a building**: That is the _foundation_ of our theory. The theory needs more _support_. The argument is _shaky_, it _falls apart_. We need to _construct_ a strong argument.
- **Argument is a battle**: Your claims are _indefensible_. He _attacked_ every _weak_ point. His criticisms were on _target_. We need to _aim_ for concision. They have a _strategy_.
- **Argument is a container**: Your argument doesn't have much _content_. That argument _has holes in it_. His objections have _less substance_. Your arguments are _empty_.
- **Argument is a journey**: We have _set out_ to prove that. _So far_, we've seen that theories will not work. We will _proceed_ in a _step-by-step_ fashion. We have _arrived_ at a conclusion. We are at a _crossroad_.
- **A problem is a body of water**: He _dived into_ the problem. He _immersed_ himself in the problem. The problem is _murky_. Finally the answer _surfaced_.
- **A problem is a region in a landscape**: We've got to _explore_ this problem. Let's _map out_ the problem before doing anything else. We're _heading_ in the right direction. The solution _lies far ahead_.
- **Ideas are food**: That's _food for thought._ We don't need to _spoon-feed_ our students. This is the _meaty_ part of the paper. There are too many facts to _digest_.
- **Ideas are cutting instruments**: That's an _incisive_ idea. That _cuts right_ to the heart of the matter. He's _sharp_. He has a _razor_ wit.
- **Ideas are plants**: The _seeds_ of his ideas were _planted_ in his youth. She has a _fertile_ imagination. Mathematics has many _branches_. Here's an idea I'd like to _plant_ in your mind.

It turns out [there are](http://www.lang.osaka-u.ac.jp/~sugimoto/MasterMetaphorList/metaphors/index.html) [catalogs](https://metaphor.icsi.berkeley.edu/pub/en/index.php/Category:Metaphor) of conceptual metaphors. We use them all the time. For example, also in the [man pages](https://jvns.ca/blog/2020/05/08/metaphors-in-man-pages/).


We think in conceptual metaphors


Today at work, we discussed when to write a script in shell and when in a language such as Python or Javascript. I realized I enjoy writing a shell script when:

-   I integrate heavily with the system, such as creating SSH tunnels, mounting a device, or copying files around. I could do so in Python, but I would need to add boilerplate code around system commands. Or I would need to find an equivalent Python/Javascript library. In most cases, I've found vanilla shell scripts easier to write.
-   I do simple text processing over a mound of files, such as searching log files or replacing text. I feel like Usain Bolt when I pipe together commands from the powerful family of `awk`, `sed`, `grep`, `cut`, `wc` and the like.
-   I run a script in an environment where I have no access to another language. Unix commands are available on any unix machine, even if Python is not. 

However, the more the script grows in complexity, the more cumbersome I feel when writing it in shell. I think it takes an extraordinary level of discipline and experience to make sure that a large shell script is easy to read and maintain. For that reason, I feel relieved when I can switch to Python or Javascript.

Another decision axis I find important is the team preference. Imagine the team doesn't need and doesn't want to learn shell. If I introduce shell scripts, I would be concerned that I would be the only one left to maintain the script.


Shell scripts vs. Python/Javascript scripts


As I've been doubling down on my writing, I started to observe situations when I hedge. Hedging should be avoided, the writing advice goes, but I've found that it depends.

> Many writers cushion their prose with wads of fluff that imply that they are not willing to stand behind what they are saying.
>
> — Steven Pinker (from [The Sense of Style](https://www.goodreads.com/book/show/20821371-the-sense-of-style), p. 43)

In one situation, a colleague asked about our website's unexpected behavior. I quickly investigated the issue and discovered that the most probable cause was a bug in the backend. So I replied: "It seems this is a bug in the backend." Such hedging, I realized, allows me to quickly communicate uncertainty in the case of the unknown. It's a shortcut I find acceptable.

![hedging-unknown](/img/hedging-unknown.png)

However, there are other situations. Once, I wrote: "It seems that the data connector is working." But this situation was different from the one above: I wrote the sentence _after_ investigating the data connector, not before. I knew the connector, I knew the facts, yet I presented the outcome with hedging.

I can do better. Instead of cushion the uncertain generic conclusion with a safety net, I can organize the facts in a logical flow. I can write: "The connector filled in the missing data for October. To declare the connector working, we need to check the connector's real-time ingestion." In this situation, the lack of hedging forces me to clarify my thoughts, which, in turn, helps others to understand the situation, the known, better.

![hedging-after](/img/hedging-after.png)

> To write well, you need first to know what you are talking about. Thus, you have to do your research. ... Then you have to be able to organize your argument, at the level of the word, the sentence, the paragraph and the essay itself. If you can do that, you can organize your thoughts and, in consequence, your brain. Then you can help organize other brains, and other structures.
>
> Jordan Peterson (from [Quora](https://www.quora.com/What-are-the-necessary-skills-for-21st-century-students-and-future-leaders/answer/Jordan-B-Peterson))


Exploring my writing: Hedging


Assume data in S3: 
```http
s3://mybucket/foo/2021/07/27/data.gz
s3://mybucket/foo/2021/07/28/data.gz
s3://mybucket/foo/2021/07/29/data.gz
s3://mybucket/foo/2021/07/30/data.gz
```

And AWS Athena table linked to the S3 bucket, with the partitions `Year`, `Month`, and `Day`.

```sql
CREATE external TABLE foos (...)  
PARTITIONED BY (year string, month string, day string)  
ROW FORMAT SERDE ...
LOCATION  's3://mybucket/foo';
```

After creating a table, partitions must be created. The docs suggests the following command:
```sql
MSCK REPAIR TABLE foos;
```
However, this won't work. The query will succeed but with warnings:
```markup
Partitions not in metastore: 
foo:2021/07/27, 
foo:2021/07/28, 
foo:2021/07/29, 
foo:2021/07/30.    
```

This is because `MSCK REPAIR TABLE foos` supports only partitioning with key-value pairs in file paths, such as:
```http
s3://mybucket/foo/year=2021/month=07/day=27/data.gz
s3://mybucket/foo/year=2021/month=07/day=28/data.gz
s3://mybucket/foo/year=2021/month=07/day=29/data.gz
s3://mybucket/foo/year=2021/month=07/day=30/data.gz
```

Partitions without the key-value pairs must be added manually:
```sql
ALTER TABLE foos ADD 
  PARTITION (year='2021',month='07',day='27') 
  	location 's3://mybucket/foo/2021/07/27'
  PARTITION (year='2021',month='07',day='28') 
  	location 's3://mybucket/foo/2021/07/28'
  PARTITION (year='2021',month='07',day='29') 
  	location 's3://mybucket/foo/2021/07/29'
  PARTITION (year='2021',month='07',day='30') 
  	location 's3://mybucket/foo/2021/07/30';
```

More info:
- [Amazon Forum answer from 2017](https://forums.aws.amazon.com/message.jspa?messageID=789078)
- [StackOverflow](https://stackoverflow.com/a/56806163)
- [AWS Partitioning Docs](https://docs.aws.amazon.com/athena/latest/ug/partitions.html)
- [Alter Table Add Partition Synopsis](https://docs.aws.amazon.com/athena/latest/ug/alter-table-add-partition.html)

AWS Athena: Adding yyyy/mm/dd partitions to an S3 table


In AWS RDS, you can provision PostgreSQL in three ways: vanilla PostgreSQL, Aurora PostgreSQL, and Aurora PostgreSQL Serverless. What are the differences? 

- The vanilla PostgreSQL is the open-source version, deployed on a fixed/provisioned instances
- The Aurora PostgreSQL is PostgreSQL-compatible implementation developed by AWS, deployed on a fixed instances 
- The Aurora PostgreSQL Serverless is the same as Aurora PostgreSQL, but deployed on scalable infrastructure. You specify minimum and maximum amount of resources and AWS scales the capacity for you based on the database load

The Aurora Serverless might be a good fit, cost-wise, for use-cases when the database load changes significantly during the day. For example, a CRM system of a high peak in the morning and no activity during the night. 




PostgreSQL vs Aurora PostgreSQL vs Aurora PostgreSQL Serverless

Within the AWS ecosystem, data stored in S3 can be queried with a business intelligence tool. The lego bricks are:



- AWS S3 for data storage
- AWS Athena as a SQL-querying layer on top of S3 (built on Presto). When creating a table in Athena, Athena uses AWS Glue Data Catalog for storing the metadata.
- AWS QuickSight as a business-intelligence tool connected to Athena.

![aws-s3-bi](/img/aws-s3-bi.png)

Business intelligence for AWS S3 data

The best practice for setting up AWS for different environments is a multi-account setup. [This article](https://aws.amazon.com/blogs/mt/best-practices-for-organizational-units-with-aws-organizations/) dives deep into the topic and uncovers the possibilities.

 

The setup in the article is comprehensive, perhaps an overkill for small dev teams. Here's a possible simpler solution:
- An empty AWS account with dev's IAM users (we can call it *root*)
- For each environment, a dedicated AWS account (we can call it *env-account*)
- In each env-account, have an Admin IAM Role with full administration access to the environment. The IAM Role must have a Trust Relationship setup with the root AWS account, that is, the users of the root AWS account can assume the Admin Role. 
- A dev logs in to the root account and assumes a IAM Role of the selected env-accounts. 

Billing could be consolidated using AWS Organizations. If more granular security is needed, different IAM Roles could be used.  

(There's also another solution: each env *and each dev* having their own AWS accounts. Pros: each dev can play in AWS independently; cons: more involved initial setup)

*Thanks to Bram and Daan for the discussion.*

Seting up AWS for test/acc/prod environments

AWS QuickSight users are structured around emails, not around IAM users. Any user of QuickSight must have an email address, no matter if they exist as an IAM user or not. However, IAM users don't have email addresses assigned to them. They have only usernames.  



So, in QuickSight: 
- Add an IAM User by filling in their IAM username *and* filling in their email. 
- Add a user without IAM by adding their email.
- Adding of an IAM *User Role* doesn't seem to be supported.

AWS QuickSight: Adding an IAM user


It turns out it is possible to read AWS Cloudwatch logs in a command-line with using `aws logs`. So I created a simple wrapper script with fzf that:

- lets me interactively choose an AWS profile from `.aws/config`
- lets me select a CloudWatch log group name

```bash
# bash function in my .rc file
awslogs() {
  set -e
  export AWS_PROFILE=$(cat ~/.aws/config | awk '/^\[profile /{print $2}' | tr -d ']' | fzf)
  local log_group=$(aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | fzf)
  aws logs tail "$log_group" --since 3h --follow --format=short
}
```


Reading AWS CloudWatch logs with CLI and fzf


If you are ensuring that a backend system runs smoothly, here are some questions/metrics that might be interesting to look at. This is a first stab, rather than an exhaustive list. 

These metrics can be communicated to C-level management.

- reliability: 
	- how stable is the system? 
	- how probable it is for an incident to happen? 
	- how fast can we recover from an incident?
- usability
	- are users satisfied with the system?
	- is the system responsive?
	- how does the system perform in the weak/problematic areas?
- cost
	- how much are the infrastructure/maintenance/license cost? 
	- what is the total cost of ownership of the system?
- speed of change
	- how long does it take for a feature to be delivered?

*Thanks to Peter for the discussion.*


Business metrics for system maintenance


Here's an example that defines input for a scheduled Lambda event based on environment, using [Conditions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html) and [Condition Functions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-conditions.html).

Two important things:
- During a stack update, you can't update conditions by themselves. You can update conditions only when you include changes that add, modify, or delete resources.
- Where to put conditions: see [How to use conditions overview](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html#conditions-section-structure-overview)


```yaml
AWSTemplateFormatVersion: '2010-09-09'  
Transform: AWS::Serverless-2016-10-31

Parameters:  
  Environment:  
    Type: String  
    AllowedValues: 
      - acc  
      - prod

Conditions: # <-- define conditions here 
  IsProd: !Equals  
    - !Ref Environment  
    - prod
	
Resources:  
  Main:  
    Type: AWS::Serverless::Function  
	Events:  
	  PeriodicTrigger:  
		Type: Schedule  
		Properties:  
		  Schedule: rate(5 minutes)  
		  Input: !If [ IsProd, '"foo"', '"bar"' ]  #  🎉
	# ... CodeUri, etc...	  
```


Defining resources conditionally based on environment


It turns out there's a linter for CloudFormation templates: [cfn-lint](https://pypi.org/project/cfn-lint/). There are IDE plugins as well. cfn-lint works well, speeding up the feedback loop. It could be setup as a git precommit hook.


Linter for CloudFormation templates


Here's an example on how to  add a Slack channel and a webhook url from SSM Parameter Store into AWS Lambda environment variable: 

```yaml
# cloudformation template
AWSTemplateFormatVersion: '2010-09-09'  
Transform: AWS::Serverless-2016-10-31

Parameters:
	SlackWebhookUrl:  
	  Type: AWS::SSM::Parameter::Value<String>  
	  Default: '/services/Slack/incomingWebhookUrl'  

	SlackChannel:  
	  Type: AWS::SSM::Parameter::Value<String>  
	  Default: '/services/Slack/channels/exampleChannel'
	  
Resources:  
  ExampleLambda:  
    Type: AWS::Serverless::Function  
    Properties:  
      Environment:  
        Variables:  
          SLACK_WEBHOOK_URL: !Ref SlackWebhookUrl  
          SLACK_CHANNEL: !Ref SlackChannel	  
	# ... CodeUri, etc...
```

in Lambda, read from env vars:
```typescript
// in the Lambda handler (Typescript example)
const slack = Slack({  
	url: process.env.SLACK_WEBHOOK_URL,  
	channel: process.env.SLACK_CHANNEL,  
})
```

For local develoment and `sam local invoke`, use [env-vars option](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-invoke.html#serverless-sam-cli-using-invoke-environment-file) or supply env vars to Lambda via bash:
```bash
# content of dev-env.sh
# better not in version control, because it contains secrets
export SLACK_CHANNEL=slack-integration-test  
export SLACK_WEBHOOK_URL=<your-slack-url>
```
followed by:
```bash 
$ source dev-env.sh && sam local invoke Main #...remaining sam args
```


Parameterize AWS Lambda with SSM Param Store


[Ditto](https://www.dittowords.com/) provides a single source of truth for product copy. Features I find interesting:

- two-way integration with Figma
- CLI to fetch copy as a json file that can be versioned controlled (similar to a headless CMS)

[HN discussion](https://news.ycombinator.com/item?id=27142930),
[demo](https://www.dittowords.com/designers).


Ditto, a single source of truth for product copy


Useful for deduplicating filtering logic: 

```typescript  
import { partition } from 'fp-ts/Array'  
  
const isEven = (n: number) => n % 2 === 0  
const separated = partition(isEven)([1,2,3])  
// even on the right, others on the left  
console.log(separated.right) // prints [2]  
console.log(separated.left) // prints [1,3]  
```



Partitioning an array




Useful to make code shorter and more expressive.  
  
```typescript  
import * as O from "fp-ts/Option"  
import { pipe } from "fp-ts/function"  
import DateTime from "luxon"  
  
const getFoo = async (): Promise<string | null> => Promise.resolve(null)  
  
const lastSyncTime = pipe(  
 O.fromNullable(await getFoo()), O.map(DateTime.fromISO), O.getOrElse(() => DateTime.local()))  
```

Basic Option creation

  
Useful to preserve types when removing nullables from a list.  
  
```typescript  
import { compact } from "fp-ts/lib/Array"  
import { some, none } from "fp-ts/lib/Option"  
const numsOptions = [none, some(1), none, some(2)]  
const nums = A.compact(numsOptions) // => [1, 2] 🎉  
```  
  
More info: https://github.com/gcanti/fp-ts/issues/34.

Remove none options from an array


Useful to make reusable sorting components.  
  
```typescript  
import { DateTime } from "luxon"  
  
// https://gcanti.github.io/fp-ts/modules/Ord.ts.html  
export const DateTimeOrd: Ord.Ord<DateTime> = {  
 equals: (x: DateTime, y: DateTime) => x.equals(y), compare: (first: DateTime, second: DateTime) => first < second ? 1 : first > second ? -1 : 0,}  
  
interface Foo {  
 bar: number modifiedAt: DateTime}  
  
const sortByModifiedAt = sort(  
 pipe( DateTimeOrd, Ord.contramap((foo: Foo) => foo.modifiedAt) // 🎉choose a field here ))  
  
const now = DateTime.local()  
  
const sorted = sortByModifiedAt([  
 { bar: 1, modifiedAt: now }, { bar: 2, modifiedAt: now.minus({ days: 1 }) },])  
```  
  
More info:  
  
- https://gcanti.github.io/fp-ts/modules/Array.ts.html#sort  
- https://gcanti.github.io/fp-ts/modules/Ord.ts.html

Sort by a specific field with custom ordering  


It turns out that in Javascript empty arrays are not falsy:

```typescript
const xs = [] as number[]
if (!xs) {
  console.log("I'm empty")
}
// doesn't print 'I'm empty'
```

To test that an array is empty in Typescript, you need to check the length:

```typescript
const xs = [] as number[]
if (!xs.length) {
  console.log("I'm empty")
}
// does print 'I'm empty'
```


empty arrays are not falsy


With [async-pool](https://github.com/rxaviers/async-pool), it's possible to run async functions in parallel, but with capped concurrency. 

For example, let's say we have 10,000 users and we need to run an API request for each user. We can run API requests sequentially, but that is too slow for our needs. We can use `Promise.all`, but the avalanche of API requests is more than the server can handle. 

What we can do is to cap the concurrency, that is, to limit the number of concurrent requests. Let's say we set the cap to 25: 
```typescript
import asyncPool from "tiny-async-pool"

const users = //... array of 10,000 user objects  
await asyncPool(25, users, async (user) => callApi(user)) // the cap is 25
```


Running async functions in parallel with capped concurrency


It turns out that `jest` CLI has [a watch mode](https://jestjs.io/docs/cli#--watch): tests are rerun automatically on code change. That's handy, especially in combination with test selection.  
  
```bash  
npx jest --watch src/foo/__tests__/bar.test.ts  
```

jest watch mode


There is a handy library for mocking function return values, called [jest-when](https://www.npmjs.com/package/jest-when):

```typescript
when(fn)
  .calledWith(1)
  .mockReturnValue("yay!")
  .calledWith(2)
  .mockReturnValue("nay!")
```

Sidenote on mocks: After seeing [[simple-components|Simple Made Easy]] talk by my engineering guru Rich Hickey, I realized that mocks most probably signal complecting of two unrelated components and thus point to accidental complexity. To remove the complexity, one can pass mocked data [[honest-function-inputs|via function arguments]] instead.


mocking with jest-when


  
- [Joi](https://github.com/sideway/joi) for input validation. For Lambda, there's [Middy Validator](https://middy.js.org/packages/validator/) with [JSON Schema](http://json-schema.org/).  
- [Luxon](https://moment.github.io/luxon/) for date/time; an immutable version of MomentJS, removing hiccups of Moment.  
- [got](https://github.com/sindresorhus/got) for HTTP requests. I like this [comparison chart with other libs](https://github.com/sindresorhus/got#comparison).  
- [big](https://github.com/MikeMcl/big.js/) for fixed-precision calculations.  
- [jest](https://jestjs.io/) for testing.  
- [middy](https://middy.js.org/) for various AWS Lambda middlewares.

some handy libraries


  
There's a [difference](https://stackoverflow.com/questions/5076944/what-is-the-difference-between-null-and-undefined-in-javascript) between `null` and `undefined`. `undefined` means a variable has been declared but has not yet been assigned a value. `null` is an assignment value; it can be assigned to a variable as **a representation of no value**. So I should use `null` to represent _no value_, such as  
  
```typescript  
interface ApiReponse {  
 createdAt: string | null}  
```  
  
[This picture](https://stackoverflow.com/a/57249968) explains the concept visually.

null vs undefined


NodeJS comes with different versions. Each version bundles the `npm` as well. For example, NodeJS v15 comes with NPM v7, while NodeJS v14 comes with NPM v6. [Here's an overview.](https://nodejs.org/en/download/releases/)  
  
When working on multiple projects, I've found it handy to switch between multiple NodeJS versions. That's when [`nvm`](https://github.com/nvm-sh/nvm#nvmrc) comes into play. `nvm` also supports `.nvmrc` to further automate the switching.

controlling NodeJS versions with nvm


Since I discovered [prettier](https://prettier.io/), I cannot imagine how I could have worked without it. Prettier automatically formats code, such as Javascript, Typescript, JSON, YAML and Markdown. It works across IDEs and very well within a team.  
  
<div class="blockquote">  
  
> By far the biggest reason for adopting Prettier is to stop all the on-going debates over styles. [It is generally accepted that having a common style guide is valuable for a project and team](https://www.smashingmagazine.com/2012/10/why-coding-style-matters/) but getting there is a very painful and unrewarding process. People get very emotional around particular ways of writing code and nobody likes spending time writing and receiving nits.  
> – Prettier docs (section [Why Prettier](https://prettier.io/docs/en/why-prettier.html))  
  
</div>

prettier: automatic code formatter  


It turns out that arrays support
`some` and `every`. Both functions take a predicate and they test if the predicate is true for some or every element of the array, respectively.
So there's no need to combine `filter` with `length` to achieve the same.
More on arrays functions [in the Javascript docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array).

```typescript
const isEven = (f: number) => f % 2 === 0

// test if an array contains an even number
console.log([1, 2, 3].some(isEven)) // prints true
console.log([1, 3, 5].some(isEven)) // prints false

// test if an array consists only of even numbers
console.log([1, 2, 3].every(isEven)) // prints false
console.log([2, 4, 6].every(isEven)) // prints true
```


arrays support some and every


In Node, [strict assertion mode](https://nodejs.org/api/assert.html#assert_strict_assertion_mode) is recommended over legacy assertion mode. Strict mode has two advantages:
- non-strict methods behave like their corresponding strict methods. For example, [`assert.deepEqual()`](https://nodejs.org/api/assert.html#assert_assert_deepequal_actual_expected_message) will behave like [`assert.deepStrictEqual()`](https://nodejs.org/api/assert.html#assert_assert_deepstrictequal_actual_expected_message).
- error messages for objects display a diff

Example usage in Typescript:
```typescript
import { strict as assert } from 'assert'

assert.deepEqual([[[1, 2, 3]], 4, 5], [[[1, 2, '3']], 4, 5])
```


Strict assertion mode is recommended


It turns out that [const assertions](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-4.html#const-assertions) are super handy to simplify typing. Code below uses const assertion on an object literal and, in fact, it compiles:

```typescript
const s = {
  1: "foo",
  2: "bar",
} as const

function fn(s: "foo" | "bar") {}

fn(s[1])
```

Without the assertion, the type of `s[1]` is `string`, and `string` is not assignable to `'foo' | 'bar'`. Deeper dive [here](https://mariusschulz.com/blog/const-assertions-in-literal-expressions-in-typescript).



const assertions are super handy


I needed to [curry](https://mostly-adequate.gitbook.io/mostly-adequate-guide/ch04) a function in Typescript to simplify code, but with the static type checks in place. I explored currying from [Rambda](https://ramdajs.com/), [Lodash](https://lodash.com/) and also the vanilla [fat-arrow syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions). Both the Lodash and the fat-arrow approach ensured proper types, Rambda didn't not:

```typescript
interface Foo {}
interface Bar {}
type Trigger = "created" | "updated"
const foos = [] as Foo[]

// function we'd like to curry...
const fooToBar = (trigger: Trigger, foo: Foo): Bar => ({})
// ...to simplify this map
const bars = foos.map((foo) => fooToBar("created", foo))

// Rambda
import R from "rambda"
const fooToBarCurriedRambda = R.curry(fooToBar)
// 😥 this compiles, but should not: number is not Foo
foos.map(fooToBarCurriedRambda(1))

// Lodash
import { curry } from "lodash"
const fooToBarCurriedLodash = curry(fooToBar)
// 🎉 compilation error: number is not assignable to Foo
foos.map(fooToBarCurriedLodash(1))
// 🎉 compilation error: Bar[] not assignable to number[]
const foosLodash: number[] = foos.map(fooToBarCurriedLodash("created"))

// vanilla Typescript with fat arrows
const fooToBarCurriedFatArrow = (trigger: Trigger) => (foo: Foo): Bar => ({})
// 🎉 compilation error: number is not assignable to Foo
foos.map(fooToBarCurriedFatArrow(1))
// 🎉 compilation error: Bar[] not assignable to number[]
const foosFatArrow: number[] = foos.map(fooToBarCurriedFatArrow("created"))
```

For now, I went with the fat arrow approach. I've also stumbled upon the [`bind` approach](https://dev.to/theagilemonkeys/currying-in-typescript-just-for-fun-5h9) that I haven't explored yet.



Function currying with static type checks


It turns out that with `typeof` it is possible to keep types dry, for example:  
  
```typescript  
const fn = async (apples: Apples[]) => {  
 const shinyApples: typeof apples = [] //...}  
```  
  
Sidenote on array mutability: Mutability complects state and time and is thus a form of complexity. Immutable values are simpler and lead to lower accidental complexity.

keeping types dry


In [indexable types](https://www.typescriptlang.org/docs/handbook/interfaces.html#indexable-types), we can reuse the index for value types as well. This is very handy. For example, below we have interfaces `Foo` and `Bar`. Type `BarToFooMap` defines mapping from `Bar` to `Foo`. With indexable types and `keyof` we can ensure the mapped values are of correct type:

```typescript
interface Foo {
  a: string
  b: number
}

interface Bar {}

type BarToFooMap = {
  [k in keyof Foo]: (bar: Bar) => Foo[k] // <-- 🎉 using index type k in value type
}

const barToFooBroken: BarToFooMap = {
  a: () => "hello",
  b: () => "world", // 🎉 compilation error: improper return type for b
}

const barToFooFixed: BarToFooMap = {
  a: () => "hello",
  b: () => 1, // this compiles 🎉
}
```


reusing index in indexable types


  
There's a handy syntactic sugar [introduced in Typescript 3.7](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html) called [optional chaining](https://en.wikipedia.org/wiki/Safe_navigation_operator#cite_note-23).  
  
```typescript  
# without optional chaining  
const baz = foo && foo.bar  && foo.bar.baz  
  
# with optional chaining  
const baz = for?.bar?.baz  
```  
  
<div class="blockquote">  
  
> The `?.` operator functions similarly to the `.` chaining operator, except that instead of causing an error if a reference is nullish (null or undefined), the expression short-circuits with a return value of undefined.  
>  
> [MDN Web docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining)  
  
</div>  
  
It works also for arrays.

optional chaining


[String literal types](https://www.typescriptlang.org/docs/handbook/2/everyday-types.html#literal-types) are not available at runtime, so how if one needs to check against them at runtime? It turns out this is possible with [const assertions](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-4.html#const-assertions):

```typescript
const Fruits = ["Apple", "Banana"] as const
type Fruit = typeof Fruits[number]
const isFruit = (s: String) => Fruits.includes(s as Fruit)
```

Deeper dive [here](https://www.damirscorner.com/blog/posts/20200619-StringLiteralTypeGuardInTypescript.html).

checking against a string literal type at runtime


  
Typescript supports [utility types ](https://www.typescriptlang.org/docs/handbook/utility-types.html) that allow to transform a type into another type. For example:  
  
- Set all properties of `Todo` to optional with `Partial<Todo>`  
- Omit a property `id` from `Todo` with `Omit<Todo, 'id'>`  
- Pick a set of properties from `Todo` with `Pick<Todo, 'id' | 'name'>`

utility types


WebStorm can be configured to import modules using absolute paths instead of relative paths:

![webstorm-import-absolute](/img/webstorm-import-absolute.png)

It works as follows:

```typescript
// if the field unchecked, WebStorm imports relative
import { queries } from "../crm/queries"

// if the field checked, WebStorm imports absolute
import { queries } from "crm/queries"
```


Marcel Krčah

Most recent articles

Most recent TILs (Today-I-Learned)

Jul 28, 2021		Lessons-learned from a 6-month Typescript/AWS Lambda project
Apr 15, 2021		Getting started with functional programming in Typescript
Apr 08, 2021		Building cathedrals
Apr 02, 2021		Adding changelogs to blog posts
Mar 29, 2021		Changelog of this blog (meta)

Aug 03, 2021		AWS Athena: Adding yyyy/mm/dd partitions to an S3 table
Aug 02, 2021		PostgreSQL vs Aurora PostgreSQL vs Aurora PostgreSQL Serverless
Jul 31, 2021		Seting up AWS for test/acc/prod environments
Jul 30, 2021		AWS QuickSight: Adding an IAM user
Jul 29, 2021		Business intelligence for AWS S3 data