Avoiding the flight forward

Replacing MySQL with ElasticSearch created unexpected new problems.

Objective: faster database queries

A start-up company gathered and analyzed social media data, which they stored in a MySql database.

The database grew steadily. It turned out that reports were kind of sluggish because the queries (the SELECT statements) executed relatively slow.

Chosen solution: replace MySql with ElasticSearch

The technical team decided to replace MySql with ElasticSearch, which is known for its fast queries. This shift must have taken quite a few man-months to implement.

Result: faster SELECTS, but slower INSERTS

This is a typical example of the flight forward (if in trouble, move towards more trouble).

By replacing the well-known MySql database by the less known and very different ElasticSearch, it opened a whole new can of worms.

The ElasticSearch interface is completely different from MySql, so it requires new tools, new functionality and new learning. More importantly, it turned out that ElasticSearch had slower(!) inserts than MySql, which posed a new problem! Goodbye problem A, Welcome problems B, C and D.

Regarding continuity, ElasticSearch poses another problem too, namely that it’s harder to find technical staff with knowledge of ElasticSearch than of a regular RDBMS like MySql.

Alternative solution: tweak and tune the MySql database

The company probably could have solved the 'slow query problem' via other means too. One obvious solution would be to use faster hardware and more memory. This is generally very inexpensive compared to man hours.

Additionally they could fine-tune the indexes and tables. If this still would not have worked they could have aggregated the data for some slow queries. By choosing these solutions, they probably would have easily saved a few man-months.