Cutting out the nonsense

On how to save time by building a Proof of concept.

The Magento Hell

Once I worked for a company that replaced its existing webshop with Magento 2.0.

Magento is huge in size (500 Mb!) and can be very slow.

This particular Magento webshop contained only about 3000 product variations. On a good day, it took at least 5 seconds for the search results to appear, even with one visitor on the site. This was acknowledged as a serious problem, since slow page speeds can easily cause customers to leave the webshop out of frustration.

Out of curiosity, I just wanted to know how fast this search process theoretically could be, once you simply aggregate the product data. So I build a Proof of Concept, during which I noticed I was overcomplicating.

1. Skip the database

Initially I thought (stupid me) I first needed to build a database to store the info, just like with Magento. I would then create a process to aggregate the data. But this was silly for I needed no database at all, I could just use a multi-dimensional, hard-coded product array to simulate it.

2. Forget data aggregation

Since I did not use a database with various tables, there was no need to aggregate the data either, so I could start with aggregated product data to begin with.

3. No real data needed

I also felt I needed real data to simulate reality. Stupid me again. Coming to my senses, I decided to mainly use gibberish data. I did however use real product names and real categories, for it feels better and wasn't a lot of work.

4. Forget images

Initially I thought I needed to show real product images, just like with a webshop. But I quickly realized that was not necessary at all. I could just show the coded data on the search results page.

A Proof of Concept generally needs no real data, no database, no fancy user-interface and no internet connection.

On the basis of the product array, I wrote two programs:

1) A small program to generate 22000 random product variations. Each product variation had ten semi-random characteristics like brand, price, size, category and weight. These random products were stored in a simple CSV file.

2) Then I wrote a program to create a user interface, scan the CSV file and select and show the qualifying product variations.

The result

It all took about two days of work to write and fine-tune the small programs. The result can be tested here. It turned out that even on shared hosting, searching through 22000 product variations took less than 0.1 seconds!!! What an impressive test result indeed.

However, this PoC was never intended to be integrated into Magento. The company had to rent many web servers (application/DB) at AWS to speed up the site and to accommodate for traffic peaks during special sales and right after sending newsletters. Total hosting costs were probably in the range of $5000 to $10000 per month! However, with proper software, even a shared(!) hosting account (at $10 a month), could have done the job much quicker!