ByteSizeWeb

Using Data for Better Customer Engagement

2018-10-08T00:00:00+00:00

Introduction

For a company that wants to increase sales, it is far more critical to keep existing customers than to find new ones. There are two reasons for it: 1) it is far cheaper to keep customers than to look for new ones, 2) in most cases, customers with a prior order history have a lower barrier to ordering again than those who have never ordered on the platform before.

Add to that the additional metrics the company prioritizes, like the sustainability of the products being purchased by the consumers. The problem now gets even more convoluted. So Imagine you are the head of a company that wants to increase sales and wants its customers to order more sustainable products while doing that. For my Insight project, I consulted for a company that faced this very problem. The company cares deeply about the sustainability of the products they sell. Without readjusting their focus on it, they also want, like any other company, to increase sales and grow their profits.

Exploring the data

The company had accumulated a lot of data over time. The data contained everything from the order history of consumers to metadata for those orders, the products themselves, and their manufacturers. This amounted to a lot of data distributed over 50 different tables interconnect with foreign keys and their respective primary key. The company’s vision for achieving their goals was to build out a ‘sustainability dashboard’ to provide company-wide, and customer specific, metrics about the sales/expenditure and the associated environmental goals. This would then be used to send out weekly/monthly updates to the consumers to encourage spending on more environmentally friendly products.

Building a histogram of the company-wide order histories made one thing clear. The company faced a challenge that most other companies face today in our modern online-ordering market that is inundated with options. It can be visualized from this figure below. It is quite evident that some customers order a lot of products, which drives up sales figures. However, the company is missing out on a lot of potential orders from customers who order only a few times but do not return to order again.

While brainstorming with the company to flesh-out the dashboard, I realized that a great value-addition to it would be a recommendation engine. All the historical data of past purchases by a consumer could be used to make customer-specific recommendations of products that they may like, but did not know existed in the catalog. This recommendation engine could serve two purposes at the same time; introduce consumers to new products they might need, and nudge them towards greener products.

Implementing the dashboard

Data from all the different tables was wrangled in python. Specifically, pandas, the python data analysis library, and numpy were extensively used to clean and pre-process the data. Sustainability estimates were generated by feature engineering to combine recycled content and the carbon footprint of each item. Pandas functions such as join, merge and other aggregating functions like groupby had to be abundantly deployed to get the required metrics.

I chose to use Bokeh to visualize these metrics. Bokeh appeared very promising for this task since it is open-source. It offers an ‘interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of basic exploratory and advanced custom graphics in the style of D3.js’. To serve these visualizations in the form of an interactive, customizable, and robust web app I decided to integrate it with Flask. Flask is a ‘micro web framework written in Python and is based on Werkzeug, Jinja 2 and good intentions’.

Implementing the recommender system

There are several possible methods to implement a recommender system. Every method offers certain advantages and disadvantages. I wanted a method that can be pushed into production rapidly in a scalable manner. I opted for the k-nearest-neighbor (KNN) method. KNN belongs to a sub-category of recommender systems called as ‘collaborative filtering’. Collaborative filtering matches users that like the same items. ‘Liking’ a product can be inferred implicitly by looking at a consumer’s purchase history. This then gives a large user-item matrix, where each row is a customer and each column is all the products that a user can purchase. This user-item matrix can then be filled with 1 if the corresponding item was purchased by its corresponding user, or 0 if it was not. Since a customer can potentially order a lot of products, this user-item matrix was very wide (many more columns than rows). Also, since a consumer only orders a small subset of these thousands of products on offer, this matrix is pretty sparse (has lots of zeros in it). Therefore, reducing the dimensions of this matrix was essential to help KNN find the optimal solution. I used scikit learn to perform principal component analysis (PCA) which reduced the dimensions of the data without much loss of its variation. I then used KNN to find the nearest-neighbors of each consumer in this reduced two-dimensional (first two principal components) space. To make a recommendation, I simply went down the list of most-ordered items by the closest neighbor until I found one that had not been ordered by the original consumer before.

This gives a good recommendation that is personalized to the customer. But, how do we account for other metrics like sustainability, which are highly prioritized by the company? To factor it in, I first calculated the weighted overall sustainability of all products of the customer. I then made only those recommendations which are at least a little greener than this previous average. This serves our two purposes. First, it introduces a customer to a personalized product they might like but were probably not aware it existed in the catalog. Additionally, it nudges them towards a greener product.

In the end, the company received a dashboard with customer-specific metrics. Additionally, the dashboard contained a personalized and environmentally conscious recommendation for each customer that would help improve sales while helping our planet.

Overall, it was a very valuable experience for me. It exposed me to real-world data, working with a real company to implement a practical and environmentally friendly solution.

US Migrations in Recent Years

2018-02-27T00:00:00+00:00

Where people are migrating says a lot about the underlying socio-economic factors that dominate a country. And for a country like USA, with more than 323 million people and an advanced economy that has a sweeping clout, these migrations can hold the key to understanding where the global economy is headed. Moving across the country is a tough decision for most people. It requires a compelling reason. For most people it is a job, or the lack thereof, that provides a significant push to make a cross-country move. The ever-changing job market, and how quickly states adapt to the turning tides, are some of the main things one can interpret from these migrations.

The interactive plot below shows how the population of individual US states has changed, relative to the county’s total population as a whole.

Interactive plot to explore these migrations can also be accessed at this link.

Surprising revelation from this interactive plot is the quite drastic progression from low migrations in the early part of the decade to the high migrations of 2016-17. Additionally, overall people have been consistently migrating to the west and the south-east. East and north-east have seen considerable decline for a large part of the last decade.

The plot also shows vividly the boom and bust of the oil rush in North Dakota. The rapidly reversing fortunes of Wyoming and Idaho in recent years stand out. More context can be gathered from this article in Washington post.

A lot of these highly contrasting migrations in the recent years also coincided with politically important times in the US. Players who better positioned themselves to ride the wake were benefitted, while others got left behind.

Countries with the best Science and Technology sector

2017-10-02T00:00:00+00:00

Scientific and technological development is everywhere around us. To say that it has completely transformed our way of life would be an understatement. Majority of people are eager to adopt any new technology if it is useful in one way or another. Others who resist in the beginning, eventually give in to the slowly rising tide and go with the flow. Technological advances have a snowball effect as well. More advanced a country becomes the faster it can advance itself.

That said, there is a lot of disparity between different countries and technologies they are good at. If we use certain indicators as measures of scientific advancement, then it is possible to divide these countries into separate groups. These groups accurately reflect the perceived scientific and technological development of these countries. The indicators used for these analysis are: 1) Charges for the use of intellectual property, 2) Research and development expenditure, 3) Patent applications (non-residents), 4) Patent applications (residents), 5) Trademark applications (direct residents), 6) Researchers in R&D, 7) Technicians in R&D, and 8) High-technology exports.

Explore the interactive ‘Principal Component Analysis’ plots below to visualize these groups by yourself.

You can also use this link to connect to the interactive plot.

Malaria – Number of Confirmed Cases by Country

2017-09-27T00:00:00+00:00

Malaria is a fatal disease that is transmitted by certain types of infected mosquitoes. The parasite that causes it has been around for several thousands of years. In fact, the very first Humans that lived in Africa were already dying from the disease. As the early Humans migrated to new land, they carried Malaria with them. Today, Malaria causes nearly half a million deaths worldwide. We have come a long way since the time of our early ancestors, but have we made any progress in eradicating this fatal disease?

Use the interactive app below to track Malaria cases by country over a 12 year period. This data was obtained from UN.

You can also use this link to connect to the interactive plot.