Recommendation engine – how to make them worth the development time and cost

Recommendation engine – how to make them worth the development time and cost

What is a Recommendation engine

A recommendation engine (also recommender system) is a system that strives to rate or recommendations based on available information. It’s easy to find an example of a successful recommendation system. Youtube, for instance, has the “watch next” and Spotify has a quite well-researched music recommender system. 

Strictly speaking, the recommendation engine is a subclass of an information filtering system. It filters the data and provides bits of information relevant to the problem or task at hand. So a crucial part of a successful recommender system is precisely defining the problem you want to solve.

Let’s take Amazon’s “Frequently bought together” recommendation that commonly appears just below the main description of a product. It has a precise and targeted problem definition: “What products are in the shopping bag when these or similar items are present”. It may sound simple, but this and other recommendation engines provide up to 35% of all sales on Amazon. And this is mostly achieved by heavily testing which recommendation engines work and then perfecting things that end up working. 

With recommendation engines, it’s essential to not attempt to just use them as a band-aid for all company’s problems. Their main value is in providing relevant recommendations that usually save 10-15 seconds of thinking, typing, and clicking. Often they tend to be an elegant way to add shortcuts in UX that might be hard to unclutter otherwise.

Since first mentioned in the 1990s recommender systems have developed into flexible solutions that can solve many problems. As such we can categorize them into a large number of non-exclusive types.

A non-complete categorization of recommender systems:

  • content-based – good if you have large amount of data about items you want to recommend but not the user. Then the model learns how to improve recommendations based on comparison on how well it predicted what user would like and what the model predicted (it solves a user-specific classification problem that classifies users attitude towards an item based on the features)
  • collaborative filtering – Asking users to help you rate recommendations. Or uses user behavior to infer what they like and dislike about your results.
  • multi-criteria – you start with the problem, the definition of success, and a lot of data. The model works on finding relevant data and learning to solve the problem form it (statisticians among you might recognize this as an “automagical” Principal Component Analysis – PCR which model does on its own and then trains itself using the relevant data)
  • risk-aware – involves adding to the model contextual information eg: inappropriate content, wrong time for notifications, not recommending a product that the user just purchased …
  • mobile – ether delivered on the smartphone or using the smartphone to actually do the data processing. An example would be Google Maps recommending you ATMs near you when you ask for it on the mobile app. 
  • hybrid – Mix and match
  • session-based – creating recommendations not based on past data but based on the current user session. An example would be Amazon offering you a comparison of similar items based on characteristics when you view an item on their store. 
  • reinforcement learning – unlike traditional supervised learning approaches, reinforcement learning can directly optimize based on metrics defined as “successful recommendation” 

A major pitfall in recommendation engines is that they can perform worse than traditional “hardcoded” or algorithmic recommendation/data retrieval systems. For example: recommend the last 10 products that a user inspected but did not buy). This is especially common with Neural Network and Machine learning-based models where a bad initial learning setup can degrade accuracy. Also, it’s quite common for companies to treat such systems as “set and forget”  instead of providing continuous tweaking, revalidating, and monitoring in order to ensure proper operation. This particular pitfall of recommender systems has been well researched, here’s an example study from 2020

How to implement a recommendation engine?

Recommendation engines are very versatile and in general. They can be used to improve the user experience or provide meaningful information in many different areas. But you need to have a problem specifically defined for it to become solvable with a recommender engine. As Hitchikers guide to the Galaxy would say: “the answer is 42, but the problem is that we don’t know what’s the question “.

High-level business goals such as more users, higher satisfaction, % increase in sales won’t do here. You need to define more specific goals for your recommender engines. Lets look at few examples.

we want our per user-generated emails to generate 20% more sales than our previous generic email to all campaign

First, create recommender models, send model generated user-specific emails to 50% of people. Also, send a baseline generic email to the other 50% of users and compare the model against this baseline. Then work on improving the model thill it significantly outperforms your baseline.

we want to increase user retention by 10% by recommending relevant new articles to logged-in users

Again create a recommendation model. Compare the recommendation against baseline algorithmic recommendations. For example the latest 5 articles or 5 top articles of the week. Only by setting such specific goals and baselines will you be able to state with accuracy that your recommender engine is working as intended.

But for the whole process to work, you have to actually know where your pain points are. Also it helps to know where your opportunities for improvement are. Amazon for instance recognized that users would often go back to buy related/compatible products. This gave birth to “users frequently bought together” feature. That feature looks for commonly purchased patterns and offers you bundles of related products. 

The next thing would be to ensure you have proper data. Once you know what your question is, it’s easy to define what data you need to answer the question. Just keep in mind that data quality control is important and you have to ensure your data is correct. As every elementary data science professor would tell you, Garbage in Garbage out. 

Then you’d go about implementing a recommender system that can solve your problem at hand. Tutorials and books on that topic are very easy to find once you master the art of the Google search. And then for good measure, you set up a process to initially validate your final solution against a well-defined baseline. And you’re done! In case you’re thinking of implementing one using neural networks, check out this article!

In short, recommender systems can make a difference in many different services or. But they only make sense if you have a problem properly defined to be solvable with such an engine. Even then you have to put work hard and provide a meaningful baseline against which to improve your engine. And that as circumstances change, so the model should adapt to the new reality. Or the very least the model should be validated to ensure it actually works in the new reality. For example, guess what happened to online shopping recommendations when the pandemic of 2020 started). 

Data center blade servers via Bob Mical/Flickr

The major issue with projects developing recommender systems

It’s perfectly fine to start developing a recommendation system without precisely defining the problem. You don’t know where the project will end when it begins. Deadlines often make projects go into refining a recommender model without ever revisiting or redefining the problem. This way projects end up perfectly solving a completely wrong problem. Or giving the best possible answer to the wrong question. This tends to be the hard part, even large IT companies fail at this. IBM famously lost $62 million developing a recommendation system for a medical expert advisor,. 

The problem is that people think they are silver bullets that you can just put into any gun and shoot at any target. Expecting recommendation engines to be easy “set and forget” solutions that just duct tapes into existing systems. In reality, they require large amounts of data, experts who take time to understand the problem, and who develop models that provide meaningful recommendations. Followed by rigorous real-world testing against a known working baseline solution. Even after validating the recommender system, you still have to periodically compare it to the baseline and tweak it to keep it relevant. On large scale, it takes a team to develop such a solution. And another team to maintain all data gathering and processing systems. 

Also, another often overlooked pitfall is not starting with a solid well-developed service. While it can be the main feature, often it’s just that bit of special sauce improving an already good product. Amazon was already pretty good at selling all sorts of products.  Spotify already had a vast library of artists that you can stream. Youtube already was the place where to upload videos. So mostly you should think about recommender systems as something that adds value. But to add value, you first have to have some base value to which you add the added value.

Further reading:

Back to top