Retailers are becoming savvier about how they use data to deliver targeted offerings to consumers. They understand, based on past interactions or demographic information, what marketing mix to employ to drive the greatest returns, engage the widest audience, or increase customer loyalty. But pricing, specifically promotion pricing, is still very much a guessing game.
Retailers set discounts based on the need to push inventory, increase shopping cart size, or ramp up foot traffic in physical stores, but the actual decision of how much to discount is not always, and for every retailer, very scientific, says Georgia Perakis, a professor at MIT Sloan School of Management. Setting retail promotions based on what has worked in the past can drive some success, but that strategy’s revenue potential is limited.
That is why Georgia began thinking about ways to apply data science to the promotion process and understand consumer demand given different products and pricing tiers. Georgia told us, “I wanted to apply the math in a useful way, and understand if we could actually drive more profits for retailers using data models.”
Georgia began researching these topics over 10 years ago and has since created data models for a variety of retailers, including department stores and grocery stores, in order to drive greater profits from promotions. She has further refined her work to target retail promotions to highly valuable “influencer shoppers” and personalize promotions to individuals.
In our conversation, Georgia describes how she developed these models in partnership with different retailers and/or data science companies and saw store profits over a quarter increase as much as 9%. And she provides valuable advice for retailers looking to implement a similar approach for their business.
Ellen Harvey: What drew you to study retail promotions and data modelling?
Georgia Perakis: I was always attracted, starting at a young age, to math but also to data science. At the same time, I was trying to do something that would be useful. I didn’t just want to do math for the sake of math, so I was drawn to different applications. And, retail fascinated me a lot. So intellectual curiosity drew me to this area, along with the fact that this is an area that definitely I have an interest in at a personal level.
EH: In your research papers, you define promotions as selling the right product at the right time to the right customer and at the right price. Can you explain how data models can address each of these issues?
GP: Yes. We always let the data drive the promotion recommendations. Based on the data we build models. First, we build data models to understand demand at different times, for different customers, for different products, and for different prices. Apart from price, we also build other features in the demand models, such as for example, the season, characteristics of the store, demographics. We do our best to build models that would predict demand at a more granular level than just a general demand for the product.
Based on those demand models, we then build models which basically decide what the promotion should be for which customers, for what products and at what time. That is, we let the data drive the demand estimation at the granular level and then we build mathematical models that say, “Given this data and this demand estimation, this is what’s optimal to do.”
EH: What is some of the important data that retailers need to collect in order to build these models?
GP: Transactional data is very important. Information on inventories and potentially more personalized information about customers are also very valuable. For example, what did they buy when and what are their characteristics? It could be demographics, it could be level of spending for what products and at what time, among others. The retailer may have more information on loyalty card holders. That is, retailers will know, for example, how much loyalty card members tend to spend on what more easily. This is some of the data that can help a lot in general.
EH: Why are loyalty card holders particularly valuable in terms of data?
GP: Retailers try to incentivize customers to use those cards to buy. This allows them to use customer purchasing data. It will allow them, on a website or in the store, to track the store location, what types of customers it has, and what type of purchases they make and when. Then the retailer can show the right assortment of products to the right customers. Because they have this information, they can do this better. It’s not just about making profit. It’s also about showing the right products to the right customers and at the right price.
EH: Could you describe some of the different stages that are involved with actually getting a data model like this off of the ground and implementing it for a retailer?
GP: The first stage is obviously understanding the key aspects behind the business model. What comes next, is to get data from the retailer, like the type of data that we talked about. Sometimes it could be an issue that the data doesn’t make sense. Very often there are a lot of missing values, in addition, companies often store data in different databases and they are not consistent. In that case, you have to iterate as well as ask a lot of questions to try to make sure that this data is “accurate,” or you may want to subset the data to the part that actually makes sense and only use this. You may even need to go back to the drawing table and access the data from the retailer in another way. So that’s the first step–make sure you collect data that makes sense.
You may realize that the information is very sparse. Then you probably are going to need to do some sort of an aggregation. Maybe you cannot work at the SKU store data level. Maybe you have to go up a level. Maybe you want to aggregate similar customers. Maybe you want to aggregate stores that are close to each other, for example.
Once you feel that your data is in good shape, then you would want to come up with a demand model that calculates the demand for certain products at different price tiers.
And then you want to build an optimization model on top of that that will say, “For this type of customers at this point of time, give them this price for this type of product.”
This is not just a number that you can plug into a computer, though. The algorithm gives you an “approximate optimal solution.” Furthermore, you want to test the findings, hopefully in some sort of pilot, if possible. You might compare some customers who do business as usual, to the similar customers who are offered the promotion for the same products. You need to make sure you do this in a fair way because you don’t want to create any bias.
EH: What are some of the barriers to data-driven promotion strategies becoming more widespread in retail?
GP: Some of the barriers have to do with the data. Collecting the right data and at the right level as well as the presence of sparsity in the data, as I mentioned before.
My advice for retailers would be to keep track of as much detailed data as possible. Starting there will already help.
EH: Could you share some of the results retailers who have implemented these demand and optimization models have seen?
GP: So typically, we see a profit increase starting at 3% up to 9% in the best situations.
EH: One of the data models you worked on focused on targeting “influencer shoppers.” Could you describe what the goals were and any lessons you took away from that project?
GP: Yeah, so that project started by asking the question, “With all the social network information, can you build this type of model at the more granular level? Then we can give this promotion to this customer, at this point of time, and for this product.” And we said, “Okay, maybe if we have social network information, and we know that this person has a lot of friends, we give them this promotion. Then hopefully they will post about it on Facebook and that will get their friends to buy.”
Soon we realized that actually, we couldn’t get access to data at this level because we would need permission. Privacy issues have become very prevalent. When we realized we didn’t have the data we wanted, we asked, “Can we infer something about trends that occur if say a group of customers buys? Do they have some cascading trend effect?”
We looked at transaction data with information from loyalty cards, including demographic information and levels of spending at which stores. That allowed us to build groups of customers with similar characteristics to see if there was a cascading trend effect. We saw that a certain type of people tended to buy certain products early and influence other groups to purchase those same products later.
So we used this information to amend our demand estimation model and get an improvement that was quite substantial. For example, we reduced the relative error in the demand estimation between 3-15%, in terms of what is called mean absolute percentage error. The mean absolute percentage error looks at what actually happened and what your model predicts relative to what happened. Clearly, you would like that to be zero so that you predict it exactly, but that’s not possible.
In fashion, this error is quite high usually, and we were able to reduce it by 3-15% (depending on the product) and get it around 30% for fashion items which was very impressive.
EH: You also worked on data models that were meant to personalize retail promotions to individuals, correct?
GP: Yes, this is totally built off of some of this work because, if I can understand my sales at the granular level, the customer level, or a small group of customers, then I can build a model off of that that will say, “For this small group of customers, this is the price I should give them. This is the promotion I should give them.”
EH: For many of these models, you optimized the data models for profit, but could you optimize for something like repeat business or customer lifetime value?
GP: Yes. Exactly. You can build data models to try to understand who are the type of customers that will come back. In fact, I actually worked on this. I did some work a few years ago where we said, “Okay, so typically many customers, especially online, see the item and don’t buy it immediately. But they tend to come back and buy it, or they might come back for something else.” So this was something that we incorporated actually in the demand estimation to try to understand this type of population.
EH: Do you think more retailers will adopt this type of data modeling in the next few years?
GP: Yes, I think that most retailers are starting to look into that. They are building their own data science teams, especially the large retailers. Without mentioning names, I know well-known big department stores are trying to build better data science teams. They try to hire people from good universities that are well-trained in data science. That shows that they are very committed.
EH: So do you think soon retail promotions will become fairly automated?
GP: Yes, definitely more retailers will adopt these models, but to be honest with you, I don’t feel that this replaces the humans and become fully automated. These recommendations from the data models need to make sense to the managers who have experience.
For example, if you are trying to space the timing of promotions, you want to try different ways of spacing them and see what the best scenario will be. You want to run different things and ask, “From my experience does this makes sense?” And obviously, you run this model in a few different ways then you make the decision.
So I don’t believe that we plug in the numbers, hit the button in the computer, and set the promotion at what it tells us. That’s a very dangerous message to give. Data models are something to aid people, and the experience and the qualitative aspect are not replaced by this, and they shouldn’t be.
EH: Yeah, the “set it and forget it” mindset is not the way that you grow your business.
GP: By no means. There’s a lot to be said for experience. And actually what’s more powerful is when you see people who have a lot of experience that they will tell you, “This strategy that your model came up with doesn’t make sense and this is why,” and you realize that you have to go back and amend the model that you built because you forgot something. And this comes from the retail experience. It’s very interesting. That experience is critical to the success of these models.