Marketing Data Science: Understanding Markets

This chapter is from the book

Marketing Data Science: Modeling Techniques in Predictive Analytics with R and Python

Learn More Buy

This chapter is from the book

This chapter is from the book 

Marketing Data Science: Modeling Techniques in Predictive Analytics with R and Python

Learn More Buy

Understanding Markets

• “What makes the elephant guard his tusk in the misty mist, or the dusky dusk? What makes a muskrat guard his musk?”
• —BERT LAHR AS COWARDLY LION IN The Wizard of Oz (1939)

While working on the first book in the Modeling Techniques series, I moved from Madison, Wisconsin to Los Angeles. I had a difficult decision to make about mobile communications. I had been a customer of U.S. Cellular for many years. I had one smartphone and two data modems (a 3G and a 4G) and was quite satisfied with U.S. Cellular services. In May of 2013, the company had no retail presence in Los Angeles and no 4G service in California. Being a data scientist in need of an example of preference and choice, I decided to assess my feelings about mobile phone services in the Los Angeles market.

The attributes in my demonstration study were the mobile provider or brand, startup and monthly costs, if the provider offered 4G services in the area, whether the provider had a retail location nearby, and whether the provider supported Apple, Samsung, or Nexus phones in addition to tablet computers. Product profiles, representing combinations of these attributes, were easily generated by computer. My consideration set included AT&T, T-Mobile, U.S. Cellular, and Verizon. I generated sixteen product profiles and presented them to myself in a random order. Product profiles, their attributes, and my ranks, are shown in table 1.1.

A linear model fit to preference rankings is an example of traditional conjoint analysis, a modeling technique designed to show how product attributes affect purchasing decisions. Conjoint analysis is really conjoint measurement. Marketing analysts present product profiles to consumers. Product profiles are defined by their attributes. By ranking, rating, or choosing products, consumers reveal their preferences for products and the corresponding attributes that define products. The computed attribute importance values and part-worths associated with levels of attributes represent measurements that are obtained as a group or jointly—thus the name conjoint analysis. The task—ranking, rating, or choosing—can take many forms.

When doing conjoint analysis, we utilize sum contrasts, so that the sum of the fitted regression coefficients across the levels of each attribute is zero. The fitted regression coefficients represent conjoint measures of utility called part-worths. Part-worths reflect the strength of individual consumer preferences for each level of each attribute in the study. Positive part-worths add to a product's value in the mind of the consumer. Negative part-worths subtract from that value. When we sum across the part-worths of a product, we obtain a measure of the utility or benefit to the consumer.

To display the results of the conjoint analysis, we use a special type of dot plot called the spine chart, shown in figure 1.1. In the spine chart, part-worths can be displayed on a common, standardized scale across attributes. The vertical line in the center, the spine, is anchored at zero.

Figure 1.1 Spine Chart of Preferences for Mobile Communication Services

The part-worth of each level of each attribute is displayed as a dot with a connecting horizontal line, extending from the spine. Preferred product or service characteristics have positive part-worths and fall to the right of the spine. Less preferred product or service characteristics fall to the left of the spine.

The spine chart shows standardized part-worths and attribute importance values. The relative importance of attributes in a conjoint analysis is defined using the ranges of part-worths within attributes. These importance values are scaled so that the sum across all attributes is 100 percent. Conjoint analysis is a measurement technology. Part-worths and attribute importance values are conjoint measures.

What does the spine chart say about this consumer's preferences? It shows that monthly cost is of considerable importance. Next in order of importance is 4G availability. Start-up cost, being a one-time cost, is much less important than monthly cost. This consumer ranks the four service providers about equally. And having a nearby retail store is not an advantage. This consumer is probably an Android user because we see higher importance for service providers that offer Samsung phones and tablets first and Nexus second, while the availability of Apple phones and tablets is of little importance.

This simple study reveals a lot about the consumer—it measures consumer preferences. Furthermore, the linear model fit to conjoint rankings can be used to predict what the consumer is likely to do about mobile communications in the future.

Traditional conjoint analysis represents a modeling technique in predictive analytics. Working with groups of consumers, we fit a linear model to each individual's ratings or rankings, thus measuring the utility or part-worth of each level of each attribute, as well as the relative importance of attributes.

The measures we obtain from conjoint studies may be analyzed to identify consumer segments. Conjoint measures can be used to predict each individual's choices in the marketplace. Furthermore, using conjoint measures, we can perform marketplace simulations, exploring alternative product designs and pricing policies. Consumers reveal their preferences in responses to surveys and ultimately in choices they make in the marketplace.

Marketing data science, a specialization of predictive analytics or data science, involves building models of seller and buyer preferences and using those models to make predictions about future marketplace behavior. Most of the examples in this book concern consumers, but the ways we conduct research—data preparation and organization, measurements, and models—are relevant to all markets, business-to-consumer and business-to-business markets alike.

Managers often ask about what drives buyer choice. They want to know what is important to choice or which factors determine choice. To the extent that buyer behavior is affected by product features, brand, and price, managers are able to influence buyer behavior, increasing demand, revenue, and profitability.

Product features, brands, and prices are part of the mobile phone choice problem in this chapter. But there are many other factors affecting buyer behavior—unmeasured factors and factors outside management control. Figure 1.2 provides a framework for understanding marketplace behavior—the choices of buyers and sellers in a market.

Figure 1.2 The Market: A Meeting Place for Buyers and Sellers

A market, as we know from economics, is the location where or channel through which buyers and sellers get together. Buyers represent the demand side, and sellers the supply side. To predict what will happen in a market—products to be sold and purchased, and the market-clearing prices of those products—we assume that sellers are profit-maximizers, and we study the past behavior and characteristics of buyers and sellers. We build models of market response. This is the job of marketing data science as we present it in this book.

Ask buyers what they want, and they may say, the best of everything. Ask them what they would like to spend, and they may say, as little as possible. There are limitations to assessing buyer willingness to pay and product preferences with direct-response rating scales, or what are sometimes called self-explicative scales. Simple rating scale items arranged as they often are, with separate questions about product attributes, brands, and prices, fail to capture tradeoffs that are fundamental to consumer choice. To learn more from buyer surveys, we provide a context for responding and then gather as much information as we can. This is what conjoint and choice studies do, and many of them do it quite well. In the appendix B (pages 312 to 337) we provide examples of consumer surveys of preference and choice.

Conjoint measurement, a critical tool of marketing data science, focuses on buyers or the demand side of markets. The method was originally developed by Luce and Tukey (1964). A comprehensive review of conjoint methods, including traditional conjoint analysis, choice-based conjoint, best-worst scaling, and menu-based choice, is provided by Bryan Orme (2013). Primary applications of conjoint analysis fall under the headings of new product design and pricing research, which we discuss later in this book.

Exhibits 1.1 and 1.2 show R and Python programs for analyzing ranking or rating data for consumer preferences. The programs perform traditional conjoint analysis. The spine chart is a customized data visualization for conjoint and choice studies. We show the R code for making spine charts in appendix D, exhibit D.1 starting on page 400. Using standard R graphics, we build this chart one point, line, and text string at a time. The precise placement of points, lines, and text is under our control.

Exhibit 1.1 Measuring and Modeling Individual Preferences (R)

# Traditional Conjoint Analysis (R)

# R preliminaries to get the user-defined function for spine chart:
# place the spine chart code file <R_utility_program_1.R>
# in your working directory and execute it by
#     source("R_utility_program_1.R")
# Or if you have the R binary file in your working directory, use
#     load(file="mtpa_spine_chart.Rdata")

# spine chart accommodates up to 45 part-worths on one page
# |part-worth| <= 40 can be plotted directly on the spine chart
# |part-worths| > 40 can be accommodated through standardization

print.digits <- 2  # set number of digits on print and spine chart

library(support.CEs)  # package for survey construction 

# generate a balanced set of product profiles for survey
provider.survey <- Lma.design(attribute.names =
  list(brand = c("AT&T","T-Mobile","US Cellular","Verizon"),
  startup = c("$100","$200","$300","$400"),
  monthly = c("$100","$200","$300","$400"),
  service = c("4G NO","4G YES"),
  retail = c("Retail NO","Retail YES"),
  apple = c("Apple NO","Apple YES"),
  samsung = c("Samsung NO","Samsung YES"),
  google = c("Nexus NO","Nexus YES")), nalternatives = 1, nblocks=1, seed=9999)
print(questionnaire(provider.survey))  # print survey design for review

sink("questions_for_survey.txt")  # send survey to external text file
questionnaire(provider.survey)
sink() # send output back to the screen

# user-defined function for plotting descriptive attribute names
effect.name.map <- function(effect.name) {
  if(effect.name=="brand") return("Mobile Service Provider")
  if(effect.name=="startup") return("Start-up Cost")
  if(effect.name=="monthly") return("Monthly Cost")
  if(effect.name=="service") return("Offers 4G Service")
  if(effect.name=="retail") return("Has Nearby Retail Store")
  if(effect.name=="apple") return("Sells Apple Products")
  if(effect.name=="samsung") return("Sells Samsung Products")
  if(effect.name=="google") return("Sells Google/Nexus Products")
  } 

# read in conjoint survey profiles with respondent ranks
conjoint.data.frame <- read.csv("mobile_services_ranking.csv")

# set up sum contrasts for effects coding as needed for conjoint analysis
options(contrasts=c("contr.sum","contr.poly"))

# main effects model specification
main.effects.model <- {ranking ~ brand + startup + monthly + service + 
  retail + apple + samsung + google}

# fit linear regression model using main effects only (no interaction terms)
main.effects.model.fit <- lm(main.effects.model, data=conjoint.data.frame)
print(summary(main.effects.model.fit))

# save key list elements of the fitted model as needed for conjoint measures
conjoint.results <-
  main.effects.model.fit[c("contrasts","xlevels","coefficients")]

conjoint.results$attributes <- names(conjoint.results$contrasts)

# compute and store part-worths in the conjoint.results list structure
part.worths <- conjoint.results$xlevels  # list of same structure as xlevels
end.index.for.coefficient <- 1  # intitialize skipping the intercept
part.worth.vector <- NULL # used for accumulation of part worths
for(index.for.attribute in seq(along=conjoint.results$contrasts)) {
  nlevels <- length(unlist(conjoint.results$xlevels[index.for.attribute]))
  begin.index.for.coefficient <- end.index.for.coefficient + 1
  end.index.for.coefficient <- begin.index.for.coefficient + nlevels -2
  last.part.worth <- -sum(conjoint.results$coefficients[
    begin.index.for.coefficient:end.index.for.coefficient])
  part.worths[index.for.attribute] <-
    list(as.numeric(c(conjoint.results$coefficients[
      begin.index.for.coefficient:end.index.for.coefficient],
      last.part.worth)))
  part.worth.vector <-
    c(part.worth.vector,unlist(part.worths[index.for.attribute]))
  } 
conjoint.results$part.worths <- part.worths
# compute standardized part-worths
standardize <- function(x) {(x - mean(x)) / sd(x)}
conjoint.results$standardized.part.worths <-
  lapply(conjoint.results$part.worths,standardize)
 
# compute and store part-worth ranges for each attribute
part.worth.ranges <- conjoint.results$contrasts
for(index.for.attribute in seq(along=conjoint.results$contrasts))
  part.worth.ranges[index.for.attribute] <-
  dist(range(conjoint.results$part.worths[index.for.attribute]))
conjoint.results$part.worth.ranges <- part.worth.ranges

sum.part.worth.ranges <- sum(as.numeric(conjoint.results$part.worth.ranges))
# compute and store importance values for each attribute
attribute.importance <- conjoint.results$contrasts
for(index.for.attribute in seq(along=conjoint.results$contrasts))
  attribute.importance[index.for.attribute] <-
  (dist(range(conjoint.results$part.worths[index.for.attribute]))/
  sum.part.worth.ranges) * 100
conjoint.results$attribute.importance <- attribute.importance
 
# data frame for ordering attribute names
attribute.name <- names(conjoint.results$contrasts)
attribute.importance <- as.numeric(attribute.importance)
temp.frame <- data.frame(attribute.name,attribute.importance)
conjoint.results$ordered.attributes <-
  as.character(temp.frame[sort.list(
  temp.frame$attribute.importance,decreasing = TRUE),"attribute.name"])

# respondent internal consistency added to list structure
conjoint.results$internal.consistency <- summary(main.effects.model.fit)$r.squared
 
# user-defined function for printing conjoint measures
if (print.digits == 2)
  pretty.print <- function(x) {sprintf("%1.2f",round(x,digits = 2))}
if (print.digits == 3)
  pretty.print <- function(x) {sprintf("%1.3f",round(x,digits = 3))}

# report conjoint measures to console
# use pretty.print to provide nicely formated output
for(k in seq(along=conjoint.results$ordered.attributes)) {
  cat("\n","\n")
  cat(conjoint.results$ordered.attributes[k],"Levels: ",
  unlist(conjoint.results$xlevels[conjoint.results$ordered.attributes[k]]))
 
  cat("\n"," Part-Worths:  ")
  cat(pretty.print(unlist(conjoint.results$part.worths
    [conjoint.results$ordered.attributes[k]])))

  cat("\n"," Standardized Part-Worths:  ")
  cat(pretty.print(unlist(conjoint.results$standardized.part.worths
    [conjoint.results$ordered.attributes[k]])))

  cat("\n"," Attribute Importance:  ")
  cat(pretty.print(unlist(conjoint.results$attribute.importance
    [conjoint.results$ordered.attributes[k]])))
  }

# plotting of spine chart begins here
# all graphical output is routed to external pdf file
pdf(file = "fig_preference_mobile_services_results.pdf", width=8.5, height=11)
spine.chart(conjoint.results)
dev.off()  # close the graphics output device

# Suggestions for the student:
# Enter your own rankings for the product profiles and generate
# conjoint measures of attribute importance and level part-worths.
# Note that the model fit to the data is a linear main-effects model.
# See if you can build a model with interaction effects for service
# provider attributes.

Exhibit 1.2 Measuring and Modeling Individual Preferences (Python)

# Traditional Conjoint Analysis (Python)

# prepare for Python version 3x features and functions
from __future__ import division, print_function

# import packages for analysis and modeling
import pandas as pd  # data frame operations
import numpy as np  # arrays and math functions
import statsmodels.api as sm  # statistical models (including regression)
import statsmodels.formula.api as smf  # R-like model specification
from patsy.contrasts import Sum

# read in conjoint survey profiles with respondent ranks
conjoint_data_frame = pd.read_csv('mobile_services_ranking.csv')

# set up sum contrasts for effects coding as needed for conjoint analysis
# using C(effect, Sum) notation within main effects model specification
main_effects_model = 'ranking ~ C(brand, Sum) + C(startup, Sum) +      C(monthly, Sum) + C(service, Sum) + C(retail, Sum) + C(apple, Sum) +     C(samsung, Sum) + C(google, Sum)'

# fit linear regression model using main effects only (no interaction terms)
main_effects_model_fit =     smf.ols(main_effects_model, data = conjoint_data_frame).fit()
print(main_effects_model_fit.summary())

conjoint_attributes = ['brand', 'startup', 'monthly', 'service',     'retail', 'apple', 'samsung', 'google']

# build part-worth information one attribute at a time
level_name = []
part_worth = []
part_worth_range = []
end = 1  # initialize index for coefficient in params

for item in conjoint_attributes:
    nlevels = len(list(unique(conjoint_data_frame[item])))
    level_name.append(list(unique(conjoint_data_frame[item])))
    begin = end
    end = begin + nlevels - 1
    new_part_worth = list(main_effects_model_fit.params[begin:end])
    new_part_worth.append((-1) * sum(new_part_worth))
    part_worth_range.append(max(new_part_worth) - min(new_part_worth))
    part_worth.append(new_part_worth)
    # end set to begin next iteration
    
# compute attribute relative importance values from ranges
attribute_importance = []
for item in part_worth_range:
    attribute_importance.append(round(100 * (item / sum(part_worth_range)),2))
    
# user-defined dictionary for printing descriptive attribute names
effect_name_dict = {'brand' : 'Mobile Service Provider',     'startup' : 'Start-up Cost', 'monthly' : 'Monthly Cost',     'service' : 'Offers 4G Service', 'retail' : 'Has Nearby Retail Store',     'apple' : 'Sells Apple Products', 'samsung' : 'Sells Samsung Products',     'google' : 'Sells Google/Nexus Products'}

# report conjoint measures to console
index = 0  # initialize for use in for-loop
for item in conjoint_attributes:
    print('\nAttribute:', effect_name_dict[item])
    print('    Importance:', attribute_importance[index])
    print('    Level Part-Worths')
    for level in range(len(level_name[index])):
        print('       ',level_name[index][level], part_worth[index][level])
    index = index + 1