- What Is Data Mining?
- What Data Mining Is Not
- The Most Common Data Mining Applications
- What Kinds of Patterns Can Data Mining Discover?
- Popular Data Mining Tools
- The Dark Side of Data Mining: Privacy Concerns
- Summary
- References
This chapter is from the book
This chapter is from the book
This chapter is from the book
The Dark Side of Data Mining: Privacy Concerns
Data that is collected, stored, and analyzed in data mining often contains information about real people. Such information may include identification data (e.g., name, address, Social Security number, driver’s license number, employee number), demographic data (e.g., age, sex, ethnicity, marital status, number of children), financial data (e.g., salary, gross family income, checking or savings account balance, home ownership, mortgage or loan account specifics, credit card limits and balances, investment account specifics), purchase history (i.e., what is bought from where and when, either from vendors’ transaction records or from credit card transaction specifics), and other personal data (e.g., anniversary, pregnancy, illness, loss in the family, bankruptcy filings). Most of these data can be accessed through some third-party data providers. The main issue here is the privacy of the person to whom the data belongs. In order to maintain the privacy and protection of individuals’ rights, data mining professionals have ethical and often legal obligations. One way to ethically handle private data is to de-identify customer records prior to applying data mining applications so that the records cannot be traced to an individual. Many publicly available data sources (e.g., CDC data, SEER data, UNOS data) are already de-identified. Prior to accessing these data sources, those mining data are often asked to agree not to try to identify the individuals behind those figures.
In a number of instances in the recent past, companies have shared their customer data with others without seeking the explicit consent of their customers. For instance, as you might recall, in 2003, JetBlue Airlines provided more than a million passenger records of its customers to Torch Concepts, a U.S. government contractor. Torch subsequently augmented the passenger data with additional information, such as family sizes and Social Security numbers—information purchased from a data broker called Acxiom. The consolidated personal database was intended to be used for a data mining project in order to develop potential terrorist profiles. All this was done without notifying or obtaining consent from passengers. When news of the activities got out, however, dozens of privacy lawsuits were filed against JetBlue, Torch, and Acxiom, and several U.S. senators called for an investigation into the incident (Wald, 2004). Similar, but not as dramatic, privacy-related news came out in the recent past about popular social networking companies, which were allegedly selling customer-specific data to other companies for personalized targeted marketing.
A peculiar story about data mining and privacy concerns made headlines in 2012. In this instance, the company did not even use any private and/or personal data. Legally speaking, there was no violation of any laws. This story, which is about Target, goes as follows. In early 2012, an infamous story broke out about Target’s practice of using predictive analytics. The story was about a teenager girl who was being sent advertising flyers and coupons by Target for the kind of things that a new mother-to-be would buy from a store like Target. An angry man went into a Target outside Minneapolis, demanding to talk to a manager: “My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?” The manager didn’t have any idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture, and pictures of smiling infants. The manager apologized and then called a few days later to apologize again. On the phone, though, the father was somewhat abashed. “I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August. I owe you an apology.”
As it turns out, Target figured out a teen girl was pregnant before her father did! Here is how they did it. Target assigns every customer a guest ID number (tied to the person’s credit card, name, or email address) that becomes a placeholder to keep a history of everything the person has bought. Target augments this data with any demographic information that it has collected from the customer or bought from other information sources. By using this type of information, Target had looked at historical buying data for all the women who had signed up for Target baby registries in the past. The company had analyzed the data from all directions, and some useful patterns emerged. For example, lotions and special vitamins were among the products with interesting purchase patterns. Lots of people buy lotion, but Target noticed that women on the baby registry were buying larger quantities of unscented lotion around the beginning of their second trimester. Another analyst noted that sometime in the first 20 weeks, pregnant women loaded up on supplements like calcium, magnesium, and zinc. Many shoppers purchase soap and cotton balls, but when someone suddenly starts buying lots of scent-free soap and extra-big bags of cotton balls, in addition to hand sanitizers and washcloths, it signals that she could be getting close to her delivery date. Target was able to identify about 25 products that, when analyzed together, allowed them to assign each shopper a “pregnancy prediction” score. Target could even estimate her due date to within a small window, so the company could send coupons timed to very specific stages of her pregnancy.
If you looked at this practice from a legality perspective, you would conclude that Target did not use any information that violates customers’ privacy rights; rather, it used transactional data that almost every other retail chain is collecting and storing (and perhaps analyzing) about customers. What was disturbing in this scenario was perhaps that pregnancy was the targeted concept. People tend to believe that certain events or concepts—such as terminal disease, divorce, and bankruptcy—should be off-limits or treated extremely cautiously.
APPLICATION EXAMPLE: DATA MINING FOR HOLLYWOOD MANAGERS
Predicting box office receipts (i.e., the financial success) for a particular motion picture is an interesting and challenging problem. According to some domain experts, the movie industry is the “land of hunches and wild guesses” due to the difficulty associated with forecasting product demand, and this makes the movie business in Hollywood a risky endeavor. In support of such observations, Jack Valenti (the longtime president and CEO of the Motion Picture Association of America) once said that “no one can tell you how a movie is going to do in the marketplace...not until the film opens in a darkened theatre and sparks fly up between the screen and the audience.” Entertainment industry trade journals and magazines have been full of examples, statements, and experiences that support this claim.
Like many other researchers who have attempted to shed light on this challenging real-world problem, Ramesh Sharda and Dursun Delen have been exploring the use of data mining to predict the financial performance of a motion picture at the box office before it even enters production (while the movie is nothing more than a conceptual idea). In their highly publicized prediction models, they convert the forecasting (or regression) problem into a classification problem; that is, rather than forecast the point estimate of box office receipts, they classify a movie based on its box office receipts in one of nine categories, ranging from “flop” to “blockbuster,” making the problem a multinomial classification problem. Table 2.2 illustrates the definitions of the nine classes in terms of the range of box office receipts.
Table 2.2 Movie Classification, Based on Receipts
Class No. |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
|---|---|---|---|---|---|---|---|---|---|
Range (in millions) |
< $1 |
> $1 |
> $10 |
> $20 |
> $40 |
> $65 |
> $100 |
> $150 |
> $200 |
(Flop) |
< $10 |
< $20 |
< $40 |
< $65 |
< $100 |
< $150 |
< $200 |
(Block-buster) |
DATA
For the movie classification model, data was collected from a variety of movie-related databases (e.g., ShowBiz, IMDb, IMSDb, All-Movie) and consolidated into a single data set. The data set for the most recently developed models contained 2,632 movies released between 1998 and 2006. A summary of the independent variables along with their specifications is provided in Table 2.3. For more descriptive details and justification for inclusion of these independent variables, see Sharda and Delen (2006).
Table 2.3 Summary of the Predictive Variables
Independent Variable Name |
Definition |
Range of Possible Values |
|---|---|---|
MPAA rating |
Indicates the rating assigned by the Motion Picture Association of America (MPAA). |
G, PG, PG-13, R, NR |
Competition |
Indicates the level at which each movie competes for the same pool of entertainment dollars against movies released at the same time. |
High, Medium, Low |
Star value |
Signifies the presence of any box office superstars in the cast. A superstar can be defined as an actor who contributes significantly to the up-front sale of the movie. |
High, Medium, Low |
Genre |
Specifies the content category the movie belongs to. Unlike with the other categorical variables, a movie can be classified in more than one content category at the same time (e.g., action as well as comedy). Therefore, each content category is represented with a separate binary variable. |
Sci-Fi, Epic Drama, Modern Drama, Thriller, Horror, Comedy, Cartoon, Action, Documentary |
Special effects |
Signifies the level of technical content and special effects (e.g., animations, sound, visual effects) used in the movie. |
High, Medium, Low |
Sequel |
Specifies whether a movie is a sequel (value of Yes or 1) or not (value of No or 0). |
Yes, No |
Number of screens |
Indicates the number of screens on which the movie is planned to be shown during its initial launch. |
A positive integer |
METHODOLOGY
Using a variety of data mining methods, including neural networks, decision trees, support vector machines, and three types of ensembles, Sharda and Delen developed the prediction models. The data from 1998 to 2005 was used as training data to build the prediction models, and the data from 2006 was used as the test data to assess and compare the models’ prediction accuracy. Figure 2.5 shows the conceptual architecture for the proposed movie forecasting system, and Figure 2.6 shows the workflow of the model-building process.
)
)
RESULTS
Table 2.4 shows the prediction results of all three data mining methods, as well as the results of the three different ensembles. The first performance measure is the percentage correct classification rate, which is called bingo. The table also reports the 1-away correct classification rate (i.e., within one category). The results indicate that SVM performed the best among the individual prediction models, followed by ANN; the worst of the three was the CART decision tree algorithm. In general, the ensemble models performed better than the individual predictions models, and of those, the fusion algorithm performed the best. What is probably more important to decision makers, and stands out in Table 2.4, is the significantly low standard deviation obtained from the ensembles compared to the individual models.
Table 2.4 Tabulated Prediction Results for Individual and Ensemble Models
Prediction Models |
||||||
|---|---|---|---|---|---|---|
Individual Models |
Ensemble Models |
|||||
Performance Measure |
SVM |
ANN |
C&RT |
Random Forest |
Boosted Trees |
Information Fusion |
Count (bingo) |
192 |
182 |
140 |
189 |
187 |
194 |
Count (1-away) |
104 |
120 |
126 |
121 |
104 |
120 |
Accuracy (% bingo) |
55.49% |
52.60% |
40.46% |
54.62% |
54.05% |
56.07% |
Accuracy (% 1-away) |
85.55% |
87.28% |
76.88% |
89.60% |
84.10% |
90.75% |
Standard deviation |
0.93 |
0.87 |
1.05 |
0.76 |
0.84 |
0.63 |
CONCLUSIONS
The researchers claim that these prediction results are better than any others reported in the published literature for this problem domain. Beyond the attractive accuracy of their prediction results of the box office receipts, these models could also be used to further analyze (and potentially optimize) the decision variables in order to maximize the financial return. Specifically, the parameters used for modeling could be altered using the already trained prediction models in order to better understand the impact of different parameters on the end results. During this process, which is commonly referred to as sensitivity analysis, the decision maker of a given entertainment firm could find out, with a fairly high accuracy level, how much value a specific actor (or a specific release date, or the addition of more technical effects, etc.) brings to the financial success of a film, making the underlying system an invaluable decision aid. Figure 2.7 shows a screenshot from the prototype implementation of the prediction system named Movie Forecast Guru.
)