Mining Big Data to Extract Patterns and Predict Real-Life Outcomes

Mining Big Data to Extract Patterns and Predict Real-Life Outcomes

By
Michal Kosinski, Yilun Wang, Himabindu Lakkaraju, Jure Leskovec
Psychological Methods. December
2016, Vol. 21, Issue 4, Pages 493-506

This article aims to introduce the reader to essential tools that can be used to obtain insights and build predictive models using large data sets. Recent user proliferation in the digital environment has led to the emergence of large samples containing a wealth of traces of human behaviors, communication, and social interactions. Such samples offer the opportunity to greatly improve our understanding of individuals, groups, and societies, but their analysis presents unique methodological challenges. In this tutorial, we discuss potential sources of such data and explain how to efficiently store them. Then, we introduce two methods that are often employed to extract patterns and reduce the dimensionality of large data sets: singular value decomposition and latent Dirichlet allocation. Finally, we demonstrate how to use dimensions or clusters extracted from data to build predictive models in a cross-validated way. The text is accompanied by examples of R code and a sample data set, allowing the reader to practice the methods discussed here. A companion website (http://dataminingtutorial.com) provides additional learning resources.

The 22nd most discussed paper of 2015 and the 12th most influential article ever published by PNAS