The Promise and Prejudice of Big Data in Intelligence Community (1610.08629v1)

Abstract: Big data holds critical importance in the current generation of information technology, with applications ranging from financial, industrial, academic to defense sectors. With the exponential rise of open source data from social media and increasing government monitoring, big data is now also linked with national security, and subsequently to the intelligence community. In this study I review the scope of big data sciences in the functioning of intelligence community. The major part of my study focuses on the inherent limitations of big data, which affects the intelligence agencies from gathering of information to anticipating surprises. The limiting factors range from technical to ethical issues connected with big data. My study concludes the need of experts with domain knowledge from intelligence community to efficiently guide big data analysis for timely filling the knowledge gaps. As a case study on limitations of using big data, I narrate some of the ongoing work in nuclear intelligence using simple analytics and argue on why big data analysis in that case would lead to unnecessary complications. For further investigation, I highlight cases of crowdsource forecasting tournaments and predicting unrest from social media.