Fear of missing out when collecting information is very real. Traditional intelligence practitioners often assume that their goal is to gather as much information as possible to formulate a more comprehensive picture of threats, and this is a common problem in cyber threat intelligence as well. However, this approach can hamper the accuracy, timeliness, and relevancy of analysis. In truth, excessive collection will likely lead to information overload on both the individual and institutional levels that can result in skewed analysis and assessments. Unfettered and undermanaged intelligence collection of raw, exploited, and production data can affect both data-driven analysis and conceptually-driven analysis. It has been shown that an analyst only needs minimum information to make an informed judgment. Common issues deriving from collection overload include overconfidence (a result of circular reporting or having too many information sets to evaluate); reinforcement of collection bias; and unchecked collection, which may cause analytic paralysis that leads to a high noise-to-signal ratio that in turn results in indecision and an inability to conduct effective structured analysis.
This presentation proposes best practices to mitigate such issues by producing a realistic collection management framework and sustainable intelligence requirements; starting with a minimal viable collection strategy; collecting what you need and growing it only as needed; conducting source review and evaluation; evaluating exploited and production data via a framework such as an admiralty system; and counting the times a source is used to enforce an assessment in order to uncover collection bias. Finally, we’ll look at upgrading analysis models as the best way to improve analysis and mitigate issues deriving from over-collection.
Sherman Chu, Cyber Intelligence Analyst, New York City Cyber Command