Predicting the future is big business. But while think tanks, corporations, and intelligence agencies have traditionally relied on hiring academics or analysts to write contingency plans, the rise of cloud computing and cheap processor time are changing things. One company, Massachusetts-based Recorded Future, is part of a growing number of predictive analytics firms—but they're one of only a few partially funded by America's intelligence agencies. In an interview with Co.Labs, founder Christopher Ahlberg said “The world actually knows a fair amount about the future. Little tidbits of information here and there about different behaviors. One company makes a corporate filing. Another person tweets something. We wondered.... what if we could get our hands on every single fact that humanity knows about the future, and actually put them together for analysis? That was the idea behind Recorded Future.”
Since Recorded Future was initially founded in 2010 with funding of less than $10 million each from Google Ventures and CIA/NSA venture capital arm In-Q-Tel, the company has sold their products to a variety of buyers. Recorded Future's products, which scour the Internet to predict upcoming news events and forecast risk, are used by intelligence agencies, Fortune 1000 companies, foreign governments, hedge funds, and energy companies. Most of them rely on open source intelligence, which—despite the name—has nothing to do with open source software. Open source intelligence is the publicly accessible content available from the Internet and newsstands; newspaper articles, television broadcasts, and publicly accessible social media posts all fall into the open source intelligence sphere. According to Ahlberg, the company only uses free and non-pay source materials for their database.
It also helps, apparently, to figure out where the next riot will be. In press materials, Recorded Future calls themselves “the world's first temporal analytics engine.” Ahlberg is Swedish and the company also maintains offices in Goteborg, Sweden; their last major funding round was $12 million from Balderton and Google Ventures in 2012.
Recorded Future's main product is a software package that claims to predict the future. The by-subscription online dashboard allows users to run queries and visualizations for upcoming events that are keyed to specific time periods. Recorded Future's spiders, which take in content from multiple languages and display original foreign-language source materials, use algorithms to parse language and correlate specific dates and times to specific events.
The web product, which licenses for $149 monthly or $1,599 per user, takes inputs from over 250,000 publicly available news and data sources and correlates them. Users can create visualizations from the data and custom maps, alongside network diagrams and event feeds. In order to attract potential users, Recorded Future also offers a free version that, while useful for dissertation or small business data forecasting work, lacks the full functionality of the $1,599 version. This web product is similar to offerings from competitors such as (also In-Q-Tel funded) Palantir, except aimed directly at predicting real-world events rather than larger detection of connections between individual data points.
Recorded Future also offers two more products. One is an API which allows Recorded Future's open source data interpretations to be ported into other apps; Ahlberg said most of their API-centric users are in the financial sector. The other is a for-government product called Foresite which applies Recorded Future's predictive algorithms to what the company calls “internal documents, third-party content, or sensitive data sources.”
Co.Labs' demo of Recorded Future's dashboard took place several days before the Egyptian coup. Ahlberg showed how Recorded Future's algorithms parsed data from different sources to make forecasts about the likelihood of disorder in several Egyptian cities. The bulk of the company's algorithms tie word choice and word frequency to the mention of specific times and dates; this data is extrapolated and visualized to help understand what different events are likely to happen. In a second demo, Co.Labs was shown how Recorded Future's dashboard could help information assurance teams in government or at large corporations analyze chatter about hacker attacks on the Internet for future threat analysis. Ahlberg noted that many of his clients look for particular combinations of events that precede another event happening—for instance, corporate resignations or large DDoS attacks.
As far as DDoS attacks and other cyberattacks, Ahlberg says, "Pretty much all cyber defense today is focused on what has already happened, inside or at the perimeter of a company. Virus scanning, firewalls, and things like that. We realized that you can find a lot of signal for upcoming attacks in social media, chat rooms, and foreign open sources by applying our temporal analytics. So instead of waiting for attacks to happen we can find early signals and try to prevent them in the first place."
Several visualizations created through Recorded Future, which show—among others—potential fracking sites all over the world, likely Egyptian protests during Ramadan, and which companies are likely to debut electric cars, are all shown above.
Here are a few other predictive exercises made using Recorded Future's platform:
Monitoring Protests And Unrest
Competitive Intelligence On Business Competitors
Predicting Cyberattacks Against Agencies Or Corporations
For all the gee-whiz futurism of Recorded Future's predictive analytics engine, the inclusion of only text for corporate and government users is a major hindrance. Because Recorded Future and their competitors rely only on text—with video, maps, audio files, and so much else pushed to the side—their event forecasts are only as good as their textual samples. Underground radio shows from Iran? YouTube samizdat from Russia? SoundClouds by the next angry young person with a gun in America? Those are missing from the equation.
This means that Recorded Future's target market—the militaries, intelligence agencies, corporations, and investment funds with a pressing need to predict the future—have to turn their products into just one part of a future predicting toolkit. For organizations which live and die by creating accurate forecasts of what will happen when, good mapping and data-extracting products are just as much part of operations as Recorded Future's software.
But where the company excels is in taking massive amounts of data and making them accessible to ordinary analysts and end users. Recorded Future's software doesn't use any amazing sci-fi magic to predict the future; it uses the same big data toolkit as hundreds of other companies. But by scouring the web's news sources and letting users quickly perform analyses on thousands of newspaper articles or message board articles at a time, they make life much easier for a lot of organizations.
[Images: Recorded Future]