Vermont scientists are developing a tool to study billions of social media messages and predict potential political and financial turmoil

For millennia, people looked into the night sky with their bare eyes – and told stories about the few visible stars. Then we invented telescopes. In 1840, the philosopher Thomas Carlyle claimed that “the history of the world is only the biography of great men”. Then we started posting on Twitter.

Now scientists have invented a tool to dig deep into the billions and billions of posts that have been published on Twitter since 2008 – and have begun to uncover the vast galaxy of stories they contain.

“We call it the story wrangler,” says Thayer Alshaabi, a graduate student at the University of Vermont who led the new research. “It’s like a telescope looking at all of this data that people are sharing on social media in real time. We hope people use it for themselves, just as you can look up to the stars and ask your own questions. “

The new tool can provide an unprecedented, up-to-the-minute view of popularity, from rising political movements to box office flops; from the overwhelming success of K-pop to signals of emerging new diseases.

The Story Wrangler’s story – a curation and analysis of over 150 billion tweets – and some of its key findings were published in the magazine on July 16 Scientific advances.


The team of eight scientists who invented story wranglers – from the University of Vermont, Charles River Analytics and MassMutual Data Science – collects about ten percent of all tweets around the globe every day. For every day they break down these tweets into individual bits as well as pairs and triplets and generate frequencies from more than a billion words, hashtags, handles, symbols and emojis, such as “Super Bowl”, “Black Lives Matter”, “gravitational” waves ” , “#Metoo”, “Coronavirus” and “Keto Diet”.

“This is the first visualization tool that allows you to look at one-, two-, and three-word phrases in 150 different languages ​​from the beginnings of Twitter to today,” says Jane Adams, co-author of the new study recently completed a three-year position as artist-in-residence for data visualization at the Complex Systems Center of the UVM.

Powered by the UVM supercomputer in the Vermont Advanced Computing Core, the online tool provides a powerful lens for viewing and analyzing the ebb and flow of words, ideas and stories among people around the world on a daily basis. “It’s important because it shows important discourses as they happen,” says Adams. “It quantifies collective attention.” Although Twitter does not represent all of humanity, it is used by a very large and diverse group of people, which means that it “codes popularity and diffusion,” the scientists write, giving a novel look at the discourse not just about famous people. like politicians and celebrities, but also the daily “expressions of the many”, the team notes.

In a powerful test of the story wrangler’s vast dataset, the team revealed that it could potentially be used to predict political and financial turmoil. They examined the percentage change in the use of the words “rebellion” and “crackdown” in different regions of the world. They found that the rise and fall of these terms were significantly related to changes in an established index of geopolitical risk for the same locations.


The global story that is now being written on social media has billions of voices – commenting and sharing, complaining and attacking – and in all cases, recordings – about world wars, strange cats, political movements, new music, food offerings, deadly diseases , Favorite soccer stars, religious hopes and dirty jokes.

“The Story Wrangler gives us a data-driven way to index what normal people talk about in everyday conversations, not just what reporters or writers have chosen; it’s not just the educated or wealthy or cultural elites, ”says applied mathematician Chris Danforth, a professor at the University of Vermont who, together with his colleague Peter Dodds, led the development of the StoryWrangler. Together they run UVM’s Computational Story Lab.

“It’s part of the evolution of science,” says Dodds, an expert on complex systems and professor at the Institute for Computer Science at the UVM. “This tool can enable new approaches to journalism, powerful ways to view natural language processing, and the development of computer history.”

How much a few powerful people shape the course of things has been debated for centuries. But if we knew what every farmer, soldier, shopkeeper, nurse and teenager had to say during the French Revolution, we would have very different stories about the rise and rule of Napoleon. “Here’s the deep question,” says Dodds, “what happened? What actually happened? “


The UVM team, with support from the National Science Foundation, is using Twitter to demonstrate how Chatter can act as some sort of global sensor system on distributed social media – what happened, how people reacted, and what might be next. But other social media streams, from Reddit to 4chan to Weibo, could theoretically be used to feed story wranglers or similar devices: following the reaction to major news events and natural disasters; follow the fame and fate of political leaders and sports stars; and opens up a glimpse of casual conversations that can provide insights into dynamics that range from racism to employment, emerging health threats to new memes.

In the new Scientific advances Study, the team presented an example from Storywrangler’s online viewer, highlighting three global events: the death of Iranian General Qasem Soleimani; the beginning of the COVID-19 pandemic; and the protests against Black Lives Matter following the murder of George Floyd by Minneapolis police. The Storywrangler record records a sudden surge in tweets and retweets with the term “Soleimani” on January 3, 2020 when the United States assassinated the general; the sharp rise in the “coronavirus” and virus emojis in spring 2020 as the disease spread; and a usage explosion of the hashtag “#BlackLivesMatter” on and after May 25, 2020, the day George Floyd was murdered.

“While I’m speaking, a hashtag is being made up,” says UVM’s Chris Danforth. “We didn’t know yesterday to look for it, but it will show up in the dates and become part of the story.”



Please enter your comment!
Please enter your name here