Ranking Airlines Based Only On Twitter Comments Is Highly Suspect

There are many ways to rank airlines. Financially, results matter and Wall Street does a good job pointing out the companies that provide a good return to shareholders and those that don’t. The Net Promoter Score (NPS), developed in part by Bain, is used by many industries and their airline applications are common and robust. The U.S. Department of Transportation compiles and reports on airline operational results, like on-time flights, cancellations, and lost baggage. Multiple universities and consumer groups also put out their own rankings of the best and worst airlines.

Many rankings are limited because they focus on consumer features but not on value, because they often don’t consider that different airlines charge different prices. Insuremytrip.com, a site that sells travel insurance, has now released a new study that ranks airlines solely based on Twitter comments. I can’t imagine a more useless and trivial ranking, and while many ranking systems can be improved, this one should just be approached with significant caution.

How The Insuremytrip Ranking Works

This new ranking uses a process called sentiment analysis. It uses artificial intelligence to scan tweets for negative and positive comments about airlines. They evaluate tens of thousands of tweets and look at changes in the amount of sentiments expressed in the messages. Based on this, they rank airlines both in the absolute amount of negative comments and on the change from a prior period. The study also scans for positive comments, and in this way the study roughly matches the main idea in the more rigorous NPS methodology.

This approach has been collected for the last four years, but with nothing for 2020 understandably, so now some trending and year-over year metrics can be viewed. If you assume that problems in the core data sample are constant, this multi-year trending is probably the most useful aspect of this study.

The Results Of This Ranking

For 2022, this study shows, for example, that among United Airlines tweets, 37% have negative comments, and 26% have positive comments. Further, the 26% positive is a six-point drop from 2021, suggesting that fewer people are tweeting positive things about United this year. Alaska Airlines has the highest percentage of positive comments at 29%, and Frontier Airlines has the highest percentage of negative comments at 61%.

As a group, for 2022 all airlines saw a decrease in positive comments and most saw an increase in negative comments. Given the operational disruptions of the industry this past summer, this is not surprising. Interestingly, the top concern when flying according to this study for 2022 is money, followed by delays, cancellations, and customer service. In other words, Twitter results suggest that customers care more about paying a low fare than being delayed, but then when they record the negative sentiment about the delay the price paid mysteriously vanishes.

What This Approach Misses

There are multiple problems with this ranking approach. The first is the overwhelmingly negative tendency of all tweets. Research from Harvard University shows that outrage spreads faster on Twitter than any other social media platform. This bias predisposes that people will tweet more when they have negative things to say, so as an absolute ranking it is not a great underlying data source. The year over year trending, as stated earlier, has some validity.

The second limitation is the lack of value implicit in the ranking. To choose the flight, people care a lot about the price they pay. But once there, the same things annoy everyone independent of what they paid. Related to this is the wide disparity of what annoys people. A 30-minute flight delay may not matter to some customers, where as others may feel this ruined their trip. Is coffee that isn’t too hot, or is too hot, the same as a lost bag or significant delay? Each would show up as a negative sentiment but there is no weighting of the severity of the negativity.

The last approach problem of this study is the limitation in the data source. The Pew Research Center estimates that only about 23% of people use Twitter or tweet with any regularity. Basing a consumer ranking on this non-random sample, with no weighting for the magnitude of the infraction, severely limits the usefulness of this study.

Better Ways To Use Social Media In Rankings

Social media is a rich environment of consumer data. Travel companies have been working on ways to identify trends in locations and other travel interests to create interesting package opportunities, or even for some to decide on new places to fly.

As a way to rank airlines, undoubtedly a robust set of social media metrics could be used to get a realistic consumer view of airlines. I don’t know that anyone has done this yet, and using a single source like this Twitter study is far from that ideal. The value metric, what you get for what you pay, is the most interesting as it is most related to how people actually purchase. Whenever looking at a source that uses only what people say, this will pale in comparison to pairing this with the decisions people actually make.