Everyone is Writing About the Orr Group Industry Report and I Am Too

A lot of people have written about the Orr Group Industry Report by now. They’ve all written basically the same thing, so take your pick of The Iron Tavern, Examiner, TGN, and Geek Native.  They follow the same basic observations, regurgitated from Orr Group’s press release: Dungeons and Dragons is popular, 5th Edition is growing, and now that players know their profile choices matter, more players are making sure their profile is accurate. In fact, I made the same observations on Saturday, and I didn’t even get a copy of the press release (Orr Group, you know where to find me.)

This report is interesting for the things that it is and the things that it is not, so we’re going to discuss it. To start, here it is:

Orr Group 2014 Q4 Industry Report Summary

Orr Group also published the detailed version of the report with Other Listed Games expanded, but that full infographic is quite large, so I’ll link to it here.

An “Industry Report?”

The most important caveat of the data is that it is not a representative sample. As Orr Group acknowledged in the Roll20 Blog post for the Q3 results, the data”only takes into account games and players who filled out relevant fields on their game page/profile.” This is less a report on the industry, and more a report on active Roll20 games and players who responded to a survey. While it does provide a view into the games people are playing, rather than what they’re buying, it’s such a narrow view and you’d need a number of assumptions to make industry-wide generalizations.

I’m willing to make those assumptions for the Games data, but not for Players.

Player Data is Nonsense

Though we’re slowly getting emerging trends, I’m still pretty unimpressed by the Player data. From the same Roll20 Blog post, the Player data is mined from the “Enjoys Playing” field on the player profile. This is a freeform, optional field that doesn’t drive any site functionality outside of this survey. Off the top of my head, some potential biases in the data:

• Respondents are more likely to enter the games at the top of mind, allowing bias towards games with recent news or increased marketing presence.
• Respondents may be biased towards inputting games they know they can play on Roll20.
• Knowing that choices are public, respondents may be biased towards games they perceive to be more popular or desirable.
• Now that players know their choices are being used for this report, they may be trying to manipulate the results. This sounds crazy, but Edition Wars are a real thing.

Evidence of this shows with less popular systems like Stars Without Number (fewer players than games) as well more popular indie titles like 13th Age (261 games but only 318 players.) This is even more problematic for non-RPG games; Settlers of Catan, Warhammer 40k, and Munchkin are certainly more popular and played more frequently than their representation in these data, because they are rather niche for Roll20.

Perhaps bigger than those issues, though, is that the Player data may just be stale. Orr Group hasn’t shared any metadata on the collection fields, but I would guess that most players fill out their profile once, even if they are active. I checked my profile for this article, and despite playing a weekly Roll20 campaign with 100+ hours, my profile hasn’t changed since the day I registered (which I know, because it’s still completely blank.)

Games Data is Pretty Good… But Only Pretty Good

I think the Games data is great for measuring relative popularity of D&D and Pathfinder. They are head-and-shoulders the most popular systems, and they ought to be representative relative to each other. However, I’m confident that Pathfinder, 3.5E, 4E, and 5E don’t total 65%+ of the roleplaying games played, period.

The reason for this overstatement lies again in the data source: the optional “Playing” field that only drives Looking For Group functionality. Most importantly, in order to use Roll20’s LFG functionality, you need to populate the field. Herein lies the bias: D&D-PF bloc are the easiest systems to play with pick-up groups, because they’re the most popular point of entry into the hobby, and everyone knows them. This popularity feeds itself. The easier it is to find groups through Roll20’s LFG, the more games will look for players using Roll20’s LFG, and the more games will have the Playing field populated.

I submit that the less popular systems on the report have communities outside Roll20, even if they commonly use Roll20 to play online. Games for these systems are less likely to be properly tagged, because groups that come to Roll20 already formed are far less likely to populate the Playing field (the 5E campaign I hosted with my established group didn’t have the field populated, for example.)

Looking Ahead

The trend data, though in its infancy, should be useful for observers like myself in the future. I’m particularly interested in looking at the Game data alongside Marketplace data, and how much Marketplace support (either publisher or third party) correlates with the Game data. I suspect that games with publisher support within the Marketplace will outpace their peers in growth over time.

Both the press release and most other coverage of the Q4 report have discussed the increase in popularity of D&D 5E. That’s fine, but I’m not willing to draw conclusions from two quarters of data. I’m hoping they continue to report these data (and expand to Marketplace, for that matter), because a year or two of trend data should take out some of the quarterly noise.