Breaking Down Roll20’s Orr Group Industry Report for Q1 of 2015

Orr Group Industry Report Q1 2015 Summary

The following article was originally posted by the author on The Mad Adventurers Society, and is reprinted here with permission. You can find the original here.

The Orr Group, the analytics arm of popular virtual tabletop app Roll20, has released its Industry Report for the first quarter of 2015. This is the third installment of the report, and as we’ve previously discussed, this is still a very noisy set of data. Don’t put much faith into the numbers, but the directional data is useful for showing trends.

First, this gives observers an update on the growth Roll20 itself. When I reached out to Nolan T. Jones, Orr Group co-founder and Roll20 spokesperson, he let slip that the site is gaining “over 1,300 new users a day.” Wait, what? 1,300 new users per day? That’s around 117,000 new users in the first three months of 2015 alone. On their February 5th blog post, Roll20 claimed “over 800,000” users. In their press release on April 8th, The Orr Group stated that Roll20 has over 850,000 users. Depending which way they’re hedging, that puts the actual user count around 875,000, and their total user growth in the quarter in the 15% range.  They should be hitting 1 million users by mid-July. (In case you’re curious, the most recent record they’ve published is 8,400 players online on Saturday, February 7th of this year.) Any way you slice it, that’s a gigantic audience.

Given that growth, you shouldn’t be surprised to see that the number of respondents has also grown significantly quarter over quarter; the total sample size of games and the total sample size of players each grew by more than 30%, respectively. As in prior quarters, Jones confirmed that they’re seeing this growth from new accounts and games filling out the optional profile survey, as well existing accounts and games updating their profile once they see the way the data is used in the Orr Group Industry Report. At this point, I should reiterate my prior complaint: we have no validation that these accounts and games are still active. Jones insists their sample “is large enough to be an accurate representation of the site”, but unless they disclose more detail of the sampling methodology, I worry that we’re looking at stale data.

The data problem will only be exacerbated as time goes on. The Pathfinder campaign I ran that ended last October shouldn’t have any bearing on the numbers for January through March of this year, much less next October, but I suspect it may. For their part, some games did decline in popularity; Settlers of Catan and Gamma World both decreased by a single game since last quarter. It’s not clear if this is because the campaigns for those games were deleted, or if they aged out of the sample set.

Click to view full Q1 2015 report.

In terms of popularity growth, the two big winners this quarter were industry heavyweight Dungeons & Dragons 5th Edition and rules-lite sci-fi RPG Stars Without Number, though I attribute their respective successes to different causes.

Dungeons & Dragons 5th Edition

D&D saw more than 70% growth quarter over quarter–more than double the growth of any of its competitors–and surpassed Pathfinder as the most popular system on the platform. We would expect that; D&D 5E was the biggest release leading into the Holiday quarter of 2014, so we’d expect to see its popularity skyrocket in the new year. In my discussion with Nolan T. Jones, he didn’t rule out my interpretation, but doesn’t believe they have the data to support it, “yet”.

The Orr Group’s analysis, on the other hand, focused on the value of map-and-mini-focused Roll20 for the Theater of the Mind-driven system. Jones suggested, “For us, though, the real kicker is this dispels the notion that people would only use our virtual tabletop for games set on a gridded map, as Fifth Edition has no such requirements. That’s a big confidence booster towards what we’re doing.” I’m not ruling out his interpretation, but I would wager they’ve got the causal relationship reversed: 5th Edition players who are using gridded maps are flocking to Roll20. Unfortunately, there’s no data on how a campaign is utilizing Roll20’s features. Yet.

Stars Without Number

Stars Without Number is a fun case study in the power of Roll20 as a platform, and perhaps a big referendum on its viability as a marketplace platform for publishers with limited resources. When Roll20 took freeform survey results to create the matching-­making systems upon which the Orr Group Industry Report was built, ​ Stars Without Number​ was not popular enough to even make the cut but now is over 1.09% of all games played via the platform.

“We know what’s brought the sudden interest in this 2011 title is [newly hired Twitch livestream producer and GM] Adam [Koebel] running the game on ​ Rollplay​,” said Jones, referring to the Twitch/YouTube series Swan Song​. The episode ran in July of 2014, but simply linking to it in their blog announcement of Koebel’s hiring was enough to launch 230 games. It would appear that, when Roll20 talks, its users listen.

A Look Forward

Because I suspect there’s a lot of inertia in the data, since it is pulled from registered games and players rather than active, I don’t expect anticipate much in the way of directional changes in the future. The more Roll20 grows, the more the entrenched systems are liable to dominate the Orr Group Industry Report, as prior months’ data accumulates to drown out current-quarter growth.

In terms of quarterly growth, D&D 5E will likely continue to outpace Pathfinder next quarter (and the quarter after.) It’s a solid system with a strong following from a major brand name that’s been well-received by critics; I doubt it’s going anywhere. What I am curious to watch for is whether or not we’ll see any impact in Roll20’s numbers owing to competitor Fantasy Grounds’ acquisition of licensed D&D 5E Core material, or if Roll20’s sheer scale will simply absorb the hit and keep chugging along. I suspect the latter, but I’ll be looking for evidence one way or the other.

Longer term, I hope the data collection is refined further, because we could truly correlate events in the marketplace (both the broader industry marketplace, as well as Roll20’s own digital goods marketplace) with active user/active game data, rather than tagged data. In particular, I suspect Roll20 could drive a lot of activity in one system or another if they provided licensed adventure content from major publishers. The simultaneous release of a new Pathfinder campaign path with the pre-made digital assets needed to run it in Roll20, for example, could be a big driver of player behavior. As someone who hopes to see publishers embrace virtual tabletops with simultaneous first-party content, I think the data would go a long way in making the case for investing their resources in release day digital content.