Pyramid Volume 3: By the numbers

Reviews of past Pyramid issues was one of the things that came up in my “what to next” blog post.

Before I take up the challenge of issue reviews of Pyramid magazine, I thought I’d do a bit of analysis and see which issues I’d review.

Of course, we can all go to the What’s Hot page and do the usual sort: show me all of the Pyramid issues sorted by either Average Customer Rating or Number of Sales.

OK, fine, take the best five and worst five, and have at.

Right? Maybe not. There have been 61 issues since the restart of the magazine in its third incarnation. In the overall sort, we have certain implicit information. How long ago it was released is a function of the issue number, and we know they come out once a month, like clockwork.

We also get the average customer rating, the number of issues sold individually. The 12-month subscriptions more or less add a certain number to this, and that number isn’t constant, but varies depending on how many subscriptions are active. Still, I’m going to neglect that, despite it being the highest sales number of all. If I really wanted to dig into things, I’m sure I could extract number of 12-month subscriptions by month, by doing 61 individual searches.

I don’t care that much. I’m just looking for overall trends in the data.

So, are there any?

Unsurprisingly, yes.

Stand Up and Be Counted

If I were to just look at the average customer rating to tell me which issues were the most popular, would that work? Well, that depends on who’s voting and how many vote. Let’s take a look at how many bother to register their approval or disapproval of each issue.

Each big purple line (hey, I’m from Minnesota) represents 12 issues, or one year. To me, it seems like I can draw two conclusions. People really got excited about the new Pyramid volume in the first six months, and after that stopped voting.

As you can see by Steven Marsh’s comment below, the real conclusion is that there were a LOT of people who got six-month subscriptions to volume 3, and that drove up the numbers.

After that, the expected number of votes on an issues could be seen to decrease by roughly one vote per four or five months. So perhaps issues of interest are those with the highest percentage increase of votes more than expected, and the issues that were disliked or ignored are those with the lower deltas.The first six issues are excluded from this, because of the “new and cool” effect.

+Steven Marsh, Pyramid Editor and e23 Guru, was foiled by Blogger’s comment page (alas). But he did email me to note:

 As a reminder (or background in case you weren’t aware), with the changeover from Volume 2 to Volume 3, we gave everyone who had ANY time left on Volume 2 a six-month subscription to Volume 3. This explains why some statistics are so much higher on those first six issues; hundreds of more people had those issues compared to any other.

What are the best and worst in this case?

Best 10
Issues: % Votes from Expected
Worst 10 issues: % Votes from
Pyramid #3/33: Low-Tech (best) Pyramid
#3/32: Fears of Days Past
Pyramid #3/36: Dungeon Fantasy Pyramid
#3/42: Noir
Pyramid #3/44: Alternate GURPS II Pyramid
#3/53: Action
Pyramid #3/34: Alternate GURPS Pyramid
#3/11: Cinematic Locations
Pyramid #3/54: Social Engineering Pyramid
#3/35: Aliens
Pyramid #3/50: Dungeon Fantasy II Pyramid
#3/46: Weird Science
Pyramid #3/13: Thaumatology Pyramid
#3/27: Monsters in Space
Pyramid #3/43: Thaumatology III Pyramid
#3/14: Martial Arts
Pyramid #3/52: Low-Tech II Pyramid
#3/49: World-Hopping
Pyramid #3/41: Fantasy World-Building Pyramid
#3/61: Way of the Warrior (worst)

So, “best” in this case really means “encouraged more people than usual to go out and vote on it at all, regardless of whether this is a good vote or a bad one. Pyramid #3/33 would thus win if a record number of people logged on to say how much they hate it. Pyramid #3/61 (the most recent, which may count against it) might be the worst because, well . . . it’s still being read, and not enough time has gone by for people to read and digest it.

One last thing: are issues that sell well more likely to be rated at all? A quick plot of Vote count vs. Sales shows that yes, the more who buy it, the more vote. Not terribly surprising there, and what you see is that while there seems to be a floor of about ten or fifteen votes, roughly one person bothers to vote on the issue for each 23 people who buy the issue.

Actual Customer Feedback

Well, number of votes might be interesting, but you get to rate an issue when you vote, from 1-5 stars. So a couple things here. No issue has received an average rating less than 3.29 (Pyramid #3/02, the Superheroes one). One is currently sitting at a perfect 5.0 (#3/60, Dungeon Fantasy III).

OK, fine, let’s just quickly look at trends

One again, we see the first year as an anomaly. The “worst 10 rated issues” and “the first ten published issues” are one and the same. The best of these 10 is #3/04: Magic on the Battlefield, and even that one lags behind all the others by a smidge.

Once you get out of the first ten issues, overall, the ratings get better by a touch over time, but it’s not a strong function. But perhaps that’s because the new issues get favorable ratings first, and the not-so-good ratings come in later. What do we think? Number of votes gets larger with time (duh), but do ratings get worse as the number of votes increase? Well, yes.

The first thing we note is that the first six issues greatly skew any potential answer. So let’s throw them out, and see what we see.

OK, so yeah, the more votes you get, the more that you expect the pool to go down. So which Pyramid issues buck this trend? Which are the most above expectations, and most below?

Best 10
Issues: % Rating Adjusted by Votes
Worst 10 issues: % Rating
Adjusted by Votes
Pyramid #3/33: Low-Tech Pyramid #3/43: Thaumatology III
Pyramid #3/60: Dungeon Fantasy III Pyramid #3/57: Gunplay
Pyramid #3/34: Alternate GURPS Pyramid #3/07: Urban Fantasy
Pyramid #3/52: Low-Tech II Pyramid #3/14: Martial Arts
Pyramid #3/31: Monster Hunters Pyramid #3/56: Prehistory
Pyramid #3/54: Social
Pyramid #3/23: Action Adventures
Pyramid #3/30: Spaceships Pyramid #3/22: Banestorm
Pyramid #3/50: Dungeon Fantasy
Pyramid #3/10: Crime and Grime
Pyramid #3/13: Thaumatology Pyramid #3/08: Cliffhangers
Pyramid #3/59: Conspiracies Pyramid #3/09: Space Opera

The blue ones appear in both “best” lists. The only issue common to the bottom of both of these analysis is poor, lamented Martial Arts.

Overall, though, it’s probably a better metric to look at deviations from the expected performance over time (the second list) than who rushes out to vote for anything (the first one).
But what is perhaps the best metric of all? Gotta be . . . 
Sales, Glorious Sales

You have to read the description, read the title, maybe read some reviews, and then run out and buy this issue on an individual basis (as opposed to a subscription) to count here, so there are some definite issues. If you see an issue that makes you so happy that you buy a subscription, well, that removes you from the running going forward.
Still, a straight-up plot of Sales versus Issue number (which is also a proxy for time) should tell us something here . . . 
The first thing, just eyeballing it, is that it takes something like 6-12 months (really 6-9, I think) for an issue to saturate. After that time, you see what’s popular, and what’s not. But a simple sort on sales is what’s going to tell the tale here.
But, before we do that . . . surely the number of sales is highest for the best-rated articles!
Pretty meaningless. In fact, rating is so uncorrelated with sales that I’ll just go ahead and call those ratings utterly pointless. I noted before that other than those first six issues, you basically get an extra vote per 23 sales (dun dun DUN!!!!!).
So all of that noise later, and what do we get? 
Best 10
Issues: Total Sales
Worst 10 issues: Total sales
Pyramid #3/34: Alternate GURPS Pyramid #3/14: Martial Arts
Pyramid #3/33: Low-Tech Pyramid #3/38: The Power of Myth
Pyramid #3/13: Thaumatology Pyramid #3/27: Monsters in Space
Pyramid #3/12: Tech and Toys Pyramid #3/17: Modern Exploration
Pyramid #3/07: Urban Fantasy Pyramid #3/23: Action Adventures
Pyramid #3/44: Alternate GURPS II Pyramid #3/42: Noir
Pyramid #3/15: Transhuman Space Pyramid #3/05: Horror & Spies
Pyramid #3/25: Epic Magic Pyramid #3/49: World-Hopping
Pyramid #3/21: Cyberpunk Pyramid #3/32: Fears of Days Past
Pyramid #3/28: Thaumatology II Pyramid #3/02: Looks Like a Job for . . . Superheroes

Which issues were taken out of the running (and how many sales)?

Pyramid #3/56:
Prehistory (165)
Pyramid #3/57: Gunplay (215)
Pyramid #3/58: Urban Fantasy II (167)
Pyramid #3/59: Conspiracies (121)
Pyramid #3/60: Dungeon Fantasy III (155)
Pyramid #3/61: Way of the Warrior (95)

So none of these break into the top 10 (yet), which starts at 325 sales. Two are “dead last,” but Gunplay has already climbed out of the 10 Lowest category. 

Ballistic’s Report

What’s the point of all this? Mainly, it was hacking at the individual sales data to see if anything popped out. Of the lists provided, the Alternate GURPS and Thaumatology titles, plus Low-Tech, showed up repeatedly.
On the other hand, five of the best-selling issues were not in either list: Tech and Toys, Urban Fantasy, Transhuman Space, Epic Magic, and Cyberpunk. Note that two of these are probably maigc-oid (Urban Fantasy and Epic Magic), while the rest are sci-fi and (especially) gear and flash.
Of the least well performing, Martial Arts, Monsters in Space, Action Adventures, Noir, World-Hopping, and Fears of Days Past showed up in multiple lists, while The Power of Myth, Modern Exploration, Horror and Spies, and Looks Like a Job for . .  Superheroes have sold very poorly but didn’t show up on any of the “normalized” lists.
Future Reviews
So, fishing around in the data, I’m going to do some whole-issue reviews. 
First, I’ll take those that sold poorly but would not have been “predicted” to do so by other metrics:
Pyramid #3/38: The Power of Myth
Pyramid #3/17: Modern Exploration
Pyramid #3/05: Horror and Spies
Pyramid #3/02: Looks Like a Job for . .  Superheroes

Then, what about the surprising (perhaps) successes?

Pyramid #3/12: Tech and Toys
Pyramid #3/07: Urban Fantasy
Pyramid #3/15: Transhuman Space
Pyramid #3/25: Epic Magic
Pyramid #3/21: Cyberpunk
It should come as no surprise that the issues that do well are Alternate GURPS (nifty rules and explorations in a mechanics-heavy generic system), and fantasy, low-tech, gear, and magic. The best-selling genre and ifs offshoots.
More surprising are the issues that have done poorly. Perhaps there are some traps to avoid in looking at those.
Mostly, though, I take from this that the Customer Rating and number of votes are mostly meaningless numbers. We need a better way to dig into issues and articles and understand why they succeed and fail, and these quantities (at the level they’re reported) aren’t it.
Finally, one interesting last-minute thought: The low ratings of the first year might be related as much to “it’s not Volume 2!” as anything particular to the first twelve issues. “Not what I’m familiar with” is a real effect.

7 thoughts on “Pyramid Volume 3: By the numbers

  1. I think one reason for the disconnect between sales and ratings is that a lot of GURPS players tend to be completists and they often buy everything that comes out whether they'll use it or not. That said I do find the newer Pyramid articles to be a lot better than the earlier ones. I think it took a while for the new format to hit its stride and for both writers and editors to get comfortable with the new format. I do miss the little "game idea" blurbs at the bottom of every page in the earlier issues.

    1. You might be right, but I'd honestly expect the completists to buy a 12-month pyramid subscription, which would potentially add to ratings and votes, but not the individual issue sales I'm tracking here.

  2. The purple lines disturb me the most because it's a bit Weirdness Magnet how many people I've talked/fought with about GURPS online who turn out to be from Minnesota. At least it gives me something to think about until the streets get plowed in April.

    1. I'm originally from NY/NJ, so my counter to passive aggressiveness is to hit people in the face with a brick. It translates well enough into text. 😛

  3. I did a weighted regression of the ratings, turning the 5-point rating into a percentage of 5 so I could get a regression coefficient. (The ratings are barely significant, r^2=0.14.) The top 10, after this regression of 162 votes, becomes:

    Pyramid #3/60: Dungeon Fantasy III
    Pyramid #3/33: Low-Tech
    Pyramid #3/34: Alternate GURPS
    Pyramid #3/52: Low-Tech II
    Pyramid #3/31: Monster Hunters
    Pyramid #3/54: Social Engineering
    Pyramid #3/30: Spaceships
    Pyramid #3/50: Dungeon Fantasy II
    Pyramid #3/44: Alternate GURPS II
    Pyramid #3/59: Conspiracies

    The bottom ten becomes:

    Pyramid #3/07: Urban Fantasy
    Pyramid #3/10: Crime and Grime
    Pyramid #3/09: Space Opera
    Pyramid #3/04: Magic on the Battlefield
    Pyramid #3/08: Cliffhangers
    Pyramid #3/06: Space Colony Alpha
    Pyramid #3/05: Horror & Spies
    Pyramid #3/01: Tools of the Trade – Wizards
    Pyramid #3/03: Venturing Into the Badlands: Post-Apocalypse
    Pyramid #3/02: Looks Like a Job for . . . Superheroes

    Again, all of this is barely significant, so the ratings aren't all that handy for quality. The biggest outliers are the bottom two, issues 2 and 3; the new format was still looking for its footing in those days; the r^2 between issue number and adjusted rating is 0.32, which is definitely significant. The average issue number of the top 20 is 40.35, of the bottom 20 is 15.6. Based on that, the biggest outlier of the top 20 is #13, Thaumatology, which comes in at 16th. The biggest outlier of the bottom 20 is Prehistory, issue #56, which is 45th. Thaumatology III, issue #43, ranks 48th, so things don't always get better.

Leave a Reply

Your email address will not be published. Required fields are marked *