Four mistakes our own scores made, and the measurement that caught each

Popularity and difficulty are estimates. In September we checked their distributions against what they should look like and found four errors. Here they are, with the numbers.

Why write this down

Google Play publishes no search volume and no difficulty. Every popularity and difficulty score, ours included, is built from signals: autocomplete, the result pages, the apps already ranking. An estimate cannot be checked against the truth, because the truth is not published. It can be checked against its own shape: does it move the way the signals say it should, does it use its whole range, does it measure the market or our own data.

In September we ran those checks on production data. Each one found something. We fixed them, re-measured, and the changes are in the release notes. This post collects them in one place, because a score you are asked to trust should come with its failures.

1. Popularity rose with the number of words

Measured on 5 September over 1,712 keywords: the average popularity was 26 for one-word terms, 44 for four-word terms and 48 for five-word terms. A five-word phrase scored almost twice a single word. Autocomplete says the opposite: the longer the phrase, the more a user has to type before Play offers it, and the fewer people get there.

Three things added up. Seed coverage was divided by the length of the term, so the same seed gave a longer term more credit. Coverage ignored position, so being suggested fourth counted the same as being suggested first. And nothing measured word count at all.

• Coverage now depends only on the length of the seed that produced the suggestion, and is weighted by the suggestion's position.
• A long-tail reduction applies by word count, and it appears as its own line in the score detail, so you can see why a long phrase scored lower.

The reduction factors are chosen, not measured; Play gives nothing to measure them against. The direction is what the data supports.

2. Difficulty never went below 32

Measured on 10 September over 2,052 keywords: the lowest difficulty was 32 and 76% of keywords sat between 60 and 79. A user who filtered for difficulty 30 or less got an empty table. The scale had a hundred points and used about forty of them.

The cause was arithmetic. Each signal was divided by a fixed ceiling: 10 million ratings, a billion installs, a 5.0 rating. Real top tens are nowhere near those ceilings on the first two, and too close on the third: for 80% of keywords the average rating of the top ten was between 4.09 and 4.69, so the rating signal added roughly nine points to every keyword, easy or hard.

The fix normalises each signal to the range actually measured, rounded from the low and high ends of the distribution. On the same 2,052 keywords:

"Light" and "heavy" here are keywords whose top-ten median ratings count is under 100 and over 100,000. The order of keywords did not change; the scale spread out. The ceilings are still chosen from one market's distribution and will need a second look as other markets fill.

3. The top ten was not ten

Measured on 12 September over 2,242 keywords: 1,353 of them (60%) had more than ten apps in the set we called the top ten. The median was 11 and the largest was 24.

The latest crawl was being chosen per app instead of per keyword. An app that had dropped out of the top ten stayed in the set with its last row, and its weight kept counting toward difficulty and toward the median ratings count. After the fix the set is the keyword's latest crawl, positions one to ten. On the first night after (14 September) the average set was 9.96 apps and the largest 10.

The median difficulty did not move. The keywords that fell by more than five points had carried an average of 12.65 apps in their old set: the drop was exactly the weight that should not have been there.

4. A low difficulty was measuring our data, not the market

Difficulty is built from the top ten apps' ratings counts, and a ratings count is known only once that app's full listing has been read. When we compared difficulty bands with how much of that data we actually had, the lowest band had a ratings count for none of its top-ten apps, and the 60 to 69 band for 79% of them. The score was low where we knew little. The hardest term in a category could look easy.

Difficulty, gap and opportunity are now shown only when enough of the top ten is known, and the panel shows a blank otherwise. A blank does not send anyone after the hardest term in the category; a small number built from missing data does. The missing listings are fetched right after a keyword is measured, so the blank usually fills within minutes.

What stays

• Every score is labelled an estimate, because it is one.
• Unknown is shown as blank, never as zero.
• Adjustments that change a score appear as their own line in its detail.
• After a formula change, the whole distribution is compared before and after, not a handful of examples.

Where the numbers come from

ersASO's production database, on the dates given in each section: the keywords our users track or discovered, almost all of them in a single store and language. They describe our scores on those keywords, not Google Play as a whole.

Related guide: Reading popularity and difficulty without fooling yourself