No updates on the dashboards since Jan 19

[FatOrangutan]

https://data.firefox.com/dashboard/user-activity or https://data.firefox.com/dashboard/usage-behavior - latest available data is from about 3 weeks back.

[openjck]

Thank you for reporting this.

@fbertsch, do you know what might be happening here? It looks like the raw data is stale.

[fbertsch]

The data stopped updating because of the test-pilot deprecation, which we dealt with in mozilla/Fx_Usage_Report@5dff993.

Currently backfilling the data.

Thanks for the bug report @FatOrangutan!

[pdehaan]

Is there a way to tell if the data is stale (apart from hovering over a chart and seeing the last updated date was Jan 18)?

Or, if it's easier to scrape one of the data endpoints and see what the latest x-axis value is: https://data.firefox.com/datasets/desktop/user-activity/Worldwide/YAU

I'm wondering if we should try adding some test on a cron job that scrapes the data.firefox.com site and makes sure the data is no older than 8 days.

[pdehaan]

Here's a cheap way of checking the last modified date (at least for YAU).

const axios = require("axios");
const ms = require("ms");

async function main(uri) {
  const res = await axios.get(uri);
  const latestDate = new Date(res.data.data.populations.default.pop().x);
  console.log(ms(latestDate - Date.now()));
}

main("https://data.firefox.com/datasets/desktop/user-activity/Worldwide/YAU"); // -24d

Not sure if there is a better way (such as including the last modified date in the payload). Interestingly the date and last-modified dates in the response headers showed today's date and not the time the upstream data was last updated.

date: Mon, 11 Feb 2019 20:05:12 GMT
last-modified: Mon, 11 Feb 2019 19:38:35 GMT

---

UPDATE: Converted the snippet above into a repo at https://github.com/pdehaan/ensemble-data-test

[openjck]

That's a good question. I would love to have some kind of alerting when the data is stale. Frank, do you think it would make more sense for us to do that testing here or in the FX_Usage_Report repo?

[openjck]

I can see the data has been updated. This issue is closed, but I'd still like to get alerts when the data is stale since this has happened a couple of times already. I opened issue #303 for that.

[pdehaan]

Interestingly, it looks like the User Activity and Usage Behavior dashboards are 6 days old, but the Hardware dashboard is 12 days old.

[-6d] https://data.firefox.com/datasets/desktop/user-activity
[-6d] https://data.firefox.com/datasets/desktop/usage-behavior
[-12d] https://data.firefox.com/datasets/desktop/hardware

Not sure if that is expected or not, but I found it interesting.

const axios = require("axios");
const ms = require("ms");

const dashboards = [
  "https://data.firefox.com/datasets/desktop/user-activity",
  "https://data.firefox.com/datasets/desktop/usage-behavior",
  "https://data.firefox.com/datasets/desktop/hardware"
];

main(dashboards);

async function main(dashboards) {
  for (const dashboardUrl of dashboards) {
    const dashboardAge = await lastModified(dashboardUrl);
    const ageNum = parseInt(dashboardAge, 10);
    console.log(`[${dashboardAge}] ${dashboardUrl}`);
    if (ageNum < -7) {
      process.exitCode = 1;
    }
  }
}

async function lastModified(uri) {
  const { data } = await axios.get(uri);
  const latestDate = new Date(data.dates[0]);
  return ms(latestDate - Date.now());
}

[openjck]

Not sure why hardware would be older than the other two, although I see that that's still the case. I'll ask @fbertsch in Slack.

[pdehaan]

I wrapped this in a basic Express app and uploaded it to https://ensemble-last-modified.now.sh/
(source at https://github.com/pdehaan/ensemble-last-modified)