Investigate what would be needed to include crawl-sites visualization #146
Comments
|
Here's an example of the output for other following along: http://lintool.github.io/warcbase/vis/crawl-sites/. I ran this on all the WALK collections, FWIW, and was able to do the full thing in a few minutes on a laptop if I remember correctly. Here's one of our 4-5TB ones: https://web-archive-group.github.io/WALK-CrawlVis/crawl-sites/ALBERTA_government_information_all_urls.html. |
|
FYI I dug back into our past workflow and am glad I did as it's a bit janky. Here's the latest workflow I was using to do this. https://github.com/web-archive-group/WALK-CrawlVis/blob/master/WORKFLOW.md Note that the major problem is the output from the domain count is different than what |
|
If we add an additional spark sub-job: We'll get output like this: Then we'll probably need to adapt After that, it'd just be following the path of the Sigmajs visualization for this implementation. |
|
This sounds promising! I'd defer to you on the implementation, but creating this file and then possibly adding it to the clean-up job is a good route forward? |
- Adds additional Spark sub-job to extract info for crawl-viz - Pre-processes crawl-viz output - This is ugly
ruebot
added
Background jobs
feature
in progress
Rails
and removed
discussion
question
labels
|
Easy part done. Now I have to port |
|
To get something out of the door here, and make sure we can work with existing data, what if we took what powers the top 10 domains table, and make a column chart out of that? That should be pretty straightforward with Chartkick. |
|
Then close the door for now on the legacy bar chart since we'd have to create a new Spark job, and update the legacy d3.js work. If there's a lot of demand for this data from Spark in the future from users, we can add the data, and possibly integrate something like this in the user interface, or maybe the notebooks. |
|
That sounds like a great idea! And yeah, maybe make it a query point in the future to see if people would really like this. We could consider changing the domain script to generate not just absolute domain frequency but domain frequency by year (that would be straightforward), and then rig something up in the Jupyter notebook. |
👍 |
|
Which yeah, as noted in your comment above, would be a simple tweak - just changing this line to (even simpler) Maybe that's worth doing anyways, which could give the domain notebook some heavier lifting to do – and in any case, make the domain derivative more useful? It'd get at the spirit of this issue, I think. |
|
Yeah, it'd definitely get to the issue, but my biggest worry with any changes to it is backwards compatibility on the 172T we've already analyzed. We'd probably just have to wrap things in a method that reads the first line of the derivative file to see if there is a date or not. If there is, continue with any we do with it. If not, break. If we're cool with that, I can update the Spark job in the work I'll do with Chartkick on this issue. |
|
Aye – that is persnickety, on both the AUK and Jupyter fronts (more the former than the latter, I think). Would Chartkick read the month/domain frequency data easily? Or would it work better with straightforward domain frequency count as is? I'm happy for you to make the final call if this is worth the effort or not, as you have a far better sense than I do. |
To explicitly mimic the d3 chart @lintool did, Not that I can see. Chartkick looks to be really great for charts with straightforward data (with my brief experience with it).
As is would be pretty easy, and give a much better look at the data than a table imo. Better? Not sure. It'd be a good question to pose to users sometime. |
|
@ianmilligan1 @SamFritz @edsu (since this was related to the datathon feedback), let me know what you think (I'm pulling the top 25 domains if they're there):
|
|
Looking great! Can't wait to see this rolled out! Is something funky happening with the labels – the columns don't always have labels, maybe when there are too many of them? And in the FCSIC screenshot above, the most frequent domains are in the middle of the graph as opposed to at the left. In general, they seem to be declining frequency left -> right? |
|
This looks great @ruebot! :) The only thing noticeable was described by @ianmilligan1 ^^ |
|
Cool. I'll work on sorting them, and see if I can get the label to display for all of them. |
|
Here we go. Let me know what you think (especially, do you want largest on left or right?)
|
|
Looking great! I would probably prefer the larger on left if possible? |
|
Nice! Since we read left to right, I think it would make sense to read the chart big (left) to small (right) |
- Update display_domains helper for chart data - Create controller methods for domains chart data feed - Add route for feed - Update Rubocop config
|
Updated production. Poke around and let me know what you think. |
|
Looks great to me! 🎉
Close issue for the time being or keep open? |
|
We gotta sort out the other bit with the additional Spark job, then we can close it. |
|
Gotta fix this formatting issue on the big collections where GraphPass won't run:
|
Sounds good - as discussed in Slack, I think, let’s chat about that on our next team call! And good catch on that formatting issue |
|
Fixed. Added a "What does this graph show?" under the bar chart. The tooltip reads: "This diagram Let me know if that text should change.
|
|
Good call on the "What does this graph show?" – text looks perfect to me, @ruebot ! |
See what is needed to add crawl-sites.
graph.jsThe text was updated successfully, but these errors were encountered: