Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove extras_rollup #3922

Open
1 task
jbrown-xentity opened this issue Aug 10, 2022 · 2 comments
Open
1 task

Remove extras_rollup #3922

jbrown-xentity opened this issue Aug 10, 2022 · 2 comments

Comments

@jbrown-xentity
Copy link
Contributor

User Story

In order to have a simple and clean backend, data.gov admins wants all extras_rollup data removed.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

  • GIVEN a dataset exists in catalog
    WHEN the data is queried in any form (SOLR, DB, etc)
    THEN the data is in a standard CKAN format of key fields and extras

Background

The fact that all extras are stored as extras_rollup caused a bug when integrating GSA/ckanext-datajson#115, and required this mind-bending PR that was only able to be tested once integrated with ckanext-geodatagov on catalog: GSA/ckanext-datajson#120.

Security Considerations (required)

None

Sketch

We believe the only location of this is in ckanext-geodatagov: https://github.com/GSA/ckanext-geodatagov/search?q=extras_rollup
We could consider implementing this in a much cleaner format, by utilizing the IDatasetForm: http://docs.ckan.org/en/2.9/extensions/adding-custom-fields.html. This would require extensive testing however, and evaluating how our other extensions parse data. Essentially I believe this could eliminate "extras", and simplify our responses via API's and make for a cleaner/simpler backend and frontend. It could also be useful in "standardizing" data, such as the fact that we have a "geospatial unique identifier" and a "datajson unique identifier" that are separate. They serve the same function and should/could be combined in CKAN, and it makes it hard for users and us to parse and requires knowing 2 different standards. See datajson example and geospatial example.

@hkdctol
Copy link
Contributor

hkdctol commented Dec 1, 2022

This will either be part of cleanup of harvester problems or be irrelevant if we adopt an entirely new approach.

@jbrown-xentity
Copy link
Contributor Author

Related to GSA/ckanext-geodatagov#178

@hkdctol hkdctol moved this to 🧊 Icebox in data.gov team board May 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🧊 Icebox
Development

No branches or pull requests

2 participants