Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use crawlers' crawlctl for automating metadata extraction from locally staged products #21

Closed
lewismc opened this issue Oct 4, 2018 · 1 comment
Assignees
Labels
crawler OODT Crawler Component help wanted
Milestone

Comments

@lewismc
Copy link
Member

lewismc commented Oct 4, 2018

As discussed on todays call, once #8 is addressed, we should automate invocation of crawler using the crawlctl.
The idea here is for products to be sent to data/staging for them to automatically be detected, for metadata extraction to kick off followed by ingestion into the file manager.

@lewismc lewismc added help wanted crawler OODT Crawler Component labels Oct 4, 2018
@lewismc lewismc added this to the 0.1 milestone Oct 4, 2018
@lewismc
Copy link
Member Author

lewismc commented Oct 19, 2018

This file can now be seen at https://github.com/capstone-coal/coal-sds/blob/master/crawler/src/main/resources/bin/crawlctl
Essentially we ensure that the Crawler runs as a daemon, checking the local directory every 2 seconds and deleting the original staging products upon successful ingest. Additionally, the products as then archived as well as ingested into the File Manager.

@lewismc lewismc closed this as completed Oct 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
crawler OODT Crawler Component help wanted
Projects
None yet
Development

No branches or pull requests

2 participants