Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for an ingest pipeline and resource file loading for configuration. #51

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

relenteny
Copy link

I've added support to have documents run through an ingestion pipeline at indexing time. My use case is to add metadata when creating the meters and parse that metadata into additional fields at index time. An example being:

The original meter name field may look like this:

field1:value1|field2:value2|field3:value3

At index time, this metadata is processed into specific fields. Fo example:

...
"attributes": {
"field1": "value1",
"field2": "value2",
"field3": "value3"
}
...

In addition, for meter index and pipeline configuration, I added support for loading these at startup time.

Note: This code does work. Unfortunately, even though they're written, I cannot get the pipeline definition and the script loading unit tests to work. I debugged into the test code, and the Elasticsearch API states that script support has not been enabled. In looking through the various test sources, I was not able to see how add this support in a straightforward manner.

Note 2: Before I started my work, I manually copied/merged the files from @mfranklin's pull request (48) in order to keep the 5.x additions from getting out of hand from a merge perspective.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant