[AWS] Add Kinesis Metrics datastream

[legoguy1000]

What does this PR do?

Add AWS Kinesis Metrics Datastream

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Requires Add ECS metricset and fix various visualizations #3799
• Closes [AWS] migrate AWS Kinesis module from beats to integrations  #3822

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-07-27T19:19:34.523+0000

• Duration: 34 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	346
Skipped	0
Total	346

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[kaiyan-sheng]

/test

[kaiyan-sheng]

@legoguy1000 Thank you for migrating this data stream!! One quick question: I think we have a dashboard for kinesis metrics in Metricbeat. Can we also move the dashboard over? Thank you!

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (11/11)	💚
Files	91.667% (11/12)	👎 -5.465
Classes	91.667% (11/12)	👎 -5.465
Methods	82.243% (176/214)	👎 -7.018
Lines	92.894% (2157/2322)	👍 2.299
Conditionals	100.0% (0/0)	💚

[legoguy1000]

@kaiyan-sheng I'll resolve the conflicts here when #3799 is merged.

[kaiyan-sheng]

/test

[kaiyan-sheng]

I think the sample_event.json file needs to be updated. But I can help regenerate that in a separate PR.

[legoguy1000]

Sounds good.

…

On Wed, Jul 27, 2022 at 2:20 PM kaiyan-sheng ***@***.***> wrote: ***@***.**** commented on this pull request. I think the sample_event.json file needs to be updated. But I can help regenerate that in a separate PR. — Reply to this email directly, view it on GitHub <#3829 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEEL4EJIUE6CT26RW5NSN3VWGDXVANCNFSM54S5XLVA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[drewdaemon]

I'm sorry that I wasn't able to review this sooner. Thank you for using Lens in your dashboard! A couple points about dashboard best practices:

• "Library visualizations" (aka "by-reference" visualizations) should generally be avoided. These are added to the user's "Visualize library" in Kibana instead of being limited to just the dashboard. You can tell that a visualization is "by-reference" because a little file icon will show up at the top right of the panel. It's easy to change these to by-value by clicking the unlink button. This issue should be resolved this particular dashboard by Inline all aws dashboards #3688

• Dashboard-native controls should be used instead of the deprecated "input controls" visualization type. Here is the documentation for dashboard-native controls.

This is what the old stuff looks like:

This is what the new stuff looks like:

This information is all available in the best practices document.

[legoguy1000]

Good to know for the future. These visualizations/dashboards were just the Metricbeat items copied straight from the beats repo.

…

On Thu, Jul 28, 2022 at 12:28 PM Andrew Tate ***@***.***> wrote: ***@***.**** commented on this pull request. I'm sorry that I wasn't able to review this sooner. Thank you for using Lens in your dashboard! A couple points about dashboard best practices: - "Library visualizations" (aka "by-reference" visualizations) should generally be avoided. These are added to the user's "Visualize library" in Kibana instead of being limited to just the dashboard. You can tell that a visualization is "by-reference" because a little file icon will show up at the top right of the panel. It's easy to change these to *by-value* by clicking the unlink button. This issue should be resolved this particular dashboard by #3688 <#3688> [image: Screen Shot 2022-07-28 at 12 11 32 PM] <https://user-images.githubusercontent.com/315764/181598593-ccefa8ef-2179-4365-a8f0-5d98429c1902.png> - Dashboard-native controls should be used instead of the deprecated "input controls" visualization type. This is what the old stuff looks like: [image: Screen Shot 2022-07-28 at 12 21 23 PM] <https://user-images.githubusercontent.com/315764/181599153-011c4580-46ec-45c9-96d6-252054996671.png> This is what the new stuff looks like: [image: Screen Shot 2022-07-28 at 12 26 31 PM] <https://user-images.githubusercontent.com/315764/181600120-fe60d2fc-e6f3-429d-97d9-e57342ca219e.png> This information is all available in the best practices document <https://docs.google.com/document/d/1uyyFGx6xA5Kvl8c-ZdvXdvBGrHTylxU9F69TGqfzdmw/edit#> . — Reply to this email directly, view it on GitHub <#3829 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEEL4E3RBDHQLDQJPJOISTVWK7NBANCNFSM54S5XLVA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[drewdaemon]

@legoguy1000 I totally get it. It's what I would do, too. It's sort of like the power of defaults.

We're pushing best practices in integrations because our customers often follow exactly the same workflow of copy-modify-publish with the integration dashboards. It's leverage to get customers using the best possible tools.

[legoguy1000]

Absolutely, totally agree!

…

On Thu, Jul 28, 2022 at 12:51 PM Andrew Tate ***@***.***> wrote: @legoguy1000 <https://github.com/legoguy1000> I totally get it. It's what I would do, too. It's sort of like the power of defaults. We're pushing best practices in the integration dashboards because our customers often follow exactly the same workflow of copy-modify-publish. It's leverage to get customers using the best possible tools. — Reply to this email directly, view it on GitHub <#3829 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEEL4BP42S3YVBIF5VDA5LVWLCA7ANCNFSM54S5XLVA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[legoguy1000]

Will do. I completely spaced on teh Visualizations.

[ruflin]

@jsoriano See above conversation and also in #4064 around by value / by reference. It would be great if there is an automated way to make our community contributors aware of these things to make their life easier. I remember we had a discussion around potentially bringing this in as a CI check or similar but couldn't find it anymore.

[jsoriano]

@jsoriano See above conversation and also in #4064 around by value / by reference. It would be great if there is an automated way to make our community contributors aware of these things to make their life easier. I remember we had a discussion around potentially bringing this in as a CI check or similar but couldn't find it anymore.

@ruflin a warning was added for this in elastic/package-spec#389. If warnings are not enough we may need to think in other strategies, or make this an error in CI.

[ruflin]

@jsoriano To see this warning, someone needs to open the CI logs? @legoguy1000 Did you ever stumble over these?

@jsoriano The more intrusive way would be with a comment as part of the coverage report or one of the other automatic comments from CI.

[jsoriano]

@jsoriano To see this warning, someone needs to open the CI logs? @legoguy1000 Did you ever stumble over these?

These warnings also appear when running elastic-package locally. This check was recently implemented, after this PR was merged, so I guess it didn't appear in this case.

@jsoriano The more intrusive way would be with a comment as part of the coverage report or one of the other automatic comments from CI.

Yeah, it'd be good if we had more easily consumable warnings, I have added a mention to this in elastic/package-spec#342.

[legoguy1000]

@jsoriano To see this warning, someone needs to open the CI logs? @legoguy1000 Did you ever stumble over these?

@jsoriano The more intrusive way would be with a comment as part of the coverage report or one of the other automatic comments from CI.

You're talking about having visualizations embedded in the dashboards instead of separate files that are referenced? If so I can't remember that for this PR but I've definitely seen it lately and I'll be honest for the dashboards that I've copied from existing beats modules I haven't really done anything with them besides copy and update the fields.