Improve Lambda sample application #205
Conversation
*Issue #, if available:* N/A. *Description of changes:* - Majority of the Python code has been put into its own module. - The Python code has been divided up into multiple functions. - The sample application directory has been restructured, with a `notebooks/` directory containing `demo.ipynb` and `src/lambda_sample/` containing the Python code for creating the Lambda function, creating the Amazon Managed Grafana workspace, and generating time series data. - Each data generator now generates its own Grafana dashboard, customized to fit its data. - The permissions the Lambda function uses to write to Timestream for LiveAnalytics is now more specific. - Instead of being a string, the code for the Lambda function is now in `lambda_function.py`. - Retries with exponential backoffs are now used when creating the Lambda's IAM role and sending data to the Lambda function, as the Lambda function can take a while to become operational. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
| prepared_records.append(prepared_record) | ||
|
|
||
| # Write to Timestream using the `write_records` API | ||
| response = timestream_client.write_records( |
There was a problem hiding this comment.
Would be better to have an alternate strategy to ingest records via BatchLoad (creating csv files, copy the generated file and data model to S3 and create BatchLoad task ) - https://docs.aws.amazon.com/timestream/latest/developerguide/code-samples.create-batch-load.html
| else: | ||
| raise e | ||
|
|
||
| def lambda_handler(event, context): |
There was a problem hiding this comment.
Post ingestion, can this lambda ingestor also run a Timestream Query to report back results of few queries like
1/ Table metadata -> "DESCRIBE db.table" https://docs.aws.amazon.com/timestream/latest/developerguide/supported-sql-constructs.DESCRIBE.html
2/ few statistics - SELECT MIN(time), MAX(time), count(*) from db.table
|
|
||
| try: | ||
| # Create table if it does not exist | ||
| timestream_client.create_table( |
There was a problem hiding this comment.
Let us also make sure that a customer defined partition key is define on newly created table - https://docs.aws.amazon.com/timestream/latest/developerguide/customer-defined-partition-keys.html
|
|
||
| def send_request(session: session, method: str, function_url: str, payload: dict, precision="MILLISECONDS"): | ||
| """ | ||
| Sends a single request to the Lambda function. |
There was a problem hiding this comment.
can comment state what kind of request with an example?
| MAX_WAIT_SECONDS = 900 # 15 minutes | ||
|
|
||
| def create_lambda(session: session, lambda_name: str, database_name: str, table_name: str, role_name: str, | ||
| mem_store_retention_period_in_hours=12, mag_store_retention_period_in_days=3653, batch_size=100) -> str: |
There was a problem hiding this comment.
can we align these defaults with TImestream console's "Create Table"'s defaults?
| """Create Timestream database and table if they do not exist.""" | ||
| try: | ||
| # Create database if it does not exist | ||
| timestream_client.create_database(DatabaseName=DATABASE_NAME) |
There was a problem hiding this comment.
create a database with customer managed key?
| @@ -0,0 +1,93 @@ | |||
| from .data_generator import DataGenerator | |||
|
|
|||
| class AirQualityDataGenerator(DataGenerator): | |||
There was a problem hiding this comment.
Can you also add unit test cases along with this change?
| @@ -21,6 +21,7 @@ The following diagram depicts the deployed Lambda function receiving generated d | |||
| ## Prerequisites | |||
There was a problem hiding this comment.
This is a data generation utility and wondering why it is names "lambda" integration?. pls rename the project folder and move it under /tools
Issue #, if available:
N/A.
Description of changes:
notebooks/directory containingdemo.ipynbandsrc/lambda_sample/containing the Python code for creating the Lambda function, creating the Amazon Managed Grafana workspace, and generating time series data.lambda_function.py.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.