Allow setting up more than one initContainer per platform #465
Comments
github-actions
bot
added
the
status/need-triage
github-actions
bot
added
the
status/need-triage
corneil
added a commit
to corneil/spring-cloud-deployer
that referenced
this issue
Jul 9, 2024
corneil
added a commit
to corneil/spring-cloud-dataflow
that referenced
this issue
Jul 9, 2024
corneil
added
area/kubernetes
status/in-progress
cppwfs
pushed a commit
to cppwfs/spring-cloud-dataflow
that referenced
this issue
Aug 23, 2024
…ng-cloud-deployer#465 [skip ci] Update paragraph for multiple Init Containers. Fix mariadb mount to be /var/lib/mysql (spring-cloud#5875) Fixes spring-cloud#5877 Add spaces after columns for clarity. This is done during merge
cppwfs
pushed a commit
to cppwfs/spring-cloud-dataflow
that referenced
this issue
Aug 23, 2024
…ng-cloud-deployer#465 [skip ci] Update paragraph for multiple Init Containers. Fix mariadb mount to be /var/lib/mysql (spring-cloud#5875) Fixes spring-cloud#5877 Add spaces after columns for clarity. This is done during merge
corneil
added a commit
to spring-cloud/spring-cloud-dataflow
that referenced
this issue
Oct 2, 2024
* Update build files for main-3 [skip-ci] * Update SCDF to use Boot 3.2.2 * Jakarta updates * Updated for Logging Date Format * Replace spring.factories with autoconfig imports * Constructor Bindings removed * Readd the yaml files that caused the migrator to fail * Update SCDF to Boot 3.2.2 as parent * Updated dependencies where needed * Added versions to some dependencies where versions are no longer in the bom * Updated hibernate usages where class names changed * Got to spring-cloud-common-security-config-web. ** The OauthSecurityCOnfiguration has @ConditionalOnMissingBean for the WebSecurityAdapter. ** Which is no longer available. We need to determine what should go here. * Update workflows to use Java 17 * Make SCDF and Skipper compilable for boot 3.x * Replaced SocketUtils with TestSocketUtils ** It was moved to the test package. * Migrated httpclient to httpclient5 ** Removed use of httpclient 4.x dependencies added yesterday * Updated Types in AuditRecord Entity to use JdbcTypeCode * Security Modules need to be compiled with Boot3 and securty 6 * Update SCDF to use deployer 3.0.x * Update SCDF pom files to create Java 17 jars * Exclude javax.annotation from deployer artifacts. * Add jakarta annotation dependencies * Update code from javax to jakarta * Replaced Entity @type with @JavaTypeCode * Update JobParam to batch 5. * Update httpclient package to httpclient5 package * Update SCDF so that it can be compiled with Boot 3.2.x. Part 3 Added dataflow specific PageQueryProvider for missing functions Batch removed some methods that are still required by dataflow. Created dataflow version of those classes so that we can implement those methods Added hibernate version Checkpoint for server core and server Updated Skipper components Update classic docs to be restful doc 3.0 compliant Update code to resolve review comments Thank you @cbono!!!! One last polish * Remove dependency overrides from 2.11.x (Boot 2.x) This commit removes the following dependency overrides that were put in place to override earlier dependencies in Boot 2.x: - logback - jackson - snakeyaml Also, the joda-time version is updated to `2.12.7` * Remove duplicated maven-source-plugin from `spring-cloud-dataflow-common/**` Update Awaitility and Semver libs This commit removes the dependency management for `com.jayway.awaitility` in favor of the Spring Boot managed version at the new `org.awaitility` coordinates. Also, the Semver lib is updated to 0.10.2 * Remove <dependencyManagement> of `testcontainers-bom` Dependency management of `testcontainers-bom` is no longer needed as Spring Boot 3.x provides the version for the `testcontainers-bom`. * Remove remaining dependency override of snakeyaml. A previous commit removed the dependency overrides for snakeyaml in `spring-cloud-dataflow-parent` (which ~50% of the modules extend from). This commit removes the dependency overrides for snakeyaml in `spring-cloud-dataflow-build-dependencies` (the remaining 50% of modules extend from this). * Remove <dependencyManagement> of `junit-jupiter-bom` Dependency management of `junit-jupiter-bom` is no longer needed as Spring Boot 3.x provides the version for the `junit-jupiter-bom`. * Remove dependency override of json-smart. The version was previously overridden to `2.4.11` due to CVE. This is no longer needed as Spring Boot provides the version of `2.5.0`. * Remove <dependencyManagement> of `postgresql` Dependency management of `postgresql` is not needed as Spring Boot 3.x provides the version for `postgresql`. * Remove unused `spring.version` property * Update DockerCompose tests to Junit5 Also had to account for some changes in Mockito w/ varargs methods. Suggested changes from PR review * Use AutoConfiguration.imports for all auto-configs Fix from PR review * Add back missing nested @configuration annotations * Fix startup and EPP ordering (#5670) * Fix tests for rest-resource (SCDF3 migration) The new HTTP client requires access to the host:port. As such, we now use @SpringBootTest to launch web server to allow test to proceed. Also include H2 test dependency to deal w/ test startup failure. * Fix tests for rest-client (SCDF3 migration) This commit excludes DataSourceAutoConfiguration from the tests. * Update SpringDoc OpenAPI to latest 2.x This commit updates to Open API 2.x which is the version that supports Boot 3. The SpringDocs migration guide was followed: https://springdoc.org/#migrating-from-springdoc-v1 * Change vscode settings Add recommendation from vscode itself not that we're a monorepo lsp needs more memory. * Use config legacy processing in Skipper (#5673) * Refactor TaskLauncherSink to SCSt Consumer Prior to this commit the sink used a PollableMessageSource with a trigger to source its inbound messages. This commit replaces that mechanism w/ a simple SCSt consumer with a @StreamRetryTemplate. * Add Autoconfiguration.imports to resource includes in pom.xmls We had an issue where some of the auto configurations were not firing. This is because when a user specifies the resource tag in maven, the boot plugin expects the user to fill the resources manually vs. Its default behavior Also removed the version from the H2 dependency so that it can be managed by the bom Removed unnecessary @configuration annotaions where a @autoConfiguration annotation is present * Add metrics exclusions till metric migration is complete Currently SCDF fails to start with errors around metrics. These will need to be re-added when metric migration begins Remove debug settings from previous commit * Stub out DataflowPagingProvider to allow Dataflow to start (#5679) * Stub out DataflowPagingProviders to allow Dataflow to start * Provide PlatformTransactionManager to JobExplorer since Batch 5 no longer provides it * Compile with `-parameters` option * What was done For each database type the following migration was implemented: Prefix the TASK V2 and BATCH V4 tables with V2_ . This allows user to determine what they wish to do with this data Remove BOOT3_prefix for TASK V2 and BATCH V5 tables Make sure that these migrations were supported by flyway Make sure that the Migration SQL scripts were added to associated yaml files. What was affected The following databases were migrated: H2 MariaDB Mysql Postgres Oracle SQLServer DB2 Types of migrations There were 3 types of migrations that occurred. Default - These types of migrations were typically alter tables and alter sequences. The following databases belong to this group: a. MariaDB b. Mysql c. Postgres d. Oracle (with some exceptions) e. Sql Server (commands different than others, but principles remained). In-Memory - For this case the Task V2 and Batch V4 tables/sequences DDL was removed and BOOT3_ prefix was removed. The only in-memory we support is H2. DB2 - Gets its own category To rename the tables db2 requires all primary and foreign keys to be dropped. In this case we decided to create the new tables and copy the contents of the original tables to the new tables. Then remove the original. This was to avoid errors when recreating the keys How to eat this Elephant This PR is a bit large so let's discuss how we can handle this review. Let's review this by sampling one database from each type of migration. This is so that we can make sure the general pattern works for folks. Default type, look at Mariadb In Memory type look at H2 DB2 Type look at (well... ) DB2 :-D After we finish this sampling review, those changes will be applied to the other databases and then a full review can be made. * Update Tests to resolve failures due to updated dependency versions (#5680) This commit does not cover all the cases where updated dependencies broke existing tests. The main ones resolved here are some basics on Hibernate dialect versions, batch 5 updates, and some basic removal of BOOT3/BOOT2 additions in 2.11.x There are other updates that are causing alot of failures. Here are the issues and a brief description: * Currently Hibernate 6 does not create sequences out of the box. More can be read here: https://github.com/hibernate/hibernate-orm/blob/6.0/migration-guide.adoc#implicit-identifier-sequence-and-table-name One test that shows this error:TabOnTapCompletionProviderTests * Hibernate Dialect. Some tests require the hibernate dialects to be updated to hibernate 6. I did resolve the ones I found. But may have missed others. * Many tests are failing trying with the following exception: `No bean named 'mvcConversionService' available` For example: SpringDocAutoConfigurationTests.enabledWithCustomSpringDocSettings * The TODO excluding wavefront are causing some errors for example: Error creating bean with name 'management.wavefront-org.springframework.boot.actuate.autoconfigure.wavefront.WavefrontProperties': Lookup method resolution failed An Example can be found here: SpringDocIntegrationTests.disabledByDefault * Add skipper to vscode launch config * Use the single hibernate_sequence strategy Hibernate 6.0 now creates a sequence per entity hierarchy instead of a single hibernate_sequence. This commit sets the `hibernate.id.db_structure_naming_strategy` property to `single` to preserve the previous behavior of using a single hibernate_sequence. * Fix SpringDocAutoConfigurationTests This commit adds a mock "mvcConversionService" bean to the app context for the SpringDocAutoConfigurationTests as SpringDoc 2.x expects this bean to be available. This commit also updates the use of Mockito verify for varargs in the SpringDocAutoConfigurationTests. * Overhaul metric deps and options Align wavefront version so that we don't have misaligned versions coming out from other parts of a metric system. Short story is that boot doesn't manage wavefront but there is an explicit dependency to wavefront sdk libs in metric system. Rename metric options within management for influx, prometheus and wavefront to align changes in boot itself. For rsocket proxy keep old management.metrics.export.prometheus.rsocket.enabled as that is going to get moved under micrometer spesific namespace when they release boot3 support. This commit was supposed to get skipper server to start but there is a new issue about missing bean org.springframework.statemachine.data.jpa.JpaStateMachineRepository which will need to get fixed in a separate commit. Fixes #5675 * Disable all tests that are failling because of schema changes. (#5688) This commit marks tests that are failing due to schema changes w/ the `TODO: Boot3x followup` so that they can be re-enabled in the future once we update the code to handle the schema changes * Remove trailing slashes from controller tests (#5693) This commit accounts for the fact that as of Spring Framework 6.0, the trailing slash matching configuration option has been deprecated and its default value set to false. The mvc tests are adjusted to remove the trailing slash from their endpoint urls. * Add DatabaseAwareLobUserType for @lob columns Hibernate 6.0 removed support for string values in @type mappings in favor of specifying UserTypes. This commit creates a DatabaseAwareLobUserType that provides the same functionality as the previous DatabaseAwareLobType. Add header+copyright * Fix DefaultPackageReader for Snakeyaml 2.0 This commit allows the PackageMetadata.class to be loaded with Snakeyaml 2.0 * Remove unused spring-boot-loader dependency This commit removes the dependency on the newer spring-boot-loader from the spring-cloud-dataflow-container-registry module. With Spring Boot 3.x, you must choose only one of these loader options. Otherwise, duplicate non-compatible classes may be loaded - for example the ZipInflaterInputStream. * Added DockerHub login to CI to prevent failure of TestContainer tests. (#5698) Updated docker/login-action to v3 * Reuse CriteriaQuery when retrieving result count. (#5702) When migrating to the latest Hibernate we saw the following exception: ``` Caused by: java.lang.IllegalArgumentException: Already registered a copy: SqmBasicValuedSimplePath ``` This is because we were recreating a CriteriaQuery for our Queries. Hibernate no longer allows us to do that, but rather allows us to use the existing CriteriaQuery, but use the convenience method createCountQuery to provide the criteria query for our createQuery. Also removed 6.1.7 versionfor hibernate core that allows us to use the Boot Bom. This also caused some downstrem issues where dataflow was using the old version brought in from skipper * Update spring-statemachine to 4.0.0 (#5707) This should make skipper to start and run. Simple ticktock stream shouldwork ok. * Update Code to use JobRepository bean directly Since we will only support BOOT 3 we do not need the JobRepositoryContainer to retrieve BOOT3 or BOOT 2 based JobRepositories. Update test code to set the default time for the local date time sample to a default of midnight Allow services to use JobServie and JobExplorer directly Currently we use containers to allocate the JobService and JobExplorer based on boot version. This is no longer necessary. So this PR removes these containers Update Project as to restore tests success percentage to original state After the updates below tests that were passing started to fail. Some because of Batch 5 updates, but others because of the removal of some of the container classes. Update code based on code review comments Removed todo Removed ExtendsWith statement * Adjust metrics replicator for Spring Boot 3 This commit adjusts the MetricsReplicationEnvironmentPostProcessor to account for the Spring Boot 3 change in the metrics config props prefix scheme from 'management.metrics.export.<meter-registry>.<property-path>' to 'management.<meter-registry>.metrics.export.<property-path>'. Update from PR review * Update ci.yml free diskspace * Update prometheus-rsocket-proxy to 1.6.0-SNAPSHOT This commit updates the Prometheus RSocket proxy to 1.6.x which in turn is updated to Spring Boot 3.2.x. * Replace job execution and task batch containers with Task implementations Replace JobExecutionDaoContainer with JdbcSearchableJobExecutionDao. Replace TaskBatchDaoContainer with TaskBatchDao Replace JdbcTaskExecutionDao with the Dao from task. Add requested changes based on code review * Use DataflowTaskExecutionMetadataDao in the place of DataflowTaskExecutionMetadataDaoContainer Replace DataflowJobExecutionDaoContainer with dataflow's JobExecutionDao. Replace DataflowTaskExecutionMetadataDaoContainer with DAO from dataflow Replace TaskRepositoryContainer with TaskRepository Move TaskRepository and SCDFDao beans from AggregateTaskConf to TaskConf This is en-leu of the next phase where we begin to remove the aggreate configurations Polish based on code review * Adjust order of DefaultEnvironmentPostProcessor This commit adjusts the change of the order attribute made in commit de58247. The default EPP in fact needs to run after the ConfigDataEnvironmentPostProcessor and before the ConfigDataMissingEnvironmentPostProcessor. * Fix skipper_manifest.data column oid to text conversion that was missed. Fixes #5715 Cherry picked from 2.11.x * Migrate Task Components to Boot 3, Task 3 This is Phase 1 of migrate the Task and Job components to Boot3/Task3/Batch5 Its purpose was to focus on migrating as much Task related features over as possible. Status: * All Task only tests in Spring Cloud Data Flow Server Core are passing * Job related migrations have not started. Nor those that are Batch/Task related * Dataflow does start and a person can register a Task app and create a task definition. * Task launches are still failing but Chris is looking to resolve that due a dependency hiccup from deployer Noteable changes * Establish property that tells which platformTransactionManager Task should use * Update Task components to run Boot3 only components. * Remove Schema related code from Task Launches and exploration * Remove AggregateExecutionSupport from project and remove usages. * AggregateTaskExplorer is still present and debating renaming it and keeping it because of some of the features it offered. * Removed the AggregateTaskExecution Updated based on code review * Update Spring Cloud Deployer to 3.0.0-SNAPSHOT * Use CLASSIC boot loader This commit does the following: * Configures the Spring Boot maven plugin to use the legacy CLASSIC boot loader when launching/building uber jars * Simplifies the BootClassLoaderFactory by removing support for legacy Boot 1.3x jar format * Remove extra Spring Cloud Task transaction manager Prior to this commit there were 2 transaction managers defined. One for Dataflow and one for Spring Cloud Task. The former was marked @primary and it was causing confusion. This commit removes the task specific one and uses the same manager for both cases. This is possible by setting the 'spring.cloud.task.transaction-manager' property. Move the Spring Cloud Task txn mgr property This commit moves the SCT 'transaction-manager' property to the dataflow-server-defaults.yml default properties file. Add TransactionManagerCustomizationAutoConfiguration to DataFlowServerConfigurationTests contextRunner * Upgrade spring-shell 3.2.2 (#5722) This commit updates spring-shell to 3.2.2 and also does the following: * Exclude spring-cloud-dataflow-common-persistence as it caused jdbc stuff in shell and then failures with datasource autoconfig * Temporarily in tests use reflection as some methods in spring-shell are not public anymore. * Add commented out shell log file settings which is a way to log hard shell startup errors * Update Job to Batch 5 and remove schema usage * Removed SchemaController/Tests/Documentation * Removed the BatchVersion * Service calls used by JobInstanceController have been migrated to batch 5.x * Migrated SQL Statements in AggregateDataFlowTaskExecutionQueryDao to use Task Execution This is a first step to retire the use of the aggregates. * DefaultTaskExecutionServiceTests - since we are no longer actively passing table prefixes the properties tests were dropped by 5 * Also Removed BATCH_ and TASK_ tests * JobExecutionControllerTests and JobExecutionThinControllerTests also had similar changes * Some tests were either @disabled or the @test was commented out because the fixes need more investigation. * they were marked with todo * We also need to revisit the use of locals over timezones. The stop gap is to use the default locale. This was marked with todo. * Give SCDF ability to compile and run tests to completion * Disable tests that need more investigation * Refresh the TODO and Disable messages to reflect an accurate message of what is required * These changes do not include those for skipper * To compile to completion comment out spring-cloud-skipper module then execute mvn clean install Update code based on review * Update DAOs to no longer use aggregate Update tests to remove Schema Version checks * Update tests so that they no longer use aggregate views * Update SimpleJobService MariaDB and PostgresDB to match was is in 2.11 tests This was to resolve the sporadic connection pool problems Remove AggregateJobQueryDao * remove unused tests * Update SimpleServiceFactory to remove use to the aggregate code Rename AggregateTaskconfigruation/AggreageTaskExplorer to an accurate name Rename aggregate package to composite Remove uses of AppBootSchema and spring-cloud-dataflow-schema-core module Re-enable more tests Remove or update tests that are Boot3 specific * Remove schema-core module from scdf-core module * Added <argLine>--add-opens java.base/java.util=ALL-UNNAMED</argLine> to surefire-plugin and failsafe-plugin configurations. (#5753) * Move DataflowTaskExplorer and associated classes to server core (#5754) * Remove aggregate-task module * Rename CompositeTaskExplorer to DataflowTaskExplorer * Update versions for Skipper. Change rest docs usage. * Switch to non-blocking call in action - UpgradeDeployTargetAppsSucceedAction can't use old non-reactive api's as we can't block within sm execution. * Setup new apps for skipper tests - Old manifests were using very old app version which don't run on jdk17 so add new definitions which uses latest app versions able to run on modern jdk's. * Remove DataflowSqlPagingQueryUtils as it is unused * SCDF does not need to support JSR-352 as of Batch 5. Remove JobName search for JSR-352. Address requested code review changees * Resolve JobExecution Search by date test failure We occassional fail to get the last entry of the test results This is a verification if the test end time is too short * Add MAX_START_WAIT_TIME property to CTR Add documentation for CTR property MAX_START_WAIT_TIME. Add description so it would appear as a properties option in the UI Small polish * stream update should not require --properties (#5785) Currently a user can not set --propertiesFile option on a stream update command This fix allows users to use --properties or --properties file with a stream update command. Added tests to verify * Added TaskExecutionThinController and TaskExecutionThinResource. Update TaskTemplate to changes in 2.11.x to improve link validation and reporting. Remove andDo(print()) * Migrate SCDF-SINGLE-STEP-BATCH-JOB to Boot3 Batch 5 * Updated representation of start time and date given timezone. * Temporarily disable Skpr auto config tests till full migration * Version updates and cleanup on Skipper files. Switch to jlumbroso/free-disk-space remove docker conditional docker login find a way to get secrets in PR build Skip tests on spring-cloud-skipper-core Removed JMockit. * Update classic doc to remove boot 3and boot schema Remove the boot 3 and boot 2 schema changes Also used queryParam when necessary per Boot3 requirements. * A document test requires generateJumpToItemQuery that is not available Disabling the test until the generateJumpToItemQuery is available * DataflowSqlPagingQueryProvider (#5757) * Implement DataflowSqlPagingQueryProvider by copying implementations from Spring Batch 4.x `org.springframework.batch.item.database.support` Removed @disabled for tests that now function as expected. * Remove docker login from ci-pr.yml * Updated for requests * Ensure docker compose will still work if docker-compose is not present. * Update DB2 docker image uris. * Updated DockerComposeTests and DockerTests. * Upgrade spring-shell 3.2.4 * Provider for trying OCI accepts header when manifest result returns no config and schemaVersion less than 1. Issue #5819 Test for DefaultContainerImageMetadataResolver needs to be updated to Jupiter and Assert4j * Update to Spring Boot 3.3.0 * Update version of the apps metadata plugin used by the sample apps to get around issue of NoClassDefFound for spring-data-jdbc config props * Modify MySQL57Database to use newly introduced `ensureSupported` API * Replace use of `OAuth2ClientPropertiesRegistrationAdapter` with newly introduced `OAuth2ClientPropertiesMapper` * Update to Flyway 10.10.0 The DB2 and Postgresql support now lives in its own module. See https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#flyway-10 * Update various dependencies * Continue using prometheus 0.x client See https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#prometheus-client-1x * the document plugin needs to use the current spring javadoc When release SCDF be sure to update javadoc-spring.version to the proper spring version Updated based on code review request * User needs ability to specify app version when creating schedule This update allows user to specify version.label=<version.number> Tests were also updated because the original settings assumed that the appregistry was real instance instead of being mocked. Thus the find would always return null. And in this case the tests returned a false positive. Now that the mocks are in place it excercises all the code. Also added explicit test if user does not set the version number. Some of the tests do this, but wanted an explicit test to verify this. resolves #5705 * [CI] Remove unused labeler workflow (#5840) As part of the Spring Project assessment tool initiative we needed to clean up our unused labels as they have grown out of control. We did this manually in the Github UI. We will manage them in the UI going forward. (cherry picked from commit 2836b64) * Add trailing slash to dashboard (#5811) - Boot 3.x changed handling of trailing slash meaning it's now adviced to define both /dashboard and /dashboard/ in UiController * Add JDK 21 and remove 8, 11 from build-images. Remove JDK pre 17 from container tests. Add JDK 21 to container tests. Update DataflowOAuthIT to dump last error output from curl. Update JDK list for containers in AbstractDataflowTests. * Remove thinTaskExecutionList from TaskTemplate (#5842) ThinTaskExecutions are not a part of SCDF-3.0.x. During the migration some of the ThinTaskExecution URL calls were left in the rest client and this caused some test failures in IT. This PR removes the API access from the client for ThinTaskExecutions. * Migrate CTR to Boot 3.x and Batch 5.x (#5839) * Migrate CTR to Boot 3.3 and Batch 5 * Remove the Batch Configurer and replace with a Configuration * Update Tests so that they will work with Boot3 * Removed EnableBatchAutoConfiguration no longer needed if using BatchAutoConfiguration * Removed schema requirements * Re-enable CTR Module build in main pom.xml * Add ability for composed task runner to use the proper JobRepository and TaskExecutor * BeanPostProcessor has been added so that CTR can use its jobRepository vs. the one provided by BatchAutoConfiguration * Update to prometheus-rsocket-proxy 2.0.0-M1 (#5888) * Update Prometheus metrics prop names (Boot 3.x) * Update Influx metrics prop names (Boot 3.x) * Update metric prop names env vars (#5894) The previous commits for updating metric names to the Boot 3.x format missed some env vars. This commit updates the env vars as well. * Update metric prop names in yaml files (#5895) The previous commits for updating metric names to the Boot 3.x format missed some entries in yaml files. This commit updates the yaml files. * Update all module except spring-cloud-dataflow-common-test-docker to JUnit 5. Converted JUnit Asserts to AssertJ Remove all Hamcrest matchers except those required by mockMvc assertions. Disabled failing tests for further investigation. * Update gitignore for generated gradle files Atleast vscode and possibly other ide's create paths .github/workflows/download-jar/.gradle and src/add-deps/.gradle which should not go to git. * Add import order to vscode settings * Remove jdk8 from spring-cloud-dataflow-server May cause ide's to use wrong jdk as boot 3.x requires jdk17. * Need to select the serialization method for JobParameters for the commandline User needs ability to set the default serialization technique for SCDF when restarting a job User needs ability to select a serialization technique for a specific job restart When user restarts a job repository from the list that is derived from thinjobexecutions it should use default technique Add restful documentation. Add support to allow user to set useJsonJobParameters for job relaunch via the shell. Note: there are not tests for the shell update in this commit. This is because the current set of tests rely on @EnableDataflowServer which does not work. But before we fix @EnableDataflowServer we need to make sure we want to carry it forward per Issue #1040 Polish PR before push to repo JobCommand should ignore identifyingParameters when deserializing JobParameters This is a generated list and and will cause deserialization to fail if not skipped SCDF needs to support any type of class for JobParameters However, if the class is not a base java type or one provided by dataflow the user has to build dataflow with their class included. Add tests for JobParameterJacksonDeserializer Remove Disabled annotation from JobExecutionController tests that are no longer needed Update per code review request. Added test for JobParamterMixin via the JobTemplateTests Removed the duration parameter from getDurationBasedEstimate method Rebased Reset the duration calculation from nanos back to millis. Optimized restartExecutionArgs routine that removes duplicates. This was per a comment in code review Remove unnecessary exclusions from AbstractShellIntegrationTest The changes are code review requests. Add warn log message when job restart id is invalid. * Initialize ObjectMapper with the tools provided by DataFlowTemplate when testing * These changes were identified during code review * Fix rest docs tests. Updated to usage of latest stream apps and tasks. * Remove .jdk8 files. (#5900) * Fix more previously disabled tests. * Update SCDF so that multipleComposedTaskWithArguments succeeds Step 1. Make sure to remove the version from the docker compose. It is no longer needed and causes older versions of docker to fail Step 2. Update compose files to use the latest version of SCDF 3.x instead of 2.11.x Step 3. Update build image script so that uses java 17 when creating containers Update the DataFlowIT and the Abstract classes it is built on so that multipleComposedTaskWithArguments test passes. Notice that JobParameterJacksonDeserializer and JobParametersJacksonMixIn have been updated. These changes mirror those in #5850. These were required for the test to pass. At the time this PR is merged we can merge accepting those from #5850. Provide docs on how SCDF images are created and pushed Also update the DEFAULT_JDK to Java 17 Update PR based on code review comments * Added log message in case a JobParameter Type is invalid * cleaned up workflow.adoc * Remove .jdk8 files. (#5900) * Update SCDF so that multipleComposedTaskWithArguments succeeds Step 1. Make sure to remove the version from the docker compose. It is no longer needed and causes older versions of docker to fail Step 2. Update compose files to use the latest version of SCDF 3.x instead of 2.11.x Step 3. Update build image script so that uses java 17 when creating containers Update the DataFlowIT and the Abstract classes it is built on so that multipleComposedTaskWithArguments test passes. Notice that JobParameterJacksonDeserializer and JobParametersJacksonMixIn have been updated. These changes mirror those in #5850. These were required for the test to pass. At the time this PR is merged we can merge accepting those from #5850. Provide docs on how SCDF images are created and pushed Also update the DEFAULT_JDK to Java 17 Update PR based on code review comments * Added log message in case a JobParameter Type is invalid * cleaned up workflow.adoc * Update apps version refs to 5.0.0 / 3.0.0 (#5896) * Update stream apps version refs to 5.0.0 * Update task apps version refs to 3.0.0 See #5897 * Update with merge from main-3 * Remove metrics auto-config exclusions (#5902) This commit removes temporary auto-config exclusions that were in place to allow compiling against Boot 3.x prior to migrating to the newer Observation API. * Update documentation for initContainers addition in spring-cloud/spring-cloud-deployer#465 [skip ci] Update paragraph for multiple Init Containers. Fix mariadb mount to be /var/lib/mysql (#5875) Fixes #5877 Add spaces after columns for clarity. This is done during merge * SCDF should be able to migrate a schema from 2.10 to 3.0 Currently the migration fails with a validation error. The cause of the validation error was that some commits from 2.11.x that contained flyway migration scripts were not ported to the main-3 branch. The following commits that contained flyway migrations were migrated to main-3 from the main branch. * 62ea6c5 * 6f97589 The goal of this PR is to resolve the validation error so DB migrations will work properly. A subsequent PR will be submitted will add the feature code for commit 6f97589. * Forward port ctr status on Thin Controller. (#5906) * Added ThinTaskExecution to store ctrStatus. Update Controller and Service to populate the ctr status. Fixes #5907 * Updated for comments. * Enable MariadbSharedDbIT (#5908) This commit re-enables the `MariadbSharedDbIT` and disables the Java 21 test variant as Java 21 container images are not yet supported. * Update tracing props (#5925) * Update Prometheus Rsocket property name This property rename was missed during the initial update to Prometheus Rsocket 2.0.0-M1 (b61c3c9). * Update tracing props in Docker Compose files This updates the Wavefront and Zipkin docker compose templates with the properly named tracing related properties to enable traces exported for each. * Fix substitution in github-release.yml [skip ci] * Fix tracing properties in docker compose (#5946) This commit adds the global tracing property 'management.tracing.enabled' to the Zipkin and Wavefront docker compose files. Also, the Wavefront docker compose file excludes the Zipkin tracing auto-configuration and the Zipkin docker compose file excludes the Wavefront tracing auto-configuration. * Updates for JUnit5 and AssertJ * CI requires Java 17. * Updated based on comments. * Updated based on comments. Added TODO on comments. * Fixed imports --------- Co-authored-by: Chris Bono <[email protected]> Co-authored-by: Glenn Renfro <[email protected]> Co-authored-by: Chris Bono <[email protected]> Co-authored-by: Janne Valkealahti <[email protected]> Co-authored-by: Clemens L <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem description:
We use several initContainers to provide some configuration files, certificates, etc., so we can mix and match what we provide to our containers. As SCDF only allows to set one initContainer per task execution we're forced to provide an additional initContainer which is the sum of the previous ones. Currently we have 3 different initContainers for our tasks, so we have to mantain the combinations of each of them with the rest, and with all of them, and then apply the appropiate one to the task execution configuration. If/As the number of available initContainer grows this becomes a larger problem.
We can narrow the amount of needed initContainers from the total number of possibilities, but we'll have the case that one initContainer will provide some file that activates some configuration that collides with the configuration provided on another initContainer (again this is workable, but it's some work we'd rather not do)
Solution description:
Allow setting several initContainers. To retain backward compatibility, introduce another property
initContainersreceiving a list of currentinitContainerand mix both of them (or prefer one over the other)Description of alternatives:
As described above, our current alternative is generate the neeed amount of initContainers and set the appropiate one to the task execution configuration.
Additional context:
N/A
The text was updated successfully, but these errors were encountered: