Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change task namespace deployment dynamically in Kubernetes without restarting scdf #4880

Open
mcisnerosb57 opened this issue Apr 8, 2022 · 9 comments
Assignees
Labels
type/enhancement Is an enhancement request

Comments

@mcisnerosb57
Copy link

We are using SCDF to perform the deployment of the BATCH services (TASK's). We have found that the only way to include a new namespace in openshift for spring-cloud-data-flow is to use the SCDF accounts in the application.yml as follows:

  cloud:
    dataflow:
      task:
        platform:
          kubernetes:
            accounts:
              default:
                namespace: sanes-nhb-serverless-dev
                entryPointStyle: exec
                limits:
                  memory: 1024Mi
                deploymentServiceAccountName: scdf-sa

The problem it present to us is the necessity of restarting the POD every time we add a new namespace in SCDF in which we want to deploy a task. This is not feasible for our model.

We propose to make an evolution in which, in addition to using the default value of the accounts, the value of the namespace can be dynamically overwritten when launching a task with the already existing property "deployer.test.kubernetes.namespace". (We do not know if the use of this property actually does something, since we have not detected any change when used, it always deploys us using the default account.)
¿it´s possible to do this?.
thanks of advance

@github-actions github-actions bot added the status/need-triage Team needs to triage and take a first look label Apr 8, 2022
@mcisnerosb57
Copy link
Author

mcisnerosb57 commented Apr 28, 2022

@onobc @markpollack What we have verified is that in large projects with multiple Openshift Namespaces we have the problem that adding an account from a new namespace puts at risk the launch of the tasks of the other namespaces, which can be hundreds. So we need to able to add accounts without a POD reboot .

@markpollack
Copy link
Contributor

We will look into this, thanks for reporting.

@markpollack markpollack added status/need-investigation Oh need to look under a hood and removed status/need-triage Team needs to triage and take a first look labels May 4, 2022
@cppwfs
Copy link
Contributor

cppwfs commented May 17, 2022

Currently each platform has a TaskLauncher, and this TaskLauncher has an assigned KubernetesClient which is assigned a namespace. These are all configured at Bean creation time. The solution would be to create a new KubernetesClient for each task launch instead of using the one that was created at TaskLauncher creation. And thus this KubernetesClient could then be assigned the namespace that is set via deployment properties by the user. This will require a change to the KubernatesDeployer.

@cppwfs cppwfs removed the status/need-investigation Oh need to look under a hood label May 17, 2022
@markpollack markpollack added this to the Priority Backlog milestone May 17, 2022
@cmiquelg
Copy link

Hi @cppwfs ,
Is this feature being built? I'm also interested in this SCDF behavior and I would like to work on it if you are not already on it.
Thanks!

@markpollack
Copy link
Contributor

markpollack commented Aug 3, 2022

Hi, currently triaging the priority backlog for the next release. We haven't started this. I believe I understand the issue as described, but not sure yet of what would be the best design. We would probably have to move away from the accounts being statically configured and create entries in the database which would also require the usual CRUD access (with protections for delete), a UI screen to manage the account. If we do it for tasks, we will also need to do it for streams. Open to your suggestions and also your contributions!

@markpollack markpollack modified the milestones: Priority Backlog, Candidate to move to 2.10 Aug 3, 2022
@markpollack markpollack modified the milestones: Priority Backlog, 2.10.0-M3 Aug 17, 2022
@cmiquelg
Copy link

Hi,

I totally agree on your proposed solution and I will start working on it, at least the 'backend' part of it. I lack of frontend knowledge and I feel like I would not do a great job.

Thanks!

@markpollack
Copy link
Contributor

can you clarify on the comment

we have the problem that adding an account from a new namespace puts at risk the launch of the tasks of the other namespaces

how does adding a namespace to the account put at risk the launch of other tasks?

While we could simply add this property to the deployer, but I believe there maybe other areas where acceptable choices are compared against the list of namespaces across the accounts, so adding it 'on the fly' would not be a good solution taking into account the whole lifecycle of the task and how other parts of the task api work.

@cmiquelg
Copy link

cmiquelg commented Oct 3, 2022

Hi @markpollack

With the current SCDF behavior, when we want to add a new account to deploy a batch in a new namespace, we have to add it to the configuration and redeploy the pod, which means the SCDF service will not be available during restart period. This means that during that period, no batches can be launched. In a production environment, with a large number of batches being launched simultaneously, this can be a problem.

The solution we are working on right now and we understood it was what you proposed is:
- Generating a new database table to contain the accounts info for deployments.
- Create a set of endpoints (CRUD) to access this new database table.
- Change every access to configuration to access either the configuration files or the new database table, depending on a configuration field. This way, we will maintain previous behavior to maintain SCDF retro compatible with past versions.
- Access to database to read accounts table can be cached in a short time cache, to avoid database access every time a batch is launched.

I don't think I completely understand your point since I think this solution fills all the requirements.

Thank you!

khaeghar pushed a commit to khaeghar/spring-cloud-dataflow that referenced this issue Nov 11, 2022
…y in Kubernetes without restarting scdf
@khaeghar
Copy link

Hi @markpollack ,

We've created a new branch with the solution @cmiquelg proposed, following each bullet.

Here's the PR: #5142

Anything else that needs to be done, please tell me :)

@cppwfs cppwfs added type/enhancement Is an enhancement request and removed area/batch-task Belongs to batch and task area/stream labels Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Is an enhancement request
Projects
None yet
Development

No branches or pull requests

5 participants