You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running the CLI in parallel frequently causes the config file to be overwritten with empty values. I instrumented the SDK and saw that sometimes when one process was writing, another process would read. Those reads returned 0bytes with no error. They then wrote empty configs back to config.json.
The 945 byte write is for an empty config struct, effectively logging out all the other processes.
Impact:
The log outs make test automation painful. E.g., they are preventing us from running a large test suite in parallel. I expect users will see more of these errors because IBM Cloud has shortened the TTL of its oauth tokens. That requires plugins to refresh and write oauth tokens more often, which will lead to more conflicts.
This is also a problem when updating plugins.
Repro:
Create a simple plugin that calls RefreshIAMToken on start. Then run the code using a script similar to
Yep - I just started to see this last week in our automated testing in which we run 5 tests in parallel all from the same machine. As it stands today this testing is broken and we can't use it - so this is kind of a big issue for us.
We've also seen these issues in the kubernetes-service plugin. Thanks @ericmillin for opening.
@weieigao Could we add some form of atomic updates to config files, no matter which process is writing or reading?
One easy way to do this, but loses a little consistency, is writing a whole new config file to a temporary file, then renaming it (mv) on top of the intended file. In Bash, that might look like this:
Description:
Running the CLI in parallel frequently causes the config file to be overwritten with empty values. I instrumented the SDK and saw that sometimes when one process was writing, another process would read. Those reads returned 0bytes with no error. They then wrote empty configs back to
config.json
.You can see it happening here.
Here's what happens in order:
The 945 byte write is for an empty config struct, effectively logging out all the other processes.
Impact:
The log outs make test automation painful. E.g., they are preventing us from running a large test suite in parallel. I expect users will see more of these errors because IBM Cloud has shortened the TTL of its oauth tokens. That requires plugins to refresh and write oauth tokens more often, which will lead to more conflicts.
This is also a problem when updating plugins.
Repro:
Create a simple plugin that calls
RefreshIAMToken
on start. Then run the code using a script similar toThe text was updated successfully, but these errors were encountered: