-
Notifications
You must be signed in to change notification settings - Fork 564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AttachContainer sometimes hangs #114
Comments
Which docker version are you using? I will try to reproduce. If you're using docker>=0.11, does it happen with the method Logs as well? It uses a different endpoint in the Docker API. Is your container running anything special? Or are you able to reproduce it with any command? |
I didn't know the Logs commands, thanks for pointing it. Will try and report in a few minutes. My container doesn't run anything special, just a shell script to get bower dependencies, and it doesn't always hangs if I restart the process with the same params, just sometimes. I'm running docker 1.1.0. By the way, the container is stopped ( |
Ok, so with Logs, it's a bit better.
(from code: var buf bytes.Buffer
opts := docker.LogsOptions{
Container: w.container.ID,
OutputStream: &buf,
Stdout: true,
Stderr: true,
}
err = w.client.Logs(opts)
if err != nil {
log.Printf("Error while fetching logs from container: %s\n", err)
}
fmt.Printf("buf.String() %+v\n", buf.String()) Where docker run simply returns:
|
@gravis that's bizarre. I'm able to reproduce the bug with the strange caracter in the method Logs, I will have a look. I'm trying to reproduce now the attach issue. Thanks for the reports! |
That's "good" news, at least it's not related to my docker install nor my image/container :) |
@gravis the bug is with the docker multiplexing capability. We need to interpret that character and find out whether it's stdout or stderr. utils.StdCopy does that for us. I'll push a fix soon. Still trying to figure out the AttachToContainer issue. |
Great! Regarding AttachToContainer, were you able to reproduce it at least once? |
@gravis nope, not yet :( |
Docker sends the output in the following format: <header identifying the stream and with the size fo the frame> <frame> It allows us to multiplex the content between the output and error streams. I wondering whether I should rewrite the StdCopy function, it's a little messy. Related to #114.
I fixed a bug in StdCopy implementation, I don't think it would hang, but it may be related. I've also fixed the Logs issue. Can you give it another try (both for |
Great! |
First test: |
Logs doesn't use hijack, so it's probably StdCopy. I'm still unable to reproduce, but I think we're closer to the solution now. Please try setting |
Ouch,
|
(we should probably create a branch to test, master is getting messy :() |
@gravis with or without RawTerminal? Seems like it's working fine here. I will run some tsuru integration tests against go-dockerclient's master. Were you able to test it with |
Definitely voting for an issue with |
It would have hanged already. Using |
Good news, I've just managed to reproduce the hang in an automated test. It fails whenever StdCopy needs to grow the buffer. I will push a fix soon :) |
You're my hero :) |
I think this may still be happening.
In github.com/grafana/scribe, I have this function: and I'm expecting the RunContainer function to return when the container exits. It seems to work pretty consistently for the container created here but not the one created here. This is what
I have a feeling this may be related to how quickly the container exits before "AttachToContainer" is called. I'll try adding a sleep before echoing and seeing what happens. Edit: confirmed, it's definitely "AttachToContainer" being called right before or right after the container has exited, causing it to hang. Do you think there might be a solution to this? I'd be happy to submit a PR. |
@kminehart hey, sorry for the delayed response. Do you think we'd be able to get a minimal reproducer? If not, I can dig into the code you shared and see if we can figure out exactly what's going on. |
I can try to get one today! |
@fsouza Much appreciated, here's a link to an example I able to reproduce it with. https://gist.github.com/kminehart/e50ae98557adc139ba9648077271367b |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Hi,
I'm fighting with a nasty bug. Sometimes, but not always (a bit hard to reproduce), AttachContainer will hangs, eating 100% of CPU.
I have profiled my program and ended up with this (profiled during the hang):
I'm using the latest go-dockerclient (master / e21b2a4), and having the same problem with older versions (tried until 992e6ab)
My code is pretty standard:
and works "most of the time" (99% of runs).
I'm not sure if it's related to docker, but here's my configuration:
Any hint would be appreciated :)
The text was updated successfully, but these errors were encountered: