Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to upload child commit to pulp server #3861

Open
yih-redhat opened this issue Dec 19, 2023 · 5 comments
Open

failed to upload child commit to pulp server #3861

yih-redhat opened this issue Dec 19, 2023 · 5 comments

Comments

@yih-redhat
Copy link
Contributor

yih-redhat commented Dec 19, 2023 •
edited
Loading

Describe the bug
ostree pulp test case https://github.com/virt-s1/rhel-edge/blob/main/ostree-pulp.sh failed frequently (after I added child commit case because pulp bug pulp/pulp_ostree#289 is fixed). If this test case failed, re-run can pass without error, but we still need to find out why.

All failures happened during upload of child commit, here is the job logs:

  1. https://github.com/virt-s1/rhel-edge/actions/runs/7176103800/job/19540481049 (waiting for task "/pulp/api/v3/tasks/018c5c15-6731-77e8-ad2e-4a72d87cdf4f/" to finish)
  2. https://github.com/virt-s1/rhel-edge/actions/runs/7178624152/job/19554398570 (waiting for task "/pulp/api/v3/tasks/018c5e17-c37b-7285-9aae-59832ff26f16/" to finish)
  3. https://github.com/virt-s1/rhel-edge/actions/runs/7254239003/job/19772602470 (failure details : [Code: 11, Reason: failed to upload file "/var/cache/osbuild-worker/output/ceb50129-914d-4ed0-ad9e-6c20245d3ae9-3083257830/commit-archive/commit.tar": 502 Bad Gateway)

From my verify result for pulp bug pulp/pulp_ostree#289, it takes 9 minutes to upload child commit. And for these failed jobs, I think maybe it took long time so osbuild-composer timed out.

Environment

  • OS version (/etc/os-release and /etc/redhat-release): RHEL 9.4
  • osbuild-composer version (rpm -qi osbuild-composer): Latest

To Reproduce
Steps to reproduce the behavior:

  • for example starting osbuild-composer.service or running composer-cli command, don't forget to include all configuration files you created

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.
@yih-redhat
Copy link
Contributor Author

@achilleas-k @lubosmj Could you please help to take a look of this issue when you have time?

@lubosmj
Copy link

lubosmj commented Dec 19, 2023

Can we get our hands on the task which has not been finished? Can we increase the timeout a bit?

Regarding the "502 Bad Gateway" error, I am not sure if this is a problem on the Pulp's side.

Dec 19 01:40:54 runner-rhos-01-rhel-9-4-large-9609 osbuild-worker[22699]: time="2023-12-19T01:40:54-05:00" level=info msg="uploading ostree commit to pulp"
+ true
+ sudo composer-cli --json compose info 3a45427f-5166-4c2f-8d5f-ce4724c8c253
+ tee /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
++ jq -r '.[0].body.queue_status' /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ COMPOSE_STATUS=RUNNING
+ [[ RUNNING != RUNNING ]]
+ sleep 5
+ true
+ tee /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ sudo composer-cli --json compose info 3a45427f-5166-4c2f-8d5f-ce4724c8c253
++ jq -r '.[0].body.queue_status' /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ COMPOSE_STATUS=RUNNING
+ [[ RUNNING != RUNNING ]]
+ sleep 5
+ true
+ sudo composer-cli --json compose info 3a45427f-5166-4c2f-8d5f-ce4724c8c253
+ tee /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
++ jq -r '.[0].body.queue_status' /tmp/tmp.sgiIl0Atqx/compose-info-ostree-commit-519b5f5a-8e2f-4560-9a8e-b195e312e75c.json
+ COMPOSE_STATUS=RUNNING
+ [[ RUNNING != RUNNING ]]
+ sleep 5
Dec 19 01:41:07 runner-rhos-01-rhel-9-4-large-9609 osbuild-worker[22699]: time="2023-12-19T01:41:07-05:00" level=error msg="osbuild job failed: at least one target failed" jobId=ceb50129-914d-4ed0-ad9e-6c20245d3ae9
Dec 19 01:41:07 runner-rhos-01-rhel-9-4-large-9609 osbuild-worker[22699]: time="2023-12-19T01:41:07-05:00" level=error msg="failure details : [Code: 11, Reason: failed to upload file \"/var/cache/osbuild-worker/output/ceb50129-914d-4ed0-ad9e-6c20245d3ae9-3083257830/commit-archive/commit.tar\": 502 Bad Gateway (<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.22.1</center>\r\n</body>\r\n</html>\r\n), Details: org.osbuild.pulp.ostree]" jobId=ceb50129-914d-4ed0-ad9e-6c20245d3ae9

@henrywang henrywang mentioned this issue Dec 19, 2023
Closed
@yih-redhat
Copy link
Contributor Author

Yes, let's just focus on unfinished task error, what's the timeout for uploading commit task?

@yih-redhat
Copy link
Contributor Author

@achilleas-k Could you please take a look of this when you have time?

@achilleas-k
Copy link
Member

A minimal reproducer would be nice. The test script is a bit too big to iterate on to experiment with this. I'll try to find some time to look into this sometime next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@achilleas-k @lubosmj @yih-redhat