Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: use fio instead of dd for performance tests #4961

Merged
merged 2 commits into from
Dec 16, 2024
Merged

test: use fio instead of dd for performance tests #4961

merged 2 commits into from
Dec 16, 2024

Conversation

roypat
Copy link
Contributor

@roypat roypat commented Dec 16, 2024

In test_drive_rate_limiter, use fio for measuring latencies of writing a fixed number of bytes to a block device, instead of dd (which the test itself notes is unreliable). Should fix intermitted failures we're been seeing in this test, hopefully.

Changes

...

Reason

...

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkstyle to verify that the PR passes the
    automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.
  • This functionality cannot be added in rust-vmm.

@roypat
test: use fio instead of dd for performance tests
576c7a3 
In test_drive_rate_limiter, use `fio` for measuring latencies of writing
a fixed number of bytes to a block device, instead of `dd` (which the
test itself notes is unreliable). Should fix intermitted failures we're
been seeing in this test, hopefully.

Signed-off-by: Patrick Roy <[email protected]>
@roypat roypat added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Dec 16, 2024
@codecov Codecov
Copy link

codecov bot commented Dec 16, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.93%. Comparing base (1709e4f) to head (73c268c).
Report is 1 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #4961   +/-   ##
=======================================
  Coverage   83.93%   83.93%           
=======================================
  Files         248      248           
  Lines       27791    27791           
=======================================
  Hits        23327    23327           
  Misses       4464     4464
Flag Coverage Δ
5.10-c5n.metal 84.51% <ø> (ø)
5.10-m5n.metal 84.49% <ø> (ø)
5.10-m6a.metal 83.78% <ø> (ø)
5.10-m6g.metal 80.61% <ø> (ø)
5.10-m6i.metal 84.49% <ø> (-0.01%) ⬇️
5.10-m7g.metal 80.61% <ø> (ø)
6.1-c5n.metal 84.51% <ø> (-0.01%) ⬇️
6.1-m5n.metal 84.49% <ø> (-0.01%) ⬇️
6.1-m6a.metal 83.78% <ø> (ø)
6.1-m6g.metal 80.61% <ø> (ø)
6.1-m6i.metal 84.49% <ø> (ø)
6.1-m7g.metal 80.61% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Manciukic
Copy link
Contributor

Should fix intermitted failures we're been seeing in this test, hopefully

Do you have any link to the intermittent failures? How far off are they from the target? Are we sure it's a dd issue? I doubt that dd is doing weird things, it just copies bytes from one place to another with a single thread, so it might be impacted by other things running at the same time.

@roypat
Copy link
Contributor Author

roypat commented Dec 16, 2024

Should fix intermitted failures we're been seeing in this test, hopefully

Do you have any link to the intermittent failures? How far off are they from the target? Are we sure it's a dd issue? I doubt that dd is doing weird things, it just copies bytes from one place to another with a single thread, so it might be impacted by other things running at the same time.

e.g. https://buildkite.com/firecracker/firecracker-pr/builds/12155#0193cfc3-5704-40ba-9ebb-9e671d9e7dbc. It seems to just be slightly above the upper target by a few ms. There shouldn't be anything running at the same time, the perf tests aren't parallelized at the pytest level, and are assigned a whole .metal (e.g. only one buildkite agent on the metal). My main point here is that dd is not a performance testing tool, its a tool for copying data around that happens to print some statistics, while fio is a proper I/O benchmark tool. E.g. on my laptop, dd has a range of around 200ms, while fio only has a range of around 100ms, while also being ~100ms faster on average.

@Manciukic
Copy link
Contributor

Makes sense, thanks for the context!

I don't know if it's karma but this same test just failed on my PR as well lol

@roypat roypat merged commit 5c3ff08 into firecracker-microvm:main Dec 16, 2024
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pb8o pb8o pb8o approved these changes

@Manciukic Manciukic Manciukic approved these changes

Assignees
No one assigned
Labels
Status: Awaiting review Indicates that a pull request is ready to be reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@roypat @Manciukic @pb8o