This repository has been archived by the owner on Jan 3, 2024. It is now read-only.
S3gw rebase 18.2.0 #222
Closed
S3gw rebase 18.2.0 #222
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Xavi Garcia <[email protected]>
This reverts commit 4b8053d, reversing changes made to 4ead447. Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Xavi Garcia <[email protected]>
`mtime` parameter was logged but that's an output variable and might be `nullptr`. Avoid logging it and check if it's `nullptr` before setting the value. Fixes: aquarist-labs/s3gw-tools#191 Signed-off-by: Xavi Garcia <[email protected]>
This patch allows a client to invoke stat_bucket call. The eventual actual implementation will be addressed on a dedicated issue. Fixes: aquarist-labs/s3gw-tools#159 Signed-off-by: Giuseppe Baccini <[email protected]>
Adds a check on metadata compatibility when the `SFStore` initialises. It creates a temporary copy of the metadata database and tries to sync the schema on it. If there are issues (exeptions) or any table is dropped due to incompatibilities between the data base and the actual metadata schema it throws a `sqlite_sync_exception`. The execption is catched when the `SFStore` is instantiated and the error is logged before exiting `rgw`. Previous database is not altered as all checks are done in the temporary database so we don't have data loss. Fixes: aquarist-labs/s3gw-tools#178 Signed-off-by: Xavi Garcia <[email protected]>
Adds code for adding a delete marker when deleting an object that doesn't exist. According to the AWS documentation https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-object.html when deleteing an object it should reply no errors even if the object doesn't exist. This means we create a delete-marker version for the object when versioning is enabled. This also fixes the s3-test test_versioning_multi_object_delete_with_marker_create which is exactly that test case. When undeleting delete markers (or deleting all versions of an object), if the object ends up having no version we just remove it from the in-memory objects map. The Garbage collector could then permanently remove those objects without a version which are, in fact, no longer reachable. Fixes: aquarist-labs/s3gw-tools#186 Signed-off-by: Xavi Garcia <[email protected]>
This adds an initial GC approach just starting a thread and running a basic remove deleted buckets operation. The implementation of how the thread is started and stopped are taken from rgw's code for consistency across stores. It uses 2 parameters from rgw: - `rgw_gc_processor_period` : defines the time GC waits to be triggered - `rgw_gc_max_objs` : defines the maximum number of items the GC can remove per iteration. Any possible actions done to the `sqlite` metadata is considered an item (object) when taking into account the maxium number of items to be processed in a single iteration as those actions block the metadata database. Fixes: aquarist-labs/s3gw-tools#165 Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Volker Theile <[email protected]>
Adds newly added unit tests so they're found when running the testcontainer. It also fixes the `unittest_rgw_sfs_gc` that was throwing an exception in certain circumstances when converting a version object. Fix is done in this same commit as the goal is to have the newly added tests and prevent other open PRs to be blocked. Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Giuseppe Baccini <[email protected]>
Add new 'status' RGW frontend serving status pages on 127.0.0.1:9999. Includes a beast-based HTTP server, a rudimentary status page registry and a metrics page. Activate by setting 'rgw frontends' (e.g `rgw frontends = beast port=7480, status`) Signed-off-by: Marcel Lauhoff <[email protected]>
Signed-off-by: Marcel Lauhoff <[email protected]>
Add and carry around the perf counter index. This makes the index available when iterating, for example to print / export the counters. This is useful when exporting histograms in the prometheus exposition format that includes a running sum for histograms. These are not included in the perf counter histograms and require a second counter. To print both perf counters at the same time the index is needed to match them from different collections. Signed-off-by: Marcel Lauhoff <[email protected]>
Add string conversion (function, <<operator) and a last element to make iteration possible. Signed-off-by: Marcel Lauhoff <[email protected]>
Add perf counter collection 'rgw_op' containing one counter per RGW operation. Increment counter during init_processing(). Signed-off-by: Marcel Lauhoff <[email protected]>
Add two new perf counter collections to capture operation service time as one-dimensional histograms and time counters. This combination allows conversion to the Prometheus histogram format that requires not only counter buckets, but also a running sum of all values. Signed-off-by: Marcel Lauhoff <[email protected]>
Add Prometheus exporter and add support for histograms and long running averages to the human readable exporter. Improve value formatting here and there. Histogram export is tied to the specific definition of the RGW operation service time (1D perf histograms + time counter) perf counter collections. Signed-off-by: Marcel Lauhoff <[email protected]>
This adds an extra build script that is called from a generic container that is built from the ceph-tools repository. The goal is to have the script that builds rgw, its tests and runs the latest from the ceph repository itself. It also changes the original build-radosgw.sh script from the s3gw-tools repository to discover the sfs unit tests so we don't need to change anything when adding a new unit test. The build container from s3gw-tools remains the same (with a change to run the entry point from the ceph volume) and it can be used to building releases and for building and running unit tests in the ceph repository. The entry point in the s3gw-tools build container uses also an environment value that can be used to run a different script so we have more flexibility and, for example, can run other tasks related to the rgw that need the same environment. All tests are now moved and executed in ceph's PR. Fixes: https://github.com/aquarist-labs/s3gw/issues/127 Signed-off-by: Xavi Garcia <[email protected]>
- Lifetime of the GCWorker is now tied to the lifetime of the SFSGC object - The garbagecollector object is created in the constructor of the SFStore - Use smart pointers to manage object lifetime and ensure deallocation - Remove initialize/finalize and start_/stop_processor methods, as their functionality was moved into constructors/destructors - Prevent the construction of a dysfunctional SFSGC object by replacing the default constructor with one that requires all necessary parameters Signed-off-by: Moritz Röhrich <[email protected]>
We were deleting some buckets during the individual tests, which led to errors when trying to delete them during teardown. Instead, force cleaning up the buckets during the individual tests, and check whether there are any left buckets in teardown to ensure correctness. Signed-off-by: Joao Eduardo Luis <[email protected]>
The test presumes a previously set up environment by the user. Because it does not set up the environment itself, at the moment it can't be automated as part of the smoke tests suite. Hence we're disabling it until a time when the test is properly fixed. Signed-off-by: Joao Eduardo Luis <[email protected]>
This commit adds an s3gw.spec file for building a standalone s3gw RPM, along with a systemd unit file and sysconfig file to tweak settings (e.g. local data directory). The systemd unit file and sysconfig file are not necessary or even desirable when running containerized, but make sense in the context of a standalone package. s3gw.spec is based on the existing ceph.spec.in, but instead of creating a ceph package plus a bunch of library packages and other subpackages, s3gw.spec creates a single RPM containing the radosgw binary, librados, libradosgw and libceph-common. The s3gw RPM is set to conflict with ceph-radosgw and librados2, to avoid accidental installation alongside (or on top of!) a regular ceph deployment. With this, we can do package builds on OBS with pretty much nothing other than a _service file, as can be currently seen at https://build.opensuse.org/package/show/home:tserong:branches:filesystems:ceph:s3gw/s3gw <?xml version="1.0"?> <services> <service name="obs_scm"> <param name="versionformat">0.7.0.@PARENT_TAG@.@TAG_OFFSET@</param> <param name="url">https://github.com/tserong/ceph.git</param> <param name="scm">git</param> <param name="revision">s3gw-obs-automation-experiment</param> <param name="extract">s3gw.spec</param> <param name="changesgenerate">enable</param> </service> <service name="set_version"/> <service name="tar" mode="buildtime"/> </services> Given the above, OBS will go grab the source from github, extract s3gw.spec, plug in a somewhat sensible version number, generate a somewhat useful changelog and build an RPM for us. Note: - The primary purpose of this work is to enable automatic builds on build.opensuse.org. That's why the version in the spec file is set to "WILL_BE_OVERWRITTEN_BY_OBS". - There are some SUSE-isms in the spec file, especially in the %pre and %post scripts. This will need adjusting if we want to try Fedora/RH builds. - s3gw is configured to use system boost, with necessary build dependencies available from filesystems:ceph:s3gw on OBS. - I've switched to using ninja instead of make for the build, because I couldn't get make to only build radsogw and deps. - I've set -DENABLE_GIT_VERSION=OFF, because this allows us to do builds straight from a git checkout, without needing to generate a source tarball with `make-dist`. - There may be other cmake flags that we can still turn off, or other things that can be stripped back further. Fixes: aquarist-labs/s3gw-tools#154 Signed-off-by: Tim Serong <[email protected]>
This fixes the CI to show the results of the unit tests. It was running the contained with `-d`, so the results were not shown in the workflow logs. Fixes: https://github.com/aquarist-labs/s3gw/issues/283 Signed-off-by: Xavi Garcia <[email protected]>
Adds support for prefix when listing objects and object versions. Right now as we're catching all the objects in memory only the filter is needed. In future versions, when we don't have all the objects cached in memory a new query to `sqlite` metadata will be needed. Fixes: https://github.com/aquarist-labs/s3gw/issues/261 Signed-off-by: Xavi Garcia <[email protected]>
Fix git version parsing in CMake. With the `ENABLE_GIT_VERSION` option set to `ON`, CMake will extract version information for building the version string from git. After extracting, it will also process the information found. Processing in this case means removing a leading `v` character. Instead of removing the first character (actually first byte) regardless of what it is, use a regex to ensure just a leading `v` is removed. This allows for version strings like `s3gw-v0.10.0` to propagate correctly into the version information compiled into radosgw without being truncated accidentally. Signed-off-by: Moritz Röhrich <[email protected]>
Fix the order of initialization and use of the SFSGC. The worker thread can not start running until the SFSGC object has been completely allocated, because the SFSGC object is the dout prefix provider and a pointer to the SFSGC instance would be invalid until the constructor has finished. Since the worker thread might want to print something out, this would resuult in an invalid pointer dereference. Upon stopping, the SFSGC must query the worker thread if it has started, because an SFSGC instance could be created without calling `SFSGC::initialize()`. In this case calling `join()` on the worker has to be prevented. Fixes bugs introduced in: f7dbf8b Signed-off-by: Moritz Röhrich <[email protected]>
This is implemented the same way that prefix support, as all objects are still cached in memory. Fixes: aquarist-labs/s3gw#306 Signed-off-by: Xavi Garcia <[email protected]>
Signed-off-by: Joao Eduardo Luis <[email protected]>
For Builds with GCC, enable the flag `-frecord-gcc-switches`, allowing recorstruction of build flags from the resulting build artifacts. Signed-off-by: Moritz Röhrich <[email protected]>
Signed-off-by: Xavi Garcia <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
18.2.0 rebase PR.
Opened for running CI tests.