Skip to content

Commit

Permalink
add big comment
Browse files Browse the repository at this point in the history
  • Loading branch information
nicklan committed Oct 11, 2024
1 parent 765b33c commit 63571a4
Showing 1 changed file with 19 additions and 1 deletion.
20 changes: 19 additions & 1 deletion kernel/src/engine/default/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,25 @@ impl<E: TaskExecutor> DefaultEngine<E> {
}

pub fn new(store: Arc<DynObjectStore>, prefix: Path, task_executor: Arc<E>) -> Self {
// HACK to check if we're using a LocalFileSystem from ObjectStore
// HACK to check if we're using a LocalFileSystem from ObjectStore. We need this because
// local filesystem doesn't return a sorted list by default. Although the `object_store`
// crate explicitly says it _does not_ return a sorted listing, in practice all the cloud
// implementations actually do:
// - AWS:
// [`ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)
// states: "For general purpose buckets, ListObjectsV2 returns objects in lexicographical
// order based on their key names." (Directory buckets are out of scope for now)
// - Azure: Docs state
// [here](https://learn.microsoft.com/en-us/rest/api/storageservices/enumerating-blob-resources):
// "A listing operation returns an XML response that contains all or part of the requested
// list. The operation returns entities in alphabetical order."
// - GCP: The [main](https://cloud.google.com/storage/docs/xml-api/get-bucket-list) doc
// doesn't indicate order, but [this
// page](https://cloud.google.com/storage/docs/xml-api/get-bucket-list) does say: "This page
// shows you how to list the [objects](https://cloud.google.com/storage/docs/objects) stored
// in your Cloud Storage buckets, which are ordered in the list lexicographically by name."
// So we just need to know if we're local and then if so, we sort the returned file list in
// `filesystem.rs`
let store_str = format!("{}", store);
let is_local = store_str.starts_with("LocalFileSystem");
Self {
Expand Down

0 comments on commit 63571a4

Please sign in to comment.