You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Wildcards will only be supported in object path components (not hosts or bucket names)
We can support initially patterns like (/dir/file-*.csv, /dir/**/*.json, /dir/subdir-*/*.parquet)
This type of ingest is aimed at initial convenience only, and should be reserved for low file counts, as the operation of listing an S3 bucket may get slow/expensive at scale
Similarly to existing FileGlob logic, the state of the source will be represented by the name of the last file ingested, with ingest proceeding in lexicographic sort order
The text was updated successfully, but these errors were encountered:
This ticket is to extend
FilesGlob
support to allow something like:It should also allow expressing paths to S3-like storages like Minio e.g. like:
Notes:
/dir/file-*.csv
,/dir/**/*.json
,/dir/subdir-*/*.parquet
)FileGlob
logic, the state of the source will be represented by the name of the last file ingested, with ingest proceeding in lexicographic sort orderThe text was updated successfully, but these errors were encountered: