Release v0.8.0
v0.8.0
Release date: May 18, 2024
👍 Highlights
- Flow Engine: which brings continuous aggregation capabilities.
- Column Type Modification: allows users to effortlessly alter the data type of columns within a table without the hassle of rebuilding the table or manually migrating data.
- Cluster Management Information Table: allows querying for information about the cluster. This functionality aids administrators in monitoring and managing the health status of the database cluster, facilitating prompt issue detection and resolution.
- Append-only Tables: Users can now create tables in Append-only mode by setting the append mode during table creation.
- DROP DATABASE: enables swift deletion of all tables and resources under a database instance.
- New Table Partitioning Methods and Syntax: considering the future need for frequent partition changes such as automatic partitioning or repartitioning, we're developing a new partitioning syntax.
EXPLAIN ANALYZE <QUERY>
: analyze and optimize query statements swiftly in distributed mode.
🚀 Features
- feat(cli): export metric physical tables first by @WenyXu in #3949
- feat(cli): prevent exporting physical table data by @WenyXu in #3978
- feat(drop_table): support to rollback table metadata by @WenyXu in #3692
- feat(flow): add types for every plan enum variant by @discord9 in #3938
- feat(flow): avg func rewrite to sum/count by @discord9 in #3955
- feat(flow): expire arrange according to time_index type by @discord9 in #3956
- feat(flow): flow node manager by @discord9 in #3954
- feat(flow): flow worker by @discord9 in #3934
- feat(flow): mirror insert req to flow node by @discord9 in #3858
- feat(flow): query table schema&refactor by @discord9 in #3943
- feat(flow): render map&related tests by @discord9 in #3581
- feat(flow): render reduce by @discord9 in #3769
- feat(flow): render source/sink by @discord9 in #3903
- feat(flow): transform substrait SELECT&WHERE&GROUP BY to Flow Plan by @discord9 in #3690
- feat(flow): tumble window func by @discord9 in #3968
- feat(fuzz): add alter logical table target by @CookiePieWw in #3818
- feat(fuzz): add create database target by @CookiePieWw in #3675
- feat(fuzz): add create logical table target by @CookiePieWw in #3756
- feat(fuzz): add insert logical table target by @CookiePieWw in #3842
- feat(fuzz): add validator for inserted rows by @CookiePieWw in #3932
- feat(fuzz): enable create-if-not-exists option by @CookiePieWw in #3732
- feat(fuzz): support to create metric table by @WenyXu in #3617
- feat(procedure): support to rollback by @WenyXu in #3625
- feat(promql): parameterize lookback by @etolbakov in #3630
- feat: Parquet reader builder supports building multiple ranges to read by @evenyag in #3841
- feat: add
ChangeColumnType
forAlterKind
by @KKould in #3757 - feat: add
filter_deleted
option to avoid removing deletion markers by @v0y4g3r in #3707 - feat: add append mode to table options by @evenyag in #3624
- feat: add checksum for checkpoint data by @tisonkun in #3651
- feat: add metasrv start time to node info by @fengjiachun in #3883
- feat: add preserve arg to sqlness runner by @waynexia in #3724
- feat: add strict mode to validate protocol strings by @irenjj in #3638
- feat: add tinytext, mediumtext and longtext data types by @CookiePieWw in #3731
- feat: add validate method to CreateExpr by @waynexia in #3772
- feat: adds information_schema cluster_info table by @killme2008 in #3832
- feat: check partition rule by @waynexia in #3711
- feat: create database with options by @tizee in #3751
- feat: create view by @killme2008 in #3807
- feat: define
CreateFlowTask
andDropFlowTask
by @WenyXu in #3801 - feat: get metasrv clusterinfo by @fengjiachun in #3696
- feat: ignore internal columns in
SHOW CREATE TABLE
by @WenyXu in #3950 - feat: impl Display for Statement by @irenjj in #3744
- feat: impl parser and operator for CREATE FLOW by @waynexia in #3806
- feat: impl show collation and show charset statements by @killme2008 in #3753
- feat: impl table_constraints table for information_schema by @killme2008 in #3698
- feat: implement FlowTaskMetadataManager by @WenyXu in #3766
- feat: implement
CacheContainer
&TableFlownodeSetCache
by @WenyXu in #3885 - feat: implement drop flow parser by @WenyXu in #3888
- feat: implement drop flow procedure by @WenyXu in #3877
- feat: implement the
CompositeTableRoute
by @WenyXu in #3918 - feat: implement the
CreateFlowProcedure
by @WenyXu in #3810 - feat: implement the
FrontendInvoker
by @WenyXu in #3824 - feat: improve error message for typo in IF NOT EXISTS by @tisonkun in #3817
- feat: introduce
TableNameCache
&TableInfoCache
&TableRouteCache
by @WenyXu in #3895 - feat: introduce
TableRouteCache
toPartitionRuleManager
by @WenyXu in #3922 - feat: introduce the
CacheRegistry
by @WenyXu in #3896 - feat: limit total rows copied in COPY TABLE FROM with LIMIT segment by @irenjj in #3910
- feat: limit total rows copied in
COPY TABLE FROM
stmt by @irenjj in #3819 - feat: limiting the size of query results to Dashboard by @realtaobo in #3901
- feat: metaclient builder options with role by @fengjiachun in #3909
- feat: metasrvs nodeinfo by @fengjiachun in #3873
- feat: migrate orc-rs to datafusion-orc by @realtaobo in #3923
- feat: pass
QueryContext
toFlowRequestHeader
by @WenyXu in #3878 - feat: physical optimizer RemoveDuplicate to remove duplicate exec plans by @waynexia in #3839
- feat: remove timeout in the channel between frontend and datanode by @waynexia in #3962
- feat: skip read-only region when trying to flush on region full by @waynexia in #3966
- feat: support PromQL scalar by @Taylor-lagrange in #3693
- feat: support
ALTER TABLE ... MODIFY COLUMN ... ...
by @KKould in #3796 - feat: support any precision in PromQL by @Taylor-lagrange in #3933
- feat: support different types for
CompatReader
by @KKould in #3745 - feat: support distributed
EXPLAIN ANALYZE
by @waynexia in #3908 - feat: support evaluate expr in range query param by @Taylor-lagrange in #3823
- feat: support invalidate schema name key cache by @realtaobo in #3725
- feat: support querying field column names in Prometheus HTTP API by @waynexia in #3880
- feat: support to create & drop flow via grpc by @WenyXu in #3915
- feat: support to drop flow by @WenyXu in #3900
- feat: support to invalidate flow cache by @WenyXu in #3926
- feat: treat all number types as field candidates by @waynexia in #3670
- feat: try get pk values from cache when applying predicate to parquet by @waynexia in #3286
- feat: update dashboard to v0.5.0 by @ZonaHex in #3768
- feat: use txn to impl cas by @fengjiachun in #3936
- feat: write manifests in background tasks by @evenyag in #3709
🐛 Bug Fixes
- fix!: use the right precision by @fengjiachun in #3794
- fix(alter_table): ignore request outdated error by @WenyXu in #3715
- fix(metric engine): label mismatch in metric engine by @v0y4g3r in #3927
- fix(readme): fix link of Ingester-js by @DiamondMofeng in #3668
- fix:
DeserializedValueWithBytes::from_inner
misusing by @WenyXu in #3676 - fix: add data type to vector cache key by @evenyag in #3876
- fix: broken link in contributing guide by @waynexia in #3831
- fix: changing column data type can't process type alias by @killme2008 in #3972
- fix: cli export "create table" with quoted names by @dimbtp in #3684
- fix: compiler warnings on Windows by @evenyag in #3844
- fix: consider both db param and extended db header in Prometheus HTTP API by @waynexia in #3776
- fix: count_wildcard_to_time_index_rule doesn't handle table reference properly by @waynexia in #3847
- fix: do not remove deletion markers when window time range overlaps by @v0y4g3r in #3773
- fix: fix fuzz test ci by @WenyXu in #3795
- fix: memory leak in unit test of telemetry by @waynexia in #3897
- fix: operating region guards should be dropped when procedure is done by @MichaelScofield in #3775
- fix: post process result on query full column name of prom labels API by @waynexia in #3793
- fix: potential deadlock by @WenyXu in #3930
- fix: prevent exporting metric physical table data by @WenyXu in #3970
- fix: prevent registering logical regions with AliveKeeper by @WenyXu in #3965
- fix: promql scalar when input empty batch by @Taylor-lagrange in #3779
- fix: prune row groups correctly for columns with the same name by @evenyag in #3802
- fix: push down order hint of the query again by @evenyag in #3797
- fix: register regions during procedure recovery by @WenyXu in #3859
- fix: remove ttl option from metadata region by @waynexia in #3726
- fix: retrieve all info instead of checking on demand by @WenyXu in #3846
- fix: return metric name instead of query in Prometheus /series API by @waynexia in #3864
- fix: set is_time_index properly on updating physical table's schema by @waynexia in #3770
- fix: sort unstable HTTP result in label values query by @waynexia in #3920
- fix: the
dropping_regions
guards should be dropped on procedure done by @MichaelScofield in #3771 - fix: tql parser hang on abnormal input by @waynexia in #3977
- fix: wrong handler implementation of prometheus remote write by @waynexia in #3826
🚜 Refactor
- refactor!: remove duration type from gRPC by @fengjiachun in #3825
- refactor!: remove opentsdb tcp server by @shuiyisong in #3828
- refactor!: remove some admin APIs by @fengjiachun in #3821
- refactor!: unify sql options into OptionMap by @tisonkun in #3792
- refactor(flow): func spec api&use Error not EvalError in mfp by @discord9 in #3657
- refactor(flow): refine comments and code by @zhongzc in #3785
- refactor(flow): split render.rs and other minor changes by @discord9 in #3750
- refactor(logstore): remove Entry::namemspace by @v0y4g3r in #3875
- refactor(meta): Ensure all moving values remain unchanged between two transactions by @WenyXu in #3727
- refactor(naming): use the better naming for pubsub by @zyy17 in #3960
- refactor: DfUdfAdapter to bridge ScalaUdf by @tisonkun in #3814
- refactor: Move manifest manager lock to
MitoRegion
by @evenyag in #3689 - refactor: Remove PhysicalPlan trait and use ExecutionPlan directly by @evenyag in #3894
- refactor: add
Configurable
trait by @zyy17 in #3917 - refactor: add
procedure_loader
macro by @WenyXu in #3906 - refactor: add the main cli entry point struct
Command{}
and simplifymain()
by @zyy17 in #3850 - refactor: add tracing options in xOptions by @zyy17 in #3919
- refactor: avoid unnecessary alloc by using unwrap_or_else by @etolbakov in #3742
- refactor: avoid unnecessary cloning by @WenyXu in #3734
- refactor: bundle the lightweight axum test client by @tisonkun in #3669
- refactor: check error type before logging by @sunng87 in #3697
- refactor: drop table procedure by @WenyXu in #3688
- refactor: log statement on error by Display by @tisonkun in #3803
- refactor: make txn easy to use by @fengjiachun in #3905
- refactor: merge RegionHandleResult into RegionHandleResponse by @tisonkun in #3721
- refactor: move
Plugins
toplugins.rs
and use rwlock by @zyy17 in #3862 - refactor: move database client to test by @dimbtp in #3820
- refactor: move mirror insertion tasks to the background runtime by @WenyXu in #3879
- refactor: move the version string to common by @MichaelScofield in #3783
- refactor: pass http method to metasrv http handler by @tisonkun in #3667
- refactor: passing QueryContext to RegionServer by @Kelvinyu1117 in #3829
- refactor: refactor
DatanodeManager
toNodeManager
by @WenyXu in #3811 - refactor: refactor alter table procedure by @WenyXu in #3678
- refactor: refactor frontend cache by @WenyXu in #3912
- refactor: remove MixOptions and use StandaloneOptions only by @zyy17 in #3893
- refactor: remove
TableMetaKey
trait by @WenyXu in #3837 - refactor: remove catalog prefix by @WenyXu in #3835
- refactor: remove location_opt and DebugFormat by @waynexia in #3830
- refactor: remove re-export from logging by @waynexia in #3865
- refactor: remove redundant try_flush invocations by @tisonkun in #3706
- refactor: rename flow task to flow by @WenyXu in #3833
- refactor: rename metasrv_addr to metasrv_addrs by @fengjiachun in #3921
- refactor: simplify the PaginationStream by @WenyXu in #3787
- refactor: use secrecy SerectString to hold secrets option by @tisonkun in #3804
- refactor: use toml2docs to generate config docs by @zyy17 in #3704
📚 Documentation
- docs: add how-to-write-fuzz-tests.md by @WenyXu in #3763
- docs: add style guide by @waynexia in #3730
- docs: correct v0.7 benchmark report by @waynexia in #3907
- docs: run fuzz tests in dev mode & disable sanitizer by @WenyXu in #3871
- docs: update memtable config example by @evenyag in #3712
🧪 Testing
- test: Add tests for KvBackend trait implement by @realtaobo in #3700
- test: give windows file watcher more time by @sunng87 in #3953
- test: gracefully shutdown postgres client in sql tests by @evenyag in #3958
- test: introduce unstable fuzz create table test by @WenyXu in #3788
🚄 Performance
- perf(operator): reuse table info from table creation by @zhongzc in #3945
- perf: optimize
count(*)
by @waynexia in #3845
⚙️ Miscellaneous Tasks
- chore(flow): more comments&lint by @discord9 in #3680
- chore(mito): set null value data size to i64 by @etolbakov in #3722
- chore: add a fix-clippy Makefile target by @tisonkun in #3677
- chore: add warn log for denied to renew region lease by @WenyXu in #3827
- chore: adding Grafana config for cluster monitor by @shuiyisong in #3781
- chore: always obsolete wal to avoid discontinuous log entries by @v0y4g3r in #3789
- chore: avoid double future by @fengjiachun in #3890
- chore: bump jobserver by @v0y4g3r in #3778
- chore: bump proto to 219b24 by @WenyXu in #3899
- chore: bump to v0.8.0 by @WenyXu in #3971
- chore: catch up label updates by @tisonkun in #3951
- chore: cleanup tech debt in #3696 by @fengjiachun in #3705
- chore: cleanup todos that should be panic by @tisonkun in #3720
- chore: correct RepeatedField origin by @tisonkun in #3838
- chore: create database api change in protobuf by @CookiePieWw in #3682
- chore: deprecate nyc-taxi benchmark by @waynexia in #3891
- chore: improve contributor click in git-cliff by @tisonkun in #3672
- chore: keep the same naming style by @fengjiachun in #3916
- chore: minor refactor on etcd kvbackend by @fengjiachun in #3940
- chore: remove TODO that has been done by @tisonkun in #3683
- chore: remove TableIdProvider by @fengjiachun in #3733
- chore: remove global_ttl config by @v0y4g3r in #3673
- chore: remove one slice copy in wal encoding by @v0y4g3r in #3861
- chore: rename all
datanode_manager
tonode_manager
by @WenyXu in #3813 - chore: respect axum test client's origin by @tisonkun in #3805
- chore: store-addr to store-addrs by @fengjiachun in #3925
- chore: try avoid one slice by @tisonkun in #3856
- chore: unify name metasrv by @fengjiachun in #3671
- chore: update greptime-proto to a11db14 by @WenyXu in #3942
- chore: update greptime-proto to main by @MichaelScofield in #3743
- ci: Use lld linker in windows tests by @evenyag in #3946
- ci: add CODEOWNERS file by @tisonkun in #3691
- ci: add fuzz_insert ci cfg by @CookiePieWw in #3840
- ci: align clippy checks and fixup by @shuiyisong in #3868
- ci: avoid trying to send slack notification in forks by @tisonkun in #3857
- ci: create cyborg for all heavy GitHub integrated actions by @tisonkun in #3852
- ci: implement docbot in cyborg by @tisonkun in #3937
- ci: reduce workflow files by merging actions by @tisonkun in #3848
- ci: replace pull-request actions with cyborg by @tisonkun in #3854
- ci: secrets.GITHUB_TOKEN not exist on self-hosted runner by @tisonkun in #3765
- ci: setup-protoc always with token by @tisonkun in #3674
- ci: temporary disable compatibility test by @waynexia in #3767
- ci: use greptimedb-ci-tester account by @zyy17 in #3719
🏗️ Build
- build(deps): bump rustls from 0.22.3 to 0.22.4 by @dependabot in #3764
- build(deps): bump snafu to 0.8 by @waynexia in #3911
- build(deps): update datafusion to latest and arrow to 51.0 by @MichaelScofield in #3661
- build: add RUSTUP_WINDOWS_PATH_ADD_BIN env by @evenyag in #3969
- build: only build amd64 image for CentOS by @evenyag in #3784
- build: run cargo gc to build binaries by @waynexia in #3874
- build: strip sqlness binary by @waynexia in #3872
- build: update toolchain to nightly-2024-04-18 by @waynexia in #3740
- build: upgrade rust toolchain to fix ci issues on Windows by @evenyag in #3898
New Contributors
- @irenjj made their first contribution in #3638
- @Kelvinyu1117 made their first contribution in #3829
- @tizee made their first contribution in #3751
Full Changelog: v0.7.2...v0.8.0