move create database to procedure

[WenyXu]

What type of enhancement is this?

Refactor, Tech debt reduction

What does the enhancement do?

We introduced multiple-level locks in the procedure, and all DDL operations should acquire all related locks during the execution. After implementing #3516, we should refactor the database creation into a procedure.

Implementation challenges

No response

[tisonkun]

Would you add some refer impl like how other procedure are implemented?

[WenyXu]

Would you add some refer impl like how other procedure are implemented?

Yes, a typical procedure looks like this:

greptimedb/src/common/meta/src/ddl/create_table.rs

Lines 316 to 350 in 623c930

	#[async_trait]
	impl Procedure for CreateTableProcedure {
	fn type_name(&self) -> &str {
	Self::TYPE_NAME
	}
	async fn execute(&mut self, _ctx: &ProcedureContext) -> ProcedureResult<Status> {
	let state = &self.creator.data.state;
	let _timer = metrics::METRIC_META_PROCEDURE_CREATE_TABLE
	.with_label_values(&[state.as_ref()])
	.start_timer();
	match state {
	CreateTableState::Prepare => self.on_prepare().await,
	CreateTableState::DatanodeCreateRegions => self.on_datanode_create_regions().await,
	CreateTableState::CreateMetadata => self.on_create_metadata().await,
	}
	.map_err(handle_retry_error)
	}
	fn dump(&self) -> ProcedureResult<String> {
	serde_json::to_string(&self.creator.data).context(ToJsonSnafu)
	}
	fn lock_key(&self) -> LockKey {
	let table_ref = &self.creator.data.table_ref();
	LockKey::new(vec![
	CatalogLock::Read(table_ref.catalog).into(),
	SchemaLock::read(table_ref.catalog, table_ref.schema).into(),
	TableNameLock::new(table_ref.catalog, table_ref.schema, table_ref.table).into(),
	])
	}
	}

And more procedures can be found here

[CookiePieWw]

Hi @WenyXu , I'd like to give it a try.

[WenyXu]

Hi @WenyXu , I'd like to give it a try.

Have fun

[CookiePieWw]

I've made some efforts in the past few days. Since It's my first time diving into greptimedb, I think it would be better to make it clear before taking actions. Here are my findings and confusions:

• Currently, the create database statement is simply executed inside create_database method in StatementExecutor. The target is to refactor it as other ddl ops, which executes as follows (here I take drop database as an example):

  • Do some checks and then forward the params to drop_database_procedure method
  • Packaging the params as a task. The packaging methods are implemented in src/common/meta/src/rpc/ddl.rs
  • Submit the task to execute. The submission methods are implemented in src/common/meta/src/ddl_manager.rs and src/common/meta/src/ddl.rs
  • The execution methods take place under src/common/meta/src/ddl, thus a file named create_database.rs should be added to implement the CreateDatabaseProcedure

• I've noticed that there are lots of methods and traits related to rpc like PbDropDatabaseTask, which lead me to greptimedb-proto finally. Is it proper to simply place a todo somewhere and leave it alone?

• I've read about the implemention of CreateTableProcedure. It seems the procedure executes in 3 stages with several concepts I'm not familiar yet (regions, memory allocation, etc.). Should I have a more complicated design for the procedure, or simply copy the original one into the execute method and acquire some locks in lock_key? Besides, it seems the from_json and dump methods are related to rpc as well.

Would you mind giving me some advice @WenyXu ? Thanks a lot.

[WenyXu]

I've made some efforts in the past few days. Since It's my first time diving into greptimedb, I think it would be better to make it clear before taking actions. Here are my findings and confusions:

• Currently, the create database statement is simply executed inside create_database method in StatementExecutor. The target is to refactor it as other ddl ops, which executes as follows (here I take drop database as an example):

  • Do some checks and then forward the params to drop_database_procedure method
  • Packaging the params as a task. The packaging methods are implemented in src/common/meta/src/rpc/ddl.rs
  • Submit the task to execute. The submission methods are implemented in src/common/meta/src/ddl_manager.rs and src/common/meta/src/ddl.rs
  • The execution methods take place under src/common/meta/src/ddl, thus a file named create_database.rs should be added to implement the CreateDatabaseProcedure

• I've noticed that there are lots of methods and traits related to rpc like PbDropDatabaseTask, which lead me to greptimedb-proto finally. Is it proper to simply place a todo somewhere and leave it alone?

• I've read about the implemention of CreateTableProcedure. It seems the procedure executes in 3 stages with several concepts I'm not familiar yet (regions, memory allocation, etc.). Should I have a more complicated design for the procedure, or simply copy the original one into the execute method and acquire some locks in lock_key? Besides, it seems the from_json and dump methods are related to rpc as well.

Would you mind giving me some advice @WenyXu ? Thanks a lot.

Is it proper to simply place a todo somewhere and leave it alone?

Nope, we should define the protobuf in the greeting-proto firstly.

I've read about the implemention of CreateTableProcedure. It seems the procedure executes in 3 stages with several concepts I'm not familiar yet (regions, memory allocation, etc.). ….

Don’t worry, creating database only requires to create the metadata. Check current logical of the database creation.