Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bulkload): fix rocksDB parameter allow_ingest_behind lose after replica migration #1651

Merged
merged 12 commits into from
Nov 29, 2023
Merged

fix(bulkload): fix rocksDB parameter allow_ingest_behind lose after replica migration #1651

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
773f202
fix ingest behind parameter lose after replica migration
Oct 19, 2023
b9e55a1
reformat code
Oct 19, 2023
b07110a
refactor code into reset_rocksdb_option
Oct 19, 2023
9e01a25
format code
Oct 19, 2023
935004b
follow Google c++ style input&output rules
Oct 19, 2023
9357333
format code
Oct 19, 2023
307ba39
Merge branch 'master' of github.com:ninsmiracle/incubator-pegasus int…
Oct 20, 2023
b939a0d
pass build
Oct 25, 2023
aac1034
Merge branch 'master' into fix_ingest_behind_lose
Oct 25, 2023
9b95a15
pass IWYU
Nov 2, 2023
668d4ce
deal with conflit
Nov 3, 2023
c624d15
pass IWYU
Nov 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions src/server/pegasus_server_impl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1666,8 +1666,8 @@ dsn::error_code pegasus_server_impl::start(int argc, char **argv)
// We don't use `loaded_data_cf_opts` directly because pointer-typed options will
// only be initialized with default values when calling 'LoadLatestOptions', see
// 'rocksdb/utilities/options_util.h'.
reset_rocksdb_options(loaded_data_cf_opts, &_table_data_cf_opts);
_db_opts.allow_ingest_behind = parse_allow_ingest_behind(envs);
reset_rocksdb_options(
loaded_data_cf_opts, &_table_data_cf_opts, loaded_db_opt, &_db_opts, envs);
}
} else {
// When create new DB, we have to create a new column family to store meta data (meta column
Expand Down Expand Up @@ -3132,7 +3132,10 @@ bool pegasus_server_impl::set_usage_scenario(const std::string &usage_scenario)
}

void pegasus_server_impl::reset_rocksdb_options(const rocksdb::ColumnFamilyOptions &base_opts,
rocksdb::ColumnFamilyOptions *target_opts)
rocksdb::ColumnFamilyOptions *target_opts,
const rocksdb::DBOptions &base_db_opt,
rocksdb::DBOptions *target_db_opt,
const std::map<std::string, std::string> &envs)
{
LOG_INFO_PREFIX("Reset rocksdb envs options");
// Reset rocksdb option includes two aspects:
Expand All @@ -3146,6 +3149,8 @@ void pegasus_server_impl::reset_rocksdb_options(const rocksdb::ColumnFamilyOptio
// aspect 2:
target_opts->num_levels = base_opts.num_levels;
target_opts->write_buffer_size = base_opts.write_buffer_size;

reset_allow_ingest_behind_option(base_db_opt, target_db_opt, envs);
}

void pegasus_server_impl::reset_usage_scenario_options(
Expand All @@ -3166,6 +3171,19 @@ void pegasus_server_impl::reset_usage_scenario_options(
target_opts->max_write_buffer_number = base_opts.max_write_buffer_number;
}

void pegasus_server_impl::reset_allow_ingest_behind_option(
const rocksdb::DBOptions &base_db_opt,
rocksdb::DBOptions *target_db_opt,
const std::map<std::string, std::string> &envs)
{
if (envs.empty()) {
// for reopen db during load balance learning
target_db_opt->allow_ingest_behind = base_db_opt.allow_ingest_behind;
} else {
target_db_opt->allow_ingest_behind = parse_allow_ingest_behind(envs);
}
ninsmiracle marked this conversation as resolved.
Show resolved Hide resolved
}

void pegasus_server_impl::recalculate_data_cf_options(
const rocksdb::ColumnFamilyOptions &cur_data_cf_opts)
{
Expand Down
9 changes: 8 additions & 1 deletion src/server/pegasus_server_impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,8 +366,15 @@ class pegasus_server_impl : public pegasus_read_service
void reset_usage_scenario_options(const rocksdb::ColumnFamilyOptions &base_opts,
rocksdb::ColumnFamilyOptions *target_opts);

void reset_allow_ingest_behind_option(const rocksdb::DBOptions &base_db_opt,
acelyc111 marked this conversation as resolved.
Show resolved Hide resolved
rocksdb::DBOptions *target_db_opt,
const std::map<std::string, std::string> &envs);

void reset_rocksdb_options(const rocksdb::ColumnFamilyOptions &base_opts,
rocksdb::ColumnFamilyOptions *target_opts);
rocksdb::ColumnFamilyOptions *target_opts,
acelyc111 marked this conversation as resolved.
Show resolved Hide resolved
const rocksdb::DBOptions &base_db_opt,
rocksdb::DBOptions *target_db_opt,
const std::map<std::string, std::string> &envs);

// return true if successfully set
bool set_options(const std::unordered_map<std::string, std::string> &new_options);
Expand Down
Loading