test_quantized_training is flaky

[StrikerRUS]

Just noticed the test failure in master with cuda 11.8.0 pip (ubuntu20.04, clang, Python 3.11) job.

=================================== FAILURES ===================================
___________________________ test_quantized_training ____________________________

    def test_quantized_training():
        X, y = make_synthetic_regression()
        ds = lgb.Dataset(X, label=y)
        bst_params = {"num_leaves": 15, "verbose": -1, "seed": 0}
        bst = lgb.train(bst_params, ds, num_boost_round=10)
        rmse = np.sqrt(np.mean((bst.predict(X) - y) ** 2))
        bst_params.update(
            {
                "use_quantized_grad": True,
                "num_grad_quant_bins": 30,
                "quant_train_renew_leaf": True,
            }
        )
        quant_bst = lgb.train(bst_params, ds, num_boost_round=10)
        quant_rmse = np.sqrt(np.mean((quant_bst.predict(X) - y) ** 2))
>       assert quant_rmse < rmse + 6.0
E       assert np.float64(2988.108396382464) < (np.float64(24.313125588236623) + 6.0)

tests/python_package_test/test_engine.py:4579: AssertionError

LightGBM/tests/python_package_test/test_engine.py

Lines 4564 to 4579 in 4a60a53

	def test_quantized_training():
	X, y = make_synthetic_regression()
	ds = lgb.Dataset(X, label=y)
	bst_params = {"num_leaves": 15, "verbose": -1, "seed": 0}
	bst = lgb.train(bst_params, ds, num_boost_round=10)
	rmse = np.sqrt(np.mean((bst.predict(X) - y) ** 2))
	bst_params.update(
	{
	"use_quantized_grad": True,
	"num_grad_quant_bins": 30,
	"quant_train_renew_leaf": True,
	}
	)
	quant_bst = lgb.train(bst_params, ds, num_boost_round=10)
	quant_rmse = np.sqrt(np.mean((quant_bst.predict(X) - y) ** 2))
	assert quant_rmse < rmse + 6.0

I don't think we should do anything right now. Just posting this issue to count future failures similarly how we do in #4074.

cc @shiyu1994 @jameslamb

[shiyu1994]

This is perhaps due to the nondeterministic result among different environments. I'll look into this and provide a more deterministic check for this test case.

[StrikerRUS]

The second failure just happened: https://github.com/microsoft/LightGBM/actions/runs/12387337319/job/34576641552?pr=6761#step:5:7146

>       assert quant_rmse < rmse + 6.0
E       assert np.float64(2988.108396382464) < (np.float64(24.313125588236623) + 6.0)

Interestingly, that numbers are the same.