diff --git a/notebooks/2024 Open Source India MariaDB Workshop .ipynb b/notebooks/2024 Open Source India MariaDB Workshop .ipynb index 1755d6c..b470256 100644 --- a/notebooks/2024 Open Source India MariaDB Workshop .ipynb +++ b/notebooks/2024 Open Source India MariaDB Workshop .ipynb @@ -3,7 +3,9 @@ { "cell_type": "markdown", "id": "8b9099f6-8b19-4a8f-8657-590712e6d006", - "metadata": {}, + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, "source": [ "# Fine Tuning and Optimising MariaDB Query\n", "\n", @@ -85,94 +87,6 @@ "(Source: Wikipedia: https://en.wikipedia.org/wiki/MariaDB and conversion by chatGPT)" ] }, - { - "cell_type": "code", - "execution_count": null, - "id": "47eae534-316d-4c29-b1ab-17c6f3d4960f", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "62fd9e24-f611-4190-b207-076bbcdb7bb1", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "bc18fb6b-cbd2-491c-9a82-8195808d23f5", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "eee7e34a-ae49-4c87-8a07-28fc414ba156", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "13669255-7193-47dd-96b5-fee3f710495e", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "40c4c39b-b4e7-4cf6-aac8-4ae7a12c7724", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "6f7cb5d9-0d11-4861-b267-9e490b343e88", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "961a0742-db90-4a7c-831a-4d8db7aa842e", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1381e994-d81c-4850-ae9e-a5cbf21a92f3", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c4cdaba8-2c29-40dc-91bb-e9b636e1cedb", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "647763f9-ae71-4673-aaf8-2b483f5aeff2", - "metadata": {}, - "outputs": [], - "source": [] - }, { "cell_type": "markdown", "id": "af51619f-2990-4c15-8265-8d1e52978192", @@ -738,6 +652,14 @@ "analyze format=json select t2.* from collate_join1 t1 left join collate_join2 t2 using (id) where t2.id is null;" ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "47eae534-316d-4c29-b1ab-17c6f3d4960f", + "metadata": {}, + "outputs": [], + "source": [] + }, { "cell_type": "markdown", "id": "7bf0e927-6585-4105-9b44-3a380609da57", @@ -2266,6 +2188,234 @@ "\n", "SET STATEMENT optimizer_switch={optimization=on} FOR SELECT ....\n" ] + }, + { + "cell_type": "markdown", + "id": "122ed4d5-7ec5-4554-b973-205fbacfce76", + "metadata": {}, + "source": [ + "# Problem Diagnosis\n", + "\n", + "Understand the problem before jumping to conclusions.\n", + "\n", + "## MariaDB Error Log\n", + "\n", + "std/stderr by default - means it goes to systemd service - so journalctl -u mariadb.service\n", + "\n", + "Take note of types:\n", + "* \\[Note\\] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB\n", + "* \\[Warning\\] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)\n", + "* \\[ERROR\\] mariadbd: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry\n", + "\n", + "To control what is logged **[log_warnings](https://mariadb.com/kb/en/server-system-variables/#log_warnings)** has levels.\n", + "\n", + "## Analyze format=json / explain\n", + "\n", + "Elements of EXPLAIN {EXTENDED}\n", + "```\n", + "MariaDB [test]> explain extended select seq from seq_1_to_30;\n", + "+------+-------------+-------------+-------+---------------+---------+---------+------+------+----------+-------------+\n", + "| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |\n", + "+------+-------------+-------------+-------+---------------+---------+---------+------+------+----------+-------------+\n", + "| 1 | SIMPLE | seq_1_to_30 | index | NULL | PRIMARY | 8 | NULL | 30 | 100.00 | Using index |\n", + "+------+-------------+-------------+-------+---------------+---------+---------+------+------+----------+-------------+\n", + "```\n", + "\n", + "This a plan. There may be changes. `ANALYZE` runs the query\n", + "\n", + "```\n", + "ANALYZE FORMAT=JSON select seq from seq_1_to_30\\G\n", + "*************************** 1. row ***************************\n", + "ANALYZE: {\n", + " \"query_optimization\": {\n", + " \"r_total_time_ms\": 0.211595358\n", + " },\n", + " \"query_block\": {\n", + " \"select_id\": 1,\n", + " \"r_loops\": 1,\n", + " \"r_total_time_ms\": 0.128058752,\n", + " \"nested_loop\": [\n", + " {\n", + " \"table\": {\n", + " \"table_name\": \"seq_1_to_30\",\n", + " \"access_type\": \"index\",\n", + " \"key\": \"PRIMARY\",\n", + " \"key_length\": \"8\",\n", + " \"used_key_parts\": [\"seq\"],\n", + " \"r_loops\": 1,\n", + " \"rows\": 30,\n", + " \"r_rows\": 30,\n", + " \"r_table_time_ms\": 0.005894714,\n", + " \"r_other_time_ms\": 0.112255834,\n", + " \"r_engine_stats\": {},\n", + " \"filtered\": 100,\n", + " \"r_filtered\": 100,\n", + " \"using_index\": true\n", + " }\n", + " }\n", + " ]\n", + " }\n", + "```\n", + "\n", + "As you can see ANALYZE has measurements, identification of what parts of the index where used.\n", + "\n", + "*r_* is the real what happened during execution. Non *r_* lows rows/filtered where an estimate. When they differ signficantly, try updating table statistics." + ] + }, + { + "cell_type": "markdown", + "id": "eed9dd6b-6bf2-41e2-9a47-56e0fc929930", + "metadata": {}, + "source": [ + "## Slow Query Log\n", + "\n", + "Since 10.11 - all dynamic, and session/global.\n", + "\n", + "* **[slow_query_log=1](https://mariadb.com/kb/en/server-system-variables/#slow_query_log)** - boolean variable to enable\n", + "* **[log_slow_query_time=0.4](https://mariadb.com/kb/en/server-system-variables/#log_slow_query_time)** - what constitues a slow query (default of 10s) is too big for many cases\n", + "* **[log_slow_verbosity=full](https://mariadb.com/kb/en/server-system-variables/#log_slow_verbosity)** - how much detail - *full* include EXPLAIN EXTENDED information.\n", + "\n", + "ref: https://mariadb.com/kb/en/slow-query-log-overview/#configuring-the-slow-query-log-time" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "eee7e34a-ae49-4c87-8a07-28fc414ba156", + "metadata": {}, + "outputs": [], + "source": [ + "## SHOW STATUS" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13669255-7193-47dd-96b5-fee3f710495e", + "metadata": {}, + "outputs": [], + "source": [ + "SHOW GLOBAL STATUS" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "40c4c39b-b4e7-4cf6-aac8-4ae7a12c7724", + "metadata": {}, + "outputs": [], + "source": [ + "SHOW SESSION STATUS" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6f7cb5d9-0d11-4861-b267-9e490b343e88", + "metadata": {}, + "outputs": [], + "source": [ + "SHOW ENGINE INNODB STATUS" + ] + }, + { + "cell_type": "markdown", + "id": "7420b9b2-5d2e-43c6-bf39-ad2e8f1be5e6", + "metadata": {}, + "source": [ + "Last Dead lock infomation\n", + "\n", + "```\n", + "=====================================\n", + "2024-10-22 16:03:46 0x7f64bad016c0 INNODB MONITOR OUTPUT\n", + "=====================================\n", + "Per second averages calculated from the last 50 seconds\n", + "-----------------\n", + "BACKGROUND THREAD\n", + "-----------------\n", + "srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 891 srv_idle\n", + "srv_master_thread log flush and writes: 891\n", + "----------\n", + "SEMAPHORES\n", + "----------\n", + "------------\n", + "TRANSACTIONS\n", + "------------\n", + "Trx id counter 111\n", + "Purge done for trx's n:o < 0 undo n:o < 0 state: running but idle\n", + "History list length 0\n", + "LIST OF TRANSACTIONS FOR EACH SESSION:\n", + "---TRANSACTION (0x7f64ca202340), not started\n", + "0 lock struct(s), heap size 1128, 0 row lock(s)\n", + "--------\n", + "FILE I/O\n", + "--------\n", + "Pending flushes (fsync): 0\n", + "187 OS file reads, 0 OS file writes, 1 OS fsyncs\n", + "0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s\n", + "---\n", + "LOG\n", + "---\n", + "Log sequence number 7704857\n", + "Log flushed up to 7704857\n", + "Pages flushed up to 7704857\n", + "Last checkpoint at 7704841\n", + "----------------------\n", + "BUFFER POOL AND MEMORY\n", + "----------------------\n", + "Total large memory allocated 167772160\n", + "Dictionary memory allocated 857728\n", + "Buffer pool size 8064\n", + "Free buffers 7756\n", + "Database pages 308\n", + "Old database pages 0\n", + "Modified db pages 0\n", + "Percent of dirty pages(LRU & free pages): 0.000\n", + "Max dirty pages percent: 90.000\n", + "Pending reads 0\n", + "Pending writes: LRU 0, flush list 0\n", + "Pages made young 0, not young 0\n", + "0.00 youngs/s, 0.00 non-youngs/s\n", + "Pages read 177, created 131, written 0\n", + "0.00 reads/s, 0.00 creates/s, 0.00 writes/s\n", + "Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000\n", + "Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s\n", + "LRU len: 308, unzip_LRU len: 0\n", + "I/O sum[0]:cur[0], unzip sum[0]:cur[0]\n", + "--------------\n", + "ROW OPERATIONS\n", + "--------------\n", + "0 read views open inside InnoDB\n", + "state: sleeping\n", + "----------------------------\n", + "END OF INNODB MONITOR OUTPUT\n", + "============================\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "79da27ff-0406-44b8-a382-74051d4280ac", + "metadata": {}, + "source": [ + "## Seeking Help\n", + "\n", + "Where to get help:\n", + " \n", + "* https://mariadb.com/kb\n", + "* https://stackoverflow.com/ (for SQL questions)\n", + "* https://dba.stackexchange.com/ (for DBA tuning and adminstration questions)\n", + "* https://mariadb.zulipchat.com" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3af1d4f0-7e63-415c-a7b6-a2e41afbe2d6", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": {