mariadb/storage/innobase/include/dict0stats.h

231 lines
8.6 KiB
C
Raw Normal View History

/*****************************************************************************
MDEV-14637: Fix hang due to persistent statistics Similar to the tables SYS_FOREIGN and SYS_FOREIGN_COLS, the tables mysql.innodb_table_stats and mysql.innodb_index_stats are updated by the InnoDB internal SQL parser, which fails to enforce the size limits of the data. Due to this, it is possible for InnoDB to hang when there are persistent statistics defined on partitioned tables where the total length of table name, partition name and subpartition name exceeds the incorrectly defined limit VARCHAR(64). That column should have been defined as VARCHAR(199). btr_node_ptr_max_size(): Interpret the VARCHAR(64) as VARCHAR(199), to prevent a hang in the case that the upgrade script has not been run. dict_table_schema_check(): Ignore difference in the length of the table_name column. ha_innobase::max_supported_key_length(): For innodb_page_size=4k, return a larger value so that the table mysql.innodb_index_stats can be created. This could allow "impossible" tables to be created, such that it is not possible to insert anything into a secondary index when both the secondary key and the primary key are long, but this is the easiest and most consistent way. The Oracle fix would only ignore the maximum length violation for the two statistics tables. os_file_get_status_posix(), os_file_get_status_win32(): Handle ENAMETOOLONG as well. This patch is based on the following change in MySQL 5.7.23. Not all changes were applied, and our variant allows persistent statistics to work without hangs even if the table definitions were not upgraded. From fdbdce701ab8145ae234c9d401109dff4e4106cb Mon Sep 17 00:00:00 2001 From: Aditya A <aditya.a@oracle.com> Date: Thu, 17 May 2018 16:11:43 +0530 Subject: [PATCH] Bug #26390736 THE FIELD TABLE_NAME (VARCHAR(64)) FROM MYSQL.INNODB_TABLE_STATS CAN OVERFLOW. In mysql.innodb_index_stats and mysql.innodb_table_stats tables the table name column didn't take into consideration partition names which can be more than varchar(64).
2018-08-02 17:35:24 +03:00
Copyright (c) 2009, 2018, Oracle and/or its affiliates. All Rights Reserved.
Copyright (c) 2017, 2018, MariaDB Corporation.
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; version 2 of the License.
This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc.,
2019-05-11 19:25:02 +03:00
51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA
*****************************************************************************/
/**************************************************//**
@file include/dict0stats.h
Code used for calculating and manipulating table statistics.
Created Jan 06, 2010 Vasil Dimov
*******************************************************/
#ifndef dict0stats_h
#define dict0stats_h
#include "dict0types.h"
#include "trx0types.h"
MDEV-14637: Fix hang due to persistent statistics Similar to the tables SYS_FOREIGN and SYS_FOREIGN_COLS, the tables mysql.innodb_table_stats and mysql.innodb_index_stats are updated by the InnoDB internal SQL parser, which fails to enforce the size limits of the data. Due to this, it is possible for InnoDB to hang when there are persistent statistics defined on partitioned tables where the total length of table name, partition name and subpartition name exceeds the incorrectly defined limit VARCHAR(64). That column should have been defined as VARCHAR(199). btr_node_ptr_max_size(): Interpret the VARCHAR(64) as VARCHAR(199), to prevent a hang in the case that the upgrade script has not been run. dict_table_schema_check(): Ignore difference in the length of the table_name column. ha_innobase::max_supported_key_length(): For innodb_page_size=4k, return a larger value so that the table mysql.innodb_index_stats can be created. This could allow "impossible" tables to be created, such that it is not possible to insert anything into a secondary index when both the secondary key and the primary key are long, but this is the easiest and most consistent way. The Oracle fix would only ignore the maximum length violation for the two statistics tables. os_file_get_status_posix(), os_file_get_status_win32(): Handle ENAMETOOLONG as well. This patch is based on the following change in MySQL 5.7.23. Not all changes were applied, and our variant allows persistent statistics to work without hangs even if the table definitions were not upgraded. From fdbdce701ab8145ae234c9d401109dff4e4106cb Mon Sep 17 00:00:00 2001 From: Aditya A <aditya.a@oracle.com> Date: Thu, 17 May 2018 16:11:43 +0530 Subject: [PATCH] Bug #26390736 THE FIELD TABLE_NAME (VARCHAR(64)) FROM MYSQL.INNODB_TABLE_STATS CAN OVERFLOW. In mysql.innodb_index_stats and mysql.innodb_table_stats tables the table name column didn't take into consideration partition names which can be more than varchar(64).
2018-08-02 17:35:24 +03:00
#define TABLE_STATS_NAME "mysql/innodb_table_stats"
#define INDEX_STATS_NAME "mysql/innodb_index_stats"
2013-03-26 00:03:13 +02:00
enum dict_stats_upd_option_t {
DICT_STATS_RECALC_PERSISTENT,/* (re) calculate the
statistics using a precise and slow
algo and save them to the persistent
storage, if the persistent storage is
not present then emit a warning and
fall back to transient stats */
DICT_STATS_RECALC_TRANSIENT,/* (re) calculate the statistics
using an imprecise quick algo
without saving the results
persistently */
2013-03-26 00:03:13 +02:00
DICT_STATS_EMPTY_TABLE, /* Write all zeros (or 1 where it makes sense)
into a table and its indexes' statistics
members. The resulting stats correspond to an
empty table. If the table is using persistent
statistics, then they are saved on disk. */
DICT_STATS_FETCH_ONLY_IF_NOT_IN_MEMORY /* fetch the stats
from the persistent storage if the in-memory
structures have not been initialized yet,
otherwise do nothing */
};
2013-03-26 00:03:13 +02:00
/*********************************************************************//**
Set the persistent statistics flag for a given table. This is set only
in the in-memory table object and is not saved on disk. It will be read
from the .frm file upon first open from MySQL after a server restart. */
UNIV_INLINE
void
dict_stats_set_persistent(
/*======================*/
dict_table_t* table, /*!< in/out: table */
ibool ps_on, /*!< in: persistent stats explicitly enabled */
ibool ps_off) /*!< in: persistent stats explicitly disabled */
2016-06-21 14:21:03 +02:00
MY_ATTRIBUTE((nonnull));
2013-03-26 00:03:13 +02:00
/** @return whether persistent statistics is enabled for a given table */
2013-03-26 00:03:13 +02:00
UNIV_INLINE
bool
dict_stats_is_persistent_enabled(const dict_table_t* table)
2016-06-21 14:21:03 +02:00
MY_ATTRIBUTE((nonnull, warn_unused_result));
2013-03-26 00:03:13 +02:00
/*********************************************************************//**
Set the auto recalc flag for a given table (only honored for a persistent
stats enabled table). The flag is set only in the in-memory table object
and is not saved in InnoDB files. It will be read from the .frm file upon
first open from MySQL after a server restart. */
UNIV_INLINE
void
dict_stats_auto_recalc_set(
/*=======================*/
dict_table_t* table, /*!< in/out: table */
ibool auto_recalc_on, /*!< in: explicitly enabled */
ibool auto_recalc_off); /*!< in: explicitly disabled */
/** @return whether auto recalc is enabled for a given table*/
2013-03-26 00:03:13 +02:00
UNIV_INLINE
bool
dict_stats_auto_recalc_is_enabled(const dict_table_t* table)
MY_ATTRIBUTE((nonnull, warn_unused_result));
2013-03-26 00:03:13 +02:00
/*********************************************************************//**
Initialize table's stats for the first time when opening a table. */
2013-03-26 00:03:13 +02:00
UNIV_INLINE
void
dict_stats_init(
/*============*/
dict_table_t* table); /*!< in/out: table */
2013-03-26 00:03:13 +02:00
/*********************************************************************//**
Deinitialize table's stats after the last close of the table. This is
used to detect "FLUSH TABLE" and refresh the stats upon next open. */
UNIV_INLINE
void
dict_stats_deinit(
/*==============*/
dict_table_t* table) /*!< in/out: table */
2016-06-21 14:21:03 +02:00
MY_ATTRIBUTE((nonnull));
2018-08-02 08:19:57 +03:00
#ifdef WITH_WSREP
/** Update the table modification counter and if necessary,
schedule new estimates for table and index statistics to be calculated.
@param[in,out] table persistent or temporary table
@param[in] thd current session */
void dict_stats_update_if_needed(dict_table_t* table, THD* thd)
MY_ATTRIBUTE((nonnull(1)));
#else
MDEV-12698 innodb.innodb_stats_del_mark test failure In my merge of the MySQL fix for Oracle Bug#23333990 / WL#9513 I overlooked some subsequent revisions to the test, and I also failed to notice that the test is actually always failing. Oracle introduced the parameter innodb_stats_include_delete_marked but failed to consistently take it into account in FOREIGN KEY constraints that involve CASCADE or SET NULL. When innodb_stats_include_delete_marked=ON, obviously the purge of delete-marked records should update the statistics as well. One more omission was that statistics were never updated on ROLLBACK. We are fixing that as well, properly taking into account the parameter innodb_stats_include_delete_marked. dict_stats_analyze_index_level(): Simplify an expression. (Using the ternary operator with a constant operand is unnecessary obfuscation.) page_scan_method_t: Revert the change done by Oracle. Instead, examine srv_stats_include_delete_marked directly where it is needed. dict_stats_update_if_needed(): Renamed from row_update_statistics_if_needed(). row_update_for_mysql_using_upd_graph(): Assert that the table statistics are initialized, as guaranteed by ha_innobase::open(). Update the statistics in a consistent way, both for FOREIGN KEY triggers and for the main table. If FOREIGN KEY constraints exist, do not dereference a freed pointer, but cache the proper value of node->is_delete so that it matches prebuilt->table. row_purge_record_func(): Update statistics if innodb_stats_include_delete_marked=ON. row_undo_ins(): Update statistics (on ROLLBACK of a fresh INSERT). This is independent of the parameter; the record is not delete-marked. row_undo_mod(): Update statistics on the ROLLBACK of updating key columns, or (if innodb_stats_include_delete_marked=OFF) updating delete-marks. innodb.innodb_stats_persistent: Renamed and extended from innodb.innodb_stats_del_mark. Reduced the unnecessarily large dataset from 262,144 to 32 rows. Test both values of the configuration parameter innodb_stats_include_delete_marked. Test that purge is updating the statistics. innodb_fts.innodb_fts_multiple_index: Adjust the result. The test is performing a ROLLBACK of an INSERT, which now affects the statistics. include/wait_all_purged.inc: Moved from innodb.innodb_truncate_debug to its own file.
2017-05-16 12:07:26 +03:00
/** Update the table modification counter and if necessary,
schedule new estimates for table and index statistics to be calculated.
@param[in,out] table persistent or temporary table */
2018-08-02 08:19:57 +03:00
void dict_stats_update_if_needed_func(dict_table_t* table)
MDEV-12698 innodb.innodb_stats_del_mark test failure In my merge of the MySQL fix for Oracle Bug#23333990 / WL#9513 I overlooked some subsequent revisions to the test, and I also failed to notice that the test is actually always failing. Oracle introduced the parameter innodb_stats_include_delete_marked but failed to consistently take it into account in FOREIGN KEY constraints that involve CASCADE or SET NULL. When innodb_stats_include_delete_marked=ON, obviously the purge of delete-marked records should update the statistics as well. One more omission was that statistics were never updated on ROLLBACK. We are fixing that as well, properly taking into account the parameter innodb_stats_include_delete_marked. dict_stats_analyze_index_level(): Simplify an expression. (Using the ternary operator with a constant operand is unnecessary obfuscation.) page_scan_method_t: Revert the change done by Oracle. Instead, examine srv_stats_include_delete_marked directly where it is needed. dict_stats_update_if_needed(): Renamed from row_update_statistics_if_needed(). row_update_for_mysql_using_upd_graph(): Assert that the table statistics are initialized, as guaranteed by ha_innobase::open(). Update the statistics in a consistent way, both for FOREIGN KEY triggers and for the main table. If FOREIGN KEY constraints exist, do not dereference a freed pointer, but cache the proper value of node->is_delete so that it matches prebuilt->table. row_purge_record_func(): Update statistics if innodb_stats_include_delete_marked=ON. row_undo_ins(): Update statistics (on ROLLBACK of a fresh INSERT). This is independent of the parameter; the record is not delete-marked. row_undo_mod(): Update statistics on the ROLLBACK of updating key columns, or (if innodb_stats_include_delete_marked=OFF) updating delete-marks. innodb.innodb_stats_persistent: Renamed and extended from innodb.innodb_stats_del_mark. Reduced the unnecessarily large dataset from 262,144 to 32 rows. Test both values of the configuration parameter innodb_stats_include_delete_marked. Test that purge is updating the statistics. innodb_fts.innodb_fts_multiple_index: Adjust the result. The test is performing a ROLLBACK of an INSERT, which now affects the statistics. include/wait_all_purged.inc: Moved from innodb.innodb_truncate_debug to its own file.
2017-05-16 12:07:26 +03:00
MY_ATTRIBUTE((nonnull));
2018-08-02 08:19:57 +03:00
# define dict_stats_update_if_needed(t,thd) dict_stats_update_if_needed_func(t)
#endif
MDEV-12698 innodb.innodb_stats_del_mark test failure In my merge of the MySQL fix for Oracle Bug#23333990 / WL#9513 I overlooked some subsequent revisions to the test, and I also failed to notice that the test is actually always failing. Oracle introduced the parameter innodb_stats_include_delete_marked but failed to consistently take it into account in FOREIGN KEY constraints that involve CASCADE or SET NULL. When innodb_stats_include_delete_marked=ON, obviously the purge of delete-marked records should update the statistics as well. One more omission was that statistics were never updated on ROLLBACK. We are fixing that as well, properly taking into account the parameter innodb_stats_include_delete_marked. dict_stats_analyze_index_level(): Simplify an expression. (Using the ternary operator with a constant operand is unnecessary obfuscation.) page_scan_method_t: Revert the change done by Oracle. Instead, examine srv_stats_include_delete_marked directly where it is needed. dict_stats_update_if_needed(): Renamed from row_update_statistics_if_needed(). row_update_for_mysql_using_upd_graph(): Assert that the table statistics are initialized, as guaranteed by ha_innobase::open(). Update the statistics in a consistent way, both for FOREIGN KEY triggers and for the main table. If FOREIGN KEY constraints exist, do not dereference a freed pointer, but cache the proper value of node->is_delete so that it matches prebuilt->table. row_purge_record_func(): Update statistics if innodb_stats_include_delete_marked=ON. row_undo_ins(): Update statistics (on ROLLBACK of a fresh INSERT). This is independent of the parameter; the record is not delete-marked. row_undo_mod(): Update statistics on the ROLLBACK of updating key columns, or (if innodb_stats_include_delete_marked=OFF) updating delete-marks. innodb.innodb_stats_persistent: Renamed and extended from innodb.innodb_stats_del_mark. Reduced the unnecessarily large dataset from 262,144 to 32 rows. Test both values of the configuration parameter innodb_stats_include_delete_marked. Test that purge is updating the statistics. innodb_fts.innodb_fts_multiple_index: Adjust the result. The test is performing a ROLLBACK of an INSERT, which now affects the statistics. include/wait_all_purged.inc: Moved from innodb.innodb_truncate_debug to its own file.
2017-05-16 12:07:26 +03:00
/*********************************************************************//**
Calculates new estimates for table and index statistics. The statistics
are used in query optimization.
@return DB_* error code or DB_SUCCESS */
2013-03-26 00:03:13 +02:00
dberr_t
dict_stats_update(
/*==============*/
dict_table_t* table, /*!< in/out: table */
dict_stats_upd_option_t stats_upd_option);
/*!< in: whether to (re) calc
the stats or to fetch them from
the persistent storage */
/*********************************************************************//**
Removes the information for a particular index's stats from the persistent
storage if it exists and if there is data stored for this index.
This function creates its own trx and commits it.
@return DB_SUCCESS or error code */
2013-03-26 00:03:13 +02:00
dberr_t
dict_stats_drop_index(
/*==================*/
const char* tname, /*!< in: table name */
const char* iname, /*!< in: index name */
char* errstr, /*!< out: error message if != DB_SUCCESS
is returned */
ulint errstr_sz);/*!< in: size of the errstr buffer */
/*********************************************************************//**
Removes the statistics for a table and all of its indexes from the
persistent storage if it exists and if there is data stored for the table.
This function creates its own transaction and commits it.
@return DB_SUCCESS or error code */
2013-03-26 00:03:13 +02:00
dberr_t
dict_stats_drop_table(
/*==================*/
const char* table_name, /*!< in: table name */
char* errstr, /*!< out: error message
if != DB_SUCCESS is returned */
ulint errstr_sz); /*!< in: size of errstr buffer */
/*********************************************************************//**
Fetches or calculates new estimates for index statistics. */
void
dict_stats_update_for_index(
/*========================*/
dict_index_t* index) /*!< in/out: index */
2016-06-21 14:21:03 +02:00
MY_ATTRIBUTE((nonnull));
2013-03-26 00:03:13 +02:00
/*********************************************************************//**
Renames a table in InnoDB persistent stats storage.
This function creates its own transaction and commits it.
2013-03-26 00:03:13 +02:00
@return DB_SUCCESS or error code */
dberr_t
dict_stats_rename_table(
/*====================*/
const char* old_name, /*!< in: old table name */
const char* new_name, /*!< in: new table name */
char* errstr, /*!< out: error string if != DB_SUCCESS
is returned */
size_t errstr_sz); /*!< in: errstr size */
/** Save an individual index's statistic into the persistent statistics
storage.
@param[in] index index to be updated
@param[in] last_update timestamp of the stat
@param[in] stat_name name of the stat
@param[in] stat_value value of the stat
@param[in] sample_size n pages sampled or NULL
@param[in] stat_description description of the stat
@param[in,out] trx in case of NULL the function will
allocate and free the trx object. If it is not NULL then it will be
rolled back only in the case of error, but not freed.
@return DB_SUCCESS or error code */
dberr_t
dict_stats_save_index_stat(
dict_index_t* index,
2017-12-08 13:10:41 +00:00
ib_time_t last_update,
const char* stat_name,
ib_uint64_t stat_value,
ib_uint64_t* sample_size,
const char* stat_description,
trx_t* trx);
Merge 10.1 into 10.2 This only merges MDEV-12253, adapting it to MDEV-12602 which is already present in 10.2 but not yet in the 10.1 revision that is being merged. TODO: Error handling in crash recovery needs to be improved. If a page cannot be decrypted (or read), we should cleanly abort the startup. If innodb_force_recovery is specified, we should ignore the problematic page and apply redo log to other pages. Currently, the test encryption.innodb-redo-badkey randomly fails like this (the last messages are from cmake -DWITH_ASAN): 2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994 2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1 2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption 2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown... i================================================================= ==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0 #0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750) #1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4 #2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17 #3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3 #4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2 #5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2 #6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3
2017-05-05 10:25:29 +03:00
/** Report an error if updating table statistics failed because
.ibd file is missing, table decryption failed or table is corrupted.
@param[in,out] table Table
@param[in] defragment true if statistics is for defragment
@retval DB_DECRYPTION_FAILED if decryption of the table failed
@retval DB_TABLESPACE_DELETED if .ibd file is missing
@retval DB_CORRUPTION if table is marked as corrupted */
dberr_t
dict_stats_report_error(dict_table_t* table, bool defragment = false)
MY_ATTRIBUTE((nonnull, warn_unused_result));
2013-03-26 00:03:13 +02:00
#include "dict0stats.ic"
#ifdef UNIV_ENABLE_UNIT_TEST_DICT_STATS
void test_dict_stats_all();
#endif /* UNIV_ENABLE_UNIT_TEST_DICT_STATS */
#endif /* dict0stats_h */