fts_cache_t::total_size_at_sync: New field, to sample total_size.
fts_add_doc_by_id(): Invoke sync if total_size has grown too much
since the previous sync request. (Maintain cache->total_size_at_sync.)
ib_wqueue_t::length: Caches ib_list_len(*items).
ib_wqueue_len(): Removed. We will refer to fts_optimize_wq->length
directly.
Based on mysql/mysql-server@bc9c46bf28
SHOW ENGINE INNODB MUTEX functionality is completely removed,
as are the InnoDB latching order checks.
We will enforce innodb_fatal_semaphore_wait_threshold
only for dict_sys.mutex and lock_sys.mutex.
dict_sys_t::mutex_lock(): A single entry point for dict_sys.mutex.
lock_sys_t::mutex_lock(): A single entry point for lock_sys.mutex.
FIXME: srv_sys should be removed altogether; it is duplicating tpool
functionality.
fil_crypt_threads_init(): To prevent SAFE_MUTEX warnings, we must
not hold fil_system.mutex.
fil_close_all_files(): To prevent SAFE_MUTEX warnings for
fil_space_destroy_crypt_data(), we must not hold fil_system.mutex
while invoking fil_space_free_low() on a detached tablespace.
Let us replace os_event_t with mysql_cond_t, and replace the
necessary ib_mutex_t with mysql_mutex_t so that they can be
used with condition variables.
Also, let us replace polling (os_thread_sleep() or timed waits)
with plain mysql_cond_wait() wherever possible.
Furthermore, we will use the lightweight srw_mutex for trx_t::mutex,
to hopefully reduce contention on lock_sys.mutex.
FIXME: Add test coverage of
mariabackup --backup --kill-long-queries-timeout
InnoDB buffer pool block and index tree latches depend on a
special kind of read-update-write lock that allows reentrant
(recursive) acquisition of the 'update' and 'write' locks
as well as an upgrade from 'update' lock to 'write' lock.
The 'update' lock allows any number of reader locks from
other threads, but no concurrent 'update' or 'write' lock.
If there were no requirement to support an upgrade from 'update'
to 'write', we could compose the lock out of two srw_lock
(implemented as any type of native rw-lock, such as SRWLOCK on
Microsoft Windows). Removing this requirement is very difficult,
so in commit f7e7f487d4b06695f91f6fbeb0396b9d87fc7bbf we
implemented an 'update' mode to our srw_lock.
Re-entrant or recursive locking is mostly needed when writing or
freeing BLOB pages, but also in crash recovery or when merging
buffered changes to an index page. The re-entrancy allows us to
attach a previously acquired page to a sub-mini-transaction that
will be committed before whatever else is holding the page latch.
The SUX lock supports Shared ('read'), Update, and eXclusive ('write')
locking modes. The S latches are not re-entrant, but a single S latch
may be acquired even if the thread already holds an U latch.
The idea of the U latch is to allow a write of something that concurrent
readers do not care about (such as the contents of BTR_SEG_LEAF,
BTR_SEG_TOP and other page allocation metadata structures, or
the MDEV-6076 PAGE_ROOT_AUTO_INC). (The PAGE_ROOT_AUTO_INC field
is only updated when a dict_table_t for the table exists, and only
read when a dict_table_t for the table is being added to dict_sys.)
block_lock::u_lock_try(bool for_io=true) is used in buf_flush_page()
to allow concurrent readers but no concurrent modifications while the
page is being written to the data file. That latch will be released
by buf_page_write_complete() in a different thread. Hence, we use
the special lock owner value FOR_IO.
The index_lock::u_lock() improves concurrency on operations that
involve non-leaf index pages.
The interface has been cleaned up a little. We will use
x_lock_recursive() instead of x_lock() when we know that a
lock is already held by the current thread. Similarly,
a lock upgrade from U to X is only allowed via u_x_upgrade()
or x_lock_upgraded() but not via x_lock().
We will disable the LatchDebug and sync_array interfaces to
InnoDB rw-locks.
The SEMAPHORES section of SHOW ENGINE INNODB STATUS output
will no longer include any information about InnoDB rw-locks,
only TTASEventMutex (cmake -DMUTEXTYPE=event) waits.
This will make a part of the 'innotop' script dead code.
The block_lock buf_block_t::lock will not be covered by any
PERFORMANCE_SCHEMA instrumentation.
SHOW ENGINE INNODB MUTEX and INFORMATION_SCHEMA.INNODB_MUTEXES
will no longer output source code file names or line numbers.
The dict_index_t::lock will be identified by index and table names,
which should be much more useful. PERFORMANCE_SCHEMA is lumping
information about all dict_index_t::lock together as
event_name='wait/synch/sxlock/innodb/index_tree_rw_lock'.
buf_page_free(): Remove the file,line parameters. The sux_lock will
not store such diagnostic information.
buf_block_dbg_add_level(): Define as empty macro, to be removed
in a subsequent commit.
Unless the build was configured with cmake -DPLUGIN_PERFSCHEMA=NO
the index_lock dict_index_t::lock will be instrumented via
PERFORMANCE_SCHEMA. Similar to
commit 1669c8890c
we will distinguish lock waits by registering shared_lock,exclusive_lock
events instead of try_shared_lock,try_exclusive_lock.
Actual 'try' operations will not be instrumented at all.
rw_lock_list: Remove. After MDEV-24167, this only covered
buf_block_t::lock and dict_index_t::lock. We will output their
information by traversing buf_pool or dict_sys.
LATCH_ID_OS_AIO_READ_MUTEX,
LATCH_ID_OS_AIO_WRITE_MUTEX,
LATCH_ID_OS_AIO_LOG_MUTEX,
LATCH_ID_OS_AIO_IBUF_MUTEX,
LATCH_ID_OS_AIO_SYNC_MUTEX: Remove. The tpool is not instrumented.
lock_set_timeout_event(): Remove.
srv_sys_mutex_key, srv_sys_t::mutex, SYNC_THREADS: Remove.
srv_slot_t::suspended: Remove. We only ever assigned this data member
true, so it is redundant.
ib_wqueue_wait(), ib_wqueue_timedwait(): Remove.
os_thread_join(): Remove.
os_thread_create(), os_thread_exit(): Remove redundant parameters.
These were missed in commit 5e62b6a5e0.
- fts_optimize_thread() uses dict_table_t object instead of table id.
So that it doesn't acquire dict_sys->mutex. It leads to remove the
hang of dict_sys->mutex between fts_optimize_thread() and other threads.
- in_queue to indicate whether the table is in fts_optimize_queue. It
is protected by fts_optimize_wq->mutex to avoid any race condition.
- fts_optimize_init() adds the fts table to the fts_optimize_wq
The function trx_purge_stop() was calling os_event_reset(purge_sys->event)
before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set()
call in srv_purge_coordinator_suspend() is protected by that X-latch.
It would seem a good idea to consistently protect both os_event_set()
and os_event_reset() calls with a common mutex or rw-lock in those
cases where os_event_set() and os_event_reset() are used
like condition variables, tied to changes of shared state.
For each os_event_t, we try to document the mutex or rw-lock that is
being used. For some events, frequent calls to os_event_set() seem to
try to avoid hangs. Some events are never waited for infinitely, only
timed waits, and os_event_set() is used for early termination of these
waits.
os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro
on other systems than Windows. TODO: remove this altogether and disable
innodb_use_native_aio on Windows.
os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0.
Contains also:
MDEV-10549 mysqld: sql/handler.cc:2692: int handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed. (branch bb-10.2-jan)
Unlike MySQL, InnoDB still uses THR_LOCK in MariaDB
MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan)
enable tests that were fixed in MDEV-10549
MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan)
fix main.innodb_mysql_sync - re-enable online alter for partitioned innodb tables
Contains also
MDEV-10547: Test multi_update_innodb fails with InnoDB 5.7
The failure happened because 5.7 has changed the signature of
the bool handler::primary_key_is_clustered() const
virtual function ("const" was added). InnoDB was using the old
signature which caused the function not to be used.
MDEV-10550: Parallel replication lock waits/deadlock handling does not work with InnoDB 5.7
Fixed mutexing problem on lock_trx_handle_wait. Note that
rpl_parallel and rpl_optimistic_parallel tests still
fail.
MDEV-10156 : Group commit tests fail on 10.2 InnoDB (branch bb-10.2-jan)
Reason: incorrect merge
MDEV-10550: Parallel replication can't sync with master in InnoDB 5.7 (branch bb-10.2-jan)
Reason: incorrect merge
Added option to disable multi-threaded flush with innodb_use_mtflush = 0
option, by default multi-threaded flush is used.
Updated innochecksum tool, still it does not support new checksums.
execution and global memory heap on shutdown.
Added a funcition to get work items from queue without waiting and
additional info when there is no work to do for a extended periods.
layout as we always had in trees containing only the builtin
2) win\configure.js WITH_INNOBASE_STORAGE_ENGINE still works.
storage/innobase/CMakeLists.txt:
fix to new directory name (and like 5.1)
storage/innobase/Makefile.am:
fix to new directory name (and like 5.1)
storage/innobase/handler/ha_innodb.cc:
fix to new directory name (and like 5.1)
storage/innobase/plug.in:
fix to new directory name (and like 5.1)
bzr branch mysql-5.1-performance-version mysql-trunk # Summit
cd mysql-trunk
bzr merge mysql-5.1-innodb_plugin # which is 5.1 + Innodb plugin
bzr rm innobase # remove the builtin
Next step: build, test fixes.
Fixed BUG#19542 "InnoDB doesn't increase the Handler_read_prev couter".
Fixed BUG#19609 "Case sensitivity of innodb_data_file_path gives stupid error".
Fixed BUG#19727 "InnoDB crashed server and crashed tables are ot recoverable".
Also:
* Remove remnants of the obsolete concept of memoryfixing tables and indexes.
* Remove unused dict_table_LRU_trim().
* Remove unused 'trx' parameter from dict_table_get_on_id_low(),
dict_table_get(), dict_table_get_and_increment_handle_count().
* Add a normal linked list implementation.
* Add a work queue implementation.
* Add 'level' parameter to mutex_create() and rw_lock_create().
Remove mutex_set_level() and rw_lock_set_level().
* Rename SYNC_LEVEL_NONE to SYNC_LEVEL_VARYING.
* Add support for bound ids in InnoDB's parser.
* Define UNIV_BTR_DEBUG for enabling consistency checks of
FIL_PAGE_NEXT and FIL_PAGE_PREV when accessing sibling
pages of B-tree indexes.
btr_validate_level(): Check the validity of the doubly linked
list formed by FIL_PAGE_NEXT and FIL_PAGE_PREV.
* Adapt InnoDB to the new tablename to filename encoding in MySQL 5.1.
ut_print_name(), ut_print_name1(): Add parameter 'table_id' for
distinguishing names of tables from other identifiers.
New: innobase_convert_from_table_id(), innobase_convert_from_id(),
innobase_convert_from_filename(), innobase_get_charset.
dict_accept(), dict_scan_id(), dict_scan_col(), dict_scan_table_name(),
dict_skip_word(), dict_create_foreign_constraints_low(): Add
parameter 'cs' so that isspace() can be replaced with my_isspace(),
whose operation depends on the connection character set.
dict_scan_id(): Convert identifier to UTF-8.
dict_str_starts_with_keyword(): New extern function, to replace
dict_accept() in row_search_for_mysql().
mysql_get_identifier_quote_char(): Replaced with innobase_print_identifier().
ha_innobase::create(): Remove the thd->convert_strin() call. Pass the
statement to InnoDB in the connection character set and let InnoDB
convert the identifier to UTF-8.
* Add max_row_size to dict_table_t.
* btr0cur.c
btr_copy_externally_stored_field(): Only set the 'offset' variable
when needed.
* buf0buf.c
buf_page_io_complete(): Write to the error log if the page number or
the space id o the disk do not match those in memory. Also write to
the error log if a page was read from the doublewrite buffer. The
doublewrite buffer should be only read by the lower-level function
fil_io() at database startup.
* dict0dict.c
dict_scan_table_name(): Remove fallback to differently encoded name
when the table is not found. The encoding is handled at a higher level.
* ha_innodb.cc
Increment statistic counter in ha_innobase::index_prev() (bug 19542).
Add innobase_convert_string wrapper function and a new file
ha_prototypes.h.
innobase_print_identifier(): Remove TODO comment before calling
get_quote_char_for_identifier(). That function apparently assumes
the identifier to be encoded in UTF-8.
* ibuf0ibuf.c|h
ibuf_count_get(), ibuf_counts[], ibuf_count_inited(): Define these
only #ifdef UNIV_IBUF_DEBUG. Previously, when compiled without
UNIV_IBUF_DEBUG, invoking ibuf_count_get() would crash InnoDB.
The function is only being called #ifdef UNIV_IBUF_DEBUG.
* innodb.result
Adjust the results for changes in the foreign key error messages.
* mem0mem.c|h
New: mem_heap_dup(), mem_heap_printf(), mem_heap_cat().
* os0file.c
Check the page trailers also after writing to disk. This improves
chances of diagnosing bug 18886.
os_file_check_page_trailers(): New function for checking that the
two copies of the LSN stamped on the page match.
os_aio_simulated_handle(): Call os_file_check_page_trailers()
before and after os_file_write().
* row0mysql.c
Move trx_commit_for_mysql(trx) calls before calls to
row_mysql_unlock_data_dictionary(trx) (bug 19727).
* row0sel.c
row_fetch_print(): Handle SQL NULL values without crashing.
row_sel_store_mysql_rec(): Remove useless call to rec_get_nth_field
when handling an externally stored column.
Fetch externally stored fields when using InnoDB's internal SQL
parser.
Optimize BLOB selects by using prebuilt->blob_heap directly instead
of first reading BLOB data to a temporary heap and then copying it
to prebuilt->blob_heap.
* srv0srv.c
srv_master_thread(): Remove unreachable code.
* srv0start.c
srv_parse_data_file_paths_and_sizes(): Accept lower-case 'm' and
'g' as abbreviations of megabyte and gigabyte (bug 19609).
srv_parse_megabytes(): New fuction.
* ut0dbg.c|h
Implement InnoDB assertions (ut_a and ut_error) with abort() when
the code is compiled with GCC 3 or later on other platforms than
Windows or Netware. Also disable the variable ut_dbg_stop_threads
and the function ut_dbg_stop_thread() i this case, unless
UNIV_SYC_DEBUG is defined. This should allow the compiler to
generate more compact code for assertions.
* ut0list.c|h
Add ib_list_create_heap().
mysql-test/r/innodb.result:
Applied innodb-5.1-ss594 snapshot.
mysql-test/t/innodb.test:
Copy the comment from the beginning of the file
to the end because MySQL developers continue
adding test cases to this file.
sql/ha_innodb.cc:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/CMakeLists.txt:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/Makefile.am:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0btr.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0cur.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0pcur.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0sea.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/buf/buf0buf.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/buf/buf0flu.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/data/data0type.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0crea.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0dict.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0load.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0mem.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/fil/fil0fil.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ha/hash0hash.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ibuf/ibuf0ibuf.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/data0type.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/dict0dict.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/dict0dict.ic:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/dict0mem.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/fil0fil.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/fsp0fsp.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ibuf0ibuf.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/mem0mem.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/pars0pars.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/pars0sym.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/pars0types.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/rem0rec.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/row0purge.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/row0undo.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/srv0srv.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/sync0rw.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/sync0sync.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/univ.i:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0dbg.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0ut.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/lock/lock0lock.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/log/log0log.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/log/log0recv.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/mem/mem0dbg.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/mem/mem0mem.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/mem/mem0pool.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/os/os0file.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/os/os0thread.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/lexyy.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/pars0lex.l:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/pars0pars.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/pars0sym.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0ins.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0mysql.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0purge.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0row.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0sel.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0upd.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/srv/srv0srv.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/srv/srv0start.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/sync/sync0arr.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/sync/sync0rw.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/sync/sync0sync.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/thr/thr0loc.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0purge.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0rec.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0roll.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0rseg.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0sys.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0trx.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/Makefile.am:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0dbg.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0ut.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ha_prototypes.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0list.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0list.ic:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0wqueue.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0list.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0wqueue.c:
Applied innodb-5.1-ss594 snapshot.
