modified: storage/connect/bsonudf.cpp
modified: storage/connect/jsonudf.cpp
- Stringify option is now a ; separated list of columns
modified: storage/connect/json.cpp
modified: storage/connect/json.h
modified: storage/connect/mongo.h
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabcmg.cpp
modified: storage/connect/tabcmg.h
modified: storage/connect/tabjmg.cpp
modified: storage/connect/tabjmg.h
modified: storage/connect/tabjson.cpp
- PrepareColist not a static function anymore (+ typo)
modified: storage/connect/taboccur.cpp
- JDVC: Recognize schema (database) from a wrapper server
modified: storage/connect/tabjdbc.cpp
The `item_func::JSON_EXTRACT_FUNC` was not handled correctly in the previous
versions on the Spider storage engine, which makes queries like
`SELECT * FROM t1 WHERE json_extract(jdoc, '$.Age')=20`
failed with syntax error.
This patch writes specific code to handle JSON_EXTRACT in the Spider Storage
Engine and fix that bug.
In commit 83d2e0841e (MDEV-24041)
we failed to notice that in addition to the bug with
DELETE and ON DELETE CASCADE, there is another bug with
UPDATE and ON UPDATE CASCADE.
row_ins_foreign_fill_virtual(): Use the correct memory heap
for everything that will be reachable from the cascade->update
that we return to the caller.
Note: It is correct to use the shorter-lived cascade->heap for
rec_get_offsets(), because that memory will be abandoned when
row_ins_foreign_fill_virtual() returns.
Problem:
Server throws OOM error when we execute twitter load with SELECTs
for UPDATE + UPDATES, and SELECT queries on tables with full-text
index.
FTS cache->total_memory store count of total memory allocated to FTS
cache (for all fulltext indexes on a table).
For each word in fts cache, we store doc-id & word position in a
node->ilist.
we increment cache->total_memory with size of doc-id & word position
whereas we allocate ilist in chuck of 16, 32 ,64 bytes or 1.2 times
of last size.
When we wil insert huge amount of data into the FTS aux index tables
then collectively these small chucks for each token become huge
unaccounted memory allocated for FTS cache.
Fix:
Incremented cache->total_memory by size of chunk allocated to
node->ilist.
RB: 25286
Reviewed by : Rahul Agarkar <rahul.agarkar@oracle.com>
mysql/mysql-server@7ab5707f1c
This patch changes it so that we do not free old BP `page_hash`, but rather modify it's parameters, during resize.
RB: 26084
Reviewed-by: Marcin Babij <marcin.babij@oracle.com>
Reviewed-by: Yasufumi Kinoshita <yasufumi.kinoshita@oracle.com>
mysql/mysql-server@ea3adc6a11
ha_innobase::prepare_inplace_alter_table(): Unless the table is
being rebuilt, determine the maximum column length based on the
current ROW_FORMAT of the table. When TABLE_SHARE (and the .frm file)
contains no explicit ROW_FORMAT, InnoDB table creation or rebuild
will use innodb_default_row_format.
Based on mysql/mysql-server@3287d33acd
InnoDB tablespace identifiers and page numbers are 32-bit numbers.
Let us use a 32-bit type for them in innochecksum.
The changes in commit 1918bdf32c
broke the build on 32-bit Windows.
Thanks to Vicențiu Ciorbaru for an initial version of this fixup.
This gives a short overview over found/missing dependencies as well
as enabled/disabled features.
Initial author Heinz Wiesinger <heinz@m2mobi.com>
Additions by Vicențiu Ciorbaru <vicentiu@mariadb.org>
* Report all plugins enabled via MYSQL_ADD_PLUGIN
* Simplify code. Eliminate duplication by making use of WITH_xxx
variable values to set feature "ON" / "OFF" state.
Reviewed by: wlad@mariadb.com (code details) serg@mariadb.com (the idea)
In commit 2e814d4702 on MariaDB 10.2
the switch case statement in trx_flush_log_if_needed_low() regressed.
Since 10.2 this code was refactored to have switches in descending
order, so value of 3 for innodb_flush_log_at_trx_commit is behaving
the same as value of 2, that is no FSYNC is being enforced during
COMMIT phase. The switch should however not be empty and cases 2 and 3
should not have the identical contents.
As per documentation, setting innodb_flush_log_at_trx_commit to 3
should do FSYNC to disk if innodb_flush_log_at_trx_commit is set to 3.
This fixes the regression so that the switch statement again does
what users expect the setting should do.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
The root cause of the bug is in `spider_db_mbase_util::open_item_func()`.
The function handles an instance of the `Item_func` class based on its
`Item_func::Functype`.
The `Functype` of `CASE WHEN ... THEN` is `CASE_SEARCHED_FUNC`.
However, the Spider SE doesn't recognize this `Functype` because
`CASE_SEARCHED_FUNC` is newly added by 4de0d92. This results in the wrong
handling of `CASE WHEN ... THEN`.
The above also applies to `CASE_SIMPLE_FUNC`.
The function spider_db_append_key_where_internal() converts
HA_READ_AFTER_KEY to '>'. The conversion seems to be correct for
single-column indexes because HA_READ_AFTER_KEY means "read the
key after the provided value."
However, how about multi-column indexes? Assume that there is a
multi-column index on c1 and c2 and we search with the condition
'c1 >= 100 AND c2 > 200'. The key_range.flag corresponds to the
search condition could be HA_READ_AFTER_KEY. In such a case,
we could not simply convert HA_READ_AFTER_KEY to '>'.
The correct conversion is to convert HA_READ_AFTER_KEY to '>'
only for the last column in key_part_map and to convert
HA_READ_AFTER_KEY to '>=' for the other column.
The similar discussion also applies to the conversion from
key_range.flag to a sign of inequality.
modified: storage/connect/ha_connect.cc
- Use NOTE instead of WARNING in connect_assisted_discovery
This because MariaDB raise an error when doing so
modified: storage/connect/ha_connect.cc
modified: storage/connect/tabrest.cpp
- Make MONGO tables recognize STRINGIFY and JsonAllPath
modified: storage/connect/mongo.cpp
modified: storage/connect/mongo.h
modified: storage/connect/tabcmg.h
modified: storage/connect/tabjmg.cpp
modified: storage/connect/tabcmg.cpp
modified: storage/connect/tabjmg.h
- Fix OBJECT option for Pretty != 2 JSN and BSON tables
Accept all syntaxes for the OBJECT option
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabjson.cpp
- Use my_snprintf in function xcurl (by vuvova)
modified: storage/connect/tabrest.cpp
- Format dates entered as integer when formatted
modified: storage/connect/value.cpp
modified: storage/connect/value.h
trx_undo_mem_create_at_db_start(): Relax too strict upgrade checks
that were introduced in
commit e46f76c974 (MDEV-15912).
On commit, pages will typically be set to TRX_UNDO_CACHED state.
Having the type TRX_UNDO_INSERT in such pages is common and
unproblematic; the type would be reset in trx_undo_reuse_cached().
trx_rseg_array_init(): On failure, clean up the rollback segments
that were initialized so far, to avoid an assertion failure later
during shutdown.
init_crypt_key(): On failure, set info->key_version to
ENCRYPTION_KEY_VERSION_INVALID.
log_crypt_101_read_block(): Refuse to attempt decryption if
info->key_version is ENCRYPTION_KEY_VERSION_INVALID.
trx_undo_report_row_operation(): Fix a race condition that was introduced
in commit f74023b955 (MDEV-15090).
We must not access undo_block after the page latch has been released
in mtr_t::commit(), because the block could be evicted or replaced.
Let us simply refuse an upgrade from earlier versions if the
upgrade procedure was not followed. This simplifies the purge,
commit, and rollback of transactions.
Before upgrading to MariaDB 10.3 or later, a clean shutdown
of the server (with innodb_fast_shutdown=1 or 0) is necessary,
to ensure that any incomplete transactions are rolled back.
The undo log format was changed in MDEV-12288. There is only
one persistent undo log for each transaction.
and *never* disable tests in suite.pm based on $::opt_big_test,
this will make the test skipped both as too big (for ./mtr)
and as too small (for ./mtr --big --big).
- During online alter conversion from compact to redundant,
virtual column field length already set during
innobase_get_computed_value(). Skip the char(n) check for
virtual column in row_merge_buf_add()
- InnoDB fails to check DB_COMPUTE_VALUE_FAILED error in
row_merge_read_clustered_index() and wrongly asserts that
the buffer shouldn't be ran out of memory. Alter table
should give warning when the column value is being
truncated.
Problem:
=======
- InnoDB iterates the fil_system space list to encrypt the
tablespace in case of key rotation. But it is not
necessary for any encryption plugin which doesn't do
key version rotation.
Solution:
=========
- Introduce a new variable called srv_encrypt_rotate to
indicate whether encryption plugin does key rotation
fil_space_crypt_t::key_get_latest_version(): Enable the
srv_encrypt_rotate only once if current key version is
higher than innodb_encyrption_rotate_key_age
fil_crypt_must_default_encrypt(): Default encryption tables
should be added to default_encryp_tables list if
innodb_encyrption_rotate_key_age is zero and encryption
plugin doesn't do key version rotation
fil_space_create(): Add the newly created space to
default_encrypt_tables list if
fil_crypt_must_default_encrypt() returns true
Removed the nondeterministic select from
innodb-key-rotation-disable test. By default,
InnoDB adds the tablespace to the rotation list and
background crypt thread does encryption of tablespace.
So these select doesn't give reliable results.
- Rename the rotation list to default_encrypt_tables in
fil_system_t. Because rotation list naming could be
misleading when it comes to key version rotation
- Rename is_in_rotation_list to is_in_default_encrypt
in fil_space_t
- Rename keyrotate_next function to default_encrypt_next
fil_system_t::default_encrypt_next(): Find the next
suitable default encrypt table if beginning of
default_encrypt_tables list has been scheduled
to be deleted
Occasionally, the test innodb.alter_copy would fail in MariaDB 10.6.1,
reporting DB_MISSING_HISTORY during CHECK TABLE. It started to occur during
the development of MDEV-25180, which introduced purge_sys.stop_SYS().
If we delay purge more during DDL operations, then the test would
almost always fail. The reason is that during startup we will restore
a purge view, and CHECK TABLE would still use REPEATABLE READ
even though innodb_read_only is set and other isolation levels
than READ UNCOMMITTED are not guaranteed to work.
ha_innobase::check(): Use READ UNCOMMITTED isolation level if
innodb_read_only is set or innodb_force_recovery exceeds 3.
dict_set_corrupted(): Do not update the persistent data dictionary
if innodb_force_recovery exceeds 3.
buf_read_ibuf_merge_pages(): If space->size is 0, invoke
fil_space_get_size() to determine the size of the tablespace
by reading the header page. Only after that proceed to delete
any entries that are beyond the end of the tablespace.
Otherwise, we could be deleting valid entries that actually
need to be applied.
This fixes a regression that had been introduced in
commit b80df9eba2 (MDEV-21069),
which aimed to avoid crashes during DROP TABLE of corrupted tables.
ha_heap::external_lock contains some consistency checks for the table,#
in a debug compilation.
This code suffers from lack of synchronization, in a rare case
where mysql_lock_tables() fail, and unlock is forced, even if lock was
not previously taken.
To workaround, require EXTRA_DEBUG compile definition in order to activate
the consistency checks.The code still might be useful in some cases - but
the audience are developers looking for errors in single-threaded scenarios,
rather than multiuser stress-tests.
- As solution `PLUGIN_CONNECT=NO` use early check to disable plugin:
Solution suggested by wlad@mariadb.com
- `JNI_FOUND` is a internal result variable and should be set with
cached library and header variables (like `JAVA_INCLUDE_PATH`) defined.
* Note: wrapper cmake/FindJNI.cmake runs first time and cmake native Find<module> returns only cached variable, like `JAVA_INCLUDE_PATH`, results variable are not cached).
Reviewed by: serg@mariadb.com
InnoDB should calculate the MBR for the first field of
spatial index and do the comparison with the clustered
index field MBR. Due to MDEV-25459 refactoring, InnoDB
calculate the length of the first field and fails with
too long column error.
The only call of the virtual member function
handler::update_table_comment() was removed in
commit 82d28fada7 (MySQL 5.5.53)
but the implementation was not removed.
The only non-trivial implementation was for InnoDB. The information
is now returned via handler::get_foreign_key_create_info() and
ha_statistics::delete_length.
FTS add index fails
Problem:
========
InnoDB double frees the table if auxiliary fts table
creation fails and fails to set the dict operation
for the transaction. It leads to failure while
dropping newly added index.
Solution:
=========
InnoDB should avoid double freeing and set the
dictionary operation of transaction in
fts_create_common_tables()
InnoDB truncate table fails to load the fts stopword table into
cache. In that case, InnoDB double frees the truncate creation
transaction. InnoDB should free the transaction which was
created inside ha_innobase::create.
ha_innobase::open(): If the table is only being opened by purge
for evaluating virtual column values, avoid invoking
initialize_auto_increment(), because the purge thread may already
be holding an shared latch on the clustered index root page.
Shared latches are not recursive. The additional request would lead
to a hang if another thread has started waiting for an exclusive latch.
ha_innobase::index_read(): If an autocommit non-locking transaction was
already started, refuse to access a SPATIAL INDEX.
Once a non-locking autocommit transaction has started, it must remain
in that mode (not acquire any locks).
This should fix one cause of the assertion failure that would occur in
DeadlockChecker::check_and_resolve() under heavy load, presumably
due to concurrent execution of trx_commit_in_memory().
modified: sql/mysqld.h
modified: storage/connect/CMakeLists.txt
modified: storage/connect/javaconn.cpp
modified: storage/connect/mycat.cc
modified: storage/connect/mycat.h
- Get a handled on a not pooled client
This to avoid a .50 delay when closing
modified: storage/connect/cmgoconn.cpp
modified: storage/connect/cmgoconn.h
- Allow FIELD_FORMAT options for DECIMAL type
modified: storage/connect/tabdos.cpp
- Update tests and result to reflect last changes
Also because Oracle password has changed
modified: storage/connect/mysql-test/connect/r/jdbc_oracle.result
modified: storage/connect/mysql-test/connect/r/json_java_2.result
modified: storage/connect/mysql-test/connect/r/json_java_3.result
modified: storage/connect/mysql-test/connect/r/json_mongo_c.result
modified: storage/connect/mysql-test/connect/r/mongo_c.result
modified: storage/connect/mysql-test/connect/r/odbc_oracle.result
modified: storage/connect/mysql-test/connect/t/jdbc_oracle.test
modified: storage/connect/mysql-test/connect/t/odbc_oracle.test
- Typo
modified: storage/connect/reldef.cpp
This reverts commit 72fa9dabad
but doesn't recover deleted jars - they still exist in
mysql-test/connect/std_data, no need to have them twice.
Also it removes a redundant copy on JavaWrappers.jar
- Limit decimals of doubles printed from MongoDB
Done in function Mini for Mongo C Driver and Java Driver
Done in function SerializeValue for Java tables using the J Driver
modified: storage/connect/cmgoconn.cpp
modified: storage/connect/json.cpp
modified: storage/connect/json.h
modified: storage/connect/tabjmg.cpp
- Fix crash when using BSON_TYPE_DECIMAL128
modified: storage/connect/cmgoconn.cpp
- Collection name default to table name
Fix it when creating tables via discovery
modified: storage/connect/ha_connect.cc
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabjson.cpp
InnoDB tries to fetch the deleted doc ids for discarded
tablespace. In i_s_fts_deleted_generic_fill(), InnoDB needs
to check whether the table is discarded or not before fetching
deleted doc ids.
fil_ibd_load(): Remove a message that is basically saying that
everything works as expected. The other "Ignoring data file" message
about the presence of an extraneous file will be retained
(and expected by the test innodb.log_file_name).
Attempt to build MariaDB server on MacOS could result in
compilation errors like the following one:
In file included from server-10.2/storage/perfschema/cursor_by_account.cc:28:
In file included from server-10.2/include/my_global.h:287:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/math.h:309:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/type_traits:418:
server-10.2/version:1:1: error: expected unqualified-id
MYSQL_VERSION_MAJOR=10
^
server-10.2/build.dir/include/my_config.h:529:29: note: expanded from macro 'MYSQL_VERSION_MAJOR'
This kind of compiler errors occur by the reson that compiler's system headers
contain the directive '#include <version>' and a compiler is invoked
with -I${CMAKE_SOURCE_DIR}.
The MariaDB source code root directory contains the file VERSION that is handled
by the compiler during processing the directive #include <version>
since file names on MacOS are case insensetive, so version and VERSION is treated as
the same file name.
To fix the issue the source code root directory should be removed from a list
of directories used by the compiler for include search path.
InnoDB startup hangs if a DDL transaction needs to be
rolled back and a recovered transaction on statistics
tables exists. In that case, InnoDB should rollback
the transaction which holds locks on innodb_table_stats
or innodb_index_stats during trx_rollback_or_clean_recovered().
InnoDB fails to fetch the index type when innodb dictionary
doesn't match with frm. InnoDB should return corrupted if it
can't find the index in ha_innobase::index_type().
The assertion is improved: storage engines like myisam always have to store
at least one field, so the assertion does not cover tables with no stored
columns.
innodb_debug_sync was introduced in commit
b393e2cb0c and reverted in
commit fc58c17216 due to memory leak reported
by valgrind, see MDEV-21336.
The leak is now fixed by adding `rw_lock_free(&slot->debug_sync_lock)`
after background thread working loop is finished, and the patch is
reapplied, with respect to c++98 fixes by Marko.
The missing DEBUG_SYNC for MDEV-18546 in row0vers.cc is also reapplied.
row_merge_is_index_usable(): Allow access to any SEQUENCE, even if it was
created after the read view. SQL sequences are no-rollback tables with no
history at all.
This is a backport of
commit fd9ca2a742 (MDEV-23295) and
commit 9a156e1a23 (MDEV-23345) to 10.3.
An instant ADD/DROP/reorder column could create a dummy table
object with the wrong ROW_FORMAT when innodb_default_row_format
was changed between CREATE TABLE and ALTER TABLE.
prepare_inplace_alter_table_dict(): If we had promised that
ALGORITHM=INPLACE is supported, we must preserve the ROW_FORMAT.
The rest of the changes are related to adding
Alter_inplace_info::inplace_supported to cache the return value of
handler::check_if_supported_inplace_alter().
It is possible that an object that was originally created by
open_purge_table() will remain cached and reused for SQL execution.
Our previous fix wrongly assumed that ha_innobase::open() would
always be called before SQL execution starts. Therefore, we must
invoke dict_stats_init() in ha_innobase::info_low() instead of
only doing it in ha_innobase::open().
Note: Concurrent execution of dict_stats_init() on the same table
is possible, but it also was possible between two calls to
ha_innobase::open(), with no ill effects observed.
This should fix the assertion failure on stat_initialized.
A possibly easy way to reproduce it would have been
to run the server with innodb_force_recovery=2 (disable the purge of
history), update a table so that an indexed virtual column will be
affected, and finally restart the server normally (purge enabled),
to observe a crash when the table is accessed from SQL.
The problem was first observed and this fix verified by
Elena Stepanova. Also Thirunarayanan Balathandayuthapani
repeated the problem.
row_sel_sec_rec_is_for_clust_rec(): If the field in the
clustered index record stored off page, always fetch it,
also when the secondary index field has been built on the
entire column. This was broken ever since the InnoDB Plugin
for MySQL Server 5.1 introduced ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPRESSED for InnoDB tables. That code was first
introduced in this tree in
commit 3945d5e554.
For the original ROW_FORMAT=REDUNDANT and the MySQL 5.0.3
ROW_FORMAT=COMPRESSED, there was no problem, because for
those tables we always stored at least a 768-byte prefix of
each column in the clustered index record.
row_sel_sec_rec_is_for_blob(): Allow prefix_len==0 for matching
the full column.
Buffer overflow in ib_push_warning() fixed by using vsnprintf().
InnoDB parser was obsoleted by MDEV-16417.
Thanks to Nikita Malyavin for review and suggestion.
to mysql interpreter
InnoDB returns uninitialized statistics to mysql interpreter
when background thread is opening the table. So it leads to
assertion failure. In that case, InnoDB avoid sending
innodb statistics information to mysql interpreter.
PROBLEM
-------
1. The customer had presented a stack which had many threads waiting on
multiple mutexes like LOCK_Status, srv_innodb_monitor_mutex, ibuf_mutex etc.
2. The root cause was that the AHI latch was held in S (shared) mode by the a thread which was
doing a truncate of a large table .
3. There was another thread which was trying to acquire the AHI latch in X (exclusive) mode
4. With our lock implementation any thread requesting a X lock ,blocks rest of the threads
requesting S(shared) locks,this caused many threads to wait for this shared lock.
5. The main reason why we hold the latches in truncate is to avoid disabling of AHI
during truncate
FIX
InnoDB purge thread locks the root page of clustered index
while accessing the undo log records and later same thread
tries to open the table, initialize statistics and tries
to lock the clustered index root page while doing virtual
column computation.
Solution:
=========
InnoDB should prevent statistics initialization when the
table is being opened by purge thread
Between btr_pcur_store_position() and btr_pcur_restore_position()
it is possible that purge empties a table and enlarges
index->n_core_fields and index->n_core_null_bytes.
Therefore, we must cache index->n_core_fields in
btr_pcur_t::old_n_core_fields so that btr_pcur_t::old_rec can be
parsed correctly.
Unfortunately, this is a huge change, because we will replace
"bool leaf" parameters with "ulint n_core"
(passing index->n_core_fields, or 0 for non-leaf pages).
For special cases where we know that index->is_instant() cannot hold,
we may also pass index->n_fields.
Problem:
========
InnoDB fails to clean the index stub if it fails to add the
virtual index which contains new virtual column. But it clears
the newly virtual column from index in clear_added_indexes()
during inplace_alter_table. On commit, InnoDB evicts and
reload the table. In case of rollback, it doesn't happen.
InnoDB clears the ABORTED index while opening the table
or doing the DDL. In the mean time, InnoDB can access
the dropped virtual index columns while creating prebuilt
or rollback of concurrent DML.
Solution:
==========
(1) InnoDB should maintain newly added virtual column while
rollbacking the newly added virtual index.
(2) InnoDB must not defer the index removal
if the alter table is executed with LOCK=EXCLUSIVE.
(3) For LOCK=SHARED, InnoDB should check whether the table
has any other transaction lock other than alter transaction
before deferring the index stub.
Replaced has_new_v_col with dict_add_vcol_info in dict_index_t to
indicate whether the index has any new virtual column.
dict_index_t::has_new_v_col(): Returns whether the index has
newly added virtual column, it doesn't say which columns are
newly added virtual column
ha_innobase_inplace_ctx::is_new_vcol(): Return whether the
given column is added as a part of the current alter.
ha_innobase_inplace_ctx::clean_new_vcol_index(): Copy the newly
added virtual column to new_vcol_info in dict_index_t. Replace
the column in the index fields with virtual column stored
in new_vcol_info.
dict_index_t::assign_new_v_col(): Store the number of virtual
column added in index as a part of alter table.
dict_index_t::get_n_new_vcol(): Get the number of newly added
virtual column
dict_index_t::assign_drop_v_col(): Allocate the memory for
adding new virtual column in new_vcol_info.
dict_index_t::add_drop_v_col(): Add the newly added virtual
column in new_vcol_info.
dict_table_t::has_lock_for_other_trx(): Whether the table has
any other transaction lock than given transaction.
row_merge_drop_indexes(): Add parameter alter_trx and check
whether the table has any other lock than alter transaction.
When a table has been evicted from dict_sys and reloaded internally by
InnoDB for FOREIGN KEY processing, statistics may not be initialized,
but nevertheless row_update_cascade_for_mysql() could invoke
dict_stats_update_if_needed(). In that case, we cannot really update
the statistics. For tables that have STATS_PERSISTENT=1 and
STATS_AUTO_RECALC=1, ANALYZE TABLE might have to be executed later.
dict_stats_update_if_needed(): Replace the assertion with
a conditional early return.
trx_sys_any_active_transactions(): Remove a bogus debug assertion.
In trx_commit_in_memory() and trx_erase_lists(), we will remove
the transaction from trx_sys->rw_trx_list and set the state to
TRX_STATE_COMMITTED_IN_MEMORY.
server failure in different, confusing ways
InnoDB fails to free the buffer pool instance mutex and zip mutex
If the allocation of buffer pool instance chunk fails. So it leads
to freeing of buffer pool before freeing the mutexes and
leads to double freeing of memory while freeing the mutex
during shutdown.
InnoDB should skip the dropped aborted FTS_DOC_ID_INDEX while
checking the existing FTS_DOC_ID_INDEX in the table. InnoDB
should able to create new FTS_DOC_ID_INDEX if the fulltext
index is being added for the first time.
modified: storage/connect/CMakeLists.txt
modified: storage/connect/javaconn.cpp
- Check privileges while creating tables with Discovery
modified: storage/connect/ha_connect.cc
- Calculate LRECL for JSON tables created with Discovery
modified: storage/connect/tabjson.cpp
- Use CreateProcess (Windows) or fork/exec (linux)
to retrieve the result from REST queries
modified: storage/connect/tabrest.cpp
- Typo
modified: storage/connect/jmgoconn.cpp
During the prepare phase of restoring backups, "mariabackup" does
not seem to allow (or recognize) the option "innodb_force_recovery"
for the embedded InnoDB server instance that it starts.
If page corruption observed during page recovery, the prepare step
fails. While this is indeed the correct behavior ideally, allowing
this option to be set in case of emergencies might be useful when
the current backup is the only copy available. Some error messages
during "--prepare" suggest to set "innodb_force_recovery" to 1:
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
For backwards compatibility, "mariabackup --innobackupex --apply-log"
should also have this option.
Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
Virtual column fields are not found in prebuilt data type, so we should
match InnoDB fields with `get_innobase_type_from_mysql_type` method.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
- Aborting of fulltext index creation fails to remove the
index from sys indexes table. When we try to reload the
table definition, InnoDB fails with index count mismatch
error. InnoDB should remove the index from sys indexes while
rollbacking the secondary index creation.
- Importing table operation fails to punch the hole in
the filesystem when page compressed table is involved.
To achieve that, InnoDB firstly punches the hole for
the IOBuffer size(1MB). After that, InnoDB should write
page by page when page compression is involved.
Add condition on trx->state == TRX_STATE_COMMITTED_IN_MEMORY in order to
avoid unnecessary work. If a transaction has already been committed or
rolled back, it will release its locks in lock_release() and let
the waiting thread(s) continue execution.
Let BF wait on lock_rec_has_to_wait and if necessary other BF
is replayed.
wsrep_trx_order_before
If BF is not even replicated yet then they are ordered
correctly.
bg_wsrep_kill_trx
Make sure victim_trx is found and check also its state. If
state is TRX_STATE_COMMITTED_IN_MEMORY transaction is
already committed or rolled back and will release it locks
soon.
wsrep_assert_no_bf_bf_wait
Transaction requesting new record lock should be TRX_STATE_ACTIVE
Conflicting transaction can be in states TRX_STATE_ACTIVE,
TRX_STATE_COMMITTED_IN_MEMORY or in TRX_STATE_PREPARED.
If conflicting transaction is already committed in memory or
prepared we should wait. When transaction is committed in memory we
held trx mutex, but not lock_sys->mutex. Therefore, we
could end here before transaction has time to do lock_release()
that is protected with lock_sys->mutex.
lock_rec_has_to_wait
We very well can let bf to wait normally as other BF will be
replayed in case of conflict. For debug builds we will do
additional sanity checks to catch unsupported bf wait if any.
wsrep_kill_victim
Check is victim already in TRX_STATE_COMMITTED_IN_MEMORY state and
if it is we can return.
lock_rec_dequeue_from_page
lock_rec_unlock
Remove unnecessary wsrep_assert_no_bf_bf_wait function calls.
We can very well let BF wait here.
The function row_upd_clust_step() is invoking several static functions,
some of which used to commit the mini-transaction in some cases.
If innobase_get_computed_value() would fail due to some reason,
we would fail to invoke mtr_t::commit() and release buffer pool
page latches. This would likely lead to a hanging server later.
This regression was introduced in
commit 97db6c15ea (MDEV-20618).
row_upd_index_is_referenced(), row_upd_sec_index_entry(),
row_upd_sec_index_entry(): Cleanup: Replace some ibool with bool.
row_upd_clust_rec_by_insert(), row_upd_clust_rec(): Guarantee that
the mini-transaction will always remain in active state.
row_upd_del_mark_clust_rec(): Guarantee that
the mini-transaction will always remain in active state.
This fixes one "leak" of mini-transaction on DB_COMPUTE_VALUE_FAILED.
row_upd_clust_step(): Use only one return path, which will always
invoke mtr.commit(). After a failed row_upd_store_row() call, we
will no longer "leak" the mini-transaction.
This fix was verified by RQG on 10.6 (depending on MDEV-371 that
was introduced in 10.4). Unfortunately, it is challenging to
create a regression test for this, and a test case could soon become
invalid as more bugs in virtual column evaluation are fixed.
As suggested by Vladislav Vaintroub, let us remove misleading
and malformatted startup messages.
Even if the global variable srv_use_atomic_writes were set, we would
still invoke my_test_if_atomic_write() to check if writes are atomic
with a particular page size.
When using the default innodb_page_size=16k, page writes should be
atomic on NTFS when using ROW_FORMAT=COMPRESSED and KEY_BLOCK_SIZE<=4.
Disabling srv_use_atomic_writes when innodb_file_per_table=OFF does
not make sense, because that is a dynamic parameter.
We also correct the documentation string of innodb_use_atomic_writes
and remove the duplicate variable innobase_use_atomic_writes.
The debug parameter innodb_simulate_comp_failures injected compression
failures for ROW_FORMAT=COMPRESSED tables, breaking the pre-existing
logic that I had implemented in the InnoDB Plugin for MySQL 5.1 to prevent
compressed page overflows. A much better check is already achieved by
defining UNIV_ZIP_COPY at the compilation time.
(Only UNIV_ZIP_DEBUG is part of cmake -DWITH_INNODB_EXTRA_DEBUG=ON.)
In commit eaeb8ec4b8 (MDEV-24653)
an incorrect debug assertion was introduced.
btr_pcur_store_position(): If the only record in the page is the
instant ALTER TABLE metadata record, we cannot expect there to be
a successor page. The situation could be improved by MDEV-24673 later.
Before MDEV-14638, there was no race condition between the
execution of fetch_data_into_cache() and transaction commit.
fetch_data_into_cache(): Acquire trx_t::mutex before checking
trx_t::state, to prevent a concurrent transition from
TRX_STATE_COMMITTED_IN_MEMORY to TRX_STATE_NOT_STARTED
in trx_commit_in_memory().
ha_innobase::info_low(): While collecting statistics for
ANALYZE TABLE, ensure that dict_stats_process_entry_from_recalc_pool()
is not executing on the same table.
We observed result differences for the test innodb.innodb_stats because
dict_stats_empty_index() was being invoked by the background statistics
calculation while ha_innobase::analyze() was executing
dict_stats_analyze_index_level().
Tests with 4096-byte sector size confirm that it is
safe to use O_DIRECT with page_compressed tables.
That had been disabled on Linux, in an attempt to fix MDEV-21584
which had been filed for the O_DIRECT problems earlier.
The fil_node_t::block_size was being set mostly correctly until
commit 10dd290b4b (MDEV-17380)
introduced a regression in MariaDB Server 10.4.4.
fil_node_t::read_page0(): Initialize fil_node_t::block_size.
This will probably make similar code in fil_space_extend_must_retry()
redundant, but we play it safe and will not remove that code.
Thanks to Vladislav Vaintroub for testing this on Microsoft Windows
using an old-fashioned rotational hard disk with 4KiB sector size.
Reviewed by: Vladislav Vaintroub
This had been originally added in
mysql/mysql-server@192bb153b6
with the motivation to disable O_DIRECT for the dedicated tablespace
for temporary tables. In MariaDB Server,
commit 5eb539555b (MDEV-12227)
should be a better solution.
The code became orphaned later in
mysql/mysql-server@c61244c0e6
and it had been applied to MariaDB Server 10.2.2 in
commit 2e814d4702 and
commit fec844aca8.
Thanks to Vladislav Vaintroub for spotting this.
This when the HTTP contains & characters
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabjson.cpp
- Make stringfy option work on only one Json item
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabbson.h
modified: storage/connect/tabjson.cpp
modified: storage/connect/tabjson.h
- Make Json/Bson DATE columns accept JSON date syntax
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabjson.cpp
- Fix bug making REST table default file not being
erased when dropping the table
modified: storage/connect/tabbson.cpp
modified: storage/connect/tabjson.cpp
modified: storage/connect/tabrest.cpp
modified: storage/connect/tabxml.cpp
- Suppress CHAR(36) --> VARCHAR(36) when DEVELOPMENT
This was fixed in MyClient
modified: storage/connect/ha_connect.cc
Keyvalue can be longer than REC_VERSION_56_MAX_INDEX_COL_LEN
and this leads out-of-array reference. Use dynamic memory
allocation using actual max length of key value.
Online log for insert operation of redundant table fails with
index->is_instant() assert. Purge can reset the n_core_fields when
alter is waiting to upgrade MDL for commit phase of DDL. In the
meantime, any insert DML tries to log the operation fails with
index is not being instant.
row_log_get_n_core_fields(): Get the n_core_fields of online log
for the given index.
rec_get_converted_size_comp_prefix_low(): Use n_core_fields of online
log when InnoDB calculates the size of data tuple during redundant
row format table rebuild.
rec_convert_dtuple_to_rec_comp(): Use n_core_fields of online log
when InnoDB does the conversion of data tuple to record during
redudant row format table rebuild.
- Adding the test case which has more than 129 instant columns.
MDEV-25105 (commit 7a4fbb55b0)
in MariaDB 10.6 will refuse the innodb_checksum_algorithm
values none, innodb, strict_none, strict_innodb.
We will issue a deprecation warning if innodb_checksum_algorithm
is set to any of these non-default unsafe values.
innodb_checksum_algorithm=crc32 was made the default in
MySQL 5.7 and MariaDB Server 10.2, and given that older versions
of the server have reached their end of life, there is no valid
reason to use anything else than innodb_checksum_algorithm=crc32
or innodb_checksum_algorithm=strict_crc32 in MariaDB 10.3.
Reviewed by: Sergei Golubchik
InnoDB set the space in dict_table_t as NULL when table
is discarded. So InnoDB shouldn't use the space present
in table to detect whether the given tablespace is
temporary tablespace.
btr_node_ptr_max_size(): Let us remove the debug assertion that was
added in MDEV-14637. The assertion assumed that no additional
indexes exist in mysql.innodb_index_stats or mysql.innodb_table_stats.
The code path is working around an incorrect definition of a table,
interpreting VARCHAR(64) as the more correct VARCHAR(199).
No test case will be added, because MDEV-24579 proves that executing
DDL on the statistics tables involves a race condition. The test
case included the following:
ALTER TABLE mysql.innodb_index_stats ADD KEY (stat_name);
CREATE TABLE t (a INT) ENGINE=InnoDB STATS_PERSISTENT=1;
Incorrect processing of an auto-incrementing field in the
WSREP-related code during applying transactions results in
a duplicate key being created. This is due to the fact that
at the beginning of the write_row() and update_row() functions,
the values of the auto-increment parameters are used, which
are read from the parameters of the current thread, but further
along the code other values are used, which are read from global
variables (when applying a transaction). This can happen when
the cluster configuration has changed while applying a transaction
(for example in the high_priority_service mode for Galera 4).
Further during IST processing duplicating key is detected, and
processing of the DB_DUPLICATE_KEY return code (inside innodb,
in the write_row() handler) results in a call to the
wsrep_thd_self_abort() function.
row_prebuilt_t::m_no_prefetch: Remove (it was always false).
row_prebuilt_t::m_read_virtual_key: Remove (it was always false).
Only ha_innopart ever set these fields.
innobase_rename_table(): Invoke dict_stats_wait_bg_to_stop_using_table()
to ensure that dict_stats_update() cannot be accessing the table name
that we will be modifying. If we are executing RENAME rather than TRUNCATE,
reset the flag at the end so that persistent statistics can be calculated
again.
The race condition was encountered with ASAN and rr.
Sorry, there is no test case, like there is for nothing related to
dict_stats_wait_bg_to_stop_using_table(). The entire code is an ugly
work-around for the failure of dict_stats_process_entry_from_recalc_pool()
to acquire MDL.
Note: It appears that an ALTER TABLE that is not rebuilding the table
will fail to reset the flag that blocks the processing of statistics.
In btr_index_rec_validate(), externally stored column
check is missing while matching the length of the field
with the length of the field data stored in record.
Fetch the length of the externally stored part and compare it
with the fixed field length.
This is a backport of commit 18535a4028
from 10.6.
lock_release(): Implement innodb_evict_tables_on_commit_debug.
Before releasing any locks, collect the identifiers of tables to
be evicted. After releasing all locks, look up for the tables and
evict them if it is safe to do so.
trx_commit_in_memory(): Invoke trx_update_mod_tables_timestamp()
before lock_release(), so that our locks will protect the tables
from being evicted.
When doing a truncate on an Innodb under lock tables, InnoDB would rename
the old table to #sql-... and recreate a new 't1' table. The table lock
would still be on the #sql-table.
When doing ALTER TABLE, Innodb would do the changes on the #sql table
(which would disappear on close).
When the SQL layer, as part of inline alter table, would close the
original t1 table (#sql in InnoDB) and then reopen the t1 table, Innodb
would notice that this does not match it's own (old) t1 table and
generate an error.
Fixed by adding code in truncate table that if we are under lock tables
and truncating an InnoDB table, we would close, reopen and lock the
table after truncate. This will remove the #sql table and ensure that
lock tables is using the new empty table.
Reviewer: Marko Mäkelä
eprintf() was missing a va_start(), which caused wrong filename to be
printed when printing recovery trace.
Added also missing new line when printing "Table is crashed" to trace file
- The commit 5fd3c7471e3e0673b50d309567c9747d36f09412(MDEV-24709)
resets the recv_no_ibuf_operations in
recv_recovery_from_checkpoint_start(), but InnoDB fails to reset
the variable recv_no_log_write() during that time and that leads
to the assert failure.
- This is caused by commit ad6171b91c
(MDEV-22456). InnoDB reloads the evicted table again from dictionary.
In that case, AHI entries and current index object mismatches
happens. When index object mismatches then InnoDB should drop
the page hash AHI entries for the block. In
btr_search_drop_page_hash_index(), InnoDB should take exclusive
lock on the AHI latch if index is already freed to avoid the
freed memory access during buf_pool_resize()
Problem was that thd::awake assumes now that you hold THD::LOCK_thd_data
so we need to keep it when we call wsrep_thd_awake from
wsrep_abort_transaction.
innobase_rename_column_try(): When renaming SYS_FIELDS records
for secondary indexes, try to use both formats of SYS_FIELDS.POS
as keys, in case the PRIMARY KEY includes a column prefix.
Without this fix, an ALTER TABLE that renames a column followed
by a server restart (or LRU eviction of the table definition
from dict_sys) would make the table inaccessible.
Add negative array indexes starting from the last
modified: storage/connect/bson.cpp
modified: storage/connect/bsonudf.cpp
modified: storage/connect/json.cpp
innodb_shutdown(): Check that fil_system.temp_space is not null
before invoking a member function.
This regression was caused by the merge
commit fa1aef39eb
of MDEV-24340 (commit 1eb59c307d).
bson.cpp:1775:3: error: case label value is less than minimum value for type [-Werror]
case TYPE_NULL:
bson.cpp:1776:7: error: statement will never be executed [-Werror=switch-unreachable]
b = true;
Analysis:
=========
Reason for test failure was a mutex deadlock between DeadlockChecker with stack
Thread 6 (Thread 0xffff70066070 (LWP 24667)):
0 0x0000ffff784e850c in __lll_lock_wait (futex=futex@entry=0xffff04002258, private=0) at lowlevellock.c:46
1 0x0000ffff784e19f0 in __GI___pthread_mutex_lock (mutex=mutex@entry=0xffff04002258) at pthread_mutex_lock.c:135
2 0x0000aaaaac8cd014 in inline_mysql_mutex_lock (src_file=0xaaaaacea0f28 "/home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_thd.cc", src_line=762, that=0xffff04002258) at /home/buildbot/buildbot/build/mariadb-10.2.37/include/mysql/psi/mysql_thread.h:675
3 wsrep_thd_is_BF (thd=0xffff040009a8, sync=sync@entry=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_thd.cc:762
4 0x0000aaaaacadce68 in lock_rec_has_to_wait (for_locking=false, lock_is_on_supremum=<optimized out>, lock2=0xffff628952d0, type_mode=291, trx=0xffff62894070) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:826
5 lock_has_to_wait (lock1=<optimized out>, lock2=0xffff628952d0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:873
6 0x0000aaaaacadd0b0 in DeadlockChecker::search (this=this@entry=0xffff70061fe8) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:7142
7 0x0000aaaaacae2dd8 in DeadlockChecker::check_and_resolve (lock=lock@entry=0xffff62894120, trx=trx@entry=0xffff62894070) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:7286
8 0x0000aaaaacae3070 in lock_rec_enqueue_waiting (c_lock=0xffff628952d0, type_mode=type_mode@entry=3, block=block@entry=0xffff62076c40, heap_no=heap_no@entry=2, index=index@entry=0xffff4c076f28, thr=thr@entry=0xffff4c078810, prdt=prdt@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:1796
9 0x0000aaaaacae3900 in lock_rec_lock_slow (thr=0xffff4c078810, index=0xffff4c076f28, heap_no=2, block=0xffff62076c40, mode=3, impl=0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:2106
10 lock_rec_lock (impl=false, mode=3, block=0xffff62076c40, heap_no=2, index=0xffff4c076f28, thr=0xffff4c078810) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:2168
11 0x0000aaaaacae3ee8 in lock_sec_rec_read_check_and_lock (flags=flags@entry=0, block=block@entry=0xffff62076c40, rec=rec@entry=0xffff6240407f "\303\221\342\200\232\303\220\302\265\303\220\302\272\303\221\302\201\303\221\342\200\232", index=index@entry=0xffff4c076f28, offsets=0xffff4c080690, offsets@entry=0xffff70062a30, mode=LOCK_X, mode@entry=1653162096, gap_mode=0, gap_mode@entry=281470749427104, thr=thr@entry=0xffff4c078810) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/lock/lock0lock.cc:6082
12 0x0000aaaaacb684c4 in sel_set_rec_lock (pcur=0xaaaac841c270, pcur@entry=0xffff4c077d58, rec=0xffff6240407f "\303\221\342\200\232\303\220\302\265\303\220\302\272\303\221\302\201\303\221\342\200\232", rec@entry=0x28 <error: Cannot access memory at address 0x28>, index=index@entry=0xffff4c076f28, offsets=0xffff70062a30, mode=281472334905456, type=281470749427104, thr=0xffff4c078810, thr@entry=0x9f, mtr=0x0, mtr@entry=0xffff70063928) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0sel.cc:1270
13 0x0000aaaaacb6bb64 in row_search_mvcc (buf=buf@entry=0xffff4c080690 "\376\026", mode=mode@entry=PAGE_CUR_GE, prebuilt=0xffff4c077b98, match_mode=match_mode@entry=1, direction=direction@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/row/row0sel.cc:5181
14 0x0000aaaaacaae568 in ha_innobase::index_read (this=0xffff4c038a80, buf=0xffff4c080690 "\376\026", key_ptr=<optimized out>, key_len=768, find_flag=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/handler/ha_innodb.cc:9393
15 0x0000aaaaac9201cc in handler::ha_index_read_map (this=0xffff4c038a80, buf=0xffff4c080690 "\376\026", key=0xffff4c07ccf8 "", keypart_map=keypart_map@entry=18446744073709551615, find_flag=find_flag@entry=HA_READ_KEY_EXACT) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/handler.cc:2718
16 0x0000aaaaac9f36b0 in Rows_log_event::find_row (this=this@entry=0xffff4c030098, rgi=rgi@entry=0xffff4c01b510) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/log_event.cc:13461
17 0x0000aaaaac9f3e44 in Update_rows_log_event::do_exec_row (this=0xffff4c030098, rgi=0xffff4c01b510) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/log_event.cc:13936
18 0x0000aaaaac9e7ee8 in Rows_log_event::do_apply_event (this=0xffff4c030098, rgi=0xffff4c01b510) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/log_event.cc:11101
19 0x0000aaaaac8ca4e8 in Log_event::apply_event (rgi=0xffff4c01b510, this=0xffff4c030098) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/log_event.h:1454
20 wsrep_apply_events (buf_len=0, events_buf=0x1, thd=0xffff4c0009a8) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_applier.cc:164
21 wsrep_apply_cb (ctx=0xffff4c0009a8, buf=0x1, buf_len=18446743528248705000, flags=<optimized out>, meta=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_applier.cc:267
22 0x0000ffff7322d29c in galera::TrxHandle::apply (this=this@entry=0xffff4c027960, recv_ctx=recv_ctx@entry=0xffff4c0009a8, apply_cb=apply_cb@entry=0xaaaaac8c9fe8 <wsrep_apply_cb(void*, void const*, size_t, uint32_t, wsrep_trx_meta_t const*)>, meta=...) at /home/buildbot/buildbot/build/galera/src/trx_handle.cpp:317
23 0x0000ffff73239664 in apply_trx_ws (recv_ctx=recv_ctx@entry=0xffff4c0009a8, apply_cb=0xaaaaac8c9fe8 <wsrep_apply_cb(void*, void const*, size_t, uint32_t, wsrep_trx_meta_t const*)>, commit_cb=0xaaaaac8ca8d0 <wsrep_commit_cb(void*, uint32_t, wsrep_trx_meta_t const*, wsrep_bool_t*, bool)>, trx=..., meta=...) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:34
24 0x0000ffff7323c0c4 in galera::ReplicatorSMM::apply_trx (this=this@entry=0xaaaac7c7ebc0, recv_ctx=recv_ctx@entry=0xffff4c0009a8, trx=trx@entry=0xffff4c027960) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:454
25 0x0000ffff7323e8b8 in galera::ReplicatorSMM::process_trx (this=0xaaaac7c7ebc0, recv_ctx=0xffff4c0009a8, trx=0xffff4c027960) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:1258
26 0x0000ffff73268f68 in galera::GcsActionSource::dispatch (this=this@entry=0xaaaac7c7f348, recv_ctx=recv_ctx@entry=0xffff4c0009a8, act=..., exit_loop=@0xffff7006535f: false) at /home/buildbot/buildbot/build/galera/src/gcs_action_source.cpp:115
27 0x0000ffff73269dd0 in galera::GcsActionSource::process (this=0xaaaac7c7f348, recv_ctx=0xffff4c0009a8, exit_loop=@0xffff7006535f: false) at /home/buildbot/buildbot/build/galera/src/gcs_action_source.cpp:180
28 0x0000ffff7323ef5c in galera::ReplicatorSMM::async_recv (this=0xaaaac7c7ebc0, recv_ctx=0xffff4c0009a8) at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:362
29 0x0000ffff73217760 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/buildbot/buildbot/build/galera/src/wsrep_provider.cpp:244
30 0x0000aaaaac8cb344 in wsrep_replication_process (thd=0xffff4c0009a8) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_thd.cc:486
31 0x0000aaaaac8bc3a0 in start_wsrep_THD (arg=arg@entry=0xaaaac7cb3e38) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/wsrep_mysqld.cc:2173
32 0x0000aaaaaca89198 in pfs_spawn_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/perfschema/pfs.cc:1869
33 0x0000ffff784defc4 in start_thread (arg=0xaaaaaca890d8 <pfs_spawn_thread(void*)>) at pthread_create.c:335
34 0x0000ffff7821c3f0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89
and background victim transaction kill with stack
Thread 28 (Thread 0xffff485fa070 (LWP 24870)):
0 0x0000ffff784e530c in __pthread_cond_wait (cond=cond@entry=0xaaaac83e98e0, mutex=mutex@entry=0xaaaac83e98b0) at pthread_cond_wait.c:186
1 0x0000aaaaacb10788 in os_event::wait (this=0xaaaac83e98a0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/os/os0event.cc:158
2 os_event::wait_low (reset_sig_count=2, this=0xaaaac83e98a0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/os/os0event.cc:325
3 os_event_wait_low (event=0xaaaac83e98a0, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/os/os0event.cc:507
4 0x0000aaaaacb98480 in sync_array_wait_event (arr=arr@entry=0xaaaac7dbb450, cell=@0xffff485f96e8: 0xaaaac7dbb560) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/sync/sync0arr.cc:471
5 0x0000aaaaacab53c8 in TTASEventMutex<GenericPolicy>::enter (line=19524, filename=0xaaaaacf2ce40 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/handler/ha_innodb.cc", max_delay=<optimized out>, max_spins=0, this=0xaaaac83cc8c0) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/include/ib0mutex.h:516
6 PolicyMutex<TTASEventMutex<GenericPolicy> >::enter (this=0xaaaac83cc8c0, n_spins=<optimized out>, n_delay=<optimized out>, name=0xaaaaacf2ce40 "/home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/handler/ha_innodb.cc", line=19524) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/include/ib0mutex.h:637
7 0x0000aaaaacaaa52c in bg_wsrep_kill_trx (void_arg=0xffff4c057430) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/innobase/handler/ha_innodb.cc:19524
8 0x0000aaaaac79e7f0 in handle_manager (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mariadb-10.2.37/sql/sql_manager.cc:112
9 0x0000aaaaaca89198 in pfs_spawn_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.37/storage/perfschema/pfs.cc:1869
10 0x0000ffff784defc4 in start_thread (arg=0xaaaaaca890d8 <pfs_spawn_thread(void*)>) at pthread_create.c:335
11 0x0000ffff7821c3f0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89
Fix:
====
Do not use THD::LOCK_thd_data mutex if we already hold lock_sys->mutex because it
will cause mutexing order violation. Victim transaction holding conflicting
locks can't be committed or rolled back while we hold lock_sys->mutex. Thus,
it is safe to do wsrep_thd_is_BF call with no additional mutexes.
recv_recovery_from_checkpoint_start(): Clear the recv_no_ibuf_operations
flag at the same time when we enabled writes to the log.
The failure to clear the flag might have caused some missed
change buffer merges, at least to the secondary index of SYS_TABLES
that were accessed by trx_resurrect_table_locks() while the last
recovery batch was in progress.
Thanks to Thirunarayanan Balathandayuthapani for suggesting this fix.
fil_rename_tablespace(): Do not write a redundant MLOG_FILE_RENAME2
record.
The recovery bug will be fixed later. The problem is that we are
invoking fil_op_replay_rename() too often, while we should skip
any 'intermediate' names of a tablespace and only apply the
very last rename for each tablespace identifier, and only if
the tablespace name is not already correct.
The assertion failed in handler::ha_reset upon SELECT under
READ UNCOMMITTED from table with index on virtual column.
This was the debug-only failure, though the problem is mush wider:
* MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw
bitmap.
* read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP
* The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE
* The pointers to the stored MY_BITMAPs, like orig_read_set etc, and
sometimes all_set and tmp_set, are assigned to the pointers.
* Sometimes tmp_use_all_columns is used to substitute the raw bitmap
directly with all_set.bitmap
* Sometimes even bitmaps are directly modified, like in
TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called.
The last three bullets in the list, when used together (which is mostly
always) make the program flow cumbersome and impossible to follow,
notwithstanding the errors they cause, like this MDEV-17556, where tmp_set
pointer was assigned to read_set, write_set and vcol_set, then its bitmap
was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call,
and then bitmap_clear_all(&tmp_set) was applied to all this.
To untangle this knot, the rule should be applied:
* Never substitute bitmaps! This patch is about this.
orig_*, all_set bitmaps are never substituted already.
This patch changes the following function prototypes:
* tmp_use_all_columns, dbug_tmp_use_all_columns
to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map*
* tmp_restore_column_map, dbug_tmp_restore_column_maps to accept
MY_BITMAP* instead of my_bitmap_map*
These functions now will substitute read_set/write_set/vcol_set directly,
and won't touch underlying bitmaps.
We may end up with an empty leaf page (containing only an ADD COLUMN
metadata record) that is not the root page.
innobase_add_instant_try(): Disable an optimization for a non-canonical
empty table that contains a metadata record somewhere else than in
the root page.
btr_pcur_store_position(): Tolerate a non-canonical empty table.
mutex order violation here.
when wsrep bf thread kills a conflicting trx, the stack is
wsrep_thd_LOCK()
wsrep_kill_victim()
lock_rec_other_has_conflicting()
lock_clust_rec_read_check_and_lock()
row_search_mvcc()
ha_innobase::index_read()
ha_innobase::rnd_pos()
handler::ha_rnd_pos()
handler::rnd_pos_by_record()
handler::ha_rnd_pos_by_record()
Rows_log_event::find_row()
Update_rows_log_event::do_exec_row()
Rows_log_event::do_apply_event()
Log_event::apply_event()
wsrep_apply_events()
and mutexes are taken in the order
lock_sys->mutex -> victim_trx->mutex -> victim_thread->LOCK_thd_data
When a normal KILL statement is executed, the stack is
innobase_kill_query()
kill_handlerton()
plugin_foreach_with_mask()
ha_kill_query()
THD::awake()
kill_one_thread()
and mutexes are
victim_thread->LOCK_thd_data -> lock_sys->mutex -> victim_trx->mutex
To fix the mutex order violation we kill the victim thd asynchronously,
from the manager thread
Ever since commit 947efe17ed
InnoDB no longer writes binlog position in one place.
It will not at all be written to the TRX_SYS page, and
instead it will be written to the undo log header page that
changes the transaction state.
trx_rseg_mem_restore(): Recover the information from the latest
written page.
I_S tables were materialized too late, an attempt to use table
statistics before the table was created caused a crash.
Let's move table creation up. it only needs read_set to
be calculated properly, this happens in JOIN::optimize_inner(),
after semijoin transformation.
Note that tables are not populated at that point, so most of the
statistics would make no sense anyway. But at least field sizes
will be correct. And it won't crash.
innodb_io_capacity_update(): When the requested innodb_io_capacity
exceeds innodb_io_capacity_max and is more than half the maximum,
do not double it for computing innodb_io_capacity_max.
This integer arithmetics overflow was introduced in
commit 0f32299437 (MDEV-7035).
No test case is added, because sizeof(ulong) varies between platforms.
Database name mismatch happens while opening the table for
virtual column computation. Because table_name_parse() returns
the length of database and table name before converting the
filename to table name. This issue is caused by
commit 8b0d4cff07 (MDEV-15855).
Fix should be that table_name_parse() should return the length of
database and table name after converting the filename to table name.
Reviewed-by: Marko Mäkelä
When online alter rollbacks due to MDL time out, it doesn't mark the
index online status as ONLINE_INDEX_ABORTED. Concurrent update fails
to update the secondary index while building the entry.
InnoDB should check the online status of the secondary index before
building the secondary index entry.
Reviewed-by: Marko Mäkelä
Some DML operations on tables having unique secondary keys cause scanning
in the secondary index, for instance to find potential unique key violations
in the seconday index. This scanning may involve GAP locking in the index.
As this locking happens also when applying replication events in high priority
applier threads, there is a probabality for lock conflicts between two wsrep
high priority threads.
This PR avoids lock conflicts of high priority wsrep threads, which do
secondary index scanning e.g. for duplicate key detection.
The actual fix is the patch in sql_class.cc:thd_need_ordering_with(), where
we allow relaxed GAP locking protocol between wsrep high priority threads.
wsrep high priority threads (replication appliers, replayers and TOI processors)
are ordered by the replication provider, and they will not need serializability
support gained by secondary index GAP locks.
PR contains also a mtr test, which exercises a scenario where two replication
applier threads have a false positive conflict in GAP of unique secondary index.
The conflicting local committing transaction has to replay, and the test verifies
also that the replaying phase will not conflict with the latter repllication applier.
Commit also contains new test scenario for galera.galera_UK_conflict.test,
where replayer starts applying after a slave applier thread, with later seqno,
has advanced to commit phase. The applier and replayer have false positive GAP
lock conflict on secondary unique index, and replayer should ignore this.
This test scenario caused crash with earlier version in this PR, and to fix this,
the secondary index uniquenes checking has been relaxed even further.
Now innodb trx_t structure has new member: bool wsrep_UK_scan, which is set to
true, when high priority thread is performing unique secondary index scanning.
The member trx_t::wsrep_UK_scan is defined inside WITH_WSREP directive, to make
it possible to prepare a MariaDB build where this additional trx_t member is
not present and is not used in the code base. trx->wsrep_UK_scan is set to true
only for the duration of function call for: lock_rec_lock() trx->wsrep_UK_scan
is used only in lock_rec_has_to_wait() function to relax the need to wait if
wsrep_UK_scan is set and conflicting transaction is also high priority.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
The parameter innodb_idle_flush_pct that was introduced in
MariaDB Server 10.1.2 by MDEV-6932 has no effect ever since
the InnoDB changes from MySQL 5.7.9 were applied in
commit 2e814d4702.
Let us declare the parameter as deprecated and having no effect.
When executing a query like "select id, 0 as const, val from ...", there are 3 columns(items) in Query->select at handlerton->create_group_by(). After that, MariaDB makes a temporary table with 2 columns. The skipped items are const item, so fixing Spider to skip const items for items at Query->select.
Server part:
kill_handlerton() was accessing thd->ha_data[] for some other thd,
while it could be concurrently modified by its owner thd.
protect thd->ha_data[] modifications with a mutex.
require this mutex when accessing thd->ha_data[] from kill_handlerton.
InnoDB part:
on close_connection, detach trx from thd before freeing the trx
with wrong data type is added
Inplace alter fails to report error when fts_doc_id column with
wrong data type is added.
prepare_inplace_alter_table_dict(): Should check whether the column
is fts_doc_id. It should be of bigint type, should accept non null
data type and it should be in capital letters.
A race condition may occur between the execution of transaction commit,
and an execution of a KILL statement that would attempt to abort that
transaction.
MDEV-17092 worked around this race condition by modifying InnoDB code.
After that issue was closed, Sergey Vojtovich pointed out that this
race condition would better be fixed above the storage engine layer:
If you look carefully into the above, you can conclude that
thd->free_connection() can be called concurrently with
KILL/thd->awake(). Which is the bug. And it is partially fixed in
THD::~THD(), that is destructor waits for KILL completion:
Fix: Add necessary mutex operations to THD::free_connection()
and move WSREP specific code also there. This ensures that no
one is using THD while we do free_connection(). These mutexes
will also ensures that there can't be concurrent KILL/THD::awake().
innobase_kill_query
We can now remove usage of trx_sys_mutex introduced on MDEV-17092.
trx_t::free()
Poison trx->state and trx->mysql_thd
This patch is validated with an RQG run similar to the one that
reproduced MDEV-17092.
The assertion failed in handler::ha_reset upon SELECT under
READ UNCOMMITTED from table with index on virtual column.
This was the debug-only failure, though the problem is mush wider:
* MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw
bitmap.
* read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP
* The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE
* The pointers to the stored MY_BITMAPs, like orig_read_set etc, and
sometimes all_set and tmp_set, are assigned to the pointers.
* Sometimes tmp_use_all_columns is used to substitute the raw bitmap
directly with all_set.bitmap
* Sometimes even bitmaps are directly modified, like in
TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called.
The last three bullets in the list, when used together (which is mostly
always) make the program flow cumbersome and impossible to follow,
notwithstanding the errors they cause, like this MDEV-17556, where tmp_set
pointer was assigned to read_set, write_set and vcol_set, then its bitmap
was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call,
and then bitmap_clear_all(&tmp_set) was applied to all this.
To untangle this knot, the rule should be applied:
* Never substitute bitmaps! This patch is about this.
orig_*, all_set bitmaps are never substituted already.
This patch changes the following function prototypes:
* tmp_use_all_columns, dbug_tmp_use_all_columns
to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map*
* tmp_restore_column_map, dbug_tmp_restore_column_maps to accept
MY_BITMAP* instead of my_bitmap_map*
These functions now will substitute read_set/write_set/vcol_set directly,
and won't touch underlying bitmaps.
mysql_col_offset was not updated after the new column has been added by an
INSTANT ALTER TABLE -- table data dictionary had been remaining the same.
When the virtual column is added or removed, table was usually evicted and
then reopened, which triggered vcol info rebuild on the next open.
However this also should be done when the usual column is added or removed:
mariadb always stores virtual field at the end of maria record,
so the shift should always happen.
Fix:
expand the eviction condition to the case when usual fields are
added/removed
Note:
this should happen only in the case of !new_clustered:
* When new_clustered is true, a new data dictionary is created, and vcol
metadata is rebuilt in `alter_rebuild_apply_log()`
* We can't do it in `new_clustered` case, because the old table is not yet
subctituted correctly
This corresponds to 10.5 commit 39378e1366.
With a patched version of the test innodb.ibuf_not_empty (so that
it would trigger crash recovery after using the change buffer),
and patched code that would modify the os_thread_sleep() in
recv_apply_hashed_log_recs() to be 1ms as well as add a sleep of
the same duration to the end of recv_recover_page() when
recv_sys->n_addrs=0, we can demonstrate a race condition.
After disabling some debug checks in buf_all_freed_instance(),
buf_pool_invalidate_instance() and buf_validate(), we managed to
trigger an assertion failure in fseg_free_step(), on the XDES_FREE_BIT.
In other words, an trx_undo_seg_free() call during
trx_rollback_resurrected() was attempting a double-free of a page.
This was repeated about once in 400 to 500 test runs. With the fix
applied, the test passed 2,000 runs.
recv_apply_hashed_log_recs(): Do not only wait for recv_sys->n_addrs
to reach 0, but also wait for buf_get_n_pending_read_ios() to reach 0,
to guarantee that buf_page_io_complete() will not be executing
ibuf_merge_or_delete_for_page().
Issues MDEV-23851 and MDEV-24229 are probably duplicates and are caused by the new self-asserting function lock0lock.cc:wsrep_assert_no_bf_bf_wait().
The criteria for asserting is too strict and does not take in consideration scenarios of "false positive" lock conflicts, which are resolved by replaying the local transaction.
As a fix, this PR is relaxing the assert criteria by two conditions, which skip assert if high priority transactions are locking in correct order or if conflicting high priority lock holder is aborting and has just not yet released the lock.
Alternative fix would be to remove wsrep_assert_no_bf_bf_wait() altogether, or remove the assert in this function and let it only print warnings in error log.
But in my high conflict rate multi-master test scenario, this relaxed asserting appears to be safe.
This PR also removes two wsrep_report_bf_lock_wait() calls in innodb lock manager, which cause mutex access assert in debug builds.
Foreign key appending missed handling of data types of float and double in INSERT execution. This is not directly related to the actual issue here but is fixed in this PR nevertheless. Missing these foreign keys values in certification could cause problems in some multi-master load scenarios.
Finally, some problem reports suggest that some of the issues reported in MDEV-23851 might relate to false positive lock conflicts over unique secondary index gaps. There is separate work for relaxing UK index gap locking of replication appliers, and separate PR will be submitted for it, with a related mtr test as well.
Add history row outside of compare_record() check. For TRX_ID
versioning we have to fail can_compare_record to force InnoDB update
which adds history row; and there in ha_innobase::update_row() is
additional "row changed" check where we force history row anyway.
First part of the fix (row0mysql.cc) addresses external columns when adding history
row on referential action. The full data must be retrieved before the
row is inserted.
Second part of the fix (the rest) avoids duplicate primary key error between
the history row generated on referential action and the history row
generated by SQL command. Both command and referential action can
happen on same table since foreign key can be self-reference (parent
and child tables are same). Moreover, the self-reference can refer
multiple rows when the key is non-unique. In such case history is
generated by referential action occured on first row but processed all
rows by a matched key. The second round is when the next row is
processed by a command but history already exists. In such case we
check TRX_ID of existing history row and if it is the same we assume
the above situation and skip adding one more history row or failing
the command.
First part (row0mysql.cc) fixes ins_node_set_new_row() usage workflow
as it is designed to operate on empty row (see row_get_prebuilt_insert_row()
for example).
Second part (row0ins.cc) fixes duplicate key error in FTS_DOC_ID_INDEX
since history rows must not generate entries in that index. We detect
FTS_DOC_ID_INDEX by a number of attributes and skip it if the row is
historical.
Misc fixes:
row_build_index_entry_low() does not accept non-NULL tuple
for FTS index (subject assertion fails), assertion (index->type !=
DICT_FTS) adds code understanding.
Now as historical_row is copied in row_update_vers_insert() there is
no need to copy the row twice: ROW_COPY_POINTERS is used to build
historical_row initially.
dbug_print_rec() debug functions.
