Problem:
=======
Test assertion fails on slave.
Assertion text: 'Last_Seen_Transaction should show .'
Assertion condition: '"0-1-1" = ""'
Assertion condition, interpolated: '"0-1-1" = ""'
Assertion result: '0'
Analysis:
========
Test case creates a table on master and it waits for it to be replicated to
slave and applied by slave applier. On completion 'Last_Seen_Transaction'
value from 'performance_schema.replication_applier_status_by_worker' table
is compared with '@@gtid_slave_pos' to ensure its correctness. The test
should ensure that user table and 'gtid_slave_pos' table are of same engine
type 'InnoDB' to get consistent test results. This guarantees that
'gtid_slave_pos' table is updated as part of transaction commit. In the of
such an engine consistency, user table gets created in default MyISAM
storage engine and 'mysql.gtid_slave_pos' table gets created in Aria
storage engine. When the test code reaches above assert there might be a
delay in an update to 'gtid_slave_pos' table, which leads to test assertion
failure.
Fix:
===
Use InnoDB engine for both user table and 'mysql.gtid_slave_pos' table.
Before we create an InnoDB data file, we must have persistently
started a DDL transaction and written a record in SYS_INDEXES
as well as a FILE_CREATE record for creating the file.
In that way, if InnoDB is killed before the DDL transaction is
committed, the rollback will be able to delete the file in
dict_drop_index_tree().
dict_build_table_def_step(): Do not create the tablespace.
At this point, we have not written any log, not even for
inserting the SYS_TABLES record.
dict_create_sys_indexes_tuple(): Relax an assertion to tolerate
a missing tablespace before the first index has been created in
dict_create_index_step().
dict_build_index_def_step(): Relax the dict_table_open_on_name()
parameter, because no tablespace may be available yet.
tab_create_graph_create(), row_create_table_for_mysql(), tab_node_t:
Remove key_id, mode.
ind_create_graph_create(), row_create_index_for_mysql(), ind_node_t:
Add key_id, mode.
dict_create_index_space(): New function, to create the tablespace
during clustered index creation.
dict_create_index_step(): After the SYS_INDEXES record has been
written, invoke dict_create_index_space() to create the tablespace
if needed.
fil_ibd_create(): Before creating the file, persistently write a
FILE_CREATE record. This will also ensure that an incomplete DDL
transaction will be recovered. After creating the file, invoke
fsp_header_init().
InnoDB used to support at most one CREATE TABLE or DROP TABLE
per transaction. This caused complications for DDL operations on
partitioned tables (where each partition is treated as a separate
table by InnoDB) and FULLTEXT INDEX (where each index is maintained
in a number of internal InnoDB tables).
dict_drop_index_tree(): Extend the MDEV-24589 logic and treat
the purge or rollback of SYS_INDEXES records of clustered indexes
specially: by dropping the tablespace if it exists. This is the only
form of recovery that we will need.
trx_undo_ddl_type: Document the DDL undo log record types better.
trx_t::dict_operation: Change the type to bool.
trx_t::ddl: Remove.
trx_t::table_id, trx_undo_t::table_id: Remove.
dict_build_table_def_step(): Remove trx_t::table_id logging.
dict_table_close_and_drop(), row_merge_drop_table(): Remove.
row_merge_lock_table(): Merged to the only callers, which can
call lock_table_for_trx() directly.
fts_aux_table_t, fts_aux_id, fts_space_set_t: Remove.
fts_drop_orphaned_tables(): Remove.
row_merge_rename_index_to_drop(): Remove. Thanks to MDEV-24589,
we can simply delete the to-be-dropped indexes from SYS_INDEXES,
while still being able to roll back the operation.
ha_innobase_inplace_ctx: Make a few data members const.
Preallocate trx.
prepare_inplace_alter_table_dict(): Simplify the logic. Let the
normal rollback take care of some cleanup.
row_undo_ins_remove_clust_rec(): Simplify the parsing of SYS_COLUMNS.
trx_rollback_active(): Remove the special DROP TABLE logic.
trx_undo_mem_create_at_db_start(), trx_undo_reuse_cached():
Always write TRX_UNDO_TABLE_ID as 0.
galera_var_wsrep_on_off : Add wait conditions to make sure DDL is
replicated before continuing.
wsrep.[variables|variables_debug] : Remove unnecessary parts
and add check to correct number of variables or skip
galera_ssl_reload: Add version check and SSL checks.
When you only need view structure, don't call handle_derived with
DT_CREATE and rely on its internal hackish check to skip DT_CREATE.
Because handle_derived is called from many different places,
and this internal hackish check is indiscriminative.
Instead, just don't ask handle_derived to do DT_CREATE
if you don't want it to do DT_CREATE.
After switching to the new mariabackup interface (instead of
the outdated innobackupex interface, which is supported for
compatibility), we need to explicitly pass a path to the datadir
directory as a parameter, since in the new interface the value
of this option is not automatically set in such a way that it
always matches the SST/IST logic. This commit adds passing this
option as an explicit parameter to mariabackup. This commit also
removed unnecessary options that are not used and not supported
by mariabackup.
Also, numerous flaws in the common wsrep_sst_common script have
been fixed:
1) There are many bash-specific constructs in the script that
may not be supported by other interpreters, which can lead
to the most unexpected errors during SST, because failures
in the interpretation of bash-specific constructs lead to
incorrect parsing of arguments;
2) There is parse_cnf() function which is often called by other
scripts for the "mysqld" or "--mysqld" group, but it does not
take into account the default group suffix, which leads to
reading values only from the default group, which then leads
to errors due to reading the default values instead of the
values for a specific group;
3) Some options such as --user, --innodb-data-home-dir or --datadir
are not removed from the --mysqld-args list, although they are
processed inside scripts (and passing of these options funther
may cause problems for mariabackup);
4) If an argument that the script understands is present in
the --mysqld-args list twice, then this causes SST to fail,
instead of reading the most recent value;
5) The "--host" parameter is technically still supported among
the arguments of the SST scripts, but in reality scripts do not
work with it as expected, especially if it has an IPv6 address;
6) If the port number is absent in the --address parameter value,
but the port number is explicitly passed through the --port
argument, then the scripts for mariabackup and xtrabackup-v2
fail;
7) If a new address interface is used (with the --address parameter),
then automatic default port substitution is not performed, although
it is supported for the legacy --host/--port interface.
8) If there are spaces in the parameter values after --mysqld_args,
then their further transfer does not occur correctly, which
causes mariabackup to fail during SST - the space splits
the argument in such a way that it breaks the parsing of the
following parameters;
9) If most of the parameters that are names or paths to the files
or directories contain spaces, then SST scripts fail in an
unpredictable way due to incorrect variable substitutions;
10) If the --log-bin option is passed among the arguments of myqlds
(--mysqld-args) without a parameter, and the --binlog option
is not specified, then the script cannot substitute the default
name for binlog and cannot construct binlog name using the
--log-basename argument (which is against server specifications);
11) Tail slashes are not removed from the directory names, which,
upon further substitution, leads to the appearance of a double
slash in the file paths;
12) The explicit --binlog parameter (which is now always transmitted
from the server side) and the "hidden" --log-bin parameter in the
list of arguments after --mysqld-args are perceived as two different
parameters in different parts of the scripts, and if they are do not
match for some reason, this will lead to failures during SST;
Also, all new changes from the 10.6 branch have been migrated here,
including the latest pull requests for authentication (only the part
that concerns SST scripts).
It also fixes dozens of other bugs in all SST scripts.
Removed numerous extra blank lines and spaces that interfere with
reading and understanding program code, making it more difficult to
find errors in scripts. I also removed all extra trailing spaces at
the ends of lines, which lead to marking extra lines as changes
(in subsequent changes). The amount of indentation in some parts
of the code has also been normalized.
Fix a race condition in the testcase. The testcase assumed that
State='Sending data' means that the thread is already in an
InnoDB lock wait. This is not case, there is a gap between the
state changing to Sending data and execution reaching the point
where it is waiting for a lock.
Use a more precise check instead, through I_S.INNODB_TRX.
The problem was caused by the following scenario:
Subquery's table has two indexes, KEY a(a), KEY a_b(a,b)
- LATERAL DERIVED optimization decides to use index a.
= The subquery uses ref access over key a.
- test_if_skip_sort_order() sees that KEY a_b satisfies the
subquery's GROUP BY clause, and attempts to switch to it.
= It fails to do so, because KEYUSE objects for index a_b
are switched off.
Fixed by disallowing to change the ref access key if it uses KEYUSE
objects injected by LATERAL DERIVED optimization.
after previous error upon multi-RENAME
- InnoDB fails to rename the foreign key constraint while
rollbacking the rename operation. In that case, InnoDB should
rename the FK constraint too.
Problem:
========
180511 11:07:58 [ERROR] Slave I/O: Unexpected master's heartbeat data:
heartbeat is not compatible with local info;the event's data: log_file_name
mysql-bin.000009 log_pos 1054262041, Error_code: 1623
Analysis:
=========
In replication setup when master server doesn't have any events to send to
slave server it sends an 'Heartbeat_log_event'. This event carries the
current binary log filename and offset details. The offset values is stored
within 4 bytes of event header. When the size of binary log is higher than
UINT32_MAX the log_pos values will not fit in 4 bytes memory. It overflows
and hence slave stops with an error.
Fix:
===
Since we cannot extend the common_header of Log_event class, a greater than
4GB value of Log_event::log_pos is made to be transported with a HeartBeat
event's sub-header. Log_event::log_pos in such case is set to zero to
indicate that the 8 byte sub-header is allocated in the event.
In case of cross version replication following behaviour is expected
OLD - Server without fix
NEW - Server with fix
OLD<->NEW : works bidirectionally as long as the binlog offset is
(normally) within 4GB.
When log_pos > UINT32_MAX
OLD->NEW : The 'log_pos' is bound to overflow and NEW slave may report
an invalid event/incompatible heart beat event error.
NEW->OLD : Since patched server sets log_pos=0 on overflow, OLD slave will
report invalid event error.
This happens during repair when a temporary table is opened
with HA_OPEN_COPY, which resets 'share->born_transactional', which
the encryption code did not like.
Fixed by resetting just share->now_transactional.
InnoDB tries to fetch the deleted doc ids for discarded
tablespace. In i_s_fts_deleted_generic_fill(), InnoDB needs
to check whether the table is discarded or not before fetching
deleted doc ids.
xdes_get_descriptor_with_space_hdr(): Use the correct mode
BUF_GET_POSSIBLY_FREED also when the tablespace is larger
than innodb_page_size pages. This function could be called by
fseg_free_step().
fsp_alloc_seg_inode(): For completeness (and for improved robustness
in case of a corrupted tablespace), use BUF_GET_POSSIBLY_FREED.
With this, the entire compilation unit fsp0fsp.cc will use that mode.
fil_ibd_load(): Remove a message that is basically saying that
everything works as expected. The other "Ignoring data file" message
about the presence of an extraneous file will be retained
(and expected by the test innodb.log_file_name).
In commit 54e2e70194
we relaxed a debug assertion in the POSIX version of
os_file_rename_func() only. Let us relax it also on Windows,
so that the test innodb.truncate_crash will pass.
In commit 91599701d0 (MDEV-25312)
some recovery code for TRUNCATE TABLE was broken
causing a regression in a case where undo log for a RENAME TABLE
operation had been durably written but the tablespace had not been
renamed yet.
row_rename_table_for_mysql(): Add a DEBUG_SYNC point for the
test case, and simplify the logic and trim the error messages.
fil_space_t::rename(): Simplify the operation. Merge the necessary
part of fil_rename_tablespace_check(). If there is no change to
the file name, do nothing.
dict_table_t::rename_tablespace(): Refactored from
dict_table_rename_in_cache().
row_undo_ins_parse_undo_rec(): On rolling back TRX_UNDO_RENAME_TABLE,
invoke dict_table_t::rename_tablespace() even if the table name matches.
os_file_rename_func(): Temporarily relax an assertion that would
fail during the recovery in the test innodb.truncate_crash.
after dfb41fddf6 tables that failed to drop are excluded from the
binlogged DROP TABLE statement. It means that the slave should not
expect any errors when executing DROP TABLE, and the binlog should
report that no error has happened, even if it was.
Do not write error code into the binlogged DROP TABLE,
and remove all code that was needed to compute it.
Cluster communication should be possible even when:
1. Node 2 is TCP
2. Node 1/3 is dynamic with SSL enabled
During test we shutdown Node 2 and enable SSL on it. It should connect
back to cluster successfully.
This commits replaces the call of the function setup_tables() with
a call of the function setup_tables_and_check_access() in the method
Multiupdate_prelocking_strategy::handle_end().
There is no known bug that would require this change. However the change
aligns this piece of code with the code existed before the patch for
MDEV-24823.
remove code duplication in Lex_input_stream::scan_ident_middle(),
make sure identifiers are always use the same code path whether
they start form an underscore or not.
Attempt to build MariaDB server on MacOS could result in
compilation errors like the following one:
In file included from server-10.2/storage/perfschema/cursor_by_account.cc:28:
In file included from server-10.2/include/my_global.h:287:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/math.h:309:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/type_traits:418:
server-10.2/version:1:1: error: expected unqualified-id
MYSQL_VERSION_MAJOR=10
^
server-10.2/build.dir/include/my_config.h:529:29: note: expanded from macro 'MYSQL_VERSION_MAJOR'
This kind of compiler errors occur by the reson that compiler's system headers
contain the directive '#include <version>' and a compiler is invoked
with -I${CMAKE_SOURCE_DIR}.
The MariaDB source code root directory contains the file VERSION that is handled
by the compiler during processing the directive #include <version>
since file names on MacOS are case insensetive, so version and VERSION is treated as
the same file name.
To fix the issue the source code root directory should be removed from a list
of directories used by the compiler for include search path.
lock_discard_for_index(): New function, to discard locks for an
index whose index tree has been purged. By definition, such indexes
must be ones for which the MDL upgrade failed in inplace ALTER TABLE
and the ADD INDEX operation was never committed.
Note: Because we do not support online ADD SPATIAL INDEX, we only
have to traverse the lock_sys.rec_hash for B-trees and not the
hash tables for R-trees.
row_purge_remove_clust_if_poss_low(): Invoke lock_discard_for_index()
if necessary before dropping a B-tree for a SYS_INDEXES record.
btr_free_if_exists(): Always use the BUF_GET_POSSIBLY_FREED mode
when accessing pages, because due to MDEV-24589 the function
fil_space_t::set_stopping(true) can be called at any time during
the execution of this function.
mtr_t::m_freeing_tree: New data member for debugging purposes.
buf_page_get_low(): Assert that the BUF_GET mode is not being used
anywhere during the execution of btr_free_if_exists().
In all code related to freeing or allocating pages, we will add some
robustness, by making more use of BUF_GET_POSSIBLY_FREED and by
reporting an error instead of crashing in some cases of corruption.
fil_check_pending_ops(), fil_check_pending_io(): Remove.
These functions were actually duplicating each other ever since
commit 118e258aaa (MDEV-23855).
fil_space_t::check_pending_operations(): Replaces
fil_check_pending_operations() and incorporates the logic of
fil_check_pending_ops(). Avoid unnecessary lookups for the tablespace.
Just wait for the reference count to drop to zero.
fil_space_t::io(): Remove an unnecessary condition. We can (and
probably better should) refuse asynchronous reads of undo tablespaces
that are being truncated.
fil_truncate_prepare(): Remove.
trx_purge_truncate_history(): Implement the necessary steps that used
to be in fil_truncate_prepare().
innodb_adaptive_flushing_lwm is hit. (possible regression)
adaptive flushing should kick in if
a. dirty_pct (dirty pages in buffer pool) > innodb_max_dirty_pages_pct_lwm
OR
b. innodb_adaptive_flushing_lwm limit is reached (default to 10%)
both conditions are mutually exclusive and whichever is first to evaluate
true should kick-start the adaptive flushing.
After recent changes to simplify the flushing algorithm logic, (b) got ignored
that introduced the said regression.
The problem is that sharing default expression among set instruction
leads to attempt access result field of function created in
other instruction runtime MEM_ROOT and already freed
(a bit different then MySQL problem).
Fix is the same as in MySQL (but no optimisation for constant), turn
DECLARE a, b, c type DEFAULT expr;
to
DECLARE a type DEFAULT expr, b type DEFAULT a, c type DEFAULT a;
