The added by MDEV-11675 assert is incorrected assuming
a replayed from binlog transaction can't contain invoke
a FD event's apply method.
In fact it can do that through BINLOG event.
The test case like
BEGIN;
INSERT INTO t1 VALUES(10);
BINLOG '
SOgWTg8BAAAAbgAAAHIAAAAAAAQANS42LjMtbTUtZGVidWctbG9nAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAABI6BZOEzgNAAgAEgAEBAQEEgAAVgAEGggAAAAICAgCAAAAAAVAYI8=';
is provided (by yet to merged atm) MDEV-27536 fixes.
The assert is removed.
This patch fixes two issues:
First, it fixes test failure due to GTID List events
having inconsistent ordering of domain ids. In
particular, this patch ensures that a GTID list log
event will have its GTIDs ordered by domain id
(ascending) followed by sequence number (ascending).
Second, it fixes an assert which could use an
unintialized variable.
Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
This commit implements two phase binloggable ALTER.
When a new
@@session.binlog_alter_two_phase = YES
ALTER query gets logged in two parts, the START ALTER and the COMMIT
or ROLLBACK ALTER. START Alter is written in binlog as soon as
necessary locks have been acquired for the table. The timing is
such that any concurrent DML:s that update the same table are either
committed, thus logged into binary log having done work on the old
version of the table, or will be queued for execution on its new
version.
The "COMPLETE" COMMIT or ROLLBACK ALTER are written at the very point
of a normal "single-piece" ALTER that is after the most of
the query work is done. When its result is positive COMMIT ALTER is
written, otherwise ROLLBACK ALTER is written with specific error
happened after START ALTER phase.
Replication of two-phase binloggable ALTER is
cross-version safe. Specifically the OLD slave merely does not
recognized the start alter part, still being able to process and
memorize its gtid.
Two phase logged ALTER is read from binlog by mysqlbinlog to produce
BINLOG 'string', where 'string' contains base64 encoded
Query_log_event containing either the start part of ALTER, or a
completion part. The Query details can be displayed with `-v` flag,
similarly to ROW format events. Notice, mysqlbinlog output containing
parts of two-phase binloggable ALTER is processable correctly only by
binlog_alter_two_phase server.
@@log_warnings > 2 can reveal details of binlogging and slave side
processing of the ALTER parts.
The current commit also carries fixes to the following list of
reported bugs:
MDEV-27511, MDEV-27471, MDEV-27349, MDEV-27628, MDEV-27528.
Thanks to all people involved into early discussion of the feature
including Kristian Nielsen, those who helped to design, implement and
test: Sergei Golubchik, Andrei Elkin who took the burden of the
implemenation completion, Sujatha Sivakumar, Brandon
Nesterenko, Alice Sherepa, Ramesh Sivaraman, Jan Lindstrom.
In commit 685d958e38 (MDEV-14425)
a bug was introduced to mariadb-backup --backup for the case when
the log is wrapping around to log_sys.START_OFFSET (12288).
This could also cause a "Missing FILE_CHECKPOINT" error during
mariadb-backup --prepare, in case the log copying resumed after
the server had produced a multiple of innodb_log_file_size-12288
bytes of more log so that the last mini-transaction would end
exactly at the end of the log file.
xtrabackup_copy_logfile(): If the log wraps around, read everything
to the end of the log file, and then the rest from log_sys.START_OFFSET.
New Feature:
===========
This commit extends the mariadb-binlog capabilities to allow events
to be filtered by GTID ranges. More specifically, the
--start-position and --stop-position arguments have been extended to
accept values formatted as a list of GTID positions, e.g.
--start-position=0-1-0,1-2-55. The following specific capabilities
are addressed:
1) GTIDs can be used to filter results on local binlog files
2) GTIDs can be used to filter results from remote servers
3) Implemented --gtid-strict-mode that ensures the GTID event
stream in each domain is monotonically increasing
4) Added new level of verbosity in mysqlbinlog -vvv to print
additional diagnostic information/warnings about invalid GTID
states
5) For a given GTID range, its start and stop position parameters
aim to mimic the behaviors of
CHANGE MASTER TO MASTER_USE_GTID=slave_pos and
START SLAVE UNTIL master_gtid_pos=<GTID>, respectively. In
particular, the start-position list expresses a gtid state of
the server, similarly to how @@global.gtid_slave_pos expresses
the gtid state of a slave server when connecting to a master
with MASTER_USE_GTID=slave_pos.
The GTID start-position list is exclusive and the
stop-position list is inclusive. This allows users to receive
events strictly after those that they already have, and is
useful in cases of point in (logical) time recovery including
1) events were received out of order and should be re-sent, or
2) specifying the gtid state of a slave to get events newer
than their current state. If a seq_no is 0 for start-position,
it means to include the entirety of the domain. If a seq_no is
0 for stop-position, it means to exclude all events from that
domain. The GTIDs provided in a start position argument must
match with the GTID state of the first processed log (i.e.
those listed in the Gtid_list event). If a stop position is
provided, the events that are output are limited to only those
with domain ids listed in the argument. When specifying
combinations of start and stop positions, the following
behaviors are expected:
[--start-position without --stop-position]: Events that have domain
ids in the start position are output if their seq_no occurs after
the respective start position. Events with domain ids that are
unspecified in the start position list are also output. Note that if
the Gtid_list event of the first binary log is populated (i.e.
non-empty), each domain in the Gtid_list must be present in the
start-position list with a seq_no at or after the listed value.
This behavior mimics how a slave only processes events after the
state provided by @@global.gtid_slave_pos when connecting to a
master with CHANGE MASTER TO MASTER_USE_GTID=slave_pos.
[--stop-position without --start-position]: Output is limited to
only events with both 1) domain ids that are present in the given
stop position list and 2) seq_nos that are less than or equal to
their respective stop GTID. Once all GTIDs in the stop position
list have been processed, the program will stop processing log
files. This behavior mimics how
START SLAVE UNTIL master_gtid_pos=<G>
has a slave only process events with domain ids present in G with
their seq_nos at or before the respective gtid.
[--start-position and --stop-position]: Output consists of the
intersection between the events permitted by both the start and stop
position rules. More concretely, the output can be defined by a
union of the following rules:
1. For domains which exist in both the start and stop position
lists, the events which exist in-between these positions
(exclusive start, inclusive stop) are output
2. For all other events, the rules of
[--stop-position without --start-position] are followed
This is due to the implicit filtering within each individual rule.
Even though the start position rule always includes events from
unspecified domains, the stop position rule takes precedence because
it always excludes events from unspecified domains. In other words,
events which the start position rule would have included would then
always be excluded by the stop position rule.
[neither --start-position nor --stop-position]: Events are not
omitted based on GTID positioning; however, --gtid-strict-mode and
-vvv can still analyze gtid correctness for warning and error
reporting.
[repeated specification of --start-position or --stop-position]:
Subsequent specifications of start and stop positions completely
override previous ones. E.g., if invoked as
mysqlbinlog --start-position=<G1> --start-position=<G2> ...
All GTIDs specified in G1 are ignored and only those specified in G2
are used for the start position.
A few additional notes:
1) this commit squashes together the commits:
f4319661120e-78a9d49907ba
2) Changed rpl.rpl_blackhole_row_annotate test because it has
out of order GTIDs in its binlog, so I added
--skip-gtid-strict-mode
3) After all binlog events have been written, the session server
id and domain id are reset to their values in the global state
Reviewed By:
===========
Andrei Elkin: <andrei.elkin@mariadb.com>
in my_print_defaults
Analysis: --defaults* option is recognized anywhere in the commandline
instead of only at the beginning because handle_options() recognizes
options in any order.
Fix: use get_defaults_options() which recognizes --defaults* options only at
the beginning. After this is done, we only want to recognize other options
given in any order which can be done using handle_options(). So only skip
--defaults* options and pass rest of them to handle_options().
Also, removed -e, -g and -c because only my_print_defaults supports them.
MyiSAM and Aria, indexes with prefix compression, where the first keypart
could be NULL - in this case they didn't expect the next key after the
not NULL key to be NULL.
Expect the first keypart of the next key to have zero length even
if store_not_null==1, this combination means keypart is NULL,
don't pack it.
also fixes MDEV-27340
ROR-index_merge relies on Rowid-ordered-retrieval property: a ROR scan,
e.g. a scan on equality range
tbl.key=const
should return rows ordered by their Rowid. Also, handler->cmp_ref() should
compare rowids according to the Rowid ordering.
When the table's primary key uses DESC keyparts, ROR scans return rows
according to the PK's ordering.
But ha_innobase::cmp_ref() compared rowids as if PK used ASC keyparts.
This caused wrong query results with index_merge.
Fixed this by making ha_innobase::cmp_ref() compare according to the PK
defintion, including keypart's DESC property.
Make QUICK_RANGE_SELECT::cmp_next() aware of reverse-ordered key parts.
(QUICK_RANGE_SELECT::cmp_prev() uses key_cmp() and so it already works
correctly)
Extend the fix for MDEV-25858 to handle non-reverse-ordered ORDER BY:
If test_if_skip_sort_order() decides to use an index to produce rows
in the required ordering, it should disable "Range Checked for Each Record".
The fix needs to be backported to earlier versions.
Make the Range Optimizer support descending index key parts.
We follow the approach taken in MySQL-8.
See HowRangeOptimizerHandlesDescKeyparts for the description.
* preserve DESC index property in the parser
* store it in the frm (only for HA_KEY_ALG_BTREE)
* read it from the frm
* show it in SHOW CREATE
* skip DESC indexes in opt_range.cc and opt_sum.cc
* ORDER BY test
This includes a fix of MDEV-27432.
This is loosely based on the InnoDB changes in
mysql/mysql-server@97fd8b1b69
that I had developed in 2015 or 2016.
For each B-tree key field, we will allow a flag ASC/DESC to be associated.
When PRIMARY KEY fields are internally appended to secondary indexes,
the ASC/DESC attribute will be inherited, so that covering index scans
will work as expected.
Note: Until the subsequent commit, the DESC attribute will be ignored
(no HA_REVERSE_SORT flag will be written to .frm files).
dict_field_t::descending: A new flag to denote descending order.
cmp_data(), cmp_dfield_dfield(): Add a new parameter descending.
cmp_dtuple_rec(), cmp_dtuple_rec_with_match(): Add a parameter "index".
dtuple_coll_eq(): Replaces dtuple_coll_cmp().
cmp_dfield_dfield_eq_prefix(): Replaces cmp_dfield_dfield_like_prefix().
dict_index_t::is_btree(): Check whether the index is a regular
B-tree index (not SPATIAL, FULLTEXT, or the ibuf.index,
or a corrupted index.
btr_cur_search_to_nth_level_func(): Only attempt to use
the adaptive hash index if index->is_btree().
This function may also be invoked on ibuf.index, and
cmp_dtuple_rec_with_match_bytes() will no longer work on ibuf.index
because it assumes that the index and record fields exactly match.
The ibuf.index is a special variadic index tree.
Thanks to Thirunarayanan Balathandayuthapani for fixing some bugs:
MDEV-27439, MDEV-27374/MDEV-27445.
innodb_flush_log_on_trx_commit=2 means we want to write
redo using filesystem buffering, and not flushing each time.
Also simplify code, remove gotos, and comment on a bug
in mariabackup that produces redo of odd sizes, that must
be worked around by forcing file to be buffered.
Changes:
1. Enabling IN/OUT/INOUT mode for sql_mode=DEFAULT,
adding tests for sql_mode=DEFAULT based by mostly
translating compat/oracle.sp-inout.test to SQL/PSM
with minor changes (e.g. testing trigger OLD.column and
NEW.column as IN/OUT parameters).
2. Removing duplicate grammar:
sp_pdparam and sp_fdparam implemented exactly the same syntax after
- the first patch for MDEV-10654 (for sql_mode=ORACLE)
- the change #1 from this patch (for sql_mode=DEFAULT)
Removing separate rules and adding a single "sp_param" rule instead,
which now covers both PRDEDURE and FUNCTION parameters
(and CURSOR parameters as well!).
3. Adding a helper rule sp_param_name_and_mode, which is a combination
of the parameter name and the IN/OUT/INOUT mode. It allows to simplify
the grammer a bit.
4. The first patch unintentionally allowed IN/OUT/INOUT mode
to be specified in CURSOR parameters.
This is good for the IN keyword - it is allowed in PL/SQL CURSORs.
This is not good the the OUT/INOUT keywords - they should not be allowed.
Adding a additional symantic post-check.
Problem: Currently stored function does not support IN/OUT/INOUT parameter qualifiers.
This is needed for Oracle compatibility (sql_mode = ORACLE).
Solution: Implemented parameter qualifier support to CREATE FUNCTION (reference: CREATE PROCEDURE)
Implemented return by reference for OUT/INOUT parameters in execute_function() (reference: execute_procedure())
Files changed:
sql/sql_yacc.yy: Added IN, OUT, INOUT parameter qualifiers for CREATE FUNCTION.
sql/sp_head.cc: Added input and output parameter binding for IN/OUT/INOUT parameters in execute_function() so that OUT/INOUT can return by reference.
sql/share/errmsg-utf8.txt: Added error message to restrict OUT/INOUT parameters while function being called from SQL query.
mysql-test/suite/compat/oracle/t/sp-inout.test: Added test cases
mysql-test/suite/compat/oracle/r/sp-inout.result: Added test results
Reviewed-by: iqbal@hasprime.com
Remove the dead-code, in Spider, which is related to the Spider's
HandlerSocket support. The code has been disabled for a long time
and it is unlikely that the code will be enabled.
We introduce engine-defined attributes to specify remote data nodes.
The engine attributes do not cover all the existing DSN parameters
because most of them need not be specified at the table level.
We introduce the following three attributes: REMOTE_SERVER,
REMOTE_DATABASE, REMOTE_TABLE.
One cannot specify both DSN parameter, in COMMENT or CONNECT, and
engine-defined attribute that are for the same SPIDER_SHARE attribute.
For example, Spider returns an error if both COMMENT='table "t1"'
and REMOTE_TABLE="t2" are specified for a single Spider table or
a single partition in a Spider table.
Make it possible to specify engine-defined attributes on partitions
as well as tables.
If an engine-defined attribute is only specified at the table level,
it applies to all the partitions in the table.
This is a backward-compatible behavior.
If the same attribute is specified both at the table level and the
partition level, the per-partition one takes precedence.
So, we can consider per-table attributes as default values.
One cannot specify engine-defined attributes on subpartitions.
Implementation details:
* We store per-partition attributes in the partition_element class
because we already have the part_comment field, which is for
per-partition comments.
* In the case of ALTER TABLE statements, the partition_elements in
table->part_info is set up by mysql_unpack_partition().
So, we parse per-partition attributes after the call of the function.
Cleaned up the log messages as suggested, with a minor code
formatting change.
On bullet point 13, I decided to not include timestamp in output
message. In most (all?) cases, the output goes to the log file,
which has timestamp already.
Implement new mini-benchmark script for simple CPU bound benchmark for the
duration of 5 minutes. The script can be run stand-alone or as part of a
CI pipeline.
Extend Gitlab-CI to run mini-benchmark on every commit to catch if there
are severe performance regressions.
Also bump MARIADB_MAJOR_VERSION to 10.8 which is needed on the 10.8 branch.
The test ./mtr --ps-protocol main.func_math
was broken in commit 5b3ad94c7b
because in that mode, one of several truncation warnings for
a single integer literal would be omitted. Those warnings are
issued by the parser somewhere outside CRC32() or CRC32C().
