New Feature:
============
Extend mariadb-binlog command-line tool to allow for filtering
events using GTID domain and server ids. The functionality mimics
that of a replica server’s DO_DOMAIN_IDS, IGNORE_DOMAIN_IDS, and
IGNORE_SERVER_IDS from CHANGE MASTER TO. For completeness, this
patch additionally adds the option --do-server-ids as an alias for
--server-id, which now accepts a list of server ids instead of a
single one.
Example usage:
mariadb-binlog --do-domain-ids=2,3,4 --do-server-ids=1,3
master-bin.000001
Functional Notes:
1. --do-domain-ids cannot be combined with --ignore-domain-ids
2. --do-server-ids cannot be combined with --ignore-server-ids
3. A domain id filter can be combined with a server id filter
4. When any new filter options are combined with the
--gtid-strict-mode option, events from excluded domains/servers are
not validated.
5. Domain/server id filters can be combined with GTID ranges (i.e.
specifications of --start-position and --stop-position). However,
because the --stop-position option implicitly undertakes filtering
to only output events within its range of domains, when combined
with --do-domain-ids or --ignore-domain-ids, output will consist of
the intersection between the filters. Specifically, with
--do-domain-ids and --stop-position, only events with domain ids
present in both argument lists will be output. Conversely, with
--ignore-domain-ids and --stop-position, only events with domain ids
present in the --stop-position and absent from the
--ignore-domain-ids options will be output.
Reviewed By
============
Andrei Elkin <andrei.elkin@mariadb.com>
The reason for this fix was that when I tried to run mysql_upgrade
at home to update an old 10.5 installation, mysql_upgrade failed
with warnings about mariadb.sys user not existing.
If the server was started with --skip-grants, there would be no warnings
from mysql_upgrade, but in some cases running mysql_upgrade again could
produce new warnings.
The reason for the warnings was that any access of the mysql.user view
will produce a warning if the mariadb.sys user does not exists.
Fixed with the following changes:
- Disable warnings about mariadb.sys user not existing
- Don't overwrite old mariadb.sys entries in tables_priv and global_priv
- Ensure that tables_priv has an entry for mariadb.sys if the user exists.
This fixes an issue that tables_priv would not be updated if there
was a failure directly after global_priv was updated.
Fix a possible crash on my_free() due to the use of strdup() versus
my_strdup(), and a memory leak.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
Problem:
========
When using mariadb-binlog with --raw and --stop-never, events from
the master's currently active log file should be written to their
respective log file specified by --result-file, and shown on-disk.
There is a bug where mariadb-binlog does not flush the result file
to disk when new events are received
Solution:
========
Add a function call to flush mariadb-binlog’s result file after
receiving an event in --raw mode.
Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
This patch also fixes:
MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition
MDEV-27853 Wrong data type on column `COLLATE DEFAULT` and table `COLLATE some_non_default_collation`
MDEV-28067 Multiple conflicting column COLLATE clauses are not rejected
MDEV-28118 Wrong collation of `CAST(.. AS CHAR COLLATE DEFAULT)`
MDEV-28119 Wrong column collation on MODIFY + CONVERT
This commit contains a test for reproducing the issue in MDEV-27649,
where a transaction, executing a prepared statment, is BF aborted.
The scenario, in MDEV-27649 has a transaction which has prepared a PS,
but not yet executed it, and this transaction is then BF aborted in this state.
When the BF aborted transaction tries to execute the PS, it will receive deadlock error.
But, when it tries to execute the PS second time, the node crashes.
Mtr test galera.galera_bf_abort_ps_bind, exercises this scenario.
However, mtr test platform does not have mechanism to control the execution of PS in required detail.
For this purpose, mysqltetst.cc was extended to contain 4 new commands:
PS_prepare - to prepare a prepared statement
PS_bind - to bind values for parameters for the PS
PS_execute - to execute the PS
PS_close - to close the PS
The support for controlling prepared statments in mtr scripts is quite minimal
in this commit. Limitations are:
* only one PS can be used by a connection, at a time
* only input parameters can be bound for the PS
* only varchar, integer or float type of parameters can be bound
added the result
fixes
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
Added ability to disable/enable (--disable_view_protocol/--enable_view_protocol) view-protocol in tests.
When the option "--disable_view_protocol" is used util connections are closed.
Added new test for checking view-protocol
Removed all dependencies of command line arguments based on positions in
an array (this kind of code should never have been written).
Instead use option names, which are stable.
Reviewer: Sergei Golubchik
This commit implements two phase binloggable ALTER.
When a new
@@session.binlog_alter_two_phase = YES
ALTER query gets logged in two parts, the START ALTER and the COMMIT
or ROLLBACK ALTER. START Alter is written in binlog as soon as
necessary locks have been acquired for the table. The timing is
such that any concurrent DML:s that update the same table are either
committed, thus logged into binary log having done work on the old
version of the table, or will be queued for execution on its new
version.
The "COMPLETE" COMMIT or ROLLBACK ALTER are written at the very point
of a normal "single-piece" ALTER that is after the most of
the query work is done. When its result is positive COMMIT ALTER is
written, otherwise ROLLBACK ALTER is written with specific error
happened after START ALTER phase.
Replication of two-phase binloggable ALTER is
cross-version safe. Specifically the OLD slave merely does not
recognized the start alter part, still being able to process and
memorize its gtid.
Two phase logged ALTER is read from binlog by mysqlbinlog to produce
BINLOG 'string', where 'string' contains base64 encoded
Query_log_event containing either the start part of ALTER, or a
completion part. The Query details can be displayed with `-v` flag,
similarly to ROW format events. Notice, mysqlbinlog output containing
parts of two-phase binloggable ALTER is processable correctly only by
binlog_alter_two_phase server.
@@log_warnings > 2 can reveal details of binlogging and slave side
processing of the ALTER parts.
The current commit also carries fixes to the following list of
reported bugs:
MDEV-27511, MDEV-27471, MDEV-27349, MDEV-27628, MDEV-27528.
Thanks to all people involved into early discussion of the feature
including Kristian Nielsen, those who helped to design, implement and
test: Sergei Golubchik, Andrei Elkin who took the burden of the
implemenation completion, Sujatha Sivakumar, Brandon
Nesterenko, Alice Sherepa, Ramesh Sivaraman, Jan Lindstrom.
New Feature:
===========
This commit extends the mariadb-binlog capabilities to allow events
to be filtered by GTID ranges. More specifically, the
--start-position and --stop-position arguments have been extended to
accept values formatted as a list of GTID positions, e.g.
--start-position=0-1-0,1-2-55. The following specific capabilities
are addressed:
1) GTIDs can be used to filter results on local binlog files
2) GTIDs can be used to filter results from remote servers
3) Implemented --gtid-strict-mode that ensures the GTID event
stream in each domain is monotonically increasing
4) Added new level of verbosity in mysqlbinlog -vvv to print
additional diagnostic information/warnings about invalid GTID
states
5) For a given GTID range, its start and stop position parameters
aim to mimic the behaviors of
CHANGE MASTER TO MASTER_USE_GTID=slave_pos and
START SLAVE UNTIL master_gtid_pos=<GTID>, respectively. In
particular, the start-position list expresses a gtid state of
the server, similarly to how @@global.gtid_slave_pos expresses
the gtid state of a slave server when connecting to a master
with MASTER_USE_GTID=slave_pos.
The GTID start-position list is exclusive and the
stop-position list is inclusive. This allows users to receive
events strictly after those that they already have, and is
useful in cases of point in (logical) time recovery including
1) events were received out of order and should be re-sent, or
2) specifying the gtid state of a slave to get events newer
than their current state. If a seq_no is 0 for start-position,
it means to include the entirety of the domain. If a seq_no is
0 for stop-position, it means to exclude all events from that
domain. The GTIDs provided in a start position argument must
match with the GTID state of the first processed log (i.e.
those listed in the Gtid_list event). If a stop position is
provided, the events that are output are limited to only those
with domain ids listed in the argument. When specifying
combinations of start and stop positions, the following
behaviors are expected:
[--start-position without --stop-position]: Events that have domain
ids in the start position are output if their seq_no occurs after
the respective start position. Events with domain ids that are
unspecified in the start position list are also output. Note that if
the Gtid_list event of the first binary log is populated (i.e.
non-empty), each domain in the Gtid_list must be present in the
start-position list with a seq_no at or after the listed value.
This behavior mimics how a slave only processes events after the
state provided by @@global.gtid_slave_pos when connecting to a
master with CHANGE MASTER TO MASTER_USE_GTID=slave_pos.
[--stop-position without --start-position]: Output is limited to
only events with both 1) domain ids that are present in the given
stop position list and 2) seq_nos that are less than or equal to
their respective stop GTID. Once all GTIDs in the stop position
list have been processed, the program will stop processing log
files. This behavior mimics how
START SLAVE UNTIL master_gtid_pos=<G>
has a slave only process events with domain ids present in G with
their seq_nos at or before the respective gtid.
[--start-position and --stop-position]: Output consists of the
intersection between the events permitted by both the start and stop
position rules. More concretely, the output can be defined by a
union of the following rules:
1. For domains which exist in both the start and stop position
lists, the events which exist in-between these positions
(exclusive start, inclusive stop) are output
2. For all other events, the rules of
[--stop-position without --start-position] are followed
This is due to the implicit filtering within each individual rule.
Even though the start position rule always includes events from
unspecified domains, the stop position rule takes precedence because
it always excludes events from unspecified domains. In other words,
events which the start position rule would have included would then
always be excluded by the stop position rule.
[neither --start-position nor --stop-position]: Events are not
omitted based on GTID positioning; however, --gtid-strict-mode and
-vvv can still analyze gtid correctness for warning and error
reporting.
[repeated specification of --start-position or --stop-position]:
Subsequent specifications of start and stop positions completely
override previous ones. E.g., if invoked as
mysqlbinlog --start-position=<G1> --start-position=<G2> ...
All GTIDs specified in G1 are ignored and only those specified in G2
are used for the start position.
A few additional notes:
1) this commit squashes together the commits:
f4319661120e-78a9d49907ba
2) Changed rpl.rpl_blackhole_row_annotate test because it has
out of order GTIDs in its binlog, so I added
--skip-gtid-strict-mode
3) After all binlog events have been written, the session server
id and domain id are reset to their values in the global state
Reviewed By:
===========
Andrei Elkin: <andrei.elkin@mariadb.com>
MDEV-27107 prevent two mariadb-upgrade running in parallel
MDEV-27279 mariadb_upgrade add --check-if-upgrade-is-needed /
restrict tests to major version
Code is based of pull request from Daniel Black, but with a several
extensions.
- mysql_upgrade now locks the mysql_upgrade file with my_lock()
(Advisory record locking). This ensures that two mysql_upgrades
cannot be run in parallel.
- Added --check-if-upgrade-is-needed to mysql_upgrade. This will return
0 if one has to run mysql_upgrade.
Other changes:
- mysql_upgrade will now immediately exit if the major version and minor
version (two first numbers in the version string) is same as last run.
Before this change mysql_upgrade was run if the version string was different
from last run.
- Better messages when there is no need to run mysql_upgrade.
- mysql_upgrade --verbose now prints out a lot more information about
the version checking.
- mysql_upgrade --debug now uses default debug arguments if there is no
option to --debug
- "MySQL" is renamed to MariaDB in the messages
- mysql_upgrade version increased to 2.0
Notes
Verifying "prevent two mariadb-upgrade running in parallel" was
done in a debugger as it would be a bit complex to do that in mtr.
Reviewer: Danial Black <daniel@mariadb.org>
Translate username, password and database from UTF8 into desired charset,
if non-auto default-character-set was used, on Windows10 1903
This change is implemented only in the command line client, and mainly to
allow users with non-UTF8 passwords to login.
The user is supposed to use the same charset that was used during setting
password (usually, console CP if used in CLI)
Add a test to document the behavior.
If someone on whatever reasons uses --default-character-set=cp850,
this will avoid incorrect display, and inserting incorrect data.
Adjusting console codepage sometimes also needs to happen with
--default-charset=auto, on older Windows. This is because autodetection
is not always exact. For example, console codepage on US editions of
Windows is 437. Client autodetects it as cp850, a rather loose
approximation, given 46 code point differences. We change the console
codepage to cp850, so that there is no discrepancy.
That fix is currently Windows-only, and serves people who used combination
of chcp to achieve WYSIWYG effect (although, this would mostly likely used
with utf8 in the past)
Now, --default-character-set would be a replacement for that.
Fix fs_character_set() detection of current codepage.
