wsrep_sst_mariabackup should use the tmpdir defined by
the user under the '[mysqld]' section of the configuration
file rather than the default '/tmp' directory.
This commit contains a fix for the code that extracts and parses
the CN (common name, domain name) record from certificates using
the openssl utility. This code is also made common to the rsync
and mariabackup scripts. There is also some systematization of
the use of 'printf' and 'echo' builtins/utilities.
This commit fixes GTID inconsistency which was injected by mariabackup SST.
Donor node now writes new info file: donor_galera_info, which is streamed
along the mariabackup donation to the joiner node. The donor_galera_info
file contains both GTID and gtid domain_id, and joiner will use these to
initialize the GTID state.
Commit has new mtr test case: galera_3nodes.galera_gtid_consistency, which
exercises potentially harmful mariabackup SST scenarios. The test has also
scenario with IST joining.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
This commit fixes a bug where IST could be rejected in favor of SST
when ssl-mode=VERIFY_CA and when mariabackup is used. It also contains
a test and small code simplifications that will make it easier to find
bugs in the future.
Due to the different command line format of the timeout
utility on FreeBSD and Linux, SST scripts for mariabackup
may not work on FreeBSD. This commit fixes this problem
by adding a different command to test options on FreeBSD
and adding proper formatting for the utility options.
SST for mariabackup may not destroy old files if datadir or
other working directory is declared as a symlink due to the lack
of the "-L" option among the find utility options, similarly SST
for rsync in some cases may not transfer data directories if they
are created as symlinks. This fix adds the missing option and
generally unifies the work with find utility options to avoid
failures in the interpretation of directories and regular
expressions.
This commit adds a new 'no-sni' option to socat which is required to
properly authenticate with newer socat versions (after version 1.7.4+).
This option is needed to disable the automatic use of the SNI feature
(Server Name Indication) since the SST script directly specifies the
commonname if necessary and automatic activation of the SNI feature
is unnecessary in such scenarios.
Renaming the default MariaDB backup directory from
xtrabackup_backupfiles to mariadb_backup_files.
Renaming files:
- xtrabackup_binlog_info to mariadb_backup_binlog_info
- xtrabackup_checkpoints to mariadb_backup_checkpoints
- xtrabackup_galera_info to mariadb_backup_galera_info
- xtrabackup_info to mariadb_backup_info
- xtrabackup_slave_info to mariadb_backup_slave_info
This commit adds support for the --aria-log-dir-path
option on the command line and for the aria-log-dir-path
option in the configuration file to the SST scripts, since
before this change these parameters were completely ignored
during SST - SST scripts assumed that aria logs files are
always located in the same directory as logs for innodb.
Tests for this change will be added as a separate commit,
along with tests for MDEV-30157 and MDEV-28669.
This commit adds even more correct handling of parameters
with paths when they contain leading or trailing spaces and/or
slashes. Also it fixes problems that occur when the user specified
explicit paths to additional directories, but these paths match
the specified path of the data directory - in this case, additional
subdirectories should be treated (in relation to the data directory)
in the same way as if these paths were not specified or as if they
are implicitly specified as "." or "./". But prior to this fix,
existing code treated any values as if they were completely
separate directories, whether or not they actually point to the
same location to which datadir points to - and this sometimes
resulted in incorrect file transfers.
This fix does not contain separate tests, as tests will be
part of the main commit(s). This fix has been made as a separate
commit to facilitate review for major substantive fixes related
to MDEV-30157 and MDEV-28669.
This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.
The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.
This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).
Also fixed broken logic of automatic generation of the Diffie-Hellman parameters
for socat version less than 1.7.3 (which defaults to 512-bit values instead of
2048-bit ones).
This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.
This commit fixes an issue with IST handling in
version 10.9 which is a regression after MDEV-26971
and related to trying to get a non-existent "total"
tag on the IST branch (this tag is only defined in
SST mode).
This commit fixes problems with parsing ipv6 addresses given via
the wsrep_sst_receive_address and wsrep_node_address options.
Also, this commit removes extra lines in the configuration files
in the mtr test suites for Galera related to these parameters.
Currenly SST script for mariabackup stops on any failure while archiving
logs, e.g. when unable to create directory, insufficient permissions, gzip
failure, etc. However, in case of such problems, the script should issue
a warning and continue without archiving, but not exit with a fatal error.
This commit adds this fix to the SST script for mariabackup.
This commit contains a fix to use modern syntax for selecting
character classes in the tr utility options.
Also one of the tests for SST via rsync (galera_sst_rysnc2) is made
more reliable (to avoid rare failures during automatic testing).
This commit adds validation of the values of the ssl-mode parameter
in SSL scripts, since now only a basic check for the presence of the
"VERIFY_" prefix is performed there to detect "VERIFY_IDENTITY" and
"VERIFY_CA", but all other values are not checked at all.
In addition, this commit removes leading and trailing spaces from
parameter values that SST scripts read from configuration files or
from the command line so that they do not interfere with parameter
checks and substitutions. Parameter substitution has been made more
robust against characters in strings that the shell might erroneously
interpret as regexp.
This commit fixes problems due to bugs and quirks in bsdtar
(the FreeBSD version of tar). Separate tests are not required,
because without these fixes, many other tests fail when tested
in the FreeBSD environment.
Also, the grep patterns for reading utility version numbers
has been made more robust. The notation of some options of
the "cut" utility has been changed.
This commit adds correct handling of binlogs for SST using rsync
or mariabackup. Before this fix, binlogs were handled incorrectly -
- only one (last) binary log file was transferred during SST, which
then led to various failures (for example, when trying to list all
events from the binary log). These bugs were long masked by flaws
in the primitive binlogs handling code in the SST scripts, which
causing binary logs files to be erased after transfer or not added
to the binlog index on the joiner node. Now the correct transfer
of all binary logs (not just the last of the binary log files) has
been implemented both for the rsync (at the script level) and for
the mariabackup (at the level of the main utility code).
This commit also adds a new sst_max_binlogs=<n> parameter, which
can be located in the [sst] section or in the [xtrabackup] section
(historically, supported for mariabackup only, not for rsync), or
in one of the server sections. This parameter specifies the number
of binary log files to be sent to the joiner node during SST. This
option is added for compatibility with old SST scripting behavior,
which can be emulated by setting the sst_max_binlogs=1 (although
in general this can cause problems for the reasons described above).
In addition, setting the sst_max_binlogs=0 can be used to suppress
the transmission of binary logs to the joiner nodes during SST
(although sometimes a single file with the current binary log can
still be transmitted to the joiner, even with sst_max_binlogs=0,
because this sometimes necessary in modes that involve the use of
GTIDs with Galera).
Also, this commit ensures correct handling of paths to various
innodb files and directories in the SST scripts, and fixes some
problems with this that existed in mariabackup utility (which
were associated with incorrect handling of the innodb_data_dir
parameter in some scenarios).
In addition, this commit contains the following enhancements:
1) Added tests for mtr, which check the correct work with binlogs
after SST (using rsync and mariabackup);
2) Added correct handling of slashes at the end of all paths that
the SST script receives as parameters;
3) Improved parsing code for --mysqld-args parameters. Now it
correctly processes the sequence "--" after the name of the
one-letter option;
4) Checking the secret signature during joiner authentication
is made independent of presence of bash (as a unix shell)
in the system and diff utility no longer needed to check
certificates compliance;
5) All directories that are necessary for the correct placement
of various logs are automatically created by SST scripts in
advance (before running mariabackup on the joiner node);
6) Removal of old binary logs on joiner is done using the binlog
index (if it exists) (not only by fixed pattern that based
on the current binlog name, as before);
7) Paths for placing binary logs are correctly processed if they
are set as relative paths (to the datadir);
8) SST scripts are made even more resistant to spaces in filenames
(now for binlogs);
9) In case of failure, SST scripts now always end with an exit
code other than zero;
10) SST script for rsync now correctly create a tar file with
the binlogs, even if the paths to them (in the binlog index
file) are specified as a mix of absolute and relative paths,
and even if they do not match with the datadir path specified
in the current configuration settings.
- Changed SST scripts to use /usr/bin/env bash instead of
/bin/bash for better portability.
- Fixed use of mktemp on non-Linux platforms to produce
temporary file instead of directory.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
This is the first part of the fixes for MDEV-24097. This commit
contains the fixes for instability when testing Galera and when
restarting nodes quickly:
1) Protection against a "stuck" old SST process during the execution
of the new SST (after restarting the node) is now implemented for
mariabackup / xtrabackup, which should help to avoid almost all
conflicts due to the use of the same ports - both during testing
with mtr, so and when restarting nodes quickly in a production
environment.
2) Added more protection to scripts against unexpected return of
the rc != 0 (in the commands for deleting temporary files, etc).
3) Added protection against unexpected crashes during binlog transfer
(in SST scripts for rsync).
4) Spaces and some special characters in binlog filenames shouldn't
be a problem now (at the script level).
5) Daemon process termination tracking has been made more robust
against crashes due to unexpected termination of the previous SST
process while new scripts are running.
6) Reading ssl encryption parameters has been moved from specific
SST scripts to a common wsrep_sst_common.sh script, which allows
unified error handling, unified diagnostics and simplifies script
revisions in the future.
7) Improved diagnostics of errors related to the use of openssl.
8) Corrections have been made for xtrabackup-v2 (both in tests and in
the script code) that restore the work of xtrabackup with updated
versions of innodb.
9) Fixed some tests for galera_3nodes, although the complete solution
for the problem of starting three nodes at the same time on fast
machines will be done in a separate commit.
No additional tests are required as this commit fixes problems with
existing tests.
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
Currently, SST scripts assume that the filename specified in
the --log-bin-index argument either does not contain an extension
or uses the standard ".index" extension. Similar assumptions are
used for the log_bin_index parameter read from the configuration
file. This commit adds support for arbitrary extensions for the
index file paths.
If the server is started with the --innodb-force-recovery argument
on the command line, then during SST this argument can be passed to
mariabackup only at the --prepare stage, and accordingly it must be
removed from the --mysqld-args list (and it is not should be passed
to mariabackup otherwise).
This commit fixes a flaw in the SST scripts and add a test that
checks the ability to run the joiner node in a configuration that
uses --innodb-force-recovery=1.
Fixed flaws with overly strict or, conversely,
overly soft verification of certificates in some
scenarios:
1. Removed the check that the 'commonname' (CN) in the
certificate matches the 'localhost' value on the side
of the joiner node, which was performed earlier, even
if the address was received by the script only as an
argument (out of the exchange via the Galera protocol) -
since for the joining node this argument always contains
its own local address, not the address of the remote host,
so it is always treated as 'localhost', which is not
necessarily true (outside of mtr testing);
2. Removed checking the domain name or IP-address of the
peer node in the encrypt=2 mode;
3. Fixed checking of compliance of certificates when
rsync SST is used;
4. Added the ability to specify CA not only as a file,
but also as a path to the directory where the certificates
are stored. To do this, the user just needs to specify the
path to this directory as the value ssl-ca or tca parameter,
ending with the '/' character.
SST scripts currently use Linux-specific construction
to create a temporary directory if the path prefix for
that directory is specified by the user. This does not
work with FreeBSD. This commit adds support for FreeBSD.
No separate test required.
This commit adds support for reading new SSL configuration
options (ssl-ca, ssl-cert and ssl-key) if the [sst] section
with old options (tca, tcert and tkey) is missing in the config
file, even if not specified authentication mode via the ssl-mode
option. Before this change, new parameters were read only if the
ssl-mode option was present in the configuration file and it was
not equal to the 'DISABLED' value.
Also added diagnostics (information level) which warns the user
that due to the presence of the tca, tcert and/or tkey parameters
in the [sst] section, new SSL configuration options will be ignored
(if their values do not match the old ones).
Fixed typo in variable name that breaks SST in some scenatios.
Also fixed one small inaccuracy after MDEV-25978 which leads
to the use of an uninitialized variable when the --log-bin
option is specified without an argument.
