From downstream bug RHEL-116757, the use of ss to map all the
usage of ports to pids and process names under selinux is rather
limited by the mysqld_exec_t limitations.
The checking of the mapping is a bit excessive in a selinux
environment where binding is limited.
We replace the use of ss under selinux to raw reading of the
/proc/tcp{,} interfaces of the linux that aren't selinux restricted.
Paths are set to "canonical" form and historical practice where
/lib comes before /usr/lib and /bin comes before /usr/bin. This
eliminates the need to skip setting LD_LIBRARY_PATH for FreeBSD,
which may help those users whose LD_LIBRARY_PATH is not configured
properly.
Bootstrapping nodes on FreeBSD 15.0 with mariadb-backup currently broken.
We should not need LD_LIBRARY_PATH trickery on FreeBSD, so skip setting
it same as on Darwin. Alternatively putting /lib before /usr/lib also
fixes it.
WSREP_SST: [INFO] Preparing the backup at /var/db/mysql/.sst (20250608 14:44:04.000)
WSREP_SST: [INFO] Evaluating /usr/local/bin/mariadb-backup --prepare ......
WSREP_SST: [ERROR] Cleanup after exit with status: 1 (20250608 14:44:04.000)
With the following error in /var/db/mysql/mariabackup.prepare.log:
ld-elf.so.1: /usr/local/bin/mariadb-backup: Undefined symbol "_ZNSt3__117bad_function_callD1Ev"
I did not look for the root cause in FreeBSD 15.0 that changed this
behaviour. Calling mariadb-backup directly from the shell works fine on
FreeBSD 14.x and 15.x when LD_LIBRARY_PATH is not set (when using
FreeBSD packages or installing from FreeBSD ports). Setting
LD_LIBRARY_PATH manually like the script does, breaks mariadb-backup
with the "Undefined symbol" error on FreeBSD 15.0 but not on 14.x.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Delaying scripts on joiner after SST/IST has been made
a common debug feature for all suitable SST/IST methods.
Also some minor fixes have been made for new tests.
Mariabackup (mariadb-backup) supports the --use-memory option that
sets the buffer pool size for innodb. However, current SST scripts
do not use this option. This commit adds support for this option,
the value for which can be specified via the "use_memory" parameter
in the configuration file in the [sst], [mariabackup] or [xtrabackup]
sections (supported only for compatibility with old configurations).
In addition, if the innodb_buffer_pool_size option is specified in
the user configuration (in the main server configuration sections)
or passed to the SST scripts or the server via arguments, its value
is also passed to mariadb-backup as the value for the --use-memory
option.
A new section name [mariabackup] has also been added, which can
be used instead of the deprecated [xtrabackup] (the section name
"mariabackup" was specified in the documentation, but was not
actually supported by SST scripts before this commit).
This commit fixes some functions in wsrep_sst_common
to ensure that now and in the future return codes from
a number of helper functions will be zero on success.
Fixed some issues in the script code, mainly related
to handling situations when a failure occurs:
1) the signal handler in the mariadb-backup SST script
was using an uninitialized variable when trying to kill
a hung streaming process;
2) inaccurate error messages were being logged sometime;
3) after completing SST, temporary or old (extra) files
could remain in database directories.
The lsof utility is prone to blocking on system calls that
it uses to obtain information about sockets (or files, devices,
etc.). This behavior is described in its own documentation.
It has a '-b' option (in combination with warnings suppression
via '-w') that reduces the probability of blocking, introducing
new problems (luckily probably not relevant for our use case).
However, there is no guarantee that it will not hang on some
distributions, with some TCP/IP stack implementations, or with
some filesystems, etc. Also, of the three utilities that are
suitable for our purposes, lsof is the slowest. So if there
are other utilities that we use during SST, such as 'ss' or
'sockstat', it is reasonable to use them instead of lsof.
This commit changes the prioritization of utilities, it does
not need additional tests (besides the numerous SST tests
already available in the galera suites). If the system still
need to use lsof, this commit adds the '-b' and '-w' options
to it command line - to reduce the likelihood of blocking.
Implement automatic creation of temporary accounts for SST and pass
account credentials to SST script via socket as opposed to environment
variables. Delete the user after the SST script returns,
Respect wsrep_sst_auth set by the adminitrator in case some additional
privilege grants are needed for particular SST method.
mysqldump SST requires significant change to make use of the new
automatic user generation facility. For now just make it compatible
by ignoring automatically generated user and rely only on wsrep_sst_auth
setting on the joiner node to keep backward compatibility.
Adapt mysqldump SST to automatic SST user generation changes:
- disable special treatment for mysqldump SST on donor
- make mysqldump SST script compatible with the new SST script
interface.
Differentiate user privileges for different SST methods:
- grant minimum required privileges for clone and xtrabackup SST
accounts
- grant all privileges to custom SST accounts as it is not known what
is needed.
- disable SST account generation for rsync SST since it is not needed.
MTR tests:
- add MTR tests for clone and xtrabackup SSTs without wsrep_sst_auth,
- add MTR test for testing masking of wsrep_sst_auth.
- don't attmept to restore original wsrep_sst_auth in MTR tests as it
is always masked.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
This commit contains a fix for the code that extracts and parses
the CN (common name, domain name) record from certificates using
the openssl utility. This code is also made common to the rsync
and mariabackup scripts. There is also some systematization of
the use of 'printf' and 'echo' builtins/utilities.
This commit adds support for the --aria-log-dir-path
option on the command line and for the aria-log-dir-path
option in the configuration file to the SST scripts, since
before this change these parameters were completely ignored
during SST - SST scripts assumed that aria logs files are
always located in the same directory as logs for innodb.
Tests for this change will be added as a separate commit,
along with tests for MDEV-30157 and MDEV-28669.
This commit adds even more correct handling of parameters
with paths when they contain leading or trailing spaces and/or
slashes. Also it fixes problems that occur when the user specified
explicit paths to additional directories, but these paths match
the specified path of the data directory - in this case, additional
subdirectories should be treated (in relation to the data directory)
in the same way as if these paths were not specified or as if they
are implicitly specified as "." or "./". But prior to this fix,
existing code treated any values as if they were completely
separate directories, whether or not they actually point to the
same location to which datadir points to - and this sometimes
resulted in incorrect file transfers.
This fix does not contain separate tests, as tests will be
part of the main commit(s). This fix has been made as a separate
commit to facilitate review for major substantive fixes related
to MDEV-30157 and MDEV-28669.
This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.
The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.
This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).
Also fixed broken logic of automatic generation of the Diffie-Hellman parameters
for socat version less than 1.7.3 (which defaults to 512-bit values instead of
2048-bit ones).
This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.
This commit adds validation of the values of the ssl-mode parameter
in SSL scripts, since now only a basic check for the presence of the
"VERIFY_" prefix is performed there to detect "VERIFY_IDENTITY" and
"VERIFY_CA", but all other values are not checked at all.
In addition, this commit removes leading and trailing spaces from
parameter values that SST scripts read from configuration files or
from the command line so that they do not interfere with parameter
checks and substitutions. Parameter substitution has been made more
robust against characters in strings that the shell might erroneously
interpret as regexp.
This commit fixes problems due to bugs and quirks in bsdtar
(the FreeBSD version of tar). Separate tests are not required,
because without these fixes, many other tests fail when tested
in the FreeBSD environment.
Also, the grep patterns for reading utility version numbers
has been made more robust. The notation of some options of
the "cut" utility has been changed.
