Another batch of changes that should make the SST process
more reliable in all scenarios:
1) Added hostname or CN verification when stunnel is used
with certificate chain verification (verifyChain = yes);
2) Added check for the absence of the stunnel utility for
mtr tests;
3) Deletion of working files before and after SST is done
more accurately;
4) rsync on joiner can be run even if the path to its
configuration file contains spaces;
5) More accurate directory creation (for data files and
for logs);
6) IST with mysqldump no longer turns off statement logging;
7) Reset password for mysqldump when password is empty but
username is specified;
8) More reliable quoting when generating statements in
wsrep_sst_mysqldump;
9) Added explicit generation of 2048-bit Diffie-Hellman
parameters for sockat < 1.7.3, by analogy with xtrabackup;
10) Compression parameters for qpress are read from all
suitable server groups in configuration file, as well as
from the [sst] and [xtrabackup] groups;
11) Added a test that checks compression using qpress;
12) Checking for optional utilities is modified to work even
if they implemented as built-in shell commands (unlikely
on real systems, but more reliable).
* Table should have primary key
* Enable wsrep_sync_wait before final selects
* Enable autocommit before final selects.
* Fix joiner monitoring in case of mysqldump.
* Add wait_conditions to stabilize
* Remove usage of wsrep_provider variable in galera_ist_restart_joiner
* Rename galera_load_provider.inc and galera_unload_provider.inc to
galera_stop_replication.inc and galera_start_replication.inc. Their
original names were no longer reflecting what these include files do.
followup for ce3a2a688d
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
The test requires adaptation for MariaDB, which is done
in this patch. In addition, this patch includes a fix for
the SST script startup code that adds escaping for special
characters on the command line (in case they are contained
in the arguments to mysqld). The fix does not require
separate tests, as the required tests are already part
of the mtr suite for Galera.
* Collect and pass apply error data to provider
* Rollback failed transaction and continue operation if provider returns
SUCCESS
* MTR tests for inconsistency voting
Currently, running mtr with an incorrect (for example, new or
obsolete) version of wsrep_provider (for example, with the 26
version of libgalera_smm.so) leads to the failure of tests in
several suites with vague error diagnostics.
As for the galera_3nodes suite, the mtr also does not effectively
check all the prerequisites after merge with MDEV-18426 fixes.
For example, tests that using mariabackup do not check for presence
of ss and socat/nc. This is due to improper handling of relative
paths in mtr scripts.
In addition, some tests in different suites can be run without
setting the environment variables such as MTR_GALERA_TFMT, XBSTREAM,
and so on.
To eliminate all these issues, this patch makes the following changes:
1. Added auxiliary wsrep_mtr_check utility (which located in the
mysql-test/lib/My/SafeProcess subdirectory), which compares the
versions of the wsrep API that used by the server and by the wsrep
provider library, and it does this comparison safely, without
accessing the API if the versions do not match.
2. All checks related to the presence of mariabackup and utilities
that necessary for its operation transferred from the local directories
of different mtr suites (from the suite.pm files) to the main suite.pm
file. This not only reduces the amount of code and eliminates duplication
of identical code fragments, but also avoids problems due to the inability
of mtr to consider relative paths to include files when checking skip
combinations.
3. Setting the values of auxiliary environment variables that
are necessary for Galera, SST scripts and mariabackup (to work
properly) is moved to the main mysql-test-run.pl script, so as
not to duplicate this code in different suites, and to avoid
partial corrections of the same errors for different suites
(while other suites remain uncorrected).
4. Fixed duplication of the have_file_key_management.inc and
have_filekeymanagement.inc files between different suites,
these checks are also transferred to the top level.
5. Added garbd presence check and garbd path variable.
https://jira.mariadb.org/browse/MDEV-18565
Currently, running mtr with an incorrect (for example, new or
obsolete) version of wsrep_provider (for example, with the 26
version of libgalera_smm.so) leads to the failure of tests in
several suites with vague error diagnostics.
As for the galera_3nodes suite, the mtr also does not effectively
check all the prerequisites after merge with MDEV-18426 fixes.
For example, tests that using mariabackup do not check for presence
of ss and socat/nc. This is due to improper handling of relative
paths in mtr scripts.
In addition, some tests in different suites can be run without
setting the environment variables such as MTR_GALERA_TFMT, XBSTREAM,
and so on.
To eliminate all these issues, this patch makes the following changes:
1. Added auxiliary wsrep_mtr_check utility (which located in the
mysql-test/lib/My/SafeProcess subdirectory), which compares the
versions of the wsrep API that used by the server and by the wsrep
provider library, and it does this comparison safely, without
accessing the API if the versions do not match.
2. All checks related to the presence of mariabackup and utilities
that necessary for its operation transferred from the local directories
of different mtr suites (from the suite.pm files) to the main suite.pm
file. This not only reduces the amount of code and eliminates duplication
of identical code fragments, but also avoids problems due to the inability
of mtr to consider relative paths to include files when checking skip
combinations.
3. Setting the values of auxiliary environment variables that
are necessary for Galera, SST scripts and mariabackup (to work
properly) is moved to the main mysql-test-run.pl script, so as
not to duplicate this code in different suites, and to avoid
partial corrections of the same errors for different suites
(while other suites remain uncorrected).
4. Fixed duplication of the have_file_key_management.inc and
have_filekeymanagement.inc files between different suites,
these checks are also transferred to the top level.
5. Added garbd presence check and garbd path variable.
https://jira.mariadb.org/browse/MDEV-18565
Most of the mtr tests in the galera_3nodes suite fail
for a variety of reasons with a variety of errors.
Some tests simply need to add the missing "connection"
lines to the result files, but many of them fail due
to substantial errors that require reworking test files.
This patch adds the missing "connection" lines to
the result files and fixes several substantial flaws
in the galera_3nodes suite tests and in the mtr framework
service files, adapting the tests from galera_3nodes
for the current version of MariaDB.
https://jira.mariadb.org/browse/MDEV-18426
* Increased timeout counter in galera_wait_ready.inc
* Replaced useless include/wait_until_ready.inc after start_mysqld.inc
in galera_st_*.inc with wait_condition on cluster size.
Test galera_sst_mysqldump often fails with error "2013: Lost connection
to MySQL server during query". The connection is lost after the test
restart one of the nodes. This happens because the server closes client
connections if it is joining a cluster through SST method mysqldump.
On unlucky runs of the test it is possible that mysqld is restarted,
and then mtr client is disconnected while it tries to determine if
galera is ready before going on with the test.
This patch rewrites galera_wait_ready.inc so that it is immune to
being disconnected.
This patch fixes a lock wait timeout error on `SELECT @@GLOBAL.WSREP_ON`
in `wait_wsrep_ready.inc`:
```
--connection node_2
...
--source include/kill_galera.inc
--connection node_1
--source include/wait_until_connected_again.inc # This includes wait_wsrep_ready.inc
```
The problem is that on node_2, kill_galera.inc may return before
the node is killed. So node_1 may still see that node_1 is alive
and will attempt to sync wait when doing those `SELECT` statements.
But sync wait is doomed to fail given that node_1 is killed, hence
the lock wait timeout.
One possible fix is to disable wsrep_sync_wait before including
wait_until_connected_again.
However, it appears that including wait_until_connected_again is
not necessary at all in node_1, so this patch removes it altogether.
kill_galera.inc can no longer rely on wait_until_connected_again.inc.
This is because wait_until_connected_again now tries to make sure
that the server it is connected eventually transition to ready
state. Whereas some tests may need to kill galera while the server
is in a non-primary view.
galera SST tests have a debug part, but we don't want to limit them
to fulltest2 builder. So, add support for test files that
have a debug part:
* add maybe_debug.inc and maybe_debug.combinations
* 'debug' combination is run when debug is available
* 'release' combination is run otherwise
* test wraps debug parts in if($with_debug) { ... }
* and creates ,debug.rdiff for debug results
* make galera.galera_sst_xtrabackup* not big
* auto-select between socat and nc, whatever available
* auto-skip xtrabackup tests if no xtrabackup or neither socat nor nc
fix galera.galera_sst_mysqldump test to work:
* must connect to 127.0.0.1, where mysqld is listening
* disable wsrep_sync_wait in wsrep_sst_mysqldump, otherwise
sst can deadlock
* allow 127.0.0.1 for bind_address and wsrep_sst_receive_address.
(it's useful in tests, or when two nodes are on the same box,
or when nodes are on different boxes, but the connection is
tunelled, or whatever. Don't judge user's setup). MDEV-14070
* don't wait for client connections to die when doing
mysqldump sst. they'll die in a due time, and if needed mysql
will wait on locks until they do. MDEV-14069
Also don't mark it big, to make sure it's sufficiently tested
The error log redirection for wsrep_recover run does not work in old version.
For the wsrep_recovery run, error logging is supposed to go into:
mysql-test/suite/galera/include/galera_wsrep_recover.inc
In old version, this works only partially, 4 first lines of error messages after mysql startup
do go into the galera_wsrep_recover.log, but after that the default error log file is enforced
and remaining error logging goes into the default error log file.
In this patch this problem is fixed by passing --log-error option in mysql startup
This fix was tested with galera_gcache_recover test, which is currently in disabled state.
Note that the test does not pass even after this fix, as there are further more issues
in later test phases.
