Commit graph

4641 commits

Author SHA1 Message Date
Julius Goryavsky
d1a4315f4c MDEV-30402: Encrypted mariabackup SST breaks on distributions with newer socat
This commit adds a new 'no-sni' option to socat which is required to
properly authenticate with newer socat versions (after version 1.7.4+).
This option is needed to disable the automatic use of the SNI feature
(Server Name Indication) since the SST script directly specifies the
commonname if necessary and automatic activation of the SNI feature
is unnecessary in such scenarios.
2023-04-13 07:49:35 +02:00
Lorna Luo
acfb5dfd97 MDEV-22683: Ensure system tables are correctly upgraded in MariaDB 10.4
Running mysql_upgrade should end up with the exact same system tables as fresh
installations have after running mysql_install_db. To ensure the upgrade is
correct and complete:

- Remove the redundant modification of thread_id`. On 5.5 version, the
  `general_log` table was created as `CREATE TABLE IF NOT EXISTS general_log
  (..., thread_id INTEGER NOT NULL, ...)`, and starting from 10.0+, the table is
  created as `CREATE TABLE IF NOT EXISTS general_log (..., thread_id BIGINT(21)
  UNSIGNED NOT NULL, ...)`, but mysql_upgrade is not properly upgrading the
  table. It modifies the `thread_id` twice in one query, which could leave the
  table not modified and lead to other potential error when upgrading from
  MariaDB 5.5 or older.

- Update `servers` to ensure `Host` and `User` has correct data type if
  upgrading from 10.1 or older. On versions 10.0 and 10.1, the `servers` table
  was created as `CREATE TABLE IF NOT EXISTS servers (..., Host char(64) NOT
  NULL DEFAULT , ..., Owner char(64) NOT NULL DEFAULT , ...)`, and starting
  from 10.2, the table is created as `CREATE TABLE IF NOT EXISTS servers (...,
  Host varchar(2048) NOT NULL DEFAULT , ..., Owner varchar(512) NOT NULL
  DEFAULT , ...)`.

All new code of the whole pull request, including one or several files that
are either new files or modified ones, are contributed under the BSD-new license.
I am contributing on behalf of my employer Amazon Web Services, Inc.
2023-03-02 09:06:35 +11:00
Ian Gilfillan
801c0b4b47 Update 10.4 HELP tables 2023-01-24 09:52:11 +11:00
Marko Mäkelä
71e8e4934d Merge 10.3 into 10.4 2023-01-13 09:28:25 +02:00
lilinjie
eb145e5ad7 fix typos
Signed-off-by: lilinjie <lilinjie@uniontech.com>
2023-01-12 14:02:20 +11:00
Julius Goryavsky
53c4be7bc0 MDEV-30220: rsync SST completely ignores aria-log-dir-path
This commit adds support for the --aria-log-dir-path
option on the command line and for the aria-log-dir-path
option in the configuration file to the SST scripts, since
before this change these parameters were completely ignored
during SST - SST scripts assumed that aria logs files are
always located in the same directory as logs for innodb.

Tests for this change will be added as a separate commit,
along with tests for MDEV-30157 and MDEV-28669.
2023-01-11 15:01:00 +01:00
Julius Goryavsky
b84f3fa769 MDEV-30157: Galera SST doesn't properly handle undo* files from innodb
This fix adds separate handling for "undo*" files that contain undo
logs as part of innodb files and adds a filter for undo* to the main
filter used when initially transferring files with rsync.
2023-01-11 15:01:00 +01:00
Julius Goryavsky
e4a4aad7cf pre-MDEV-30157 & pre-MDEV-28669: fixes before the main corrections
This commit adds even more correct handling of parameters
with paths when they contain leading or trailing spaces and/or
slashes. Also it fixes problems that occur when the user specified
explicit paths to additional directories, but these paths match
the specified path of the data directory - in this case, additional
subdirectories should be treated (in relation to the data directory)
in the same way as if these paths were not specified or as if they
are implicitly specified as "." or "./". But prior to this fix,
existing code treated any values as if they were completely
separate directories, whether or not they actually point to the
same location to which datadir points to - and this sometimes
resulted in incorrect file transfers.

This fix does not contain separate tests, as tests will be
part of the main commit(s). This fix has been made as a separate
commit to facilitate review for major substantive fixes related
to MDEV-30157 and MDEV-28669.
2023-01-11 15:01:00 +01:00
Sergei Golubchik
fdcfc25127 Merge branch '10.3' into 10.4 2023-01-10 21:04:17 +01:00
Andrew Hutchings
e51a1d6fc0 MDEV-30329: mariadb-service-convert resets systemd service to default User=root
If mariadb-service-convert is run and the user variable is unset then
this sets `User=` in `[Service]`, which then tries to run mariadb as
root, which in-turn fails. This only happens when mysqld_safe is missing
which is all the time now. So don't set `User=` if there is no user variable.

Reviewer: Sergei Golubchik <serg@mariadb.org> (in PR #2382)
2023-01-03 11:45:24 +11:00
Marko Mäkelä
f97f6955bd Merge 10.3 into 10.4 2022-12-14 06:20:04 +02:00
Daniel Black
acfaa04587
MDEV-18591: mysql_install_db - pass --log-error to mysqld in install (#2363)
Previously we parsed it out in mysql_install_db for use in the error
message, but failed to pass it to mysqld in the bootstrap.

Also match log_error as it might appear in the .cnf files.

Thanks Michal Schorm for the test case.

Reviewed by: Faustin Lammler
2022-12-13 15:17:44 +01:00
Marko Mäkelä
fdf43b5c78 Merge 10.3 into 10.4 2022-12-13 11:37:33 +02:00
Christian Gonzalez
112870d7b4 MDEV-16735 Ensure mysql_upgrade works when changing alter_algorithm
MDEV-16735 describes how mysql_upgrade fails when alter_algorithm
is set to a value different than 'DEFAULT'/'COPY'. It was marked as
fixed by 0ee0868, but the fix didn't covered the possibility of having
the global value of alter_algorithm set to something different than
'DEFAULT'/'COPY'. To ensure that the upgrade process works properly
regardless the global value of alter_altorithm, this commit force it's
value to 'DEFAULT' (note the quotes) for the mysql_upgrade session.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2022-11-29 10:55:17 +11:00
Ian Gilfillan
f19e8559aa MDEV-28701 Update 10.4 HELP tables 2022-10-25 09:40:57 +11:00
Ian Gilfillan
34ff5ca895 MDEV-28701 Update 10.3 HELP tables 2022-10-25 09:40:38 +11:00
Ian Gilfillan
ef784c4ea2 Update 10.4 HELP contents 2022-09-19 16:57:10 +10:00
Oleksandr Byelkin
48e35b8cf6 Merge branch '10.3' into 10.4 2022-08-02 14:15:39 +02:00
Julius Goryavsky
7fb1f919d0 MDEV-28758: Mariabackup copies binary logs to backup directory
This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.

The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.
2022-08-01 15:53:14 +02:00
Marko Mäkelä
392ee571c1 Merge 10.3 into 10.4 2022-07-01 13:10:36 +03:00
Marko Mäkelä
045771c050 Fix most clang-15 -Wunused-but-set-variable
Also, refactor trx_i_s_common_fill_table() to remove dead code.

Warnings about yynerrs in Bison-generated yyparse() will remain for now.
2022-07-01 09:48:36 +03:00
Marko Mäkelä
01d757036f Merge 10.3 into 10.4 2022-06-27 10:14:37 +03:00
Julius Goryavsky
3e09c6199d MDEV-26562: galera-sst-mariabackup is failing due to missing xtrabackup_checkpoints
This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).

Also fixed broken logic of automatic generation of the Diffie-Hellman parameters
for socat version less than 1.7.3 (which defaults to 512-bit values instead of
2048-bit ones).
2022-06-21 13:45:20 +02:00
Julius Goryavsky
124326d810 MDEV-28656: Inability to roll upgrade without stopping the Galera cluster 2022-06-14 12:29:14 +02:00
Marko Mäkelä
c89e3b70a7 Merge 10.3 into 10.4 2022-06-09 11:53:46 +03:00
Sergei Golubchik
7c4efab903 typo fixed: [[space]] -> [[:space:]] 2022-06-07 18:26:05 +02:00
Marko Mäkelä
96f4b4a55b Merge 10.3 into 10.4 2022-06-02 16:34:17 +03:00
Julius Goryavsky
7d3d3838c1 MDEV-28583: post-merge fixes 2022-05-23 13:11:14 +02:00
Sergei Golubchik
23ddc3518f Merge branch '10.3' into 10.4 2022-05-18 01:25:30 +02:00
Julius Goryavsky
d388e7eb86 MDEV-28583: Galera: binlogs disappear after rsync IST
This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.
2022-05-18 01:22:29 +02:00
Julius Goryavsky
b081ad8c65 MDEV-28423: Galera IST is failing on Joiner node
This commit fixes an issue with IST handling in
version 10.9 which is a regression after MDEV-26971
and related to trying to get a non-existent "total"
tag on the IST branch (this tag is only defined in
SST mode).
2022-05-18 01:22:29 +02:00
Hartmut Holzgraefe
480323f7d6 MDEV-19161: Let galera_new_cluster use "restart" instead of "start"
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-05-12 07:46:26 +03:00
Julius Goryavsky
fccca49997 MDEV-28377: galera_as_slave_nonprim bind: Address already in use
This commit fixes a crash reported as MDEV-28377 and a number
of other crashes in automated tests with mtr that are related
to broken .cnf files in galera and galera_3nodes suites, which
happened when automatically migrating MDEV-26171 from 10.3 to
subsequent higher versions.
2022-04-26 20:44:20 +02:00
Marko Mäkelä
394784095e Merge 10.3 into 10.4 2022-04-21 11:33:59 +03:00
Julius Goryavsky
42908dc5fb MDEV-26171: wsrep_sst_receive_address does not parse IPv6 address correctly
This commit fixes problems with parsing ipv6 addresses given via
the wsrep_sst_receive_address and wsrep_node_address options.

Also, this commit removes extra lines in the configuration files
in the mtr test suites for Galera related to these parameters.
2022-04-12 17:14:39 +02:00
Julius Goryavsky
7a03128faf MDEV-28205: SST via mariabackup stops on failure while archiving logs
Currenly SST script for mariabackup stops on any failure while archiving
logs, e.g. when unable to create directory, insufficient permissions, gzip
failure, etc. However, in case of such problems, the script should issue
a warning and continue without archiving, but not exit with a fatal error.

This commit adds this fix to the SST script for mariabackup.
2022-04-07 13:26:36 +02:00
Marko Mäkelä
d172df9913 MDEV-25975: Merge 10.3 into 10.4 2022-04-06 09:18:38 +03:00
Marko Mäkelä
d6d66c6e90 Merge 10.3 into 10.4 2022-04-06 08:59:09 +03:00
Marko Mäkelä
e9735a8185 MDEV-25975 innodb_disallow_writes causes shutdown to hang
We will remove the parameter innodb_disallow_writes because it is badly
designed and implemented. The parameter was never allowed at startup.
It was only internally used by Galera snapshot transfer.
If a user executed
SET GLOBAL innodb_disallow_writes=ON;
the server could hang even on subsequent read operations.

During Galera snapshot transfer, we will block writes
to implement an rsync friendly snapshot, as follows:

sst_flush_tables() will acquire a global lock by executing
FLUSH TABLES WITH READ LOCK, which will block any writes
at the high level.

sst_disable_innodb_writes(), invoked via ha_disable_internal_writes(true),
will suspend or disable InnoDB background tasks or threads that could
initiate writes. As part of this, log_make_checkpoint() will be invoked
to ensure that anything in the InnoDB buf_pool.flush_list will be written
to the data files. This has the nice side effect that the Galera joiner
will avoid crash recovery.

The changes to sql/wsrep.cc and to the tests are based on a prototype
that was developed by Jan Lindström.

Reviewed by: Jan Lindström
2022-04-06 08:06:49 +03:00
Monty
c4ebb2bd04 Fixed that mysql_upgrade doesn't give errors about mariadb.sys
The reason for this fix was that when I tried to run mysql_upgrade
at home to update an old 10.5 installation, mysql_upgrade failed
with warnings about mariadb.sys user not existing.

If the server was started with --skip-grants, there would be no warnings
from mysql_upgrade, but in some cases running mysql_upgrade again could
produce new warnings.

The reason for the warnings was that any access of the mysql.user view
will produce a warning if the mariadb.sys user does not exists.

Fixed with the following changes:
- Disable warnings about mariadb.sys user not existing
- Don't overwrite old mariadb.sys entries in tables_priv and global_priv
- Ensure that tables_priv has an entry for mariadb.sys if the user exists.
  This fixes an issue that tables_priv would not be updated if there
  was a failure directly after global_priv was updated.
2022-04-04 16:14:22 +03:00
Julius Goryavsky
daed558b2c MDEV-28204: The tr utility does not work as expected on rsync SST
This commit contains a fix to use modern syntax for selecting
character classes in the tr utility options.

Also one of the tests for SST via rsync (galera_sst_rysnc2) is made
more reliable (to avoid rare failures during automatic testing).
2022-04-04 11:46:06 +02:00
Marko Mäkelä
ae6e214fd8 Merge 10.3 into 10.4 2022-03-29 11:13:18 +03:00
Marko Mäkelä
020e7d89eb Merge 10.2 into 10.3 2022-03-29 09:53:15 +03:00
Rucha Deodhar
549a71e74b MDEV-21873: 10.2 to 10.3 upgrade doesn't remove semi-sync reference
from mysql.plugin table
Fix: Since mysql_upgrade runs commands from mysql_system_tables.fix,
added sql commands to check for semisync plugins in
INFORMATION_SCHEMA.PLUGINS and if they aren't there then delete them
from mysql.plugin.
2022-03-25 18:34:58 +05:30
sjaakola
9b2fa2ae8e MDEV-24845 Oddities around innodb_fatal_semaphore_wait_threshold and global.innodb_disallow_writes
This commit adds a mtr test for reproducing a test scenario where despite of
innodb_disallow_writes blocking, writes to file system can still happen.

The test launches a garbd node, which triggers one of the cluster node to switch to
SST donor state. In this state, all disk activity should be halted, and e.g.
innodb_disallow_writes has been set. The test records md5sum aggregate over mariadb
data directory when the node enters the donor state, and records another md5sum
when the node leaves the donor state. If there is no IO activity in data directory, these
hashes should be equal.

For this test, the Donor state processing, has beeen instrumented so that, SST donor thread can be
stopped when entering the donor state. The test uses this new dbug sync point,
to control when to record the md5sums.

New SST script was added: wsrep_sst_backup, and garbd uses backup method to lauch the donor
node to call this script, and to enter in donor state.

The backup script could be later extended as general purpose backup method for the cluster.

This commit fixes also one race condition happening in wsrep_sst_rsync, like this:
* wsrep_rsync_sst script requests for flush tables,
  and then waits in a loop until mariadbd has created file tables_flushed,
  as confirmation that FLUSH TABLES has completed
* mariadbd's SST donor thread, wakes for the flush table request and then performs FTWRL,
  and after this it creates the tables_flushed file
* note that SST script will now continue to startup rsync sending
* mariadbd's SST donor thread now calls for sst_disallow_writes(),
  so that innodb would setup disk IO blockage, however rsyncing may already be ongoing at this point

This race condition is fixed in this commit, by performing all disk IO blocking before
creating the tables_flushed file.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-03-25 10:04:15 +02:00
Sergei Golubchik
421153848f MDEV-27980 file-key-management plugin disabled in mysql_install_db breaks automated deployments (and container initialization)
Revert "Silence the file-key-management plugin during mysql_install_db"

This reverts commit e99d3da638.
2022-03-20 21:38:48 +01:00
Marko Mäkelä
7b97020d40 Merge 10.3 into 10.4 2022-03-07 09:05:36 +02:00
Marko Mäkelä
02da00a98c Merge 10.2 into 10.3 2022-03-04 14:29:36 +02:00
Monty
3fd79a04b6 MMDEV-27823 mariadb-install-db --group fails
Fixed by not sending --group option to the server (for now)

Reviwer: Sergei Golubchik
2022-03-01 12:54:47 +02:00
Marko Mäkelä
f5ff7d09c7 Merge 10.3 into 10.4 2022-02-25 13:00:48 +02:00