If you are running mariadb-install-db from a source tree instead of
installation it was executing `mysqld` instead of `mariadb` which showed
the deprecation warning. This patch fixes that as well as fixing
messages and links to other things that have been renamed.
- Remove DBUG calls from my_winfile.c where call and parameters
are already printed by mysys.
- Remove DBUG from my_get_osfhandle() and my_get_open_flags() to remove
DBUG noise.
- Updated convert-debug-for-diff to take into account windows.
- Changed some DBUG_RETURN(function()) to tmp=function(); DBUG_RETURN(tmp);
This is needed as Visual C++ prints for DBUG binaries a trace for
func_a()
{
DBUG_ENTER("func_a");
DBUG_RETURN(func_b())
}
as
>func_a
<func_a
>func_b
<func_b
instead of when using gcc:
>func_a
| >func_b
| <func_b
<func_a
Running mysql_upgrade should end up with the exact same system tables as fresh
installations have after running mysql_install_db. To ensure the upgrade is
correct and complete:
- Remove the redundant modification of thread_id`. On 5.5 version, the
`general_log` table was created as `CREATE TABLE IF NOT EXISTS general_log
(..., thread_id INTEGER NOT NULL, ...)`, and starting from 10.0+, the table is
created as `CREATE TABLE IF NOT EXISTS general_log (..., thread_id BIGINT(21)
UNSIGNED NOT NULL, ...)`, but mysql_upgrade is not properly upgrading the
table. It modifies the `thread_id` twice in one query, which could leave the
table not modified and lead to other potential error when upgrading from
MariaDB 5.5 or older.
- Update `servers` to ensure `Host` and `User` has correct data type if
upgrading from 10.1 or older. On versions 10.0 and 10.1, the `servers` table
was created as `CREATE TABLE IF NOT EXISTS servers (..., Host char(64) NOT
NULL DEFAULT , ..., Owner char(64) NOT NULL DEFAULT , ...)`, and starting
from 10.2, the table is created as `CREATE TABLE IF NOT EXISTS servers (...,
Host varchar(2048) NOT NULL DEFAULT , ..., Owner varchar(512) NOT NULL
DEFAULT , ...)`.
All new code of the whole pull request, including one or several files that
are either new files or modified ones, are contributed under the BSD-new license.
I am contributing on behalf of my employer Amazon Web Services, Inc.
For both Deb and RPM, create mariadb-client-compat and
mariadb-server-compat containing the mysql links to the mariadb
named executables/scripts.
The mariadb-client-core mysqlcheck was moved to mariadb-client-compat.
The symlinks in MYSQL_ADD_EXECUTABLE is tagged as a
{Client,Server}Symlinks component and placed in
the symlinks packages.
Man pages are restructured be installed into compat package
if that matches the executable.
Columnstore has a workaround as it doesn't use the cmake/plugin.cmake.
Scripts likewise have compatibility symlinks are in
the {server,client}-compat packages.
Co-author: Andrew Hutchings <andrew@linuxjedi.co.uk>
Closes#2390
don't include my_progname in the error message, my_error starts from it
automatically, resulting in, like
/usr/bin/mysqladmin: Notice: /usr/bin/mysqladmin is deprecated and will be removed in a future release, use command 'mariadb-admin'
and remove "Notice" so that the problem description would directly
follow the executable name.
make the check to work when the executable is in the PATH
(so, invoked simply like 'mysql' and thus readlink cannot find it)
fix the check in mysql_install_db and mysql_secure_installation to not
print the warning if the intermediate path contains "mysql" substring
add this message also to
* mysql_waitpid
* mysql_convert_table_format
* mysql_find_rows
* mysql_setpermissions
* mysqlaccess
* mysqld_multi
* mysqld_safe
* mysqldumpslow
* mysqlhotcopy
* mysql_ldb
Closes#2273
Eventually mysql symlinks will go away, as MariaDB and MySQL keep
diverging and we do not want to make it impossible to install
MariaDB and MySQL side-by-side when users want it.
It also useful if people start using MariaDB tools with MariaDB.
If the exe doesn't begine with "mariadb" or is a symlink,
print a warning to use the resolved name.
In my_readlink, add check on my_thread_var as its used by comp_err
and other build utils that also use my_init.
Renames the upgrade state file, and ensures the old
file is properly removed when `mariadb-upgrade` tool is executed.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
try to make them less confusing for users.
Hopefully, if the version string will be changed like
- mariadb Ver 15.1 Distrib 10.11.2-MariaDB for Linux (x86_64)
+ mariadb from 10.11.2-MariaDB, client 15.1 for Linux (x86_64)
users will be less inclined to reply "15.1" to the question
"what mariadb version are you using?"
This commit adds support for the --aria-log-dir-path
option on the command line and for the aria-log-dir-path
option in the configuration file to the SST scripts, since
before this change these parameters were completely ignored
during SST - SST scripts assumed that aria logs files are
always located in the same directory as logs for innodb.
Tests for this change will be added as a separate commit,
along with tests for MDEV-30157 and MDEV-28669.
This fix adds separate handling for "undo*" files that contain undo
logs as part of innodb files and adds a filter for undo* to the main
filter used when initially transferring files with rsync.
This commit adds even more correct handling of parameters
with paths when they contain leading or trailing spaces and/or
slashes. Also it fixes problems that occur when the user specified
explicit paths to additional directories, but these paths match
the specified path of the data directory - in this case, additional
subdirectories should be treated (in relation to the data directory)
in the same way as if these paths were not specified or as if they
are implicitly specified as "." or "./". But prior to this fix,
existing code treated any values as if they were completely
separate directories, whether or not they actually point to the
same location to which datadir points to - and this sometimes
resulted in incorrect file transfers.
This fix does not contain separate tests, as tests will be
part of the main commit(s). This fix has been made as a separate
commit to facilitate review for major substantive fixes related
to MDEV-30157 and MDEV-28669.
The MDEV-25004 test innodb_fts.versioning is omitted because ever since
commit 685d958e38 InnoDB would not allow
writes to a database where the redo log file ib_logfile0 is missing.
If mariadb-service-convert is run and the user variable is unset then
this sets `User=` in `[Service]`, which then tries to run mariadb as
root, which in-turn fails. This only happens when mysqld_safe is missing
which is all the time now. So don't set `User=` if there is no user variable.
Reviewer: Sergei Golubchik <serg@mariadb.org> (in PR #2382)
* mariadb-service-convert to use mariadbd-safe
* galera_recovery to use mariadbd
* mtr - wsrep use mariadb executables
* debian/mariadb-server.mariadb.init use mariadbd-safe
* debian/tests/smoke uses mariadb instead of mysql as client.
Co-Author: Daniel Black <daniel@mariadb.org>
Previously we parsed it out in mysql_install_db for use in the error
message, but failed to pass it to mysqld in the bootstrap.
Also match log_error as it might appear in the .cnf files.
Thanks Michal Schorm for the test case.
Reviewed by: Faustin Lammler
MDEV-16735 describes how mysql_upgrade fails when alter_algorithm
is set to a value different than 'DEFAULT'/'COPY'. It was marked as
fixed by 0ee0868, but the fix didn't covered the possibility of having
the global value of alter_algorithm set to something different than
'DEFAULT'/'COPY'. To ensure that the upgrade process works properly
regardless the global value of alter_altorithm, this commit force it's
value to 'DEFAULT' (note the quotes) for the mysql_upgrade session.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
The mysqld_multi script template used @datadir@ as default log
destination, this is not the MariaDB datadir in this context though
but rather the -- typically write-only -- /share dir.
The correct placeholder to use here is @localstatedir@ which gets
replaced with the actual MariaDB datadir
Co-authored-by: Hartmut Holzgraefe <hartmut@php.net>
Increasing the data type from char(32) to char(64) for
the following system columns:
- mysql.event.collation_connection
- mysql.event.db_collation
- mysql.proc.collation_connection
- mysql.proc.db_collation
This change was forgotten during MDEV-27009.
Also fixing the comment "Start/End of 10.9 tests" in ctype_utf8_uca.test
to "Start/End of 10.10 tests", as MDEV-27009 was actually
released in 10.10.
The mysqld_safe script was using bad grep options.
The line that was fixed was likely meant to be 2 separate grep commands,
piped into each other, with each one removing any lines that matched.
The `-E` option was unneeded, as the command is not using regex.
This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.
The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.
The incorrect type of mysql.column_stats caused the server during the
upgrade of every other table to complain:
[ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9
and expected column 'histogram' at position 10 to have type longblob.
To prevent these verbose server errors, we upgrade the
mysql.column_stats table first.
Consequently limit "Incorrect definition of table mysql.*" to the appropriate
set of limited test cases.
The rpl_gtid_errorhandling.result changes the GTID number by one
because of the added early suppression (adding a table row).
Reviewer: Vicențiu Ciorbaru
FixesMariaDB/mariadb-docker#438
This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).
Also fixed broken logic of automatic generation of the Diffie-Hellman parameters
for socat version less than 1.7.3 (which defaults to 512-bit values instead of
2048-bit ones).
This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.
This commit fixes an issue with IST handling in
version 10.9 which is a regression after MDEV-26971
and related to trying to get a non-existent "total"
tag on the IST branch (this tag is only defined in
SST mode).
- This commit rely on MDEV-28391
- When temporary table shadows the base table, error is raised (it can be changed if needed),
since the procedure is relying on creating the views and view cannot be created from the temporary table.
- Reviewed by: <wlad@mariadb.com>
- When arguments to the procedure contain quote in the name, procedure fails with parsing error.
The reason was because additional quoting is done when testing TEMPORARY table with the same name.
- Reviewed by: <wlad@mariadb.com>
- The problem:
==============
- Commit f7216fa63d created the check function for
default temporary storage engine and in case the SE doesn't support temporary tables
the error `ER_ILLEGAL_HA_CREATE_OPTION` is raised.
Before that commit in such cases temporary tables were created by silently substituting
default SE (RocksDB, Connect, PerfSchema) with MyISAM.
- The test `pr_diagnostics.test` was modified in that commit with raising the error,
since I didn't check the root cause of test itself.
- The solution:
===============
- This commit update the root case: procedure `ps_setup_save()` that uses temporary
tables created from performance schema tables definition using `LIKE`, what is not supported.
The suggested fix is to use InnoDB table by using `AS SELECT`.
- Note that test `pr_diagnostics` will raise this error for `medium/full` third argument,
but not for `current` value of third argument.
- Additionally this patch updates the test case of commit f7216fa, by adding missing relation
between temporary tables and Performance schema in `perfschema.misc` test.
- Reviewed by: <wlad@mariadb.com>
This commit fixes a crash reported as MDEV-28377 and a number
of other crashes in automated tests with mtr that are related
to broken .cnf files in galera and galera_3nodes suites, which
happened when automatically migrating MDEV-26171 from 10.3 to
subsequent higher versions.
This commit fixes problems with parsing ipv6 addresses given via
the wsrep_sst_receive_address and wsrep_node_address options.
Also, this commit removes extra lines in the configuration files
in the mtr test suites for Galera related to these parameters.
Currenly SST script for mariabackup stops on any failure while archiving
logs, e.g. when unable to create directory, insufficient permissions, gzip
failure, etc. However, in case of such problems, the script should issue
a warning and continue without archiving, but not exit with a fatal error.
This commit adds this fix to the SST script for mariabackup.
We will remove the parameter innodb_disallow_writes because it is badly
designed and implemented. The parameter was never allowed at startup.
It was only internally used by Galera snapshot transfer.
If a user executed
SET GLOBAL innodb_disallow_writes=ON;
the server could hang even on subsequent read operations.
During Galera snapshot transfer, we will block writes
to implement an rsync friendly snapshot, as follows:
sst_flush_tables() will acquire a global lock by executing
FLUSH TABLES WITH READ LOCK, which will block any writes
at the high level.
sst_disable_innodb_writes(), invoked via ha_disable_internal_writes(true),
will suspend or disable InnoDB background tasks or threads that could
initiate writes. As part of this, log_make_checkpoint() will be invoked
to ensure that anything in the InnoDB buf_pool.flush_list will be written
to the data files. This has the nice side effect that the Galera joiner
will avoid crash recovery.
The changes to sql/wsrep.cc and to the tests are based on a prototype
that was developed by Jan Lindström.
Reviewed by: Jan Lindström
The reason for this fix was that when I tried to run mysql_upgrade
at home to update an old 10.5 installation, mysql_upgrade failed
with warnings about mariadb.sys user not existing.
If the server was started with --skip-grants, there would be no warnings
from mysql_upgrade, but in some cases running mysql_upgrade again could
produce new warnings.
The reason for the warnings was that any access of the mysql.user view
will produce a warning if the mariadb.sys user does not exists.
Fixed with the following changes:
- Disable warnings about mariadb.sys user not existing
- Don't overwrite old mariadb.sys entries in tables_priv and global_priv
- Ensure that tables_priv has an entry for mariadb.sys if the user exists.
This fixes an issue that tables_priv would not be updated if there
was a failure directly after global_priv was updated.
This commit contains a fix to use modern syntax for selecting
character classes in the tr utility options.
Also one of the tests for SST via rsync (galera_sst_rysnc2) is made
more reliable (to avoid rare failures during automatic testing).
from mysql.plugin table
Fix: Since mysql_upgrade runs commands from mysql_system_tables.fix,
added sql commands to check for semisync plugins in
INFORMATION_SCHEMA.PLUGINS and if they aren't there then delete them
from mysql.plugin.
This commit adds a mtr test for reproducing a test scenario where despite of
innodb_disallow_writes blocking, writes to file system can still happen.
The test launches a garbd node, which triggers one of the cluster node to switch to
SST donor state. In this state, all disk activity should be halted, and e.g.
innodb_disallow_writes has been set. The test records md5sum aggregate over mariadb
data directory when the node enters the donor state, and records another md5sum
when the node leaves the donor state. If there is no IO activity in data directory, these
hashes should be equal.
For this test, the Donor state processing, has beeen instrumented so that, SST donor thread can be
stopped when entering the donor state. The test uses this new dbug sync point,
to control when to record the md5sums.
New SST script was added: wsrep_sst_backup, and garbd uses backup method to lauch the donor
node to call this script, and to enter in donor state.
The backup script could be later extended as general purpose backup method for the cluster.
This commit fixes also one race condition happening in wsrep_sst_rsync, like this:
* wsrep_rsync_sst script requests for flush tables,
and then waits in a loop until mariadbd has created file tables_flushed,
as confirmation that FLUSH TABLES has completed
* mariadbd's SST donor thread, wakes for the flush table request and then performs FTWRL,
and after this it creates the tables_flushed file
* note that SST script will now continue to startup rsync sending
* mariadbd's SST donor thread now calls for sst_disallow_writes(),
so that innodb would setup disk IO blockage, however rsyncing may already be ongoing at this point
This race condition is fixed in this commit, by performing all disk IO blocking before
creating the tables_flushed file.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
This commit adds validation of the values of the ssl-mode parameter
in SSL scripts, since now only a basic check for the presence of the
"VERIFY_" prefix is performed there to detect "VERIFY_IDENTITY" and
"VERIFY_CA", but all other values are not checked at all.
In addition, this commit removes leading and trailing spaces from
parameter values that SST scripts read from configuration files or
from the command line so that they do not interfere with parameter
checks and substitutions. Parameter substitution has been made more
robust against characters in strings that the shell might erroneously
interpret as regexp.
This commit fixes problems due to bugs and quirks in bsdtar
(the FreeBSD version of tar). Separate tests are not required,
because without these fixes, many other tests fail when tested
in the FreeBSD environment.
Also, the grep patterns for reading utility version numbers
has been made more robust. The notation of some options of
the "cut" utility has been changed.
This commit adds correct handling of binlogs for SST using rsync
or mariabackup. Before this fix, binlogs were handled incorrectly -
- only one (last) binary log file was transferred during SST, which
then led to various failures (for example, when trying to list all
events from the binary log). These bugs were long masked by flaws
in the primitive binlogs handling code in the SST scripts, which
causing binary logs files to be erased after transfer or not added
to the binlog index on the joiner node. Now the correct transfer
of all binary logs (not just the last of the binary log files) has
been implemented both for the rsync (at the script level) and for
the mariabackup (at the level of the main utility code).
This commit also adds a new sst_max_binlogs=<n> parameter, which
can be located in the [sst] section or in the [xtrabackup] section
(historically, supported for mariabackup only, not for rsync), or
in one of the server sections. This parameter specifies the number
of binary log files to be sent to the joiner node during SST. This
option is added for compatibility with old SST scripting behavior,
which can be emulated by setting the sst_max_binlogs=1 (although
in general this can cause problems for the reasons described above).
In addition, setting the sst_max_binlogs=0 can be used to suppress
the transmission of binary logs to the joiner nodes during SST
(although sometimes a single file with the current binary log can
still be transmitted to the joiner, even with sst_max_binlogs=0,
because this sometimes necessary in modes that involve the use of
GTIDs with Galera).
Also, this commit ensures correct handling of paths to various
innodb files and directories in the SST scripts, and fixes some
problems with this that existed in mariabackup utility (which
were associated with incorrect handling of the innodb_data_dir
parameter in some scenarios).
In addition, this commit contains the following enhancements:
1) Added tests for mtr, which check the correct work with binlogs
after SST (using rsync and mariabackup);
2) Added correct handling of slashes at the end of all paths that
the SST script receives as parameters;
3) Improved parsing code for --mysqld-args parameters. Now it
correctly processes the sequence "--" after the name of the
one-letter option;
4) Checking the secret signature during joiner authentication
is made independent of presence of bash (as a unix shell)
in the system and diff utility no longer needed to check
certificates compliance;
5) All directories that are necessary for the correct placement
of various logs are automatically created by SST scripts in
advance (before running mariabackup on the joiner node);
6) Removal of old binary logs on joiner is done using the binlog
index (if it exists) (not only by fixed pattern that based
on the current binlog name, as before);
7) Paths for placing binary logs are correctly processed if they
are set as relative paths (to the datadir);
8) SST scripts are made even more resistant to spaces in filenames
(now for binlogs);
9) In case of failure, SST scripts now always end with an exit
code other than zero;
10) SST script for rsync now correctly create a tar file with
the binlogs, even if the paths to them (in the binlog index
file) are specified as a mix of absolute and relative paths,
and even if they do not match with the datadir path specified
in the current configuration settings.
- Changed SST scripts to use /usr/bin/env bash instead of
/bin/bash for better portability.
- Fixed use of mktemp on non-Linux platforms to produce
temporary file instead of directory.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
This avoids printing the error
"mysqld: file-key-management-filename is not set"
which can happen if the file-key-management pluging is statically compiled
Removed the option as it safe to always create the file when we have
created the MariaDB data directories. This fixes this issue not only
for debian but for all MariaDB users.
For compatibility this is under an extra option --upgrade-info
The goal here is to install a data directory with the required
info to let mysql_upgrade know that an upgrade isn't required.
The InnoDB redo log used to be formatted in blocks of 512 bytes.
The log blocks were encrypted and the checksum was calculated while
holding log_sys.mutex, creating a serious scalability bottleneck.
We remove the fixed-size redo log block structure altogether and
essentially turn every mini-transaction into a log block of its own.
This allows encryption and checksum calculations to be performed
on local mtr_t::m_log buffers, before acquiring log_sys.mutex.
The mutex only protects a memcpy() of the data to the shared
log_sys.buf, as well as the padding of the log, in case the
to-be-written part of the log would not end in a block boundary of
the underlying storage. For now, the "padding" consists of writing
a single NUL byte, to allow recovery and mariadb-backup to detect
the end of the circular log faster.
Like the previous implementation, we will overwrite the last log block
over and over again, until it has been completely filled. It would be
possible to write only up to the last completed block (if no more
recent write was requested), or to write dummy FILE_CHECKPOINT records
to fill the incomplete block, by invoking the currently disabled
function log_pad(). This would require adjustments to some logic around
log checkpoints, page flushing, and shutdown.
An upgrade after a crash of any previous version is not supported.
Logically empty log files from a previous version will be upgraded.
An attempt to start up InnoDB without a valid ib_logfile0 will be
refused. Previously, the redo log used to be created automatically
if it was missing. Only with with innodb_force_recovery=6, it is
possible to start InnoDB in read-only mode even if the log file
does not exist. This allows the contents of a possibly corrupted
database to be dumped.
Because a prepared backup from an earlier version of mariadb-backup
will create a 0-sized log file, we will allow an upgrade from such
log files, provided that the FIL_PAGE_FILE_FLUSH_LSN in the system
tablespace looks valid.
The 512-byte log checkpoint blocks at 0x200 and 0x600 will be replaced
with 64-byte log checkpoint blocks at 0x1000 and 0x2000.
The start of log records will move from 0x800 to 0x3000. This allows us
to use 4096-byte aligned blocks for all I/O in a future revision.
We extend the MDEV-12353 redo log record format as follows.
(1) Empty mini-transactions or extra NUL bytes will not be allowed.
(2) The end-of-minitransaction marker (a NUL byte) will be replaced
with a 1-bit sequence number, which will be toggled each time when the
circular log file wraps back to the beginning.
(3) After the sequence bit, a CRC-32C checksum of all data
(excluding the sequence bit) will written.
(4) If the log is encrypted, 8 bytes will be written before
the checksum and included in it. This is part of the
initialization vector (IV) of encrypted log data.
(5) File names, page numbers, and checkpoint information will not be
encrypted. Only the payload bytes of page-level log will be encrypted.
The tablespace ID and page number will form part of the IV.
(6) For padding, arbitrary-length FILE_CHECKPOINT records may be written,
with all-zero payload, and with the normal end marker and checksum.
The minimum size is 7 bytes, or 7+8 with innodb_encrypt_log=ON.
In mariadb-backup and in Galera snapshot transfer (SST) scripts, we will
no longer remove ib_logfile0 or create an empty ib_logfile0. Server startup
will require a valid log file. When resizing the log, we will create
a logically empty ib_logfile101 at the current LSN and use an atomic rename
to replace ib_logfile0 with it. See the test innodb.log_file_size.
Because there is no mandatory padding in the log file, we are able
to create a dummy log file as of an arbitrary log sequence number.
See the test mariabackup.huge_lsn.
The parameter innodb_log_write_ahead_size and the
INFORMATION_SCHEMA.INNODB_METRICS counter log_padded will be removed.
The minimum value of innodb_log_buffer_size will be increased to 2MiB
(because log_sys.buf will replace recv_sys.buf) and the increment
adjusted to 4096 bytes (the maximum log block size).
The following INFORMATION_SCHEMA.INNODB_METRICS counters will be removed:
os_log_fsyncs
os_log_pending_fsyncs
log_pending_log_flushes
log_pending_checkpoint_writes
The following status variables will be removed:
Innodb_os_log_fsyncs (this is included in Innodb_data_fsyncs)
Innodb_os_log_pending_fsyncs (this was limited to at most 1 by design)
log_sys.get_block_size(): Return the physical block size of the log file.
This is only implemented on Linux and Microsoft Windows for now, and for
the power-of-2 block sizes between 64 and 4096 bytes (the minimum and
maximum size of a checkpoint block). If the block size is anything else,
the traditional 512-byte size will be used via normal file system
buffering.
If the file system buffers can be bypassed, a message like the following
will be issued:
InnoDB: File system buffers for log disabled (block size=512 bytes)
InnoDB: File system buffers for log disabled (block size=4096 bytes)
This has been tested on Linux and Microsoft Windows with both sizes.
On Linux, only enable O_DIRECT on the log for innodb_flush_method=O_DSYNC.
Tests in 3 different environments where the log is stored in a device
with a physical block size of 512 bytes are yielding better throughput
without O_DIRECT. This could be due to the fact that in the event the
last log block is being overwritten (if multiple transactions would
become durable at the same time, and each of will write a small
number of bytes to the last log block), it should be faster to re-copy
data from log_sys.buf or log_sys.flush_buf to the kernel buffer,
to be finally written at fdatasync() time.
The parameter innodb_flush_method=O_DSYNC will imply O_DIRECT for
data files. This option will enable O_DIRECT on the log file on Linux.
It may be unsafe to use when the storage device does not support
FUA (Force Unit Access) mode.
When the server is compiled WITH_PMEM=ON, we will use memory-mapped
I/O for the log file if the log resides on a "mount -o dax" device.
We will identify PMEM in a start-up message:
InnoDB: log sequence number 0 (memory-mapped); transaction id 3
On Linux, we will also invoke mmap() on any ib_logfile0 that resides
in /dev/shm, effectively treating the log file as persistent memory.
This should speed up "./mtr --mem" and increase the test coverage of
PMEM on non-PMEM hardware. It also allows users to estimate how much
the performance would be improved by installing persistent memory.
On other tmpfs file systems such as /run, we will not use mmap().
mariadb-backup: Eliminated several variables. We will refer
directly to recv_sys and log_sys.
backup_wait_for_lsn(): Detect non-progress of
xtrabackup_copy_logfile(). In this new log format with
arbitrary-sized blocks, we can only detect log file overrun
indirectly, by observing that the scanned log sequence number
is not advancing.
xtrabackup_copy_logfile(): On PMEM, do not modify the sequence bit,
because we are not allowed to modify the server's log file, and our
memory mapping is read-only.
trx_flush_log_if_needed_low(): Do not use the callback on pmem.
Using neither flush_lock nor write_lock around PMEM writes seems
to yield the best performance. The pmem_persist() calls may
still be somewhat slower than the pwrite() and fdatasync() based
interface (PMEM mounted without -o dax).
recv_sys_t::buf: Remove. We will use log_sys.buf for parsing.
recv_sys_t::MTR_SIZE_MAX: Replaces RECV_SCAN_SIZE.
recv_sys_t::file_checkpoint: Renamed from mlog_checkpoint_lsn.
recv_sys_t, log_sys_t: Removed many data members.
recv_sys.lsn: Renamed from recv_sys.recovered_lsn.
recv_sys.offset: Renamed from recv_sys.recovered_offset.
log_sys.buf_size: Replaces srv_log_buffer_size.
recv_buf: A smart pointer that wraps log_sys.buf[recv_sys.offset]
when the buffer is being allocated from the memory heap.
recv_ring: A smart pointer that wraps a circular log_sys.buf[] that is
backed by ib_logfile0. The pointer will wrap from recv_sys.len
(log_sys.file_size) to log_sys.START_OFFSET. For the record that
wraps around, we may copy file name or record payload data to
the auxiliary buffer decrypt_buf in order to have a contiguous
block of memory. The maximum size of a record is less than
innodb_page_size bytes.
recv_sys_t::parse(): Take the smart pointer as a template parameter.
Do not temporarily add a trailing NUL byte to FILE_ records, because
we are not supposed to modify the memory-mapped log file. (It is
attached in read-write mode already during recovery.)
recv_sys_t::parse_mtr(): Wrapper for recv_sys_t::parse().
recv_sys_t::parse_pmem(): Like parse_mtr(), but if PREMATURE_EOF would be
returned on PMEM, use recv_ring to wrap around the buffer to the start.
mtr_t::finish_write(), log_close(): Do not enforce log_sys.max_buf_free
on PMEM, because it has no meaning on the mmap-based log.
log_sys.write_to_buf: Count writes to log_sys.buf. Replaces
srv_stats.log_write_requests and export_vars.innodb_log_write_requests.
Protected by log_sys.mutex. Updated consistently in log_close().
Previously, mtr_t::commit() conditionally updated the count,
which was inconsistent.
log_sys.write_to_log: Count swaps of log_sys.buf and log_sys.flush_buf,
for writing to log_sys.log (the ib_logfile0). Replaces
srv_stats.log_writes and export_vars.innodb_log_writes.
Protected by log_sys.mutex.
log_sys.waits: Count waits in append_prepare(). Replaces
srv_stats.log_waits and export_vars.innodb_log_waits.
recv_recover_page(): Do not unnecessarily acquire
log_sys.flush_order_mutex. We are inserting the blocks in arbitary
order anyway, to be adjusted in recv_sys.apply(true).
We will change the definition of flush_lock and write_lock to
avoid potential false sharing. Depending on sizeof(log_sys) and
CPU_LEVEL1_DCACHE_LINESIZE, the flush_lock and write_lock could
share a cache line with each other or with the last data members
of log_sys.
Thanks to Matthias Leich for providing https://rr-project.org traces
for various failures during the development, and to
Thirunarayanan Balathandayuthapani for his help in debugging
some of the recovery code. And thanks to the developers of the
rr debugger for a tool without which extensive changes to InnoDB
would be very challenging to get right.
Thanks to Vladislav Vaintroub for useful feedback and
to him, Axel Schwenke and Krunal Bauskar for testing the performance.
The mysql_install_db script is used to initialize the data directory.
However, if the user needs to apply some customized init commands (for
example to change user or password) they have to start the database
with `--skip-grant-tables` after the install script, and then restart
the database with normal mode.
To make it easier to include customization in the mysql_install_db
script, in this commit, a new parameter `extra-file` is added in
mysql_install_db script to support some extra customized init commands.
With this option we can support applying extra file containing sql for
mysql_install_db command line at runtime, which reduces the complexity
of customized initialization process.
This script is only used for install and is not included in the mtr
bootstrap file.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
This is the first part of the fixes for MDEV-24097. This commit
contains the fixes for instability when testing Galera and when
restarting nodes quickly:
1) Protection against a "stuck" old SST process during the execution
of the new SST (after restarting the node) is now implemented for
mariabackup / xtrabackup, which should help to avoid almost all
conflicts due to the use of the same ports - both during testing
with mtr, so and when restarting nodes quickly in a production
environment.
2) Added more protection to scripts against unexpected return of
the rc != 0 (in the commands for deleting temporary files, etc).
3) Added protection against unexpected crashes during binlog transfer
(in SST scripts for rsync).
4) Spaces and some special characters in binlog filenames shouldn't
be a problem now (at the script level).
5) Daemon process termination tracking has been made more robust
against crashes due to unexpected termination of the previous SST
process while new scripts are running.
6) Reading ssl encryption parameters has been moved from specific
SST scripts to a common wsrep_sst_common.sh script, which allows
unified error handling, unified diagnostics and simplifies script
revisions in the future.
7) Improved diagnostics of errors related to the use of openssl.
8) Corrections have been made for xtrabackup-v2 (both in tests and in
the script code) that restore the work of xtrabackup with updated
versions of innodb.
9) Fixed some tests for galera_3nodes, although the complete solution
for the problem of starting three nodes at the same time on fast
machines will be done in a separate commit.
No additional tests are required as this commit fixes problems with
existing tests.
1) Removed symlinks that are not very well supported in tar under Windows.
2) Added comment + changed code formatting in viosslfactories.c
3) Fixed a small bug in the yassl code.
4) Fixed a typo in the script code.
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
From 10.4.13, the `mariadb.sys` user was created to replace `root` definers.
- In commit 0253ea7f22, definer of
Add/DropGeometryColumn procedures was changed to `mariadb.sys`, in
`scripts/maria_add_gis_sp.sql.in`.
However, maria_add_gis_sp.sql only applies to new databases created by
installation script. Databases upgraded from old versions will miss this
change.
- In addition, according to commit
0d6d801e5886208b2632247d88da106a543e1032(MDEV-23102), in some scenarios
when root user is replaced it will skip creating `mariadb.sys` user.
This commit is to update the definer from `root` to `mariadb.sys` during
upgrade. It only makes the change if the original definers are root.
Doesn't choose to execute `maria_add_gis_sp.sql` in upgrade script to
recreate the procedures is because of considering the scenarios of
MDEV-23102 that `root` user is replaced and `mariadb.sys` is not created.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Currently, SST scripts assume that the filename specified in
the --log-bin-index argument either does not contain an extension
or uses the standard ".index" extension. Similar assumptions are
used for the log_bin_index parameter read from the configuration
file. This commit adds support for arbitrary extensions for the
index file paths.
If the server is started with the --innodb-force-recovery argument
on the command line, then during SST this argument can be passed to
mariabackup only at the --prepare stage, and accordingly it must be
removed from the --mysqld-args list (and it is not should be passed
to mariabackup otherwise).
This commit fixes a flaw in the SST scripts and add a test that
checks the ability to run the joiner node in a configuration that
uses --innodb-force-recovery=1.
Currently, SST scripts assume that the filename specified in
the --log-bin-index argument either does not contain an extension
or uses the standard ".index" extension. Similar assumptions are
used for the log_bin_index parameter read from the configuration
file. This commit adds support for arbitrary extensions for the
index file paths.
If the server is started with the --innodb-force-recovery argument
on the command line, then during SST this argument can be passed to
mariabackup only at the --prepare stage, and accordingly it must be
removed from the --mysqld-args list (and it is not should be passed
to mariabackup otherwise).
This commit fixes a flaw in the SST scripts and add a test that
checks the ability to run the joiner node in a configuration that
uses --innodb-force-recovery=1.
`mytop` and `my_print_defaults` for RPM
- Add `mytop` to client package
- Add man page of `my_print_defaults` to client package
- Add dependencies for RPMs
- Remove old comment
- Remove dead link
Reviewed by: serg@mariadb.com
MySQL-5.7 mysql.user tables have a last_password_changed field.
Because before MariaDB-10.4 remained oblivious to this, the act of creating
users or otherwise changing a users row left the last_password_field with 0.
Running a MariaDB-10.4 instance on this would work correctly, until mysql_upgrade
is run, when this 0 value immediately translates to password expired
state.
MySQL-5.7 relied on the password_expired enum to indicate password
expiry so we aren't going to activate password that were expired in
MySQL-5.7.
Thanks Hans Borresen for the bug report and review of the fix.
Fixed flaws with overly strict or, conversely,
overly soft verification of certificates in some
scenarios:
1. Removed the check that the 'commonname' (CN) in the
certificate matches the 'localhost' value on the side
of the joiner node, which was performed earlier, even
if the address was received by the script only as an
argument (out of the exchange via the Galera protocol) -
since for the joining node this argument always contains
its own local address, not the address of the remote host,
so it is always treated as 'localhost', which is not
necessarily true (outside of mtr testing);
2. Removed checking the domain name or IP-address of the
peer node in the encrypt=2 mode;
3. Fixed checking of compliance of certificates when
rsync SST is used;
4. Added the ability to specify CA not only as a file,
but also as a path to the directory where the certificates
are stored. To do this, the user just needs to specify the
path to this directory as the value ssl-ca or tca parameter,
ending with the '/' character.
SST scripts currently use Linux-specific construction
to create a temporary directory if the path prefix for
that directory is specified by the user. This does not
work with FreeBSD. This commit adds support for FreeBSD.
No separate test required.
Changes:
- Don't include port in connection parameters with 'localhost' hostname
More info:
The hostname, if not specified or specified as '' or 'localhost', will default
to a MySQL server running on the local machine using the default for the UNIX socket.
To connect to a MySQL server on the local machine via TCP, you must specify the
loopback IP address (127.0.0.1) as the host.
To comply with what the perl module expects, we omit the port for a
host=localhost connection.
Reported issue: https://bugzilla.redhat.com/show_bug.cgi?id=1976224
