Issue: During mtr_t:commit, if there is not enough space available in
redo log buffer, we flush the buffer. During flush, the LSN lock is
released allowing other concurrent mtr to commit. After flush we
reacquire the lock but use the old LSN obtained before check. It could
lead to redo log corruption. As the LSN moves backwards with the
possibility of data loss and unrecoverable server if the server aborts
for any reason or if server is shutdown with innodb_fast_shutdown=2.
With normal shutdown, recovery fails to map the checkpoint LSN to
correct offset.
In debug mode it hits log0log.cc:863: lsn_t log_t::write_buf()
Assertion `new_buf_free == ((lsn - first_lsn) & write_size_1)' failed.
In release mode, after normal shutdown, restart fails.
[ERROR] InnoDB: Missing FILE_CHECKPOINT(8416546) at 8416546
[ERROR] InnoDB: Log scan aborted at LSN 8416546
Backup fails reading the corrupt redo log.
[00] 2024-07-31 20:59:10 Retrying read of log at LSN=7334851
[00] FATAL ERROR: 2024-07-31 20:59:11 Was only able to copy log from
7334851 to 7334851, not 8416446; try increasing innodb_log_file_size
Unless a backup is tried or the server is shutdown or killed
immediately, the corrupt redo part is eventually truncated and there
may not be any visible issues seen in release mode.
This issue was introduced by the following commit.
commit a635c40648
MDEV-27774 Reduce scalability bottlenecks in mtr_t::commit()
Fix: If we need to release latch and flush redo before writing mtr
logs, make sure to get the latest system LSN after reacquiring the
redo system latch.
In commit bf0b82d24b (MDEV-33515)
the function log_t::init_lsn_lock() was removed. This was fine on
those platforms where InnoDB uses futex-based mutexes (Linux, FreeBSD,
OpenBSD, NetBSD, DragonflyBSD).
Dave Gosselin debugged this on Apple macOS and submitted a fix where
pthread_mutex_wrapper::pthread_mutex_wrapper() would invoke init().
We do not really need that; we only need to invoke lsn_lock.init()
like we used to do before commit bf0b82d24b.
This should be a no-op for the futex based mutexes, which intentionally
rely on zero initialization.
The missing pthread_mutex_init() call would cause race conditions
and corruption of log_sys.buf because multiple threads could
apparently hold log_sys.lsn_lock concurrently in
log_t::append_prepare(). The error would be caught by a debug
assertion in log_t::write_buf(), or in non-debug builds by the
fact that the server cannot be restarted due to an apparently
missing FILE_CHECKPOINT record (because it had been written
to wrong offset in log_sys.buf).
The failure in log_t::append_prepare() was caught on Microsoft Windows
after enabling SUX_LOCK_GENERIC and therefore forcing the use of
pthread_mutex_wrapper for the log_sys.lsn_lock. It appears to be fine
to omit the pthread_mutex_init() call on GNU/Linux.
log_t::create(): Invoke lsn_lock.init().
log_t::close(): Invoke lsn_lock.destroy().
To better catch this kind of issues in the future by simply defining
SUX_LOCK_GENERIC on any platform, a separate debug instrumentation patch
will be applied to the 10.6 branch later.
Reviewed by: Debarun Banerjee
- During copy algorithm, InnoDB should use bulk insert operation
for row by row insert operation. By doing this, copy algorithm
can effectively build indexes. This optimization is disabled
for temporary table, versioning table and table which has
foreign key relation.
Introduced the variable innodb_alter_copy_bulk to allow
the bulk insert operation for copy alter operation
inside InnoDB. This is enabled by default
ha_innobase::extra(): HA_EXTRA_END_ALTER_COPY mode tries to apply
the buffered bulk insert operation, updates the non-persistent
table stats.
row_merge_bulk_t::write_to_index(): Update stat_n_rows after
applying the bulk insert operation
row_ins_clust_index_entry_low(): In case of copy algorithm,
switch to bulk insert operation.
copy_data_error_ignore(): Handles the error while copying
the data from source to target file.
When mysqldump is run to dump the `mysql` system database, it generates
INSERT statements into the table `mysql.gtid_slave_pos`.
After running the backup script
those inserts did not produce the expected gtid state on slave. In
particular the maximum of mysql.gtid_slave_pos.sub_id did not make
into
rpl_global_gtid_slave_state.last_sub_id
an in-memory object that is supposed to match the current state of the
table. And that was regardless of whether --gtid option was specified
or not. Later when the backup recipient server starts as slave
in *non-gtid* mode this desychronization may lead to a duplicate key
error.
This effect is corrected for --gtid mode mysqldump/mariadb-dump only
as the following. The fixes ensure the insert block of the dump
script is followed with a "summing-up" SET @global.gtid_slave_pos
assignment.
For the implemenation part, note a deferred print-out of
SET-gtid_slave_pos and associated comments is prefered over relocating
of the entire blocks if (opt_master,slave_data &&
do_show_master,slave_status) ... because of compatiblity
concern. Namely an error inside do_show_*() is handled in the new code
the same way, as early as, as before.
A regression test can be run in how-to-reproduce mode as well.
One affected mtr test observed.
rpl_mysqldump_slave.result "mismatch" shows now the new deferring print
of SET-gtid_slave_pos policy in action.
The test was missing a save_master_gtid.inc on the master,
leading to the slave thinking it was in sync after executing
sync_with_master_gtid.inc, despite not having executed the
latest transaction. This skipped transaction, XA COMMIT,
was supposed to error-to-be-ignored because its XID could not
be found, but be thrown out because the replication filters
would filter out the target database. However, if the slave
was able to stop before executing the transaction, then
the replication filer is reset (to empty), and when the
slave is later restarted, that transactions error would
no longer be ignored.
Additionally, as the test cases added in MDEV-33921 rely
on GTID synchronization, the test cases now force
master_use_gtid=slave_pos for consistency
for ALTER_PARTITION_ADMIN (CHECK/REPAIR/LOAD INDEX/CACHE INDEX/etc)
partitioning marks affected partitions with PART_ADMIN state.
The assumption is that the server will call a corresponding
method of ha_partition which will reset the state back to PART_NORMAL.
This assumption is invalid, the server is not required to do so,
indeed, in CHECK ... FOR UPGRADE the server might decide early that
the table is fine and won't call ha_partition::check(), leaving
partitions in the wrong state. It will thus leak into the next
statement confusing the engine about what it is doing (see
ha_partition::create_handler_file()), causing a crash later.
Let's force all partitions into PART_NORMAL state after the admin
operation succeeded, in case it did so without consulting the engine.
There are 3 diff in result:
1) NULL value from SELECT
Due to incorrect truncating of the hex value, incorrect value is
written instead of original value to the view frm. This results in reading
incorrect value from frm, so eventual result is NULL.
2) 'Name_exp1' in column name (in gis.test)
This was because the identifier in SELECT is longer than 64 characters,
so 'Name_exp1' alias is also written to the view frm.
3)diff in explain extended
This was because the query plan for view protocol doesn't
contain database name. As a fix, disable view protocol for that particular
query.
Import only the required functions instead of all the functions from the
module to reduce the unnecessary functions in the namespace and prevent
shadowing. Note: All code changes are non-functional.
All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.
> After May 31, 2024, CentOS Stream 8 will be archived and no further
updates will be provided. [1]
CentOS Stream 8 is now EOL and should be updated to using CentOS Stream
9 for compatibility testing in GitLab CI.
[1] https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/https://www.centos.org/centos-linux-eol/
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services.
This patch fixes two problems:
- The code inside my_strtod_int() in strings/dtoa.c could test the byte
behind the end of the string when processing the mantissa.
Rewriting the code to avoid this.
- The code in test_if_number() in sql/sql_analyse.cc called my_atof()
which is unsafe and makes the called my_strtod_int() look behind
the end of the string if the input string is not 0-terminated.
Fixing test_if_number() to use my_strtod() instead, passing the correct
end pointer.
As is everywhere in mariadbd is used it more than
convienient to use mariadbd-safe than mysql_safe
in init script also in upstream test use output
mariadb-test-run-junit.xml than mysql-test-run-junit.xml
Adapted from upstream commit 8171f9da87 but separated only the datadir
section from the commit and wrote it in a way that does not trigger
Shellcheck or English grammar nags.
This check is intentionally not added to the preinst script as was done
upstream in 30fb72ca6e as the preinst script will always create the
data directory if missing, and thus checking for it right after the
creation is moot.
Fix a large amount of minor fixes to maintainer scripts and other done
downstream in the official Debian packaging.
Changes include:
38198d0b9e
> Limit check of running mysqld/mariadbd to system users (Closes: #1032047)
>
> If a random user has their own copy of mysqld/mariadbd running, the
> dpkg maintainer script should not care about it.
8116354d22
> Make error more helpful in case server restart fails (Related: #1033234)
>
> Bugs such as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033234
> and https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/2011293
> show that currently dpkg stopping on service stop/start does not have
> a very helpful error message.
8675e97202
> Complement upstream commits with more complete mysql->mariadb conversion
>
> The upstream commit 952af4a1 missed some places where 'mysql' or
> 'MySQL' can and should be converted to use 'mariadb' or 'MariaDB'.
c983613300
> Fix indentation in Debian post and pre scripts
>
> There is several misindentation inside Debian post and pre
> installation scripts. False indentation with space as indent space
> should be 2 and indentation with tabs.
>
> Adopt upstream commit 7cbb45d1 in Debian by conserving customizations
> in:
> - debian/mariadb-server.postinst
> - debian/mariadb-server.postrm
> - debian/mariadb-server.preinst
d0bcab443f
> Ensure spaces are used everywhere instead of tabs for indentation
0300a9157c
> Complement previous upstream commits to fix Shellcheck issues
>
> - Unify if/then and while/do on separate lines
> - Fix indentation to be consistent
> - Use "$()" instead of backticks for subshells
> - Exit code cannot be -1, must be 0-255
> - Remove unused variables MYCHECK and MYCHECK_PARAMS
> - Rewrite messy command-line database calls to an easier to read form
> that does exactly the same
> - Use 'command -v' test instead of 'which'
>
> With this commit, all of debian/* is Shellcheck clean.
Also
* Update mariadb.conf.d template to tell users where to create logdir
if they are not using journald
* Remove use of work 'slave'
* Add minor workaround for Debian Bug #1022994 if TMPDIR is empty
* Make start/stop in maintainer scripts correctly check mariadbd
ownership and only start/stop processes owned by root or 'mysql'
* Remove obsolete 'NO_UPDATE_BUILD_VERSION=1' as it did not affect the
RocksDB build reproducibility as previously assumed
* Run 'wrap-and-sort -av'
- Unify on MTR_SKIP_TEST_LIST in both d/rules and autopkgtests
- Unify MTR command in both d/rules and autopkgtests
- Make d/rules section more verbose to help debugging why tests
sometimes ran and sometimes not
- If MTR fails, make the log a bit more verbose
(inspired by https://github.com/MariaDB/buildbot/pull/76/files)
The way DPKG_GENSYMBOLS_CHECK_LEVEL was exported did not actually
have any effect on the build. Fix the syntax so that build will
indeed fail if there there are new symbols in new upstream version.
Based on riscv64 build logs the RocksDB plugin currently builds fine on
it, and the riscv64 platform is 64-bit and has correct endianness for
RocksDB, so all the pre-requisites for it working exist, so it should
work.
Specifying filters (filter_status, filter_user, etc) in the mytop config
previously wouldn't work, because any filter specified here was added to
the config hash as a literal string.
This change fixes that - if filter_* is defined in the config and matches
an existing filter_* key, then run the value through StringOrRegex() and
assign to the config hash.
Make sure that Debian respects systemd disabled by bumping
to deb compat 11 which is available from Debian 10 and Ubuntu 20.04
and it provides better integration with systemd.
Start using dh_installsystemd which is new recommended way
in compat 11
As mariadb.pc contains mostly the same than
libmariadb.pc and it mainly only creates distortion
for client developers. They use libmariadb.pc not mariadb.pc
(which is for embbeded use mainly).
Move mariadb.pc to not-installed from libmariadbd-dev
to clear out this situation
There is some package-contains-documentation-outside-usr-share-doc that
are better to be there than move somewhere else. They are:
* mariadb-server: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/errmsg-utf8.txt]
* mariadb-server-core: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/charsets/README]
* mariadb-test: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/mysql-test/README]
Also fix Mroonga
* mariadb-plugin-mroonga: extra-license-file [usr/share/mysql/mroonga/COPYING]
There is couple spare-manual-page problems which means that
there is man page but no binary for that.
wsrep_sst_backup is in not-installed and man page is in
mariadb-server package. Move man page also to not-installed
mysql-test-run.pl is in unusual location which makes
lintian think that it not available.
There is unused override which is not needed an polluting
output
* unused-override arch-dependent-file-not-in-arch-specific-directory usr/bin/mariadb_config [usr/share/lintian/overrides/libmariadb-dev:2]
