Commit graph

201522 commits

Author SHA1 Message Date
Tuukka Pasanen
91336f6a18 MDEV-33750: Update Salsa-CI file
Update changes from Debian Salsa that they are
in sync
2024-07-17 14:32:50 +10:00
Tuukka Pasanen
74aea60d92 MDEV-33750: Update few Debian Salsa-CI tests from upstream to make smoke test pass
autopkgtests from package are not passing currently and make them pass with
upgrading Salsa-CI YAML file.
2024-07-17 14:32:50 +10:00
Tuukka Pasanen
89a638f4b8 MDEV-33750: Sync smoke test from newer version of Debian Salsa-CI
For making smoke test work sync current Debian Salsa-CI version
for making needed changes
2024-07-17 14:32:50 +10:00
Tuukka Pasanen
4c1e4ba62e MDEV-33750: Remove seq in Debian init.d for-loop
Make all init.d script for loops to use new
{1..5} syntax and rework one not to use seq as
all the rest use new Bash syntax.
2024-07-17 14:32:50 +10:00
Tuukka Pasanen
7ae9505106 MDEV-33750: Rework MyISAM recovery script
Make small adjustment to MyISAM recovery function
SQL statement and how to handle it.
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
af124c4f1c MDEV-33750: Make SysV init more verbose in case of MariaDB start failures
MariaDB installs/upgrades in Docker containers (and elsewhere where
systemd is not used) occasionally fail with output like:

  Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
  invoke-rc.d: initscript mariadb, action "start" failed.
  dpkg: error processing package mariadb-server-10.5 (--configure):
   installed mariadb-server-10.5 package post-installation script subprocess returned error exit status 1

This is not very helpful. Thus extend the init script to try the server
start/restart one more time but with error log defined separately,
and then print out the error log contents of this single start attempt.

  ...
  Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  230103 01:06:48 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
  230103 01:06:48 mysqld_safe Logging to '/tmp/tmp.JlE4sdUMZz.err'.
  230103 01:06:49 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
  Running '/etc/init.d/mariadb start' failed with error log:
  230103 01:06:49 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
  2023-01-03  1:06:49 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.18-MariaDB-0+deb11u1) starting as process 10417 ...
  2023-01-03  1:06:49 0 [Note] InnoDB: Uses event mutexes
  2023-01-03  1:06:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
  2023-01-03  1:06:49 0 [Note] InnoDB: Number of pools: 1
  2023-01-03  1:06:49 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
  2023-01-03  1:06:49 0 [Note] InnoDB: Using Linux native AIO
  2023-01-03  1:06:49 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
  2023-01-03  1:06:49 0 [Note] InnoDB: Completed initialization of buffer pool
  2023-01-03  1:06:49 0 [ERROR] InnoDB: Invalid flags 0x4800 in ./ibdata1
  ...
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
3f44efaa17 MDEV-33750: Make sure that datadir always has some value and exists
Adapted from upstream commit 8171f9da87 but separated only the datadir
section from the commit and wrote it in a way that does not trigger
Shellcheck or English grammar nags.

This check is intentionally not added to the preinst script as was done
upstream in 30fb72ca6e as the preinst script will always create the
data directory if missing, and thus checking for it right after the
creation is moot.
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
2adaf5c261 MDEV-33750: Sync maintainer scripts etc with latest downstream 10.11.5 in Debian
Fix a large amount of minor fixes to maintainer scripts and other done
downstream in the official Debian packaging.

Changes include:

38198d0b9e
> Limit check of running mysqld/mariadbd to system users (Closes: #1032047)
>
> If a random user has their own copy of mysqld/mariadbd running, the
> dpkg maintainer script should not care about it.

8116354d22
> Make error more helpful in case server restart fails (Related: #1033234)
>
> Bugs such as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033234
> and https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/2011293
> show that currently dpkg stopping on service stop/start does not have
> a very helpful error message.

8675e97202
> Complement upstream commits with more complete mysql->mariadb conversion
>
> The upstream commit 952af4a1 missed some places where 'mysql' or
> 'MySQL' can and should be converted to use 'mariadb' or 'MariaDB'.

c983613300
> Fix indentation in Debian post and pre scripts
>
> There is several misindentation inside Debian post and pre
> installation scripts. False indentation with space as indent space
> should be 2 and indentation with tabs.
>
> Adopt upstream commit 7cbb45d1 in Debian by conserving customizations
> in:
> - debian/mariadb-server.postinst
> - debian/mariadb-server.postrm
> - debian/mariadb-server.preinst

d0bcab443f
> Ensure spaces are used everywhere instead of tabs for indentation

0300a9157c
> Complement previous upstream commits to fix Shellcheck issues
>
> - Unify if/then and while/do on separate lines
> - Fix indentation to be consistent
> - Use "$()" instead of backticks for subshells
> - Exit code cannot be -1, must be 0-255
> - Remove unused variables MYCHECK and MYCHECK_PARAMS
> - Rewrite messy command-line database calls to an easier to read form
>   that does exactly the same
> - Use 'command -v' test instead of 'which'
>
> With this commit, all of debian/* is Shellcheck clean.

Also
* Update mariadb.conf.d template to tell users where to create logdir
  if they are not using journald
* Remove use of work 'slave'
* Add minor workaround for Debian Bug #1022994 if TMPDIR is empty
* Make start/stop in maintainer scripts correctly check mariadbd
  ownership and only start/stop processes owned by root or 'mysql'
* Remove obsolete 'NO_UPDATE_BUILD_VERSION=1' as it did not affect the
  RocksDB build reproducibility as previously assumed
* Run 'wrap-and-sort -av'
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
5000d1ba6e MDEV-33750: Increase MTR verbosity and unify skip test lists usage
- Unify on MTR_SKIP_TEST_LIST in both d/rules and autopkgtests
- Unify MTR command in both d/rules and autopkgtests
- Make d/rules section more verbose to help debugging why tests
  sometimes ran and sometimes not
- If MTR fails, make the log a bit more verbose
  (inspired by https://github.com/MariaDB/buildbot/pull/76/files)
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
f79f3ada24 MDEV-33750: Fix DPKG_GENSYMBOLS_CHECK_LEVEL so it actually takes effect
The way DPKG_GENSYMBOLS_CHECK_LEVEL was exported did not actually
have any effect on the build. Fix the syntax so that build will
indeed fail if there there are new symbols in new upstream version.
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
b11892c9fb MDEV-33750: Fix Lintian warnings
- mariadb-server: recursive-privilege-change "chown -R"
- use correct URL https://jira.mariadb.org everywhere
- dependency-is-not-multi-archified libmariadb-dev-compat depends on libmariadb-dev (multi-arch: no)
- dependency-is-not-multi-archified mariadb-plugin-gssapi-client depends on mariadb-client (multi-arch: no)
2024-07-17 14:32:50 +10:00
Otto Kekäläinen
9e9211215c MDEV-33750: Enable mariadb-plugin-rocksdb for riscv64
Based on riscv64 build logs the RocksDB plugin currently builds fine on
it, and the riscv64 platform is 64-bit and has correct endianness for
RocksDB, so all the pre-requisites for it working exist, so it should
work.
2024-07-17 14:32:50 +10:00
PinkFreud
49dff5a4b6 MDEV-34604 mytop - fix specifying filters in .mytop
Specifying filters (filter_status, filter_user, etc) in the mytop config
previously wouldn't work, because any filter specified here was added to
the config hash as a literal string.

This change fixes that - if filter_* is defined in the config and matches
an existing filter_* key, then run the value through StringOrRegex() and
assign to the config hash.
2024-07-17 14:19:53 +10:00
Tuukka Pasanen
bccdc72536 MDEV-34191: Make sure that Debian respects systemd disabled
Make sure that Debian respects systemd disabled by bumping
to deb compat 11 which is available from Debian 10 and Ubuntu 20.04
and it provides better integration with systemd.

Start using dh_installsystemd which is new recommended way
in compat 11
2024-07-17 13:58:44 +10:00
Tuukka Pasanen
d83742622d MDEV-33750: Rename mysql to mariadb in Debian directory
As this is MariaDB then also variable names in
mariadb-server.*inst should correlate when possible
this change variable and directory names
2024-07-17 12:32:03 +10:00
Tuukka Pasanen
70e3c144b3 MDEV-34456: Move mariadb.pc to not-installed
As mariadb.pc contains mostly the same than
libmariadb.pc and it mainly only creates distortion
for client developers. They use libmariadb.pc not mariadb.pc
(which is for embbeded use mainly).

Move mariadb.pc to not-installed from libmariadbd-dev
to clear out this situation
2024-07-17 12:19:08 +10:00
Tuukka Pasanen
b41168c6b5 MDEV-33837: Suspend package-contains-documentation-outside-usr-share-doc warnings
There is some package-contains-documentation-outside-usr-share-doc that
are better to be there than move somewhere else. They are:

 * mariadb-server: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/errmsg-utf8.txt]
 * mariadb-server-core: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/charsets/README]
 * mariadb-test: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/mysql-test/README]

Also fix Mroonga

 * mariadb-plugin-mroonga: extra-license-file [usr/share/mysql/mroonga/COPYING]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
56087d0d15 MDEV-33837: Fix spare-manual-page warnings in Debian lintian
There is couple spare-manual-page problems which means that
there is man page but no binary for that.

wsrep_sst_backup is in not-installed and man page is in
mariadb-server package. Move man page also to not-installed

mysql-test-run.pl is in unusual location which makes
lintian think that it not available.
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
3e384d830e MDEV-33837: Remove unneeded lintian override in libmariadb-dev
There is unused override which is not needed an polluting
output

 * unused-override arch-dependent-file-not-in-arch-specific-directory usr/bin/mariadb_config [usr/share/lintian/overrides/libmariadb-dev:2]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
cdeb30647f MDEV-33837: Suspend lintian warnings for description is synopsis
Lintian warn that mariadb-common and mysql-common descriptions
are sypnosis as they contain dot but dots are used in other
purposes so they are just false-positives:

 * mysql-common: synopsis-is-a-sentence "MariaDB client common configuration files package (e.g. /etc/mysql/my.cnf)"
 * mariadb-server: unused-debconf-template mariadb-server/old_data_directory_saved [templates:2]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
280035bf82 MDEV-33837: Remove purposed spelling errors
Lot's of binaries have 'spelling errors' which are there in
purpose and they are simply false positives some list of them
are:

 * I: libmariadbd19: spelling-error-in-binary noone no one [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: libmariadbd19: spelling-error-in-binary thats that's [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: libmariadbd19: spelling-error-in-binary theres there's [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: libmariadbd19: spelling-error-in-binary yuR your [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: mariadb-backup: spelling-error-in-binary exising existing [usr/bin/mariadb-backup]
 * I: mariadb-backup: spelling-error-in-binary noone no one [usr/bin/mariadb-backup]
 ...
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
659a596ca5 MDEV-33837: Lintian warns there are typos in manuals
which are being handled

When new man pages are generated for libmariadb13 then manpage
typos are fixed and now they just pollute Lintian output

 * typo-in-manual-page handshak handshake [usr/share/man/man3/mariadb_get_infov.3.gz:173]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_ping.3.gz:39]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_set_character_set.3.gz:37]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_stmt_error.3.gz:30]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_stmt_reset.3.gz:27]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_store_result.3.gz:34]
 * typo-in-manual-page releated related [usr/share/man/man3/mariadb_get_infov.3.gz:8]
 * typo-in-manual-page reponse response [usr/share/man/man3/mysql_send_query.3.gz:8]
 * typo-in-manual-page seperated separated [usr/share/man/man3/mysql_real_connect.3.gz:144]
 * typo-in-manual-page seperated separated [usr/share/man/man3/mysql_real_connect.3.gz:148]
 * typo-in-manual-page sucess success [usr/share/man/man3/mariadb_reconnect.3.gz:25]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
3607ecdfb9 MDEV-33837: Remove autogenerated files in columnstore warnings
There is lot of warnings like this:

 * source-contains-autogenerated-visual-c++-file [extra/wolfssl/wolfssl/IDE/WIN10/resource.h]
 * source-contains-autogenerated-visual-c++-file [extra/wolfssl/wolfssl/IDE/WIN10/wolfssl-fips.rc]
 * source-contains-autogenerated-visual-c++-file [extra/wolfssl/wolfssl/resource.h]
 * source-contains-autogenerated-visual-c++-file [storage/columnstore/columnstore/dbcon/ddlpackageproc/libddlpackageproc.rc]
 * source-contains-autogenerated-visual-c++-file [storage/columnstore/columnstore/dbcon/ddlpackageproc/resource.h]
 * source-contains-autogenerated-visual-c++-file [storage/columnstore/columnstore/dbcon/dmlpackageproc/libdmlpackageproc.rc]

Which are needed and should be there. They are just false-positives

Also MariaDB rebuilds these Java arhives but they are available for testing purposes still
so they are also false-positives

 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/JavaWrappers.jar]
 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/JdbcMariaDB.jar]
 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/Mongo2.jar]
 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/Mongo3.jar]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
8f87f9c745 MDEV-33837: Remove conflicts warnings from lintian
Some packages are needed to conflict packages that are not in
this package and they emerge some warnings. Remove them from
lintian output as they are not errors or something that should
be handled.

I: libmariadb-dev: conflicts-with-version libmariadb-dev-compat (<< 3.0.0)
I: libmariadb-dev-compat: conflicts-with-version libmariadbclient-dev (<< 1:10.11.8+maria~deb11)
I: libmariadb3: conflicts-with-version libmariadbclient18 (<< 10.2.0)
I: mariadb-client: conflicts-with-version mysql-client (<< 5.0.51)
I: mariadb-client-core: conflicts-with-version mysql-client (<< 5.0.51)
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
517d9515f1 MDEV-33837: Remove false-positive Bash warnings from
lintian output

Lintian have many warnings that there is shebang '/bin/sh' which can
be populary /bin/bash or /bin/zsh but also many others like Ksh
or even Fish.

Warned lintian problems are testing that are run under Bash or some other
shell so they are or they are words in comments like let

 * I: mariadb-client: bash-term-in-posix-shell '`echo "testing\c"' [usr/bin/mariadb-secure-installation:191]
 * I: mariadb-server: bash-term-in-posix-shell '${BASH_VERSION' [usr/share/mysql/wsrep_notify:86]
 * I: mariadb-server: bash-term-in-posix-shell '[ "$url" ==' [usr/bin/mariadbd-safe:216]
 * I: mariadb-server: bash-term-in-posix-shell 'let' [usr/bin/mariadbd-safe:41]
 * I: mariadb-server: bash-term-in-posix-shell 'ulimit' [usr/bin/mariadbd-safe:712]
 * I: mariadb-server: bash-term-in-posix-shell 'ulimit' [usr/bin/mariadbd-safe:832]
 * I: mariadb-server-core: bash-term-in-posix-shell 'source tree,' [usr/bin/mariadb-install-db:93]
 * I: mariadb-test-data: bash-term-in-posix-shell '${BASH_VERSION' [usr/share/mysql/mysql-test/std_data/wsrep_notify.sh:87]
 * I: mariadb-test-data: bash-term-in-posix-shell '${BASH_VERSION' [usr/share/mysql/mysql-test/std_data/wsrep_notify_ssl.sh:87]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
ddefc59bc8 MDEV-33837: Fix some miscellaneous lintian warnings
These are not big ones.

These are needed in *BSD family

 * incorrect-path-for-interpreter /usr/bin/env perl != /usr/bin/perl [usr/share/mysql/mysql-test/mariadb-stress-test.pl]
 * incorrect-path-for-interpreter /usr/bin/env perl != /usr/bin/perl [usr/share/mysql/mysql-test/mariadb-test-run.pl]

and these are needed to make sure that these packages get removed if they exist:

 * version-substvar-for-external-package Replaces ${source:Version} libmariadb-dev -> libmysqlclient-dev [debian/control:*]
 * version-substvar-for-external-package Replaces ${source:Version} libmariadb-dev -> libmysqld-dev [debian/control:*]
 * version-substvar-for-external-package Replaces ${source:Version} libmariadbd-dev -> libmariadbclient-dev [debian/control:*]
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
cfec45db14 MDEV-33837: Fix unwanted-path-too-specific specific warning
There is not-installed static libraries which only
have x86-64 path so they will be installed in other
architectures like ARM. Fix them replacing with '*'
char

 * W: mariadb source: unwanted-path-too-specific
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
263064c020 MDEV-33837: Add and fix lintian overrides
Lintian overdrives tends to rot and with Debian 12
Lintian got overhaul with syntax changes which made
most of the old overrides obsoleted. Change old ones
to new ones and remove unneeded onews also add some
needed overrides.

 * W: mariadb-test-data: mismatched-override *
   - There was serveral old syntax overrides which should be just removed

 * W: mariadb-test-data: national-encoding  *
   - There is lot's test files which are in ISO/IEC 8859-1 encoding
     and not UTF-8 for purpose. Remove then from polluting lintian

 * W: mariadb-plugin-provider-bzip2: mismatched-override
 * W: mariadb-plugin-provider-lz4: mismatched-override
 * W: mariadb-plugin-provider-lzma: mismatched-override
 * W: mariadb-plugin-provider-lzo: mismatched-override
 * W: mariadb-plugin-provider-snappy: mismatched-override
   - Remove old style overrides from packages

 * W: mariadb-test: shared-library-lacks-prerequisites [usr/lib/mysql/plugin/auth_0x0100.so]
 * W: mariadb-test: shared-library-lacks-prerequisites [usr/lib/mysql/plugin/debug_key_management.so]
 * W: mariadb-test: shared-library-lacks-prerequisites [usr/lib/mysql/plugin/test_sql_service.so]
   - These libraries are like that for a purpose
2024-07-17 11:25:51 +10:00
Tuukka Pasanen
13ee9417c9 MDEV-33837: Fix lintian warning that are against debian/control
Debian control file has few lintian warning level notifications

 * mariadb-test: breaks-without-version mariadb-server-5.5
   - Several packages conflicts with mariadb-server-5.5 but
     on mariadb-test breaks without it so it's controversial
     and can be resolved only with conflicting in mariadb-test

 * W: libmariadbd-dev: extended-description-line-too-long line 6
 * W: mariadb-plugin-cracklib-password-check: extended-description-line-too-long line 4
   - Lines are too long (over 80 chars) and they are just separated
     to new line.
2024-07-17 11:25:51 +10:00
Yuchen Pei
03a350378a
MDEV-30408 Reset explicit_limit in exists2in
Item_exists_subselect::fix_length_and_dec() sets explicit_limit to 1.
In the exists2in transformation it resets select_limit to NULL. For
consistency we should reset explicity_limit too.

This fixes a bug where spider table returns wrong results for queries
that gets through exists2in transformation when semijoin is off.
2024-07-17 08:54:40 +08:00
Yuchen Pei
8416fd323c
MDEV-32627 [fixup] Spider: Fix conn key length
To avoid off-by-one in spider_get_share.

And document conn key and conn key length.
2024-07-17 08:52:35 +08:00
Yuchen Pei
20d99f3f86
MDEV-32627 Distinguish between absence of a keyword and empty value for the keyword
Distinguish them in two place:

when constructing connection key in create_conn_key and
spider_create_conn for both ordinary queries and spider_direct_sql

For spider_create_conn and spider_udf_direct_sql_create_conn, the
created conn gets assigned a field from the source object if and only
if source->field is non-null.

For spider_create_conn_keys and spider_udf_direct_sql_create_conn_key,
we update the encoding so that absence of keyword and keyword with an
empty value result in different keys. More specifically, if the i-th
keyword has a value, the corresponding part in the conn key begins
with the char \i, followed by the possibly empty value and ends with a
\0. If the i-th keyword is not specified, then it does not get a
mention in the conn key.

As part of this change, we also update table param / option parsing to
recognise a singleton empty string when creating an string list,
instead of writing it off as NULL.
2024-07-17 08:51:44 +08:00
Yuchen Pei
6f3baec4f5
MDEV-32627 Spider: add tests for connection param overriding
When connection parameters are specified in COMMENT, CONNECTION and
SERVER, they are overriden in the following order:

COMMENT > CONNECTION > SERVER
2024-07-17 08:51:44 +08:00
Daniel Black
75d354a23a MDEV-33988 DELETE single table to support table aliases
Gain MySQL compatibility by allowing table aliases in a single
table statement.

This now supports the syntax of:

DELETE [delete_opts] FROM tbl_name [[AS] tbl_alias] [PARTITION (partition_name [, partition_name] ...)] ....

The delete.test is from MySQL commit 1a72b69778a9791be44525501960b08856833b8d
/ Change-Id: Iac3a2b5ed993f65b7f91acdfd60013c2344db5c0.

Co-Author: Gleb Shchepa <gleb.shchepa@oracle.com> (for delete.test)

Reviewed by Igor Babaev (igor@mariadb.com)
2024-07-17 09:50:53 +10:00
Tuukka Pasanen
62dfd0c09d MDEV-33837: Workaround chown warnings
Blindly recursive chown is not way to do it.
This Workaround is not much better than just chown -R but
there is small adjustment just chown MariaDB statedir and logdir
then with find only chown those files that are not correctly
owned.

Fixes lintian warnings:
 * W: mariadb-server: recursive-privilege-change "chown -R" [postinst:*]
 * W: mariadb-server: recursive-privilege-change "chown -R" [postinst:*]
2024-07-17 09:39:49 +10:00
Ian Gilfillan
2ee061c258 Add missing options to the mariadb-dump man page 2024-07-17 09:32:29 +10:00
Brandon Nesterenko
0cd20e3aaf MDEV-34571: Fix funcs_1.is_columns_is_embedded
Result file needed re-recording to account for the
new information_schema columns
2024-07-16 09:32:42 -06:00
Vladislav Vaintroub
9e25d6f0cc MDEV-33627 : Implement option --dir in mariadb-import
With that, it is possible to restore the full "instance" from a backup
made with mariadb-dump --dir

The patch implements executing DDL (tables, views, triggers) using
statements that are stored in .sql file, created by mariadb-dump
--dir .

Care is taken of creating triggers correctly after the data is loaded,
disabling foreign keys and unique key checks etc.

The files are loaded in descending order by datafile size -
to ensure better work distribution when running with --parallel option.

In addition to --dir option, following options are implemented for
partial restore

include-only options:
--database             -  import one or several databases
--table                -  import one or several tables

exclude options:
--ignore-database      -. ignore one or several databases when importing
--ignore-table         -  to ignore one or several tables when importing

All options above are only valid together with --dir option,
and can be specified multiple times.
2024-07-16 15:16:29 +02:00
Vladislav Vaintroub
04988d87aa MDEV-33627 refactor threading in mariadb-import
Use threadpool, instead of one-thread-and-connection-per-table
2024-07-16 15:14:28 +02:00
Vladislav Vaintroub
c483c5ca56 MDEV-33627 preparation - tpool fix
Fix tpool to not use maintenance timer for fixed pool size.
2024-07-16 15:00:29 +02:00
Vladislav Vaintroub
59167c567b MDEV-33627 preparation - compile mysqlimport as C++ 2024-07-16 14:56:01 +02:00
Sergei Petrunia
e644e130b0 MDEV-30623: Fix the testcase
- Fix view-protocol: long expressions in SELECT
  list should have "expr AS column_name".

- Also, moved the test from subselect*test to
  suite/json/t/json_table.test.
2024-07-16 12:52:31 +03:00
Oleg Smirnov
972879f413 MDEV-33010 Crash when pushing condition with CHARSET()/COERCIBILITY() into derived table
Based on the current logic, objects of classes Item_func_charset and
Item_func_coercibility (responsible for CHARSET() and COERCIBILITY()
functions) are always considered constant.
However, SQL syntax allows their use in a non-constant manner, such as
CHARSET(t1.a), COERCIBILITY(t1.a).

In these cases, the `used_tables()` parameter corresponds to table names
in the function parameters, creating an inconsistency: the item is marked
as constant but accesses tables. This leads to crashes when
conditions with CHARSET()/COERCIBILITY() are pushed into derived tables.

This commit addresses the issue by setting `used_tables()` to 0 for
`Item_func_charset` and `Item_func_coercibility`. Additionally, the items
now store the return values during the preparation phase and return
them during the execution phase. This ensures that the items do not call
its arguments methods during the execution and are truly constant.

Reviewer: Alexander Barkov <bar@mariadb.com>
2024-07-16 16:20:17 +07:00
Yuchen Pei
384ec03e48
MDEV-34421 Check the SQL command when resolving storage engine
ENGINE_SUBSTITUTION only applies to CREATE TABLE and ALTER TABLE, and
Storage_engine_name::resolve_storage_engine_with_error() could be
called when executing any sql command.
2024-07-16 16:33:05 +08:00
Yuchen Pei
132270d3de
MDEV-34541 Clean up spider self reference check
SPIDER_CONN::loop_check_meraged_first is useless, because all
SPIDER_CONN_LOOP_CHECKs are in SPIDER_CONN::loop_check_queue, which in
spider_db_conn::fin_loop_check() is iterated over.

This fixes the use-after-free issue when there are three spider tables
sharing the same remote, and their corresponding
SPIDER_CONN_LOOP_CHECKs getting merged in
spider_conn_queue_and_merge_loop_check()

This also fixes MDEV-34555
2024-07-16 16:33:05 +08:00
Yuchen Pei
0bb9862888
MDEV-29962 Spider: creates connections if needed before lock_tables()
Same cause as MDEV-31996. This prevents reuse of conns freed
previously, e.g. during the commit of previous statement.

It does not occur in 10.4 because of the commit for MDEV-19002 removed
the call to spider_check_trx_and_get_conn().
2024-07-16 16:33:05 +08:00
Yuchen Pei
696c2497fc
MDEV-27902 Spider check trx and get conn in rnd_next()
This allows creation of SPIDER_CONN on demand, if the previous one was
freed.

Also, the first_link_idx's are reset during
spider_check_trx_and_get_conn(), which in the case of remote HANDLER
commands, might not match the link to use correct first_link_idx for
remote HANDLER statement that was later set in
ha_spider::rnd_handler_init() (causing testing regressions in the
spider/handler suite). Therefore we fix the first_link_idx there.
2024-07-16 16:33:05 +08:00
Yuchen Pei
85a36958e3
MDEV-27902 Some Spider code documentation regarding first_link_idx and remote HANDLER commands 2024-07-16 16:33:04 +08:00
Yuchen Pei
14c4050992
MDEV-32492 Delete and remove trx_ha on spider share mismatch
A SPIDER_TRX_HA associated with a SPIDER_TRX could have longer
lifetime than its associated SPIDER_SHARE. And it is identified with
the associated table name. When the SPIDER_SHARE no longer valid, e.g.
when the associated spider table has been dropped and recreated, the
SPIDER_TRX_HA should be reset too.

Since spider could create a new SPIDER_SHARE with the exact same
address of a freed SPIDER_SHARE, we try to mark all SPIDER_TRX_HAs
associated with a SPIDER_SHARE invalid when the SPIDER_SHARE is about
to be freed.
2024-07-16 16:33:04 +08:00
Yuchen Pei
653050ac84
MDEV-32492 MDEV-29676 Spider: some code documentation and cleanup
Some documentations come from commit for MDEV-29676, and the rest are
mainly related to SPIDER_TRX_HA.

Removed SPIDER_TRX_HA::trx as it is unused.
2024-07-16 16:33:04 +08:00