Commit graph

9292 commits

Author SHA1 Message Date
Vladislav Vaintroub
9e25d6f0cc MDEV-33627 : Implement option --dir in mariadb-import
With that, it is possible to restore the full "instance" from a backup
made with mariadb-dump --dir

The patch implements executing DDL (tables, views, triggers) using
statements that are stored in .sql file, created by mariadb-dump
--dir .

Care is taken of creating triggers correctly after the data is loaded,
disabling foreign keys and unique key checks etc.

The files are loaded in descending order by datafile size -
to ensure better work distribution when running with --parallel option.

In addition to --dir option, following options are implemented for
partial restore

include-only options:
--database             -  import one or several databases
--table                -  import one or several tables

exclude options:
--ignore-database      -. ignore one or several databases when importing
--ignore-table         -  to ignore one or several tables when importing

All options above are only valid together with --dir option,
and can be specified multiple times.
2024-07-16 15:16:29 +02:00
Vladislav Vaintroub
04988d87aa MDEV-33627 refactor threading in mariadb-import
Use threadpool, instead of one-thread-and-connection-per-table
2024-07-16 15:14:28 +02:00
Vladislav Vaintroub
59167c567b MDEV-33627 preparation - compile mysqlimport as C++ 2024-07-16 14:56:01 +02:00
Alexander Barkov
8f4ec79d09 Merge remote-tracking branch 'origin/11.4' into 11.5 2024-07-08 12:25:04 +04:00
Alexander Barkov
c4bf4ce948 Merge remote-tracking branch 'origin/11.2' into 11.4 2024-06-17 15:46:39 +04:00
Marko Mäkelä
a21e49cbcc Merge 11.1 into 11.2 2024-06-17 12:02:03 +03:00
Marko Mäkelä
d34289a3e2 Merge 10.11 into 11.1 2024-06-17 09:21:50 +03:00
Marko Mäkelä
346a0c1402 Merge 10.6 into 10.11 2024-06-17 09:08:07 +03:00
Marko Mäkelä
e60acae655 Merge 10.5 into 10.6 2024-06-17 08:40:07 +03:00
Monty
956bcf8f49 Change mysqldump to use DO instead of 'SELECT' for storing sequences.
This avoids a lot of SETVAL() results when applying a mysqldump with
sequences.
2024-06-16 10:51:33 +03:00
Monty
fef32fd9ad MDEV-34406 Enhance mariadb_upgrade to print failing query in case of error
To make this possible, it was also necessary to enhance the mariadb
client with the option --print-query-on-error.
This option can also be very useful when running a batch of queries
through the mariadb client and one wants to find out where things goes
wrong.

TODO: It would be good to enhance mariadb_upgrade to not call the mariadb
client for executing queries but instead do this internally.  This
would have made this patch much easier!

Reviewed by: Sergei Golubchik <serg@mariadb.com>
2024-06-16 10:51:33 +03:00
Marko Mäkelä
5b89cab44f Merge 10.6 into 10.11 2024-06-13 08:16:49 +03:00
Marko Mäkelä
fc9005adc4 Merge 10.5 into 10.6 2024-06-12 07:51:28 +03:00
Marko Mäkelä
b81d717387 Merge 10.6 into 10.11 2024-06-11 12:50:10 +03:00
Vladislav Vaintroub
f2eda61579 MDEV-33616 workaround libmariadb bug : mysql_errno = 0 on failed connection
The bug can happens on macOS, if server closes the socket without sending
error packet to client. Closing the socket on server side is legitimate,
and happen e.g when write timeout occurs, perhaps also other situations.

However mysqltest is not prepared to handle mysql_errno 0, and erroneously
thinks connection was successfully established.

The fix/workaround in mysqltest is to treat client failure with
mysql_errno 0 the same as CR_SERVER_LOST (generic client-side
communication error)

The real fix in client library would ensure that mysql_errno is set
on errors.
2024-06-11 09:15:32 +02:00
Marko Mäkelä
27834ebc91 Merge 10.5 into 10.6 2024-06-10 15:22:15 +03:00
Marko Mäkelä
e255837eaf MDEV-34266 safe_strcpy() includes an unnecessary conditional branch
The strncpy() wrapper that was introduced in
commit 567b681299
is checking whether the output was truncated even in cases
where the caller does not care about it.

Let us introduce a separate function safe_strcpy_truncated() that
indidates whether the output was truncated.
2024-06-07 19:24:36 +03:00
Oleksandr Byelkin
77c4c0f256 MDEV-34203 Sandbox mode \- is not compatible with --binary-mode
"Process" sandbox short command put by masqldump to avoid an error.
2024-06-07 14:07:54 +02:00
Yuchen Pei
2d3e2c58b6
Merge branch '10.11' into 11.1 2024-05-31 10:54:31 +10:00
Marko Mäkelä
22ba7e4ff8 Merge 10.6 into 10.11 2024-05-30 16:04:00 +03:00
Marko Mäkelä
5ba542e9ee Merge 10.5 into 10.6 2024-05-30 14:27:07 +03:00
Vladislav Vaintroub
99f6684ba0 Fix mismerge.
mysqldump --dir produces unexpected sql output on stdout.
All sql should go into .sql files (and data to .txt files)
2024-05-28 22:37:40 +02:00
Souradeep Saha
83a04be84a Fix Various Typos
Fix various typos, in comments and DEBUG statements, and code changes
are non-functional.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
2024-05-28 11:31:49 +10:00
Monty
775cba4d0f MDEV-33145 Add FLUSH GLOBAL STATUS
- FLUSH GLOBAL STATUS now resets most global_status_vars.
  At this stage, this is mainly to be used for testing.
- FLUSH SESSION STATUS added as an alias for FLUSH STATUS.
- FLUSH STATUS does not require any privilege (before required RELOAD).
- FLUSH GLOBAL STATUS requires RELOAD privilege.
- All global status reset moved to FLUSH GLOBAL STATUS.
- Replication semisync status variables are now reset by
  FLUSH GLOBAL STATUS.
- In test cases, the only changes are:
  - Replace FLUSH STATUS with FLUSH GLOBAL STATUS
  - Replace FLUSH STATUS with FLUSH STATUS; FLUSH GLOBAL STATUS.
    This was only done in a few tests where the test was using SHOW STATUS
    for both local and global variables.
- Uptime_since_flush_status is now always provided, independent if
  ENABLED_PROFILING is enabled when compiling MariaDB.
- @@global.Uptime_since_flush_status is reset on FLUSH GLOBAL STATUS
  and @@session.Uptime_since_flush_status is reset on FLUSH SESSION STATUS.
- When connected, @@session.Uptime_since_flush_status is set to 0.
2024-05-27 12:39:03 +02:00
Monty
24c57165d5 ALTER TABLE and replication should convert old row_end timestamps to new timestamp range
MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range

- Added --update-history option to mariadb-dump to change 2038
  row_end timestamp to 2106.
- Updated ALTER TABLE ... to convert old row_end timestamps to
  2106 timestamp for tables created before MariaDB 11.4.0.
- Fixed bug in CHECK TABLE where we wrongly suggested to USE REPAIR
  TABLE when ALTER TABLE...FORCE is needed.
- mariadb-check printed table names that where used with REPAIR TABLE but
  did not print table names used with ALTER TABLE or with name repair.
  Fixed by always printing a table that is fixed if --silent is not
  used.
- Added TABLE::vers_fix_old_timestamp() that will change max-timestamp
  for versioned tables when replication from a pre-11.4.0 server.

A few test cases changed. This is caused by:
- CHECK TABLE now prints 'Please do ALTER TABLE... instead of
  'Please do REPAIR TABLE' when there is a problem with the information
  in the .frm file (for example a very old frm file).
- mariadb-check now prints repaired table names.
- mariadb-check also now prints nicer error message in case ALTER TABLE
  is needed to repair a table.
2024-05-27 12:39:03 +02:00
Christian Gonzalez
4186fa72fb MDEV-28671 Enable var deprecation for mysqld help output
Currently there are mechanism to mark a system variable as
deprecated, but they are only used to print warning messages
when a deprecated variable is set.

Leverage the existing mechanisms in order to make the
deprecation information available at the --help output of mysqld by:

* Moving the deprecation information (i.e `deprecation_substitute`
  attribute) from the `sys_var` class into the `my_option` struct.
  As every `sys_var` contains its own `my_option` struct, the access
  to the deprecation information remains available to `sys_var`
  objects. `my_getotp` functions, which works directly with
  `my_option` structs, gain access to this information while building
  the --help output.

* For plugin variables, leverages the `PLUGIN_VAR_DEPRECATED` flag
  and set the `deprecation_substitute` attribute  accordingly when
  building the `my_option` objects.

* Change the `option_cmp` function to use the `deprecation_substitute`
  attribute instead of the name when sorting the options. This way
  deprecated options and the substitutes will be grouped together.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2024-05-27 12:39:02 +02:00
Oleksandr Byelkin
dd7d9d7fb1 Merge branch '11.4' into 11.5 2024-05-23 17:01:43 +02:00
Sergei Golubchik
19f7edf420 mysqltest: support MARIADB_OPT_RESTRICTED_AUTH
C/C 3.4 disables mysql_old_password by default, so

add an option for the `connect` command to support specifying
allowed authentication plugins (MARIADB_OPT_RESTRICTED_AUTH).

use it to enable mysql_old_password when needed for testing
2024-05-21 19:40:03 +02:00
Sergei Golubchik
1588e61566 small cleanup: mysqltest 2024-05-21 19:39:54 +02:00
Oleksandr Byelkin
99b370e023 Merge branch '11.2' into 11.4 2024-05-21 19:38:51 +02:00
Sergei Golubchik
bf5da43e50 Merge branch '11.1' into 11.2 2024-05-13 10:00:26 +02:00
Sergei Golubchik
f0a5412037 Merge branch '11.0' into 11.1 2024-05-13 09:52:30 +02:00
Sergei Golubchik
f9807aadef Merge branch '10.11' into 11.0 2024-05-12 12:18:28 +02:00
Sergei Golubchik
a6b2f820e0 Merge branch '10.6' into 10.11 2024-05-10 20:02:18 +02:00
Sergei Golubchik
7b53672c63 Merge branch '10.5' into 10.6 2024-05-08 20:06:00 +02:00
Sergei Golubchik
13663cb5c4 MDEV-33727 mariadb-dump trusts the server and does not validate the data
safety first - tell mariadb client not to execute dangerous
cli commands, they cannot be present in the dump anyway.

wrapping the command in /*!999999 ..... */ guarantees that
if a non-mariadb-cli client loads the dump and sends it to the
server - the server will ignore the command it doesn't understand
2024-05-06 17:16:10 +02:00
Sergei Golubchik
2025597c0b MDEV-21778 Disable system commands in mysql/mariadb client
mysql --sandbox

disables system (\!), tee (\T), pager with an argument(\P foo), source (\.)

does *not* disable edit (\e). Use EDITOR=/bin/false to disable
or, for example, EDITOR=rnano for something more useful

does *not* disable pager (\P) without an argument. Use
PAGER=cat or, for example PAGER=less LESSSECURE=1 for something
more useful

using a disabled command is an error, which can be ignored with --force

Also, a "sandbox" command (\-) - enables the sandbox mode until EOF
(current file or the session, if interactive)
2024-05-06 17:16:10 +02:00
Sergei Golubchik
83aedeacc4 cleanup: remove __attribute__((unused)) from mysql.cc
the client is C++, use a much more concise C++ syntax

as a bonus, arguments that are used, are no longer marked "unused"
2024-05-06 17:16:10 +02:00
Sergei Golubchik
994a0de96b cleanup: reorder mysql help output 2024-05-06 17:16:10 +02:00
Sergei Golubchik
98b157fdee cleanup: unused OPT_xxx and client_priv.h 2024-05-06 17:16:10 +02:00
Vladislav Vaintroub
029e2a5fd9 MDEV-33876 CMake, zlib - use names compatible with official FindZLIB.cmake
- ZLIB_LIBRARIES, not ZLIB_LIBRARY
- ZLIB_INCLUDE_DIRS, not ZLIB_INCLUDE_DIR

For building libmariadb, ZLIB_LIBRARY/ZLIB_INCLUDE_DIR are still defined
This workaround will be removed later.
2024-05-03 21:48:47 +02:00
Sergei Golubchik
0aae11ac28 Merge branch '10.6' into 10.11 2024-04-30 16:56:49 +02:00
Sergei Golubchik
c1f3eff53f Merge branch '10.5' into 10.6 2024-04-29 10:08:58 +02:00
Hugo Wen
3d41747625 MDEV-33574 Improve mysqlbinlog error message
Previously, when running mysqlbinlog without providing a binlog file, it
would print the entire help text, which was very verbose and made it
difficult to identify the actual issue.

Now change the behavior to print a more concise error message instead:

    "ERROR: Please provide the log file(s). Run with '--help' for usage instructions."

This makes the error output more user-friendly and easier to understand,
especially when running the tool in scripts or automated processes.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2024-04-26 12:27:28 +01:00
Oleksandr Byelkin
c9b1ebee2f Merge branch '10.6' into 10.11 2024-04-26 08:02:49 +02:00
Monty
0ccdf54b64 Check and remove high stack usage
I checked all stack overflow potential problems found with
gcc -Wstack-usage=16384
and
clang -Wframe-larger-than=16384 -no-inline

Fixes:
Added '#pragma clang diagnostic ignored "-Wframe-larger-than="'
  to a lot of function to where stack usage large but resonable.
- Added stack check warnings to BUILD scrips when using clang and debug.

Function changed to use malloc instead allocating things on stack:
- read_bootstrap_query() now allocates line_buffer (20000 bytes) with
  malloc() instead of using stack. This has a small performance impact
  but this is not releant for bootstrap.
- mroonga grn_select() used 65856 bytes on stack. Changed it to use
  malloc().
- Wsrep_schema::replay_transaction() and
  Wsrep_schema::recover_sr_transactions().
- Connect zipOpen3()

Not fixed:
- mroonga/vendor/groonga/lib/expr.c grn_proc_call() uses
  43712 byte on stack.  However this is not easy to fix as the stack
  used is caused by a lot of code generated by defines.
- Most changes in mroonga/groonga where only adding of pragmas to disable
  stack warnings.
- rocksdb/options/options_helper.cc uses 20288 of stack space.
  (no reason to fix except to get rid of the compiler warning)
- Causes using alloca() where the allocation size is resonable.
- An issue in libmariadb (reported to connectors).
2024-04-23 14:12:31 +03:00
Markus Staab
361b790392 Remove unnecessary whitespace in mysqldump 2024-04-22 21:41:29 +01:00
Sergei Golubchik
018d537ec1 Merge branch '10.6' into 10.11 2024-04-22 15:23:10 +02:00
Sergei Golubchik
3f9182126c mysqltest: support MARIADB_OPT_RESTRICTED_AUTH
C/C 3.4 disables mysql_old_password by default, so

add an option for the `connect` command to support specifying
allowed authentication plugins (MARIADB_OPT_RESTRICTED_AUTH).

use it to enable mysql_old_password when needed for testing
2024-04-22 14:59:05 +02:00
Sergei Golubchik
5d74e43914 small cleanup: mysqltest 2024-04-22 14:59:00 +02:00