Fix various typos, in comments and DEBUG statements, and code changes
are non-functional.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
- FLUSH GLOBAL STATUS now resets most global_status_vars.
At this stage, this is mainly to be used for testing.
- FLUSH SESSION STATUS added as an alias for FLUSH STATUS.
- FLUSH STATUS does not require any privilege (before required RELOAD).
- FLUSH GLOBAL STATUS requires RELOAD privilege.
- All global status reset moved to FLUSH GLOBAL STATUS.
- Replication semisync status variables are now reset by
FLUSH GLOBAL STATUS.
- In test cases, the only changes are:
- Replace FLUSH STATUS with FLUSH GLOBAL STATUS
- Replace FLUSH STATUS with FLUSH STATUS; FLUSH GLOBAL STATUS.
This was only done in a few tests where the test was using SHOW STATUS
for both local and global variables.
- Uptime_since_flush_status is now always provided, independent if
ENABLED_PROFILING is enabled when compiling MariaDB.
- @@global.Uptime_since_flush_status is reset on FLUSH GLOBAL STATUS
and @@session.Uptime_since_flush_status is reset on FLUSH SESSION STATUS.
- When connected, @@session.Uptime_since_flush_status is set to 0.
MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range
- Added --update-history option to mariadb-dump to change 2038
row_end timestamp to 2106.
- Updated ALTER TABLE ... to convert old row_end timestamps to
2106 timestamp for tables created before MariaDB 11.4.0.
- Fixed bug in CHECK TABLE where we wrongly suggested to USE REPAIR
TABLE when ALTER TABLE...FORCE is needed.
- mariadb-check printed table names that where used with REPAIR TABLE but
did not print table names used with ALTER TABLE or with name repair.
Fixed by always printing a table that is fixed if --silent is not
used.
- Added TABLE::vers_fix_old_timestamp() that will change max-timestamp
for versioned tables when replication from a pre-11.4.0 server.
A few test cases changed. This is caused by:
- CHECK TABLE now prints 'Please do ALTER TABLE... instead of
'Please do REPAIR TABLE' when there is a problem with the information
in the .frm file (for example a very old frm file).
- mariadb-check now prints repaired table names.
- mariadb-check also now prints nicer error message in case ALTER TABLE
is needed to repair a table.
Currently there are mechanism to mark a system variable as
deprecated, but they are only used to print warning messages
when a deprecated variable is set.
Leverage the existing mechanisms in order to make the
deprecation information available at the --help output of mysqld by:
* Moving the deprecation information (i.e `deprecation_substitute`
attribute) from the `sys_var` class into the `my_option` struct.
As every `sys_var` contains its own `my_option` struct, the access
to the deprecation information remains available to `sys_var`
objects. `my_getotp` functions, which works directly with
`my_option` structs, gain access to this information while building
the --help output.
* For plugin variables, leverages the `PLUGIN_VAR_DEPRECATED` flag
and set the `deprecation_substitute` attribute accordingly when
building the `my_option` objects.
* Change the `option_cmp` function to use the `deprecation_substitute`
attribute instead of the name when sorting the options. This way
deprecated options and the substitutes will be grouped together.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
C/C 3.4 disables mysql_old_password by default, so
add an option for the `connect` command to support specifying
allowed authentication plugins (MARIADB_OPT_RESTRICTED_AUTH).
use it to enable mysql_old_password when needed for testing
safety first - tell mariadb client not to execute dangerous
cli commands, they cannot be present in the dump anyway.
wrapping the command in /*!999999 ..... */ guarantees that
if a non-mariadb-cli client loads the dump and sends it to the
server - the server will ignore the command it doesn't understand
mysql --sandbox
disables system (\!), tee (\T), pager with an argument(\P foo), source (\.)
does *not* disable edit (\e). Use EDITOR=/bin/false to disable
or, for example, EDITOR=rnano for something more useful
does *not* disable pager (\P) without an argument. Use
PAGER=cat or, for example PAGER=less LESSSECURE=1 for something
more useful
using a disabled command is an error, which can be ignored with --force
Also, a "sandbox" command (\-) - enables the sandbox mode until EOF
(current file or the session, if interactive)
- ZLIB_LIBRARIES, not ZLIB_LIBRARY
- ZLIB_INCLUDE_DIRS, not ZLIB_INCLUDE_DIR
For building libmariadb, ZLIB_LIBRARY/ZLIB_INCLUDE_DIR are still defined
This workaround will be removed later.
Previously, when running mysqlbinlog without providing a binlog file, it
would print the entire help text, which was very verbose and made it
difficult to identify the actual issue.
Now change the behavior to print a more concise error message instead:
"ERROR: Please provide the log file(s). Run with '--help' for usage instructions."
This makes the error output more user-friendly and easier to understand,
especially when running the tool in scripts or automated processes.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
I checked all stack overflow potential problems found with
gcc -Wstack-usage=16384
and
clang -Wframe-larger-than=16384 -no-inline
Fixes:
Added '#pragma clang diagnostic ignored "-Wframe-larger-than="'
to a lot of function to where stack usage large but resonable.
- Added stack check warnings to BUILD scrips when using clang and debug.
Function changed to use malloc instead allocating things on stack:
- read_bootstrap_query() now allocates line_buffer (20000 bytes) with
malloc() instead of using stack. This has a small performance impact
but this is not releant for bootstrap.
- mroonga grn_select() used 65856 bytes on stack. Changed it to use
malloc().
- Wsrep_schema::replay_transaction() and
Wsrep_schema::recover_sr_transactions().
- Connect zipOpen3()
Not fixed:
- mroonga/vendor/groonga/lib/expr.c grn_proc_call() uses
43712 byte on stack. However this is not easy to fix as the stack
used is caused by a lot of code generated by defines.
- Most changes in mroonga/groonga where only adding of pragmas to disable
stack warnings.
- rocksdb/options/options_helper.cc uses 20288 of stack space.
(no reason to fix except to get rid of the compiler warning)
- Causes using alloca() where the allocation size is resonable.
- An issue in libmariadb (reported to connectors).
C/C 3.4 disables mysql_old_password by default, so
add an option for the `connect` command to support specifying
allowed authentication plugins (MARIADB_OPT_RESTRICTED_AUTH).
use it to enable mysql_old_password when needed for testing
New option works just like --tab, wrt output (sql file for table definition
and tab-separated for data, same options, e.g --parallel)
Compared to --tab it allows --databases and --all-databases.
When --dir is used , it creates directory structure in the output directory,
pointed to by --dir. For every database to be dumped, there will be a
directory with database name.
All options that --tab supports, are also supported by --dir, in particular
--parallel
This patch also fixes:
MDEV-33050 Build-in schemas like oracle_schema are accent insensitive
MDEV-33084 LASTVAL(t1) and LASTVAL(T1) do not work well with lower-case-table-names=0
MDEV-33085 Tables T1 and t1 do not work well with ENGINE=CSV and lower-case-table-names=0
MDEV-33086 SHOW OPEN TABLES IN DB1 -- is case insensitive with lower-case-table-names=0
MDEV-33088 Cannot create triggers in the database `MYSQL`
MDEV-33103 LOCK TABLE t1 AS t2 -- alias is not case sensitive with lower-case-table-names=0
MDEV-33109 DROP DATABASE MYSQL -- does not drop SP with lower-case-table-names=0
MDEV-33110 HANDLER commands are case insensitive with lower-case-table-names=0
MDEV-33119 User is case insensitive in INFORMATION_SCHEMA.VIEWS
MDEV-33120 System log table names are case insensitive with lower-cast-table-names=0
- Removing the virtual function strnncoll() from MY_COLLATION_HANDLER
- Adding a wrapper function CHARSET_INFO::streq(), to compare
two strings for equality. For now it calls strnncoll() internally.
In the future it will turn into a virtual function.
- Adding new accent sensitive case insensitive collations:
- utf8mb4_general1400_as_ci
- utf8mb3_general1400_as_ci
They implement accent sensitive case insensitive comparison.
The weight of a character is equal to the code point of its
upper case variant. These collations use Unicode-14.0.0 casefolding data.
The result of
my_charset_utf8mb3_general1400_as_ci.strcoll()
is very close to the former
my_charset_utf8mb3_general_ci.strcasecmp()
There is only a difference in a couple dozen rare characters, because:
- the switch from "tolower" to "toupper" comparison, to make
utf8mb3_general1400_as_ci closer to utf8mb3_general_ci
- the switch from Unicode-3.0.0 to Unicode-14.0.0
This difference should be tolarable. See the list of affected
characters in the MDEV description.
Note, utf8mb4_general1400_as_ci correctly handles non-BMP characters!
Unlike utf8mb4_general_ci, it does not treat all BMP characters
as equal.
- Adding classes representing names of the file based database objects:
Lex_ident_db
Lex_ident_table
Lex_ident_trigger
Their comparison collation depends on the underlying
file system case sensitivity and on --lower-case-table-names
and can be either my_charset_bin or my_charset_utf8mb3_general1400_as_ci.
- Adding classes representing names of other database objects,
whose names have case insensitive comparison style,
using my_charset_utf8mb3_general1400_as_ci:
Lex_ident_column
Lex_ident_sys_var
Lex_ident_user_var
Lex_ident_sp_var
Lex_ident_ps
Lex_ident_i_s_table
Lex_ident_window
Lex_ident_func
Lex_ident_partition
Lex_ident_with_element
Lex_ident_rpl_filter
Lex_ident_master_info
Lex_ident_host
Lex_ident_locale
Lex_ident_plugin
Lex_ident_engine
Lex_ident_server
Lex_ident_savepoint
Lex_ident_charset
engine_option_value::Name
- All the mentioned Lex_ident_xxx classes implement a method streq():
if (ident1.streq(ident2))
do_equal();
This method works as a wrapper for CHARSET_INFO::streq().
- Changing a lot of "LEX_CSTRING name" to "Lex_ident_xxx name"
in class members and in function/method parameters.
- Replacing all calls like
system_charset_info->coll->strcasecmp(ident1, ident2)
to
ident1.streq(ident2)
- Taking advantage of the c++11 user defined literal operator
for LEX_CSTRING (see m_strings.h) and Lex_ident_xxx (see lex_ident.h)
data types. Use example:
const Lex_ident_column primary_key_name= "PRIMARY"_Lex_ident_column;
is now a shorter version of:
const Lex_ident_column primary_key_name=
Lex_ident_column({STRING_WITH_LEN("PRIMARY")});
On Windows systems, occurrences of ERROR_SHARING_VIOLATION due to
conflicting share modes between processes accessing the same file can
result in CreateFile failures.
mysys' my_open() already incorporates a workaround by implementing
wait/retry logic on Windows.
But this does not help if files are opened using shell redirection like
mysqltest traditionally did it, i.e via
--echo exec "some text" > output_file
In such cases, it is cmd.exe, that opens the output_file, and it
won't do any sharing-violation retries.
This commit addresses the issue by introducing a new built-in command,
'write_line', in mysqltest. This new command serves as a brief alternative
to 'write_file', with a single line output, that also resolves variables
like "exec" would.
Internally, this command will use my_open(), and therefore retry-on-error
logic.
Hopefully this will eliminate the very sporadic "can't open file because
it is used by another process" error on CI.
Values of all session tracking system variables will be sent in the
first ok packet upon connection after successful authentication.
Also updated mtr to print session track info on connection (h/t Sergei
Golubchik) so that we can write mtr tests for this change.
We do this by checking server status. By doing this we avoid printing
session tracking info from previous (but not the last) statement.
The change is from Sergei Golubchik
This commit fixes the following issues:
- memory leak checking enabled for mysqltest. This cover all cases except
calls to 'die()' that only happens in case of internal failures in
mysqltest. die() is not called anymore in the result files differs.
- One can now run mtr --embedded without failures (this crashed or hang
before)
- cleanup_and_exit() has a new parameter that indicates that it is called
from die(), in which case we should not do memory leak checks. We now
always call cleanup_and_exit() instead of exit() to be able to free up
memory and discover memory leaks.
- Lots of new assert to catch error conditions
- More DBUG statements.
- Fixed that all results are freed in mysqltest (Fixed a memory leak in
mysqltest when using prepared statements).
- Fixed race condition in do_stmt_close() that caused embedded server
to not free memory. (Memory leak in mysqltest with embedded server).
- Fixed two memory leaks in embedded server when using prepared statements.
These memory leaks caused timeout hangs in mtr when server was compiled
with safemalloc. This issue was not noticed (except as timeouts) as
memory report checking was done but output of it was disabled.
This commit fixes the following issues:
- memory leak checking enabled for mysqltest. This cover all cases except
calls to 'die()' that only happens in case of internal failures in
mysqltest. die() is not called anymore in the result files differs.
- One can now run mtr --embedded without failures (this crashed or hang
before)
- cleanup_and_exit() has a new parameter that indicates that it is called
from die(), in which case we should not do memory leak checks. We now
always call cleanup_and_exit() instead of exit() to be able to free up
memory and discover memory leaks.
- Lots of new assert to catch error conditions
- More DBUG statements.
- Fixed that all results are freed in mysqltest (Fixed a memory leak in
mysqltest when using prepared statements).
- Fixed race condition in do_stmt_close() that caused embedded server
to not free memory. (Memory leak in mysqltest with embedded server).
- Fixed two memory leaks in embedded server when using prepared statements.
These memory leaks caused timeout hangs in mtr when server was compiled
with safemalloc. This issue was not noticed (except as timeouts) as
memory report checking was done but output of it was disabled.
This commit fixes the following issues:
- memory leak checking enabled for mysqltest. This cover all cases except
calls to 'die()' that only happens in case of internal failures in
mysqltest. die() is not called anymore in the result files differs.
- One can now run mtr --embedded without failures (this crashed or hang
before)
- cleanup_and_exit() has a new parameter that indicates that it is called
from die(), in which case we should not do memory leak checks. We now
always call cleanup_and_exit() instead of exit() to be able to free up
memory and discover memory leaks.
- Lots of new assert to catch error conditions
- More DBUG statements.
- Fixed that all results are freed in mysqltest (Fixed a memory leak in
mysqltest when using prepared statements).
- Fixed race condition in do_stmt_close() that caused embedded server
to not free memory. (Memory leak in mysqltest with embedded server).
- Fixed two memory leaks in embedded server when using prepared statements.
These memory leaks caused timeout hangs in mtr when server was compiled
with safemalloc. This issue was not noticed (except as timeouts) as
memory report checking was done but output of it was disabled.
In case of failure, the something like the following is now printed:
Slave position: file: binary.000004 position: 3647
Master position: file: binary.000004 position: 3647
Recording both is useful on a replication relay when the backup
can be used to replace the server, or ack as a new replica to the
server.
If an option=2, commented is selected, allow the alternate option
to exist.
This still disables --dump-slave=1 --master-data=1 as having the
a CHANGE MASTER TO and START SLAVE on different positions would be
confusing and dangerious to the try to execute the output. The
previous behaviour of silently disabling --master-data occurs in
this case.
The commented code related to --dump-slave/--master-data is greatly
expanded for human consumption.
A redundant opt_slave_data= 0 was removed from get_opts. If
--dump-slave=1 or 2, then the only possible value of --master-data
is a valid one.
Re-order to preference gtid based replication.
Based of code from Elena Stepanova.
Review by: Brandon Nesterenko and Anel Husakovic
* --ssl-verify-server-cert was not enabled explicitly, and
* CA was not specified, and
* fingerprint was not specified, and
* protocol is TCP, and
* no password was provided
insecure passwordless logins are common in test environment, let's
not break them. practically, it hardly makes sense to have strong
MitM protection if an attacker can simply login without a password.
Covers mariadb, mariadb-admin, mariadb-binlog, mariadb-dump
implement --ssl-fp and --ssl-fplist for all clients.
--ssl-fp takes one certificate fingerprint, for example,
00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33
--ssl-fplist takes a path to a file with one fingerprint per line.
if the server's certificate fingerprint matches ssl-fp or is found
in the file - the certificate is considered verified.
If the fingerprint is specified but doesn't match - the connection
is aborted independently from the --ssl-verify-server-cert
This is done for symmetry with mariadb-dump, which does not use threads
but allows parallelism via --parallel
Traditional --use-threads can still be used, it is synonymous
with --parallel
Parallelism is achieved by using mysql_send_query on multiple connections
without waiting for results, and using IO multiplexing (poll/IOCP) to
wait for completions.
Refresh libmariadb to pick up CONC-676 (fixes for IOCP use with named pipe)
- make connect_to_db() return MYSQL*, we'll reuse the function for
connection pool.
- Remove variable 'mysql_connection', duplicated by variable 'mysql'
- do not attempt to start slave if connection did not succeed,#
and fix mysqldump.result
Testing exit code from popen(), comparing it with 1, and deciding that
perl.exe is not there, is a) wrong conclusion, and b) uninteresting,
because MTR always runs with perl, and with MTR_PERL set.
Background:
Recent change in 7af50e4df4 introduced
exit code 1 from perl snippet, that broke Windows CI. Do not want
to debug this ever again.
This patch introduces the following behaviour for Linux while
maintaining old behaviour for Windows:
Ctrl + C (sigint) clears the current buffer and redraws the prompt.
Ctrl-C no longer exits the client if no query is running.
Ctrl-C kills the current running query if there is one. If there is an
error communicating with the server while trying to issue a KILL QUERY,
the client exits. This is in line with the past behaviour of Ctrl-C.
On Linux Ctrl-D can be used to close the client.
On Windows Ctrl-C and Ctrl-BREAK still exits the client if no query is running.
Windows can also exit the client via \q<enter> or exit<enter>.
== Implementation details ==
The Linux implementation has two corner cases, based on which library is
used: libreadline or libedit, both are handled in code to achieve the
same user experience.
Additional code is taken from MySQL, ensuring there is identical
behaviour on Windows, to MySQL's mysql client implementation for other
CTRL- related signals.
* The CTRL_CLOSE, CTRL_LOGOFF, CTRL_SHUTDOWN will issue the equivalent
of CTRL-C and "end" the program. This ensures that the query is killed
when the client is closed by closing the terminal, logging off the
user or shutting down the system. The latter two signals are not sent
for interactive applications, but it handles the case when a user has
defined a service to use mysql client to issue a command. See
https://learn.microsoft.com/en-us/windows/console/handlerroutine
This patch is built on top of the initial work done by Anel Husakovic
<anel@mariadb.org>.
Closes#2815
What is shown in this case is the result from the following query:
select * from information_schema.processlist where id != connection_id()
This allows easy monitoring of for example MAX_MEMORY_USED
sets of the first and second execution. The results of the first
and second execution are compared only if result logging is enabled
Comparing two result sets is done as comparing two strings.
Added a new method 'read_stmt_results' to get a result set
after execution of prepare statement.
- Prevent opening of any user tables in case `upgrade-system-table`
option is used.
- Still there may be uninstalled data types in `mysql` system table so
allow it to perform.
- Closes PR #2790
- Reviewer: <daniel@mariadb.org>, <vicentiu@mariadb.org>
The MDEV-29693 conflict resolution is from Monty, as well as is
a bug fix where ANALYZE TABLE wrongly built histograms for
single-column PRIMARY KEY.
Also includes a fix for safe_malloc error reporting.
Other things:
- Copied main.log_slow from 10.4 to avoid mtr issue
Disabled test:
- spider/bugfix.mdev_27239 because we started to get
+Error 1429 Unable to connect to foreign data source: localhost
-Error 1158 Got an error reading communication packets
- main.delayed
- Bug#54332 Deadlock with two connections doing LOCK TABLE+INSERT DELAYED
This part is disabled for now as it fails randomly with different
warnings/errors (no corruption).
Flush stdout on finalizing of mysqldump/mysqlbinlog output
to avoid truncation.
The same patch has been applied to the mysqltest.cc code with
commit 34ff714b0d
Author: Magnus Svensson <msvensson@mysql.com>
Date: Fri Nov 14 11:06:56 2008 +0100
WL#4189 Make mysqltest flush log file at close if logfile is stdout
but not to mysqldump.c/mysqlbinlog.cc
mysqladmin's presumption about the cause of an error by looking
at the error code was presumptious. Server knows best, so pass
the error along.
Avoid returning -1 as a exit code, Linux makes this 255 and Windows
keeps this as -1.
Raise notes if indexes cannot be used:
- in case of data type or collation mismatch (diferent error messages).
- in case if a table field was replaced to something else
(e.g. Item_func_conv_charset) during a condition rewrite.
Added option to write warnings and notes to the slow query log for
slow queries.
New variables added/changed:
- note_verbosity, with is a set of the following options:
basic - All old notes
unusable_keys - Print warnings about keys that cannot be used
for select, delete or update.
explain - Print unusable_keys warnings for EXPLAIN querys.
The default is 'basic,explain'. This means that for old installations
the only notable new behavior is that one will get notes about
unusable keys when one does an EXPLAIN for a query. One can turn all
of all notes by either setting note_verbosity to "" or setting sql_notes=0.
- log_slow_verbosity has a new option 'warnings'. If this is set
then warnings and notes generated are printed in the slow query log
(up to log_slow_max_warnings times per statement).
- log_slow_max_warnings - Max number of warnings written to
slow query log.
Other things:
- One can now use =ALL for any 'set' variable to set all options at once.
For example using "note_verbosity=ALL" in a config file or
"SET @@note_verbosity=ALL' in SQL.
- mysqldump will in the future use @@note_verbosity=""' instead of
@sql_notes=0 to disable notes.
- Added "enum class Data_type_compatibility" and changing the return type
of all Field::can_optimize*() methods from "bool" to this new data type.
Reviewer & Co-author: Alexander Barkov <bar@mariadb.com>
- The code that prints out the notes comes mainly from Alexander
This patch adds for "--ps-protocol" second execution
of queries "SELECT".
Also in this patch it is added ability to disable/enable
(--disable_ps2_protocol/--enable_ps2_protocol) second
execution for "--ps-prototocol" in testcases.
This patch adds a way to override default collations
(or "character set collations") for desired character sets.
The SQL standard says:
> Each collation known in an SQL-environment is applicable to one
> or more character sets, and for each character set, one or more
> collations are applicable to it, one of which is associated with
> it as its character set collation.
In MariaDB, character set collations has been hard-coded so far,
e.g. utf8mb4_general_ci has been a hard-coded character set collation
for utf8mb4.
This patch allows to override (globally per server, or per session)
character set collations, so for example, uca1400_ai_ci can be set as a
character set collation for Unicode character sets
(instead of compiled xxx_general_ci).
The array of overridden character set collations is stored in a new
(session and global) system variable @@character_set_collations and
can be set as a comma separated list of charset=collation pairs, e.g.:
SET @@character_set_collations='utf8mb3=uca1400_ai_ci,utf8mb4=uca1400_ai_ci';
The variable is empty by default, which mean use the hard-coded
character set collations (e.g. utf8mb4_general_ci for utf8mb4).
The variable can also be set globally by passing to the server startup command
line, and/or in my.cnf.
The `--vertical` option does not appear to behave as documented and
research shows it never appears to have behaved as intended.
As this appears to be a completely unused feature, let's remove it.
