MDEV-27106 added REMOTE_TABLE, REMOTE_DATABASE, REMOTE_SERVER spider
table options. In this commit, we add all remaining options for table
params that are not marked to be deprecated.
All these options are parsed as strings from sql statements and have
string values at the sql level, so that we can determine whether it is
specified by checking its nullness.
The string values are further parsed by Spider into their actual types
in the SPIDER_SHARE, including string list, bounded nonnegative int,
bounded nonnegative int list, nonnegative longlong, boolean, and key
hints. Except for string lists, all other types are validated during
this parsing process.
Most of the options are backward compatible, i.e. they accept any
values that is accepted by there corresponding param parser. The only
exception is the index hint IDX which corresponds to the idxNNN param
name. For example,
'idx000 "f PRIMARY", idx001 "u k1"'
translates to
IDX="f PRIMARY u k1".
We include a test with all options specified, and tests involving
spider table options of all actual types.
Any table options, if present, will cause comments to be ignored with
a warning. The warning can be disabled by setting a new spider
global/session system variable spider_suppress_comment_ignored_warning
to 1.
Another global/session variable introduced is spider_ignore_comments,
which if set to 1, will cause COMMENT and CONNECTION strings to be
ignored unconditionally, whether or not table options are specified.
- Introduced the variable "innodb_truncate_temporary_tablespace_now"
to shrink the temporary tablespace.
Steps for shrinking the temporary tablespace:
1) Find the last used extent in temporary tablespace
by iterating through the BITMAP in extent descriptor pages
2) If the last used extent is lesser than user specified size
then set desired target size to user specified size
3) Store the page contents of "to be modified" extent
descriptor pages, latches the "to be modified" extent
descriptor pages and check for buffer pool memory availability
4) Update the FSP_SIZE and FSP_FREE_LIMIT in header page
5) Remove the "to be truncated" pages from FSP_FREE and
FSP_FREE_FRAG list
6) Reset the bitmap in the last descriptor pages for the
"to be truncated" pages.
7) Clear the freed range in temporary tablespace which
are to be truncated.
8) Evict the "to be truncated" temporary tablespace pages
from LRU list.
9) In case of multiple files, calculate the truncated last
file size and do truncation in last file
10) Commit the mini-transaction for shrinking the tablespace
Like sql_mode, we factor out of ON_CHECK function for export, to be
used in get_options() during server startup, for validation of
--redirect_url value.
Adding a global/session var `redirect_url' of string type. The initial
value is empty. Can be supplied in mysqld with --redirect-url or set
in --init-connect. A valid redirect_url should be of the format
{mysql,mariadb}://host[:port]
where <host> is an arbitrary string not containing colons, and <port>
is a number between 0 and 65535 inclusive.
The variable will be used by the server to notify clients that they
should connect to another server, specified by the value of the
variable, if not empty.
The notification is done by the inclusion of the variable in
session_track_system_variable.
Merge sys_var_charptr with sys_var_charptr_base, as well as merge
Sys_var_session_lexstring into Sys_var_lexstring. Also refactored
update methods of sys_var_charptr accordingly.
Because the class is more generic, session_update() calls
sys_var_charptr::session_update() which does not assume a buffer field
associated with THD, but instead call strdup/free, we get rid of
THD::default_master_connection_buff accordingly. This also makes THD
smaller by ~192 bytes, and there can be many thousands of concurrent
THDs.
New Feature:
============
This patch extends the START SLAVE UNTIL command with options
SQL_BEFORE_GTIDS and SQL_AFTER_GTIDS to allow user control of
whether the replica stops before or after a provided GTID state. Its
syntax is:
START SLAVE UNTIL (SQL_BEFORE_GTIDS|SQL_AFTER_GTIDS)=”<gtid_list>”
When providing SQL_BEFORE_GTIDS=”<gtid_list>”, for each domain
specified in the gtid_list, the replica will execute transactions up
to the GTID found, and immediately stop processing events in that
domain (without executing the transaction of the specified GTID).
Once all domains have stopped, the replica will stop. Events
originating from domains that are not specified in the list are not
replicated.
START SLAVE UNTIL SQL_AFTER_GTIDS=”<gtid_list>” is an alias to the
default behavior of START SLAVE UNTIL master_gtid_pos=”<gtid_list>”.
That is, the replica will only execute transactions originating from
domain ids provided in the list, and will stop once all transactions
provided in the UNTIL list have all been executed.
Example:
=========
If a primary server has a binary log consisting of the following GTIDs:
0-1-1
1-1-1
0-1-2
1-1-2
0-1-3
1-1-3
If a fresh replica (i.e. one with an empty GTID position,
@@gtid_slave_pos='') is started with SQL_BEFORE_GTIDS, i.e.
START SLAVE UNTIL SQL_BEFORE_GTIDS=”1-1-2”
The resulting gtid_slave_pos of the replica will be “1-1-1”.
This is because the replica will execute only events from domain 1
until it sees the transaction with sequence number 2, and
immediately stop without executing it.
If the replica is started with SQL_AFTER_GTIDS, i.e.
START SLAVE UNTIL SQL_AFTER_GTIDS=”1-1-2”
then the resulting gtid_slave_pos of the replica will be “1-1-2”.
This is because it will only execute events from domain 1 until it
has executed the provided GTID.
Reviewed By:
============
Kristian Nielson <knielsen@knielsen-hq.org>
utf8mb4_general_ci has been outdated for a while and contained loosely
standardized collations.
UCA-14.0.0 has a more defined collation with multiple benefit that new
users may not immediately consider, or may assume to be default.
By defining default collation for utf8mb4 to be uc1400_ai_ci newly
created tables will have a modern standard collation.
In MemorySanitizer builds of 10.10 and 10.11, we would rather often
have the assertion fail in innodb_init() during mariadb-backup --prepare.
The assertion could also fail during InnoDB startup, but less often.
Before commit 685d958e38 in 10.8 the
log file cleanup after a successfully applied backup is different,
and the os_aio_pending_writes() assertion is in srv0start.cc.
IORequest::write_complete(): Invoke node->complete_write() before
releasing the page latch, so that a log checkpoint that is about to
execute concurrently will not miss a fdatasync() or fsync() on the
file, in case this was the first write since the last such call.
create_log_file(), srv_start(): Replace the debug assertion with
a debug check. For all intents and purposes, all writes could have
been completed but some write_io_callback() may not have invoked
io_slots::release() yet.
This was introduced in last merge with 10.6
The reason is that 10.6 does not need anything special to free histograms
as everything is allocated on a memroot.
In 10.10 histograms is using the vector class, which has some problems:
- No automatic free
- No memory usage accounting
(we should at some point remove vector usage because of the above problem)
Fixed by expliciting freeing histograms when freeing TABLE_STATISTICS
objects.
- InnoDB fails to check the overflow buffer while applying
the operation to the table that was rebuilt. This is caused
by commit 3cef4f8f0f (MDEV-515).
Fixed missing initialization of Alter_info()
This could cause crashes in some create table like scenarios
where some generated indexes where automatically dropped.
I also added a test that we do not try to drop from index_stats for
temporary tables.
The intentention was always to not create histograms for single value
unique keys (as histograms is not useful in this case), but because of
a bug in the code this was still done.
The changes in the test cases was mainly because hist_size is now NULL
for these kind of columns.
The MDEV-29693 conflict resolution is from Monty, as well as is
a bug fix where ANALYZE TABLE wrongly built histograms for
single-column PRIMARY KEY.
Also includes a fix for safe_malloc error reporting.
Other things:
- Copied main.log_slow from 10.4 to avoid mtr issue
Disabled test:
- spider/bugfix.mdev_27239 because we started to get
+Error 1429 Unable to connect to foreign data source: localhost
-Error 1158 Got an error reading communication packets
- main.delayed
- Bug#54332 Deadlock with two connections doing LOCK TABLE+INSERT DELAYED
This part is disabled for now as it fails randomly with different
warnings/errors (no corruption).
The error is caused by MDEV-30165 fix with the following commit:
d13a57ae81
There is logical error in lock_release_on_prepare_try():
if (supremum_bit)
lock_rec_unlock_supremum(*cell, lock);
else
lock_rec_dequeue_from_page(lock, false);
Because there can be other bits set in the lock's bitmap, and the lock
type can be suitable for releasing criteria, but the above logic
releases only supremum bit of the lock.
The fix is to release lock if it suits for releasing criteria and unlock
supremum if supremum is locked otherwise.
Tere is also the test for the case, which was reported by QA team. I
placed it in a separate files, because it requires debug build.
Reviewed by: Marko Mäkelä
- InnoDB fails to report the error when encryption configuration
wasn't passed. This patch addresses the issue by adding
the error while loading the tablespace and deferring the
tablespace creation.
There are many filesystem related errors that can occur with
MariaBackup. These already outputed to stderr with a good description of
the error. Many of these are permission or resource (file descriptor)
limits where the assertion and resulting core crash doesn't offer
developers anything more than the log message. To the user, assertions
and core crashes come across as poor error handling.
As such we return an error and handle this all the way up the stack.
Before starting to go over the format string, prepare the current time
zone information incase '%z' or '%Z' is encountered.
This information can be obtained as given below:
A) If timezone is not set ( meaning we are working with system timezone):
Get the MYSQL_TIME representation for current time and GMT time using
current thread variable for timezone and timezone variable for UTC
respectively. This MYSQL_TIME variable will be used to calculate time
difference. Also convert current time in second to tm structure to
get system timezone information.
B) If timezone is set as offset:
Get timezone information using current timezone information and store
in appropriate variable.
C) If timezone is set as some place (example: Europe/Berlin)
Get timezone information by searching the timezone. During internal
timezone search, information like timeoffset from UTC and abbrevation
is stored in another relevant structure. Hence use the same information.
log_t::create(), log_t::attach(): Return whether the initialisation
succeeded. It may fail if too large an innodb_log_buffer_size is specified.
recv_sys_t::close_files(): Actually close the data files so that the
test mariabackup.huge_lsn,strict_crc32 will not fail on Microsoft Windows
when renaming ib_logfile101 due to a leaked file handle of ib_logfile0.
recv_sys_t::find_checkpoint(): Register recv_sys.files[0] as OS_FILE_CLOSED
because the file handle has already been attached to log_sys.log and
we do not want to close the file twice.
recv_sys_t::read(): Access the first log file via log_sys.log.
This is a port of commit 6e9b421f77
adapted to commit 685d958e38 (MDEV-14425).
The test case is omitted, because it would fail to fail when the log
is stored in persistent memory (or "fake PMEM" on Linux /dev/shm).
In case the option require_secure_transport is on the user can't
establish a secure ssl connection over TCP protocol. Inability to set up
a ssl session over TCP was caused by the fact that a type of client's
connection was checked before ssl handshake performed (ssl handshake
happens at the function acl_authenticate()). At that moment vio type has
the value VIO_TYPE_TCPIP for client connection that uses TCP transport.
In result, checking for allowable vio type for fails despite the fact
that SSL session being established. To fix the issue move checking of
vio type for allowable values inside the function
parse_client_handshake_packet()
right after client's capabilities discovered that SSL is not requested
by the client.
The system variable spider_disable_group_by_handler, if on, will
disable the spider group by handler (gbh), and such disablement serves
as workaround to bugs caused by gbh, labelled with spider-gbh on jira,
including MDEV-26247, MDEV-28998, MDEV-29163, MDEV-30392, MDEV-31645.
Tests for these tickets are added accordingly with the workaround in
place.
In commit 5ea5291 @sanja-byelkin for unknown reason switched the file mode
for 3 Galera tzinfo related test files from 644 -> 755. This exists only
from branch 10.6 onward:
$ git checkout 10.5
$ find mysql-test -executable -name *.test -or -executable -name *.result
(no results)
$ git checkout 10.6
$ find mysql-test -executable -name *.test -or -executable -name *.result
mysql-test/suite/galera/t/mysql_tzmysql-test/suite/galera/t/mysql_tzinfo_to_sql.test
mysql-test/suite/galera/t/mariadb_tzinfo_to_sql.test
mysql-test/suite/galera/r/mariadb_tzinfo_to_sql.resultinfo_to_sql.test
mysql-test/suite/galera/t/mariadb_tzinfo_to_sql.test
mysql-test/suite/galera/r/mariadb_tzinfo_to_sql.result
No test file nor test result file should be executable, so run chmod -x
on them.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
copy_back(): Also copy the dummy empty ib_logfile0 so that
MariaDB Server 10.8 or later can be started after
--copy-back or --move-back.
Thanks to Daniel Black for reporting this.
This is a 10.5 version of
commit ebf3649259
Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)
Two things are changed:
First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595
Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.
Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.
A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.
