The --gtid-ignore-duplicates option was not working correctly with row-based
replication. When a row event was completed, but before committing, there
was a small window where another multi-source SQL thread could wrongly try
to re-execute the same transaction, without properly ignoring the duplicate
GTID. This would lead to duplicate key error or out-of-order GTID error or
similar.
Thanks to Matt Neth for reporting this and giving an easy way to reproduce
the issue.
Introduce a new dummy INFORMATION_SCHEMA.CHANGED_PAGE_BITMAPS table to XtraDB
with reset_table callback to allow FLUSH NO_WRITE_TO_BINLOG CHANGED_PAGE_BITMAPS
to be called from innobackupex.
Moved Apc_target::destroy(), Apc_target::enable() and Apc_targe::disable()
definitions to my_apc.h so that they can be inlined.
Apc_targe::disable() now calls Apc_target::process_apc_requests() only if
there're APC requests. This saves one pthread_mutex_lock() call.
Overhead change:
Apc_target::disable 0.04% -> out of radar
Apc_target::enable 0.03% -> out of radar
Apc_target::process_apc_requests 0.02% -> out of radar
pthread_mutex_lock 0.43% -> 0.42%
pthread_mutex_unlock 0.26% -> 0.25%
* Wait for aborted thd (victim) to release MDL locks
* Skip aborting an already aborted thd
* Defer setting OK status in case of CTAS
* Minor cosmetic changes
* Added a test case
Problem:
If we add a referential integrity constraint with a duplicate
name, an error occurs. The foreign key object would not have
been added to the dictionary cache. In the error path, there
is an attempt to remove this foreign key object. Since this
object is not there, the search returns a NULL result.
De-referencing the null object results in this crash.
Solution:
If the search to the foreign key object failed, then don't
attempt to access it.
rb#9309 approved by Marko.
Added new dynamic configuration variable innodb_buf_dump_status_frequency
to configure how often buffer pool dump status is printed in the logs.
A number between [0, 100] that tells how oftern buffer pool dump status
in percentages should be printed. E.g. 10 means that buffer pool dump
status is printed when every 10% of number of buffer pool pages are
dumped. Default is 0 (only start and end status is printed).
in ha_delete_table()
* only convert ENOENT and HA_ERR_NO_SUCH_TABLE to warnings
* only return real error codes (that is, not ENOENT and
not HA_ERR_NO_SUCH_TABLE)
* intercept HA_ERR_ROW_IS_REFERENCED to generate backward
compatible ER_ROW_IS_REFERENCED
in mysql_rm_table_no_locks()
* no special code to handle HA_ERR_ROW_IS_REFERENCED
* no special code to handle ENOENT and HA_ERR_NO_SUCH_TABLE
* return multi-table error ER_BAD_TABLE_ERROR <table list> only
when there were many errors, not when there were many
tables to drop (but only one table generated an error)
Follow-up patch to temporarily avoid a sporadic failure in the test
rpl.rpl_000011 due to MDEV-8301.
There is a window during thread exit where the global status is
counted incorrectly - the contribution for the exiting thread is
counted twice. The patch for MDEV-8294 made this window visible to the
test case rpl.rpl_000011, causing it to sporadically fail. Temporarily
silence this with a wait for the expected value; can be removed once
MDEV-8294 is fixed.
Add log_bin_index, log_bin_basename and relay_log_basename system
variables. Also, convert relay_log_index system variable to
NO_CMD_LINE and implement --relay-log-index as a command line
option.
Changing the error message to:
"...from type 'decimal(0,?)/*old*/' to type ' 'decimal(10,7)'..."
So it's now clear that the master data type is OLD decimal.
If the SET PASSWORD query doesn't have the password string,
the parsing of it can fail. It manifested first in MySQL 5.6 as
it started to hide password lines sent to the plugins.
Fixed by checking for that case.
Analysis: Problem is that both encrypted tables and compressed tables use
FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION to store
required metadata. Furhermore, for only compressed tables currently
code skips compression.
Fixes:
- Only encrypted pages store key_version to FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION,
no need to fix
- Only compressed pages store compression algorithm to FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION,
no need to fix as they have different page type FIL_PAGE_PAGE_COMPRESSED
- Compressed and encrypted pages now use a new page type FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED and
key_version is stored on FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION and compression
method is stored after FIL header similar way as compressed size, so that first
FIL_PAGE_COMPRESSED_SIZE is stored followed by FIL_PAGE_COMPRESSION_METHOD
- Fix buf_page_encrypt_before_write function to really compress pages if compression is enabled
- Fix buf_page_decrypt_after_read function to really decompress pages if compression is used
- Small style fixes
* mysqld_safe: Since wsrep_on variable is mandatory in 10.1, skip wsrep
position recovery if its OFF.
* mysqld: Remove "-wsrep" from server version
* mysqld: Remove wsrep patch version from @@version_comment
* mysqld: Introduce @@wsrep_patch_version
that was apparently lost in 20c23048:
commit 20c23048c1
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Sun May 17 14:14:16 2015 +0300
MDEV-8164: Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing
This also reverts 8635c4b4:
commit 8635c4b4e6
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Thu May 21 11:02:03 2015 +0300
Fix test failure.
When the slave processes the master restart format_description event,
parallel replication needs to complete any prior events before processing
the restart event (which closes temporary tables and such stuff).
This happens in wait_for_workers_idle(), however it was not waiting long
enough. The wait was using wait_for_prior_commit(), but at that points table
can still be open. This lead to assertion in this case.
So change wait_for_workers_idle() to wait until all worker threads have
reached finish_event_group(), at which point all tables should have been
closed.
Make sure that when we publish the crypt_data we access the
memory cache of the tablespace crypt_data. Make sure that
crypt_data is stored whenever it is really needed.
All this is not yet enough in my opinion because:
sql/encryption.cc has DBUG_ASSERT(scheme->type == 1) i.e.
crypt_data->type == CRYPT_SCHEME_1
However, for InnoDB point of view we have global crypt_data
for every tablespace. When we change variables on crypt_data
we take mutex. However, when we use crypt_data for
encryption/decryption we use pointer to this global
structure and no mutex to protect against changes on
crypt_data.
Tablespace encryption starts in fil_crypt_start_encrypting_space
from crypt_data that has crypt_data->type = CRYPT_SCHEME_UNENCRYPTED
and later we write page 0 CRYPT_SCHEME_1 and finally whe publish
that to memory cache.
Analysis: Problem was that tablespaces not encrypted might not have
crypt_data stored on disk.
Fixed by always creating crypt_data to memory cache of the tablespace.
MDEV-8138: strange results from encrypt-and-grep test
Analysis: crypt_data->type is not updated correctly on memory
cache. This caused problem with state tranfer on
encrypted => unencrypted => encrypted.
Fixed by updating memory cache of crypt_data->type correctly based on
current srv_encrypt_tables value to either CRYPT_SCHEME_1 or
CRYPT_SCHEME_UNENCRYPTED.
that is, after
commit 2300fe2e0e
Author: Sergei Golubchik <serg@mariadb.org>
Date: Wed May 13 21:57:24 2015 +0200
Identical key derivation code in XtraDB/InnoDB/Aria
* Extract it into the "encryption_scheme" service.
* Make these engines to use the service, remove duplicate code.
* Change MY_AES_xxx error codes, to return them safely
from encryption_scheme_encrypt/decrypt without conflicting
with ENCRYPTION_SCHEME_KEY_INVALID error
Analysis: Problem was that we did create crypt data for encrypted table but
this new crypt data was not written to page 0. Instead a default crypt data
was written to page 0 at table creation.
Fixed by explicitly writing new crypt data to page 0 after successfull
table creation.
fix encryption of the last partial block
* now really encrypt it, using key and iv
* support the case of very short plaintext (less than one block)
* recommend aes_ctr over aes_cbc, because the former
doesn't have problems with partial blocks
.. wsrep.binlog_format, wsrep.mdev_6832 fail in buildbot
Galera-3.9 logs an additional warning in the error log if
it fails to find gvwstate.dat file. Update wsrep/suite.pm.
In optimistic parallel replication, it is not safe to try to run a following
transaction in parallel with a DDL statement, and there is code to prevent
this.
However, the code was missing the case where the DDL is the very first event
after slave start. In this case, following transactions could run in
parallel with the DDL, which can cause the slave to hang or even corrupt
slave in unlucky cases.
AVOID DEADLOCK AFTER RESTORE
Analysis
--------
Accessing the restored NDB table in an active multi-statement
transaction was resulting in deadlock found error.
MySQL Server needs to discover metadata of NDB table from
data nodes after table is restored from backup. Metadata
discovery happens on the first access to restored table.
Current code mandates this statement to be the first one
in the transaction. This is because discover needs exclusive
metadata lock on the table. Lock upgrade at this point can
lead to MDL deadlock and the code was written at the time
when MDL deadlock detector was not present. In case when
discovery attempted in the statement other than the first
one in transaction ER_LOCK_DEADLOCK error is reported
pessimistically.
Fix:
---
Removed the constraint as any potential deadlock will be
handled by deadlock detector. Also changed code in discover
to keep metadata locks of active transaction.
Same issue was present in table auto repair scenario. Same
fix is added in repair path also.
The following FLUSH commands are now executed under total
order isolation:
* FLUSH DES_KEY_FILE
* FLUSH HOSTS
* FLUSH PRIVILEGES
* FLUSH QUERY CACHE
* FLUSH STATUS
* FLUSH USER_RESOURCES
table
Performance schema discovery fails if connection has no active database set.
This happened due to restriction in SQL parser: table name with no database name
is ambiguous in such case.
Fixed by temporary substitution of default database with being discovered table
database.
Problem was that information schema tables innodb_tablespaces_encryption and
innodb_tablespaces_scrubbing where missing required check is InnoDB enabled
or not.
including the big commit
commit 305130361bf72726de220f3d2b2787395e10be61
Author: Marc Alff <marc.alff@oracle.com>
Date: Tue Feb 10 11:31:32 2015 +0100
WL#8354 BACKPORT DIGEST IMPROVEMENTS TO MYSQL 5.6
(with the following commits) and related changes in sql/
1. After a period of wait (where last_master_timestamp=0)
do NOT restore the last_master_timestamp to the timestamp
of the last executed event (which would mean we've just
executed it, and we're that much behind the master).
2. Update last_master_timestamp before executing the event,
not after.
Take the approach from the this commit (but with a different test
case that actually makes sense):
commit 0c75ab453fb8c5439576af8fe5add7a1b89f1569
Author: Luis Soares <luis.soares@sun.com>
Date: Thu Apr 15 17:39:31 2010 +0100
BUG#52166: Seconds_Behind_Master spikes after long idle period
The slave SQL thread was clearing serial_rgi->thd before deleting
serial_rgi, which could cause access to NULL THD.
The clearing was introduced in commit
2e100cc5a4 and is just plain wrong. So revert
that part (single line) of that commit.
Thanks to Daniel Black for bug analysis and test case.
on disconnect THD must clean user_var_events array before
dropping temporary tables. Otherwise when binlogging a DROP,
it'll access user_var_events, but they were allocated
in the already freed memroot.
Three-way deadlock:
T1: SHOW GLOBAL STATUS
-> acquire LOCK_status
T2: STOP SLAVE
-> acquire LOCK_active_mi
-> terminate_slave_thread()
-> -> cond_timedwait for handle_slave_sql to stop
T3: sql slave thread (same applies to io thread)
-> handle_slave_sql(), when exiting
-> -> THD::add_status_to_global()
-> -> -> wait for LOCK_status...
T1: SHOW GLOBAL STATUS
-> for "Slave_heartbeat_period" status variable
-> -> show_heartbeat_period()
-> -> -> wait for LOCK_active_mi
cherry-pick from 5.6:
commit fc8b395898f40387b3468122bd0dae31e29a6fde
Author: Venkatesh Duggirala <venkatesh.duggirala@oracle.com>
Date: Wed Jun 12 21:41:05 2013 +0530
BUG#16904035-SHOW STATUS - EXCESSIVE LOCKING ON LOCK_ACTIVE_MI AND
ACTIVE_MI->RLI->DATA_LOCK
Problem: Excessive locking on lock_active_mi and rli->data_lock
while executing any `show status like 'X'` command.
Analysis: SHOW_FUNCs for Slave_running, Slave_retried_transactions,
Slave_heartbeat_period, Slave_received_heartbeats,
Slave_last_heartbeat are acquiring lock_active_mi and rli->data_lock
to show their variable value. It is ok to show stale data while showing
the status variables i.e., even if they miss one update, it will
not cause any great trouble.
Fix: Remove the locks from the above mentioned SHOW_FUNC functions.
Add a test case
There was a rare race, where a deadlock error might not be correctly
handled, causing the slave to stop with something like this in the error
log:
150423 14:04:10 [ERROR] Slave SQL: Connection was killed, Gtid 0-1-2, Internal MariaDB error code: 1927
150423 14:04:10 [Warning] Slave: Connection was killed Error_code: 1927
150423 14:04:10 [Warning] Slave: Deadlock found when trying to get lock; try restarting transaction Error_code: 1213
150423 14:04:10 [Warning] Slave: Connection was killed Error_code: 1927
150423 14:04:10 [Warning] Slave: Connection was killed Error_code: 1927
150423 14:04:10 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'master-bin.000001 position 1234
The problem was incorrect error handling. When a deadlock is detected, it
causes a KILL CONNECTION on the offending thread. This error is then later
converted to a deadlock error, and the transaction is retried.
However, the deadlock error was not cleared at the start of the retry, nor
was the lingering kill signal. So it was possible to get another deadlock
kill early during retry. If this happened with particular thread
scheduling/timing, it was possible that the new KILL CONNECTION error was
masked by the earlier deadlock error, so that the second kill was not
properly converted into a deadlock error and retry.
This patch adds code that clears the old error and killed flag before
starting the retry. It also adds code to handle a deadlock kill caught in a
couple of places where it was not handled before.
This bug is a side-effect of fix for MDEV-6924, where we completely
stopped a statement-based event from getting into the binlog cache when
binary logging is not enabled (and thus, wsrep_emulate_binlog mode = 1).
As a result, the SBR events were not replicated.
Fixed by allowing the SBR events to be written into the binlog cache.
Note: Only DMLs were affected as DDLs are replicated via TOI.
Merged galera_create_trigger.test from github.com/codership/mysql-wsrep.
This was a regression from the patch for MDEV-7668.
A test was incorrect, so the slave would not properly handle re-using
temporary tables, which lead to replication failure in this case.
THD's lex->definer initially holds "*current_user" for CURRENT_USER().
As a result when wsrep tries to create the trigger query, it incorrectly
uses "*current_user". Fixed by calling get_current_user() to get the
real current user.
Merged galera_create_trigger.test from github.com/codership/mysql-wsrep.
Make sure that in parallel replication, we execute wait_for_prior_commit()
before setting table->in_use for a temporary table. Otherwise we can end up
with two parallel replication worker threads competing with each other for
use of a temporary table.
Re-factor the use of find_temporary_table() to be able to handle errors
in the caller (as wait_for_prior_commit() can return error in case of
deadlock kill).
[This commit cherry-picked to be able to merge MDEV-7936, of which it
is a pre-requisite, into both 10.0 and 10.1.]
Parallel replication depends on locking (table locks, row locks, etc.) to
prevent two conflicting transactions from running and committing in parallel.
But temporary tables are designed to be visible only to one thread, and have
no such locking.
In the concrete issue, an intermediate master could commit a CREATE TEMPORARY
TABLE in the same group commit as in INSERT into that table. Thus, a
lower-level master could attempt to run them in parallel and get an error.
More generally, we need protection from parallel replication trying to run
transactions in parallel that access a common temporary table.
This patch simply causes use of a temporary table from parallel replication
to wait for all previous transactions to commit, serialising the replication
at that point.
(A more fine-grained locking could be added later, possibly. However,
using temporary tables in statement-based replication is in any case
normally undesirable; for example a restart of the server will lose
temporary tables and can break replication).
Note that row-based replication is not affected, as it does not do any
temporary tables on the slave-side.
This patch also cleans up the locking around protecting the list of
temporary tables in Relay_log_info. This used to take the
rli->data_lock at the end of every statement, which is very bad for
concurrency. With this patch, the lock is not taken unless temporary
tables (with statement-based binlogging) are in use on the slave.
do not *always* add them to the create table definition,
but only when a sysvar value is different from a default.
also, when adding them - don't quote numbers
With changes:
* update tests to pass (new encryption/encryption_key_id syntax).
* not merged the code that makes engine aware of the encryption mode
(CRYPT_SCHEME_1_CBC, CRYPT_SCHEME_1_CTR, storing it on disk, etc),
because now the encryption plugin is handling it.
* compression+encryption did not work in either branch before the
merge - and it does not work after the merge. it might be more
broken after the merge though - some of that code was not merged.
* page checksumming code was not moved (moving of page checksumming
from fil_space_encrypt() to fil_space_decrypt was not merged).
* restored deleted lines in buf_page_get_frame(), otherwise
innodb_scrub test failed.
