order with Galera and encrypt-tmp-files=1
Problem:- If trans_cache (IO_CACHE) uses encrypted tmp file
then on next DML server will crash.
Case:-
Lets take a case , we have a table t1 , We try to do 2 inserts in t1
1. A really long insert so that trans_cache has to use temp_file
2. Just a small insert
Analysis:- Actually server crashes from inside of galera
library.
/lib64/libc.so.6(abort+0x175)[0x7fb5ba779dc5]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5State...
mysys/stacktrace.c:247(my_print_stacktrace)[0x7fb5a714940e]
sql/signal_handler.cc:160(handle_fatal_signal)[0x7fb5a715c1bd]
sql/wsrep_hton.cc:257(wsrep_rollback)[0x7fb5bcce923a]
sql/wsrep_hton.cc:268(wsrep_rollback)[0x7fb5bcce9368]
sql/handler.cc:1658(ha_rollback_trans(THD*, bool))[0x7fb5bcd4f41a]
sql/handler.cc:1483(ha_commit_trans(THD*, bool))[0x7fb5bcd4f804]
but actual issue is not in galera but in mariadb, because for 2nd
insert we should never call rollback. We are calling rollback because
log_and_order fails it fails because write_cache fails , It fails
because after reinit_io_cache(trans_cache) , my_b_bytes_in_cache says 0
so we look into tmp_file for data , which is obviously wrong since temp
was used for previous insert and it no longer exist.
wsrep_write_cache_inc() reads the IO_CACHE in a loop, filling it with
my_b_fill() until it returns "0 bytes read". Later
MYSQL_BIN_LOG::write_cache() does the same. wsrep_write_cache_inc()
assumes that reading a zero bytes past EOF leaves the old data in the
cache
Solution:- There is two issue in my_b_encr_read
1st we should never equal read_end to info->buffer. I mean this
does not make sense read_end should always point to end of buffer.
2nd For most of the case(apart from async IO_CACHE) info->pos_in_file
should be equal to info->buffer position wrt to temp file , since
in this case we are not changing info->buffer it should remain
unchanged.
Added a test which verifies that if the gcomm background thread
is forced to quit via exception, the error is propagated all the
way up the stack and wsrep_ready becomes 0.
The kills the two nodes that compose the cluster and then recovers them.
The first node that is recovered is expected to recover its gcache, so
that when the second node is recovered, it is expected that it rejoins
the cluster via IST.
However, it is possible that if the second node is killed while it is
applying in TOI mode, its local state is marked unsafe. In which case,
SST is the only option to rejoin the cluster.
codership/mysql-wsrep#323
The test relies on the false assumption that node_1 is always
"assigned" autoincrement value 1 for a given key.
However that is no necessarily the case when wsrep autoincrement
control is enabled.
This patch changed the test so that it does not make use of a
autoincrement key, which is not fundamental for the test itself.
codership/mysql-wsrep#322
The test assumes that `@@global.auto_increment_offset` is equal to
`wsrep_local_index + 1`. Which is normally the case if galera runs
with option `wsrep_auto_increment_control` enabled.
However, if some prior test performs a restart of a server, then
`wsrep_local_index` may change, and galera will set the value of
`auto_increment_offset` accordingly.
However, if `auto_increment_offset` changes during a test run, then
mtr will complain. To avoid that, tests that perform restarts include
`auto_increment_offset_save.inc` and `auto_increment_offset_restore.inc`.
Which reset the value of `auto_increment_offset`. And when that
happens, `auto_increment_offset` will no longer be equal to
`wsrep_local_index + 1`, and the test fails.
To avoid this problem, simply check that the offsets are different
on the nodes that compose the cluster.
The test was not deterministic and would occasionally fail, due to the
use of `sleep`.
This patch is a complete rewrite of the test using proper sync points.
The test performs a restart and while doing so it occasionally
fails with "Server did not transition to READY state" in
`include/start_mysqld.inc".
Fix is to wait for the cluster to shrink before restarting
the node.
* Increased timeout counter in galera_wait_ready.inc
* Replaced useless include/wait_until_ready.inc after start_mysqld.inc
in galera_st_*.inc with wait_condition on cluster size.
Test galera_sst_mysqldump often fails with error "2013: Lost connection
to MySQL server during query". The connection is lost after the test
restart one of the nodes. This happens because the server closes client
connections if it is joining a cluster through SST method mysqldump.
On unlucky runs of the test it is possible that mysqld is restarted,
and then mtr client is disconnected while it tries to determine if
galera is ready before going on with the test.
This patch rewrites galera_wait_ready.inc so that it is immune to
being disconnected.
slave node killed himself.
Problem:- If we try to delete table with foreign key and table whom it is
referring with wsrep_slave_threads>1 then galera tries to execute both
Delete_rows_log-event in parallel, which should not happen.
Solution:- This is happening because we do not have foreign key info in
write set. Upto version 10.2.7 it used to work fine. Actually it happening
because of issue in commit 2f342c4. wsrep_must_process_fk has changed to
make it similar to original condition.
This patch fixes a lock wait timeout error on `SELECT @@GLOBAL.WSREP_ON`
in `wait_wsrep_ready.inc`:
```
--connection node_2
...
--source include/kill_galera.inc
--connection node_1
--source include/wait_until_connected_again.inc # This includes wait_wsrep_ready.inc
```
The problem is that on node_2, kill_galera.inc may return before
the node is killed. So node_1 may still see that node_1 is alive
and will attempt to sync wait when doing those `SELECT` statements.
But sync wait is doomed to fail given that node_1 is killed, hence
the lock wait timeout.
One possible fix is to disable wsrep_sync_wait before including
wait_until_connected_again.
However, it appears that including wait_until_connected_again is
not necessary at all in node_1, so this patch removes it altogether.
This patch makes two changes:
* Remove unnecessary `set SESSION wsrep_sync_wait=0`.
Disabling `wsrep_sync_wait` caused the test to fail occasionally,
due to subsequent `SELECT`s observing stale values.
* Remove redundant `--source include/galera_wait_ready.inc`,
`galera_wait_ready.inc` is already included in the above
`wait_until_connected_again.inc`
Test would occasionally fail as follows:
```
mysqltest: At line 20: query 'SELECT COUNT(*) = 0 FROM t1' failed:
1317: Query execution was interrupted
```
This was due to a `CREATE TABLE` applied concurrently that would
occasionally cause the `SELECT` to be BF aborted, due to MDL
conflict.
The `SELECT` is executed while `wsrep_on=OFF`. The change makes sure
it is executed only after wsrep is enabled again.
The test tends to fail if many parallel instances of it are executed:
```
mysqltest: At line 23: query 'ALTER TABLE t1 ADD PRIMARY KEY (f1)' failed:
1317: Query execution was interrupted
```
The `ALTER` fails because it is BF aborted due to an earlier `INSERT SELECT`
that is being applied:
```
INSERT INTO t1 (f1) SELECT ...
--connection node_2
SET GLOBAL wsrep_desync = TRUE;
SET SESSION wsrep_on = FALSE;
ALTER TABLE t1 ADD PRIMARY KEY (f1);
SET SESSION wsrep_on = TRUE;
SET GLOBAL wsrep_desync = FALSE;
```
And because the `ALTER` is executed with `wsrep_on = OFF`, it does not
run in total order isolation.
To avoid the problem it must be ensured that the `ALTER` only after the
large `INSERT SELECT` is done. To do so it is sufficient to issue
`SELECT COUNT(*) FROM t1;` from `node_2` before turning off wsrep.
The `SELECT` will trigger `wsrep_sync_wait` and proceed only after the
`INSERT SELECT` from node_1 is done.
