compiled for debugging, when the server goes down
This happens in the following scenario:
- Server gets a shutdown message
- Servers sends error ER_CONNECTION_KILLED to the clients connection
- The client sends a query to the server, before the server has time to
close the connection to the client
- Client reads the ER_CONNECTION_KILLED error message
In the above case, the packet number for the reply is one less than
what the client expected and the client prints "Packets out of order".
Fixed the following way:
- The client accepts now error packages with a packet number
one less than expected.
- To ensure that this issue can be detected early in my_real_read(), error
messages sent to the client are not compressed, even when compressed protocol is used.
There were so many changes into mtr (this is the new mtr coming) that I rather
copied mtr from 6.0-main here (at least this one knows how to run Maria tests).
I also fixed suite/maria tests to be accepted by the new mtr.
mysys/thr_mutex.c:
adding DBUG_PRINT here, so that we can locate where the warning is issued.
mysql-test/include/maria_empty_logs.inc:
move mysqladmin call to include file for reusal (so that others don't have to hit the same bug in their tests);
this include file additionally waits for disconnection. Reasons for BUG#41102:
- in maria-no-logging.test, we shut down mysqld in maria_empty_logs.inc, remove logs, tell mtr that it can
restart mysqld, then wait for being connected (wait_until_connected_again.inc), then send commands to new mysqld.
- but under Windows "mysqladmin shutdown" does not wait for mysqld to be down, so we accidentally removed logs
while it was shutting down, wait_until_connected_again found that we were connected (because it was shutting
down slowly, connections were not yet broken), sent new commands, and one of them failed because mysqld finally
closed connections.
- the fix is to just wait for disconnection after shutdown.
- in a sense, the catch is the "again" in wait_until_connected_again.inc...
mysql-test/include/mysqladmin_shutdown.inc:
shutdown with mysqladmin AND waiting for shutdown to complete (needed on Windows)
mysql-test/include/wait_until_connected_again.inc:
comment
mysql-test/include/wait_until_disconnected.inc:
script which waits until connection is lost
the server dissapears
Check return code of 'mysql_ping'
Add "shutdown", call 'mysql_shutdown' on the current connection
mysql-test/include/restart_mysqld.inc:
Use "kill_server"
mysql-test/include/wait_until_connected_again.inc:
Init $mysql_errno to 9999
Add workaround for BUG#36228
mysql-test/include/wait_until_connected_again.inc:
On Windows, mysqladmin does not wait on pid file when it has shut down
mysqld. Thus, in maria_empty_logs.inc, wait_until_connected_again.inc
can run while mysqld is shutting down (has not finished) and so
get "shutdown in progress" instead of "can't connect".
mysql-test/t/maria-recovery-big.test:
it is a big test
storage/maria/unittest/ma_test_recovery.pl:
We want to see the result of 'cmp', like in the shell version
Memory leak fixed.
Some other small cahnges.
mysql-test/include/wait_until_connected_again.inc:
increased time for recovery (to be able use with --debug)
mysql-test/r/maria-recovery.result:
Result fixed.
storage/maria/ma_loghandler.c:
Multigroup record write fixed.
Function for skipping to the next page while filling buffer with
unlocked handler added.
Removed possible memory leaks.
More debug.
storage/maria/ma_recovery.c:
Memory leak fixed.
See test maria-recovery.test for a model; all include scripts have
an "API" section at start if they do take parameters from outside.
* Fixing bug reported by Jani and Monty (when two REDOs about the same
page in one group, see ma_blockrec.c).
* Fixing small bugs in recovery
mysql-test/include/wait_until_connected_again.inc:
be sure to enter the loop (the previous query by the caller may not have
failed: it could be
query;
mysqladmin shutdown;
call this script).
mysql-test/lib/mtr_process.pl:
* Through the "expect" file a test can tell mtr that a server crash
is expected. What the file contains is irrelevant. Now if its last
line starts with "wait", mtr will wait before restarting (it will
wait for the last line to not start with "wait"). This is for
tests which need to mangle files under the feet of a dead mysqld.
* Remove "expect" file before restarting; otherwise there could be a
race condition: tests sees server restarted, does something, writes
an "expect" file, and then mtr removes that file, then
test kills mysqld, and then mtr will never restart it.
storage/maria/ma_blockrec.c:
- when applying a REDO in recovery, we don't anymore put UNDO's LSN on the page
at once; indeed if in this REDO's group there comes another REDO
for the same page it would be wrongly skipped. Instead, we keep
pages pinned, don't change their LSN. When done with all REDOs
of the group we unpin them and stamp them with UNDO's LSN.
- fixing bug in applying of REDO_PURGE_BLOCKS in recovery: page_range
sometimes has TAIL_BIT set, need to turn it down to know the real page
range.
- Both bugs are covered in maria-recovery.test
storage/maria/ma_checkpoint.c:
Capability to, in debug builds only, do some special operations
(flush all bitmap and data pages, flush state, flush log)
and crash mysqld, to later test recovery.
Driven by some --debug=d, symbols.
storage/maria/ma_open.c:
debugging info
storage/maria/ma_pagecache.c:
Now that we can _ma_unpin_all_pages() during the REDO phase
to set page's LSN, the assertion needs to be relaxed.
storage/maria/ma_recovery.c:
- open trace file in append mode (useful when a test triggers several
recoveries, we see them all).
- fixing wrong error detection, it's possible that during recovery
we want to open an already open table.
- when applying a REDO in recovery, we don't anymore put UNDO's LSN on the page
at once; indeed if in this REDO's group there comes another REDO
for the same page it would be wrongly skipped. Instead, we keep
pages pinned, don't change their LSN. When done with all REDOs
of the group we unpin them and stamp them with UNDO's LSN.
- we verify that all log records of a group are about the same table,
for debugging.
mysql-test/r/maria-recovery.result:
result
mysql-test/t/maria-recovery-master.opt:
crash is expected, core file would take room, stack trace would
wake pushbuild up.
mysql-test/t/maria-recovery.test:
Test of recovery from mysql-test (it is already tested as unit tests
in ma_test_recovery) (WL#4137)
- test that, if recovery is made to start on an empty table it can
replay the effects of committed and uncommitted statements (having only
the committed ones in the end result). This should be the first test
for someone writing code of new REDOs.
- test that, if mysqld is crashed and recovery runs we have only
committed statements in the end result. Crashes are done in different
ways: flush nothing (so, uncommitted statement is often missing
from the log => no rollback to do); flush pagecache (implicitely flushes
log (WAL)) and flush log, both causes rollbacks; flush log can also
flush state (state.records etc) to test recovery of the state
(not tested well now as we repair the index anyway).
- test of bug found by Jani and Monty in recovery (two REDO about
the same page in one group).
mysql-test/include/maria_empty_logs.inc:
removes logs, to have a clean sheet for testing recovery.
mysql-test/include/maria_make_snapshot.inc:
copies a table to another directory, or back, or compares both
(comparison is not implemented as physical comparison is impossible
if an UNDO phase happened).
mysql-test/include/maria_make_snapshot_for_comparison.inc:
copies tables to another directory so that they can later
serve as a comparison reference (they are the good tables,
recovery should produce similar ones).
mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc:
When we want to force recovery to start on old tables, we prepare
old tables with this script: we put them in a spare directory.
They are later copied back over mysqltest tables while mysqld is dead.
We also need to copy back the control file, otherwise mysqld,
in recovery, would start from the latest checkpoint: latest
checkpoint plus old tables is not a recovery-possible scenario of course.
mysql-test/include/maria_verify_recovery.inc:
causes mysqld to crash, restores old tables if requested,
lets recovery run, compares resulting tables with reference tables
by using CHECKSUM TABLE.
We don't do any sanity checks on page's LSN in resulting tables, yet.
- Make mysql-test-run.pl restart a crashed/stopped process if it was expected it should crash.
- Added testcase for "crash_commit_before", which tests behaviour when server crashes just before commit.
* The testcase first write a small var/tmp/master0.expect file indicating expected crash
* Then sets "DEBUG" variable to for example "d,crash_commit_before" and then executes a commit.
* The server will crash and be brought back up by mysql-test-run.pl
* Test case will then wait for the process to come back online before continuing.
client/mysqltest.c:
Add "die" command used to abort a failing test case
Remove unused "exit" command
mysql-test/lib/mtr_process.pl:
When a process crash during execution of mysqltest, check if it was expected and restart the process in that case.
mysql-test/mysql-test-run.pl:
Remmeber options used when ndbd's and mysqld's was started, to be used by restart code
mysql-test/r/crash_commit_before.result:
New BitKeeper file ``mysql-test/r/crash_commit_before.result''
mysql-test/include/wait_until_connected_again.inc:
New BitKeeper file ``mysql-test/include/wait_until_connected_again.inc''
mysql-test/t/crash_commit_before.test:
New BitKeeper file ``mysql-test/t/crash_commit_before.test''
