mariadb/mysql-test
Monty c4cad8d50c MDEV-33449 improving repair of tables
This task is to ensure we have a clear definition and rules of how to
repair or optimize a table.

The rules are:

- REPAIR should be used with tables that are crashed and are
  unreadable (hardware issues with not readable blocks, blocks with
  'unexpected data' etc)
- OPTIMIZE table should be used to optimize the storage layout for the
  table (recover space for delete rows and optimize the index
  structure.
- ALTER TABLE table_name FORCE should be used to rebuild the .frm file
  (the table definition) and the table (with the original table row
  format). If the table is from and older MariaDB/MySQL release with a
  different storage format, it will convert the data to the new
  format. ALTER TABLE ... FORCE is used as part of mariadb-upgrade

Here follows some more background:

The 3 ways to repair a table are:
1) ALTER TABLE table_name FORCE" (not other options).
   As an alias we allow: "ALTER TABLE table_name ENGINE=original_engine"
2) "REPAIR TABLE" (without FORCE)
3) "OPTIMIZE TABLE"

All of the above commands will optimize row space usage (which means that
space will be needed to hold a temporary copy of the table) and
re-generate all indexes. They will also try to replicate the original
table definition as exact as possible.

For ALTER TABLE and "REPAIR TABLE without FORCE", the following will hold:
If the table is from an older MariaDB version and data conversion is
needed (for example for old type HASH columns, MySQL JSON type or new
TIMESTAMP format) "ALTER TABLE table_name FORCE, algorithm=COPY" will be
used.

The differences between the algorithms are
1) Will use the fastest algorithm the engine supports to do a full repair
   of the table (except if data conversions are is needed).
2) Will use the storage engine internal REPAIR facility (MyISAM, Aria).
   If the engine does not support REPAIR then
   "ALTER TABLE FORCE, ALGORITHM=COPY" will be used.
   If there was data incompatibilities (which means that FORCE was used)
   then there will be a warning after REPAIR that ALTER TABLE FORCE is
   still needed.
   The reason for this is that REPAIR may be able to go around data
   errors (wrong incompatible data, crashed or unreadable sectors) that
   ALTER TABLE cannot do.
3) Will use the storage engine internal OPTIMIZE. If engine does not
   support optimize, then "ALTER TABLE FORCE" is used.

The above will ensure that ALTER TABLE FORCE is able to
correct almost any errors in the row or index data.  In case of
corrupted blocks then REPAIR possible followed by ALTER TABLE is needed.
This is important as mariadb-upgrade executes ALTER TABLE table_name
FORCE for any table that must be re-created.

Bugs fixed with InnoDB tables when using ALTER TABLE FORCE:
- No error for INNODB_DEFAULT_ROW_FORMAT=COMPACT even if row length
  would be too wide. (Independent of innodb_strict_mode).
- Tables using symlinks will be symlinked after any of the above commands
  (Independent of the setting of --symbolic-links)

If one specifies an algorithm together with ALTER TABLE FORCE, things
will work as before (except if data conversion is required as then
the COPY algorithm is enforced).

ALTER TABLE .. OPTIMIZE ALL PARTITIONS will work as before.

Other things:
- FORCE argument added to REPAIR to allow one to first run internal
  repair to fix damaged blocks and then follow it with ALTER TABLE.
- REPAIR will not update frm_version if ha_check_for_upgrade() finds
  that table is still incompatible with current version. In this case the
  REPAIR will end with an error.
- REPAIR for storage engines that does not have native repair, like InnoDB,
  is now using ALTER TABLE FORCE.
- REPAIR csv-table USE_FRM now works.
  - It did not work before as CSV tables had extension list in wrong
    order.
- Default error messages length for %M increased from 128 to 256 to not
  cut information from REPAIR.
- Documented HA_ADMIN_XX variables related to repair.
- Added HA_ADMIN_NEEDS_DATA_CONVERSION to signal that we have to
  do data conversions when converting the table (and thus ALTER TABLE
  copy algorithm is needed).
- Fixed typo in error message (caused test changes).
2024-05-27 12:39:03 +02:00
..
collections Allow tests to be run without debug when possible, and on Windows 2024-05-27 12:39:02 +02:00
include MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range 2024-05-27 12:39:02 +02:00
lib Merge branch '11.2' into 11.4 2024-05-21 19:38:51 +02:00
main MDEV-33449 improving repair of tables 2024-05-27 12:39:03 +02:00
std_data Merge branch '11.2' into 11.4 2024-05-21 19:38:51 +02:00
suite MDEV-33449 improving repair of tables 2024-05-27 12:39:03 +02:00
asan.supp
CMakeLists.txt
dgcov.pl Merge branch '10.6' into 10.11 2024-05-10 20:02:18 +02:00
lsan.supp Added asan options to mysql-test-run 2019-08-23 22:06:30 +02:00
mariadb-stress-test.pl
mariadb-test-run.pl Merge branch '11.2' into 11.4 2024-05-21 19:38:51 +02:00
mtr.out-of-source
purify.supp
README
README-gcov
README.stress
suite.pm Merge branch '11.2' into 11.4 2024-05-21 19:38:51 +02:00
valgrind.supp

This directory contains test suites for the MariaDB server. To run
currently existing test cases, execute ./mysql-test-run in this directory.

Some tests are known to fail on some platforms or be otherwise unreliable.
In the file collections/smoke_test there is a list of tests that are
expected to be stable.

In general you do not have to have to do "make install", and you can have
a co-existing MariaDB installation, the tests will not conflict with it.
To run the tests in a source directory, you must do "make" first.

In Red Hat distributions, you should run the script as user "mysql".
The user is created with nologin shell, so the best bet is something like
  # su -
  # cd /usr/share/mariadb-test
  # su -s /bin/bash mysql -c ./mysql-test-run

This will use the installed MariaDB executables, but will run a private
copy of the server process (using data files within /usr/share/mariadb-test),
so you need not start the mysqld service beforehand.

You can omit --skip-test-list option if you want to check whether
the listed failures occur for you.

To clean up afterwards, remove the created "var" subdirectory, e.g.
  # su -s /bin/bash - mysql -c "rm -rf /usr/share/mariadb-test/var"

If tests fail on your system, please read the following manual section
for instructions on how to report the problem:

https://mariadb.com/kb/en/reporting-bugs

If you want to use an already running MySQL server for specific tests,
use the --extern option to mysql-test-run. Please note that in this mode,
you are expected to provide names of the tests to run.

For example, here is the command to run the "alias" and "analyze" tests
with an external server:

  # mariadb-test-run --extern socket=/tmp/mysql.sock alias analyze

To match your setup, you might need to provide other relevant options.

With no test names on the command line, mysql-test-run will attempt
to execute the default set of tests, which will certainly fail, because
many tests cannot run with an external server (they need to control the
options with which the server is started, restart the server during
execution, etc.)

You can create your own test cases. To create a test case, create a new
file in the main subdirectory using a text editor. The file should have a .test
extension. For example:

  # xemacs t/test_case_name.test

In the file, put a set of SQL statements that create some tables,
load test data, and run some queries to manipulate it.

Your test should begin by dropping the tables you are going to create and
end by dropping them again. This ensures that you can run the test over
and over again.

If you are using mysqltest commands in your test case, you should create
the result file as follows:

  # mariadb-test-run --record test_case_name

  or

  # mariadb-test --record < t/test_case_name.test

If you only have a simple test case consisting of SQL statements and
comments, you can create the result file in one of the following ways:

  # mariadb-test-run --record test_case_name

  # mariadb test < t/test_case_name.test > r/test_case_name.result

  # mariadb-test --record --database test --result-file=r/test_case_name.result < t/test_case_name.test

When this is done, take a look at r/test_case_name.result.
If the result is incorrect, you have found a bug. In this case, you should
edit the test result to the correct results so that we can verify that
the bug is corrected in future releases.

If you want to submit your test case you can send it
to developers@lists.mariadb.org or attach it to a bug report on
http://mariadb.org/jira/.

If the test case is really big or if it contains 'not public' data,
then put your .test file and .result file(s) into a tar.gz archive,
add a README that explains the problem, ftp the archive to
ftp://ftp.mariadb.org/private and submit a report to
https://mariadb.org/jira about it.

The latest information about mysql-test-run can be found at:
https://mariadb.com/kb/en/mariadb/mysqltest/

If you want to create .rdiff files, check
https://mariadb.com/kb/en/mariadb/mysql-test-auxiliary-files/