Commit graph

30 commits

Author SHA1 Message Date
Marko Mäkelä
f2b4972bd4 Merge 10.11 into 11.0 2023-07-26 15:13:06 +03:00
Marko Mäkelä
bce3ee704f Merge 10.10 into 10.11 2023-07-26 14:44:43 +03:00
Marko Mäkelä
864bbd4d09 Merge 10.6 into 10.9 2023-07-26 13:42:23 +03:00
Alexey Botchkov
a03ce7b99c MDEV-31521 bzero wipes more bytes than necessary in set_global_from_ddl_log_entry.
'defaults' made into 'constexpr' variables.
2023-07-13 19:54:39 +04:00
Monty
a4e103ad82 Fixed bug in ddl log
This issue could only be noticed in very extraordinary circumstances
when trying to rename a temporary table that is not in the file system.

The issue was found when I temporarly disabled check_if_frm_exists()
while searching for another bug.
2023-07-13 14:20:02 +03:00
Monty
6418c24c94 Set thd->query() for internal (startup) transactions
This helps with debugging as 'Query: ' in DBUG traces will show something
useful, for internal transactions, instead of just "".
2023-02-03 10:34:49 +03:00
Sergei Golubchik
2bd41fc5bf Revert MDEV-25292 Atomic CREATE OR REPLACE TABLE
Specifically:

Revert "MDEV-29664 Assertion `!n_mysql_tables_in_use' failed in innobase_close_connection"
This reverts commit ba875e9396.

Revert "MDEV-29620 Assertion `next_insert_id == 0' failed in handler::ha_external_lock"
This reverts commit aa08a7442a.

Revert "MDEV-29628 Memory leak after CREATE OR REPLACE with foreign key"
This reverts commit c579d66ba6.

Revert "MDEV-29609 create_not_windows test fails with different result"
This reverts commit cb583b2f1b.

Revert "MDEV-29544 SIGSEGV in HA_CREATE_INFO::finalize_locked_tables"
This reverts commit dcd66c3814.

Revert "MDEV-28933 CREATE OR REPLACE fails to recreate same constraint name"
This reverts commit cf6c517632.

Revert "MDEV-28933 Moved RENAME_CONSTRAINT_IDS to include/sql_funcs.h"
This reverts commit f1e1c1335b.

Revert "MDEV-28956 Locking is broken if CREATE OR REPLACE fails under LOCK TABLES"
This reverts commit a228ec80e3.

Revert "MDEV-25292 gcol.gcol_bugfixes --ps fix"
This reverts commit 24fff8267d.

Revert "MDEV-25292 Disable atomic replace for slave-generated or-replace"
This reverts commit 2af15914cb.

Revert "MDEV-25292 backup_log improved"
This reverts commit 34398a20b5.

Revert "MDEV-25292 Atomic CREATE OR REPLACE TABLE"
This reverts commit 93c8252f02.

Revert "MDEV-25292 Table_name class for (db, table_name, alias)"
This reverts commit d145dda9c7.

Revert "MDEV-25292 ha_table_exists() cleanup and improvement"
This reverts commit 409b8a86de.

Revert "MDEV-25292 Cleanups"
This reverts commit 595dad83ad.

Revert "MDEV-25292 Refactoring: moved select_field_count into Alter_info."
This reverts commit f02af1d229.
2022-10-27 23:13:41 +02:00
Aleksey Midenkov
93c8252f02 MDEV-25292 Atomic CREATE OR REPLACE TABLE
Atomic CREATE OR REPLACE allows to keep an old table intact if the
command fails or during the crash. That is done through creating
a table with a temporary name and filling it with the data
(for CREATE OR REPLACE .. SELECT), then renaming the original table
to another temporary (backup) name and renaming the replacement table
to original table. The backup table is kept until the last chance of
failure and if that happens, the replacement table is thrown off and
backup recovered. When the command is complete and logged the backup
table is deleted.

Atomic replace algorithm

  Two DDL chains are used for CREATE OR REPLACE:
  ddl_log_state_create (C) and ddl_log_state_rm (D).

  1. (C) Log CREATE_TABLE_ACTION of TMP table (drops TMP table);
  2. Create new table as TMP;
  3. Do everything with TMP (like insert data);

  finalize_atomic_replace():
  4. Link chains: (D) is executed only if (C) is closed;
  5. (D) Log DROP_ACTION of BACKUP;
  6. (C) Log RENAME_TABLE_ACTION from ORIG to BACKUP (replays BACKUP -> ORIG);
  7. Rename ORIG to BACKUP;
  8. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG);
  9. Rename TMP to ORIG;

  finalize_ddl() in case of success:
  10. Close (C);
  11. Replay (D): BACKUP is dropped.

  finalize_ddl() in case of error:
  10. Close (D);
  11. Replay (C):
    1) ORIG is dropped (only after finalize_atomic_replace());
    2) BACKUP renamed to ORIG (only after finalize_atomic_replace());
    3) drop TMP.

  If crash happens (C) or (D) is replayed in reverse order. (C) is
  replayed if crash happens before it is closed, otherwise (D) is
  replayed.

Temporary table for CREATE OR REPLACE

  Before dropping "old" table, CREATE OR REPLACE creates "tmp" table.
  ddl_log_state_create holds the drop of the "tmp" table.  When
  everything is OK (data is inserted, "tmp" is ready) ddl_log_state_rm
  is written to replace "old" with "tmp". Until ddl_log_state_create
  is closed ddl_log_state_rm is not executed.

  After the binlogging is done ddl_log_state_create is closed. At that
  point ddl_log_state_rm is executed and "tmp" is replaced with
  "old". That is: final rename is done by the DDL log.

  With that important role of DDL log for CREATE OR REPLACE operation
  replay of ddl_log_state_rm must fail at the first hit error and
  print the error message if possible. F.ex. foreign key error is
  discovered at this phase: InnoDB rejects to drop the "old" table and
  returns corresponding foreign key error code.

Additional notes

  - CREATE TABLE without REPLACE is not affected by this commit.

  - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by
    this commit.

  - CREATE TABLE .. SELECT XID usage is fixed and now there is no need
    to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in
    do_postlock()). XID is now correctly updated so it disables
    DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the
    final stage when the table is ready. So if we have XID in the
    binary log we don't need to drop the table.

  - Three variations of CREATE OR REPLACE handled:

    1. CREATE OR REPLACE TABLE t1 (..);
    2. CREATE OR REPLACE TABLE t1 LIKE t2;
    3. CREATE OR REPLACE TABLE t1 SELECT ..;

  - Test case uses 6 combinations for engines (aria, aria_notrans,
    myisam, ib, lock_tables, expensive_rename) and 2 combinations for
    binlog types (row, stmt). Combinations help to check differences
    between the results. Error failures are tested for the above three
    variations.

  - expensive_rename tests CREATE OR REPLACE without atomic
    replace. The effect should be the same as with the old behaviour
    before this commit.

  - Triggers mechanism is unaffected by this change. This is tested in
    create_replace.test.

  - LOCK TABLES is affected. Lock restoration must be done after "rm"
    chain is replayed.

  - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This
    checkpoint was not executed before for normal CREATE TABLE but is
    executed now.

  - CREATE TABLE will now rollback also if writing to the binary
    logging failed. See rpl_gtid_strict.test

Rename and drop via DDL log

  We replay ddl_log_state_rm to drop the old table and rename the
  temporary table. In that case we must throw the correct error
  message if ddl_log_revert() fails (f.ex. on FK error).

  If table is deleted earlier and not via DDL log and the crash
  happened, the create chain is not closed. Linked drop chain is not
  executed and the new table is not installed. But the old table is
  already deleted.

ddl_log.cc changes

  Now we can place action before DDL_LOG_DROP_INIT_ACTION and it will
  be replayed after DDL_LOG_DROP_TABLE_ACTION.

  report_error parameter for ddl_log_revert() allows to fail at first
  error and print the error message if possible.
  ddl_log_execute_action() now can print error message.

  Since we now can handle errors from ddl_log_execute_action() (in
  case of non-recovery execution) unconditional setting "error= TRUE"
  is wrong (it was wrong anyway because it was overwritten at the end
  of the function).

On XID usage

  Like with all other atomic DDL operations XID is used to avoid
  inconsistency between master and slave in the case of a crash after
  binary log is written and before ddl_log_state_create is closed. On
  recovery XIDs are taken from binary log and corresponding DDL log
  events get disabled.  That is done by
  ddl_log_close_binlogged_events().

On linking two chains together

  Chains are executed in the ascending order of entry_pos of execute
  entries. But entry_pos assignment order is undefined: it may assign
  bigger number for the first chain and then smaller number for the
  second chain. So the execution order in that case will be reverse:
  second chain will be executed first.

  To avoid that we link one chain to another. While the base chain
  (ddl_log_state_create) is active the secondary chain
  (ddl_log_state_rm) is not executed. That is: only one chain can be
  executed in two linked chains.

  The interface ddl_log_link_chains() was done in "MDEV-22166
  ddl_log_write_execute_entry() extension".

More on CREATE OR REPLACE .. SELECT

  We use create_and_open_tmp_table() like in ALTER TABLE to create
  temporary TABLE object (tmp_table is (NON_)TRANSACTIONAL_TMP_TABLE).

  After we created such TABLE object we use create_info->tmp_table()
  instead of table->s->tmp_table when we need to check for
  parser-requested tmp-table.

  External locking is required for temporary table created by
  create_and_open_tmp_table(). F.ex. that disables logging for Aria
  transactional tables and without that (when no mysql_lock_tables()
  is done) it cannot work correctly.

  For making external lock the patch requires Aria table to work in
  non-transactional mode. That is usually done by
  ha_enable_transaction(false). But we cannot disable transaction
  completely because: 1. binlog rollback removes pending row events
  (binlog_remove_pending_rows_event()). The row events are added
  during CREATE .. SELECT data insertion phase. 2. replication slave
  highly depends on transaction and cannot work without it.

  So we put temporary Aria table into non-transactional mode with
  "thd->transaction->on hack". See comment for on_save variable.

  Note that Aria table has internal_table mode. But we cannot use it
  because:

  if (!internal_table)
  {
    mysql_mutex_lock(&THR_LOCK_myisam);
    old_info= test_if_reopen(name_buff);
  }

  For internal_table test_if_reopen() is not called and we get a new
  MARIA_SHARE for each file handler. In that case duplicate errors are
  missed because insert and lookup in CREATE .. SELECT is done via two
  different handlers (see create_lookup_handler()).

  For temporary table before dropping TABLE_SHARE by
  drop_temporary_table() we must do ha_reset(). ha_reset() releases
  storage share. Without that the share is kept and the second CREATE
  OR REPLACE .. SELECT fails with:

    HA_ERR_TABLE_EXIST (156): MyISAM table '#sql-create-b5377-4-t2' is
    in use (most likely by a MERGE table). Try FLUSH TABLES.

    HA_EXTRA_PREPARE_FOR_DROP also removes MYISAM_SHARE, but that is
    not needed as ha_reset() does the job.

  ha_reset() is usually done by
  mark_tmp_table_as_free_for_reuse(). But we don't need that mechanism
  for our temporary table.

Atomic_info in HA_CREATE_INFO

  Many functions in CREATE TABLE pass the same parameters. These
  parameters are part of table creation info and should be in
  HA_CREATE_INFO (or whatever). Passing parameters via single
  structure is much easier for adding new data and
  refactoring.

InnoDB changes (revised by Marko Mäkelä)

  row_rename_table_for_mysql(): Specify the treatment of FOREIGN KEY
  constraints in a 4-valued enum parameter. In cases where FOREIGN KEY
  constraints cannot exist (partitioned tables, or internal tables of
  FULLTEXT INDEX), we can use the mode RENAME_IGNORE_FK.
  The mod RENAME_REBUILD is for any DDL operation that rebuilds the
  table inside InnoDB, such as TRUNCATE and native ALTER TABLE
  (or OPTIMIZE TABLE). The mode RENAME_ALTER_COPY is used solely
  during non-native ALTER TABLE in ha_innobase::rename_table().
  Normal ha_innobase::rename_table() will use the mode RENAME_FK.

  CREATE OR REPLACE will rename the old table (if one exists) along
  with its FOREIGN KEY constraints into a temporary name. The replacement
  table will be initially created with another temporary name.
  Unlike in ALTER TABLE, all FOREIGN KEY constraints must be renamed
  and not inherited as part of these operations, using the mode RENAME_FK.

  dict_get_referenced_table(): Let the callers convert names when needed.

  create_table_info_t::create_foreign_keys(): CREATE OR REPLACE creates
  the replacement table with a temporary name table, so for
  self-references foreign->referenced_table will be a table with
  temporary name and charset conversion must be skipped for it.

Reviewed by:

  Michael Widenius <monty@mariadb.org>
2022-08-31 11:55:04 +03:00
Aleksey Midenkov
65e0d0ea66 MDEV-25292 Removed thd argument in ddl_log functions 2022-08-31 11:55:03 +03:00
Aleksey Midenkov
595dad83ad MDEV-25292 Cleanups
ddl_log_write_execute_entry() cleanup

Rename functions renamed:
  do_rename() -> rename_table_and_triggers()
  do_rename_temporary() -> rename_temporary_table()
  check_rename() -> rename_check_preconditions()
2022-08-31 11:55:03 +03:00
Marko Mäkelä
e67d46e4a1 Merge 10.6 into 10.7 2022-03-14 11:30:32 +02:00
Sergei Golubchik
e0dc22b2d4 MDEV-27753 Incorrect ENGINE type of table after crash for CONNECT table
fix two null pointer dereferences
2022-03-12 13:13:58 +01:00
Aleksey Midenkov
32e8e84795 MDEV-27048 UBSAN: runtime error: shift exponent 32 is too large for 32-bit type 'unsigned int'
32-bit variable must be expanded to 64-bit before shift left.
2021-11-19 18:06:48 +03:00
Aleksey Midenkov
7da721be31 Review and crash-safety fix 2021-10-26 17:07:46 +02:00
Aleksey Midenkov
d324c03d0c Vanilla cleanups and refactorings
Dead code cleanup:

part_info->num_parts usage was wrong and working incorrectly in
mysql_drop_partitions() because num_parts is already updated in
prep_alter_part_table(). We don't have to update part_info->partitions
because part_info is destroyed at alter_partition_lock_handling().

Cleanups:

- DBUG_EVALUATE_IF() macro replaced by shorter form DBUG_IF();
- Typo in ER_KEY_COLUMN_DOES_NOT_EXITS.

Refactorings:

- Splitted write_log_replace_delete_frm() into write_log_delete_frm()
  and write_log_replace_frm();
- partition_info via DDL_LOG_STATE;
- set_part_info_exec_log_entry() removed.

DBUG_EVALUATE removed

DBUG_EVALUTATE was only added for consistency together with
DBUG_EVALUATE_IF. It is not used anywhere in the code.

DBUG_SUICIDE() fix on release build

On release DBUG_SUICIDE() was statement. It was wrong as
DBUG_SUICIDE() is used in expression context.
2021-10-26 17:07:46 +02:00
Aleksey Midenkov
2dc3c32070 MDEV-25292 Better debug trace
Improves readability of DDL log debug traces.
2021-10-26 17:07:46 +02:00
Monty
7e31cfda64 Errors from failed drop tables where ignored by atomic ddl. 2021-09-07 13:50:59 +03:00
Monty
87de0e2129 Minor cleanups of atomic ddl code
- Rename DDL_IGNORE_LOG_ENTRY_CODE to DDL_LOG_IGNORE_ENTRY_CODE
  This makes it similar to all other ddl_log_entry_code's.
- Added some new comments
- ddl_log_revert() now returns != 0 if revert didn't succeed
2021-06-19 03:46:00 +03:00
Monty
ad02d53a71 Create a backup file of ddl_recovery.log before starting recovery
This is done by prefixing -backup.log to the --log-ddl-recovery file.
The reason for this is to have a copy of the original ddl log file
if ddl recovery does not succeed.
2021-05-19 22:54:13 +02:00
Monty
7762ee5dbe MDEV-25180 Atomic ALTER TABLE
MDEV-25604 Atomic DDL: Binlog event written upon recovery does not
           have default database

The purpose of this task is to ensure that ALTER TABLE is atomic even if
the MariaDB server would be killed at any point of the alter table.
This means that either the ALTER TABLE succeeds (including that triggers,
the status tables and the binary log are updated) or things should be
reverted to their original state.

If the server crashes before the new version is fully up to date and
commited, it will revert to the original table and remove all
temporary files and tables.
If the new version is commited, crash recovery will use the new version,
and update triggers, the status tables and the binary log.
The one execption is ALTER TABLE .. RENAME .. where no changes are done
to table definition. This one will work as RENAME and roll back unless
the whole statement completed, including updating the binary log (if
enabled).

Other changes:
- Added handlerton->check_version() function to allow the ddl recovery
  code to check, in case of inplace alter table, if the table in the
  storage engine is of the new or old version.
- Added handler->table_version() so that an engine can report the current
  version of the table. This should be changed each time the table
  definition changes.
- Added  ha_signal_ddl_recovery_done() and
  handlerton::signal_ddl_recovery_done() to inform all handlers when
  ddl recovery has been done. (Needed by InnoDB).
- Added handlerton call inplace_alter_table_committed, to signal engine
  that ddl_log has been closed for the alter table query.
- Added new handerton flag
  HTON_REQUIRES_NOTIFY_TABLEDEF_CHANGED_AFTER_COMMIT to signal when we
  should call hton->notify_tabledef_changed() during
  mysql_inplace_alter_table. This was required as MyRocks and InnoDB
  needed the call at different times.
- Added function server_uuid_value() to be able to generate a temporary
  xid when ddl recovery writes the query to the binary log. This is
  needed to be able to handle crashes during ddl log recovery.
- Moved freeing of the frm definition to end of mysql_alter_table() to
  remove duplicate code and have a common exit strategy.

-------
InnoDB part of atomic ALTER TABLE
(Implemented by Marko Mäkelä)
innodb_check_version(): Compare the saved dict_table_t::def_trx_id
to determine whether an ALTER TABLE operation was committed.

We must correctly recover dict_table_t::def_trx_id for this to work.
Before purge removes any trace of DB_TRX_ID from system tables, it
will make an effort to load the user table into the cache, so that
the dict_table_t::def_trx_id can be recovered.

ha_innobase::table_version(): return garbage, or the trx_id that would
be used for committing an ALTER TABLE operation.

In InnoDB, table names starting with #sql-ib will remain special:
they will be dropped on startup. This may be revisited later in
MDEV-18518 when we implement proper undo logging and rollback
for creating or dropping multiple tables in a transaction.

Table names starting with #sql will retain some special meaning:
dict_table_t::parse_name() will not consider such names for
MDL acquisition, and dict_table_rename_in_cache() will treat such
names specially when handling FOREIGN KEY constraints.

Simplify InnoDB DROP INDEX.
Prevent purge wakeup

To ensure that dict_table_t::def_trx_id will be recovered correctly
in case the server is killed before ddl_log_complete(), we will block
the purge of any history in SYS_TABLES, SYS_INDEXES, SYS_COLUMNS
between ha_innobase::commit_inplace_alter_table(commit=true)
(purge_sys.stop_SYS()) and purge_sys.resume_SYS().
The completion callback purge_sys.resume_SYS() must be between
ddl_log_complete() and MDL release.

--------

MyRocks support for atomic ALTER TABLE
(Implemented by Sergui Petrunia)

Implement these SE API functions:
- ha_rocksdb::table_version()
- hton->check_version = rocksdb_check_versionMyRocks data dictionary
  now stores table version for each table.
  (Absence of table version record is interpreted as table_version=0,
  that is, which means no upgrade changes are needed)
- For inplace alter table of a partitioned table, call the underlying
  handlerton when checking if the table is ok. This assumes that the
  partition engine commits all changes at once.
2021-05-19 22:54:13 +02:00
Monty
3c578b0a96 Check if we can rename triggers before doing an ALTER TABLE ... RENAME
ALTER TABLE .. RENAME, when used with the inplace algorithm, does:
- Do an inplace or online alter to the new definition
- Rename to new name
- Update triggers.

If update triggers would fail, we would rename the table back.
The problem with this approach is that the table would have the new
definition but the rename would fail.  The binary log would also not be
updated.

The solution to this is to very early check if we can rename triggers
and give an error if this would fail.
Both ALTER TABLE ... RENAME and RENAME TABLE is fixed.

This was implemented by moving the pre-check of rename table in triggers
from Table_triggers_list::change_table_name() to
Table_triggers_list::prepare_for_rename().
2021-05-19 22:54:13 +02:00
Monty
ffe7f19fa6 MDEV-24746 Atomic CREATE TRIGGER
The purpose of this task is to ensure that CREATE TRIGGER is atomic

When a trigger is created, we first create a trigger_name.TRN file and then
create or update the table_name.TRG files.
This is done by creating .TRN~ and .TRG~ files and replacing (or creating)
the result files.

The new logic is

- Log CREATE TRIGGER to DDL log, with a marker if old trigger existsted
- If old .TRN or .TRG files exists, make backup copies of these
- Create the new .TRN and .TRG files as before
- Remove the backups

Crash recovery
- If query has been logged to binary log:
  - delete any left over backup files
- else
   - Delete any old .TRN~ or .TRG~ files
   - If there was orignally some triggers (old .TRG file existed)
      - If we crashed before creating all backup files
         - Delete existing backup files
      - else
         - Restore backup files
      - end
   - Delete .TRN and .TRG file (as there was no triggers before

One benefit of the new code is that CREATE OR REPLACE TRIGGER is now
totally atomic even if there existed an old trigger: Either the old
trigger will be replaced or the old one will be left untouched.

Other things:
- If sql_create_definition_file() would fail, there could be memory leaks
  in CREATE TRIGGER, DROP TRIGGER or CREATE OR REPLACE TRIGGER.  This
  is now fixed.
2021-05-19 22:54:13 +02:00
Monty
d494abd175 MDEV-24607 Atomic CREATE VIEW
The logic of the new code is:
- Log CREATE view to DDL log, with a marker if old view existed
- If old view exists (in case of CREATE or REPLACE view), make a copy
  of the old view as view_name.frm-
- Create the new view definition file
- Delete copy of view if it was created.

Crash recovery:
- Delete view_name.frm~ file (Temporary file for view definition)
- If query was logged to binary log
  - Delete copy of view if it exists
- else
   -rename the copy of the view over the .frm file (restoring the
    old definition)

One benefit of the new code is that CREATE OR REPLACE VIEW for an
existing view is no fully atomic: Either the view will be replaced or
the old one will be left unchanged.
2021-05-19 22:54:13 +02:00
Monty
6aa9a552c2 MDEV-24576 Atomic CREATE TABLE
There are a few different cases to consider

Logging of CREATE TABLE and CREATE TABLE ... LIKE
- If REPLACE is used and there was an existing table, DDL log the drop of
  the table.
- If discovery of table is to be done
    - DDL LOG create table
  else
    - DDL log create table (with engine type)
    - create the table
- If table was created
  - Log entry to binary log with xid
  - Mark DDL log completed

Crash recovery:
- If query was in binary log do nothing and exit
- If discoverted table
   - Delete the .frm file
-else
   - Drop created table and frm file
- If table was dropped, write a DROP TABLE statement in binary log

CREATE TABLE ... SELECT required a little more work as when one is using
statement logging the query is written to the binary log before commit is
done.
This was fixed by adding a DROP TABLE to the binary log during crash
recovery if the ddl log entry was not closed. In this case the binary log
will contain:
CREATE TABLE xxx ... SELECT ....
DROP TABLE xxx;

Other things:
- Added debug_crash_here() functionality to Aria to be able to test
  crash in create table between the creation of the .MAI and the .MAD files.
2021-05-19 22:54:13 +02:00
Monty
7a588c30b1 MDEV-24408 Crash-safe DROP DATABASE
Description of how DROP DATABASE works after this patch

- Collect list of tables
- DDL log tables as they are dropped
- DDL log drop database
- Delete db.opt
- Delete data directory
- Log either DROP TABLE or DROP DATABASE to binary log
- De active ddl log entry

This is in line of how things where before (minus ddl logging) except that
we delete db.opt file last to not loose it if DROP DATABASE fails.

On recovery we have to ensure that all dropped tables are logged in
binary log and that they are properly dropped (as with atomic drop
table).
No new tables be dropped as part of recovery.

Recovery of active drop database ddl log entry:

- If drop database was logged to ddl log but was not found in the binary
  log:
  - drop the db.opt file and database directory.
  - Log DROP DATABASE to binary log
- If drop database was not logged to ddl log
  - Update binary log with DROP TABLE of the dropped tables. If table list
    is longer than max_allowed_packet, then the query will be split into
    multiple DROP TABLE/VIEW queries.

Other things:
- Added DDL_LOG_STATE and 'current database' as arguments to
  mysql_rm_table_no_locks(). This was needed to be able to combine
  ddl logging of DROP DATABASE and DROP TABLE and make the generated
  DROP TABLE statements shorter.
- To make the DROP TABLE statement created by ddl log shorter, I changed
  the binlogged query to use current directory and omit the directory
  part for all tables in the current directory.
- Merged some DROP TABLE and DROP VIEW code in ddl logger.  This was done
  to be able get separate DROP VIEW and DROP TABLE statements in the binary
  log.
- Added a 'recovery_state' variable to remember the state of dropped
  tables and views.
- Moved out code that drops database objects (stored procedures) from
  mysql_rm_db_internal() to drop_database_objects() for better code reuse.
- Made mysql_rm_db_internal() global so that could be used by the ddl
  recovery code.
2021-05-19 22:54:13 +02:00
Monty
407e9b78cf MDEV-24395 Atomic DROP TRIGGER
The purpose of this task is to ensure that DROP TRIGGER is atomic.

Description of how atomic drop trigger works:

Logging of DROP TRIGGER
    Log the following information:
    db
    table name
    trigger name
    xid /* Used to check if query was already logged to binary log */
    initial length of the .TRG file
    query if there is space for it, if not log a zero length query.

Recovery operations:
- Delete if exists 'database/trigger_name.TRN~'
  - If this file existed, it means that we crashed before the trigger
    was deleted and there is nothing else to do.
- Get length of .TRG file
  - If file length is unchanged, trigger was not dropped. Nothing else to
    do.
- Log original query to binary, if it was stored in the ddl log. If it was
  not stored (long query string), log the following query to binary log:
  use `database` ; DROP TRIGGER IF EXISTS `trigger_name`
  /* generated by ddl log */;

Other things:
- Added trigger name and DDL_LOG_STATE to drop_trigger()
  Trigger name was added to make the interface more consistent and
  more general.
2021-05-19 22:54:13 +02:00
Monty
e3cfb7c803 MDEV-23844 Atomic DROP TABLE (single table)
Logging logic:
- Log tables, just before they are dropped, to the ddl log
- After the last table for the statement is dropped, log an xid for the
  whole ddl log event

In case of crash:
- Remove first any active DROP TABLE events from the ddl log that matches
  xids found in binary log (this mean the drop was successful and was
  propery logged).
- Loop over all active DROP TABLE events
  - Ensure that the table is completely dropped
- Write a DROP TABLE entry to the binary log with the dropped tables.

Other things:
- Added code to ha_drop_table() to be able to tell the difference if
  a get_new_handler() failed because of out-of-memory or because the
  handler refused/was not able to create a a handler. This was needed
  to get sequences to work as sequences needs a share object to be passed
  to get_new_handler()
- TC_LOG_BINLOG::recover() was changed to always collect Xid's from the
  binary log and always call ddl_log_close_binlogged_events(). This was
  needed to be able to collect DROP TABLE events with embedded Xid's
  (used by ddl log).
- Added a new variable "$grep_script" to binlog filter to be able to find
  only rows that matches a regexp.
- Had to adjust some test that changed because drop statements are a bit
  larger in the binary log than before (as we have to store the xid)

Other things:
- MDEV-25588 Atomic DDL: Binlog query event written upon recovery is corrupt
  fixed (in the original commit).
2021-05-19 22:54:12 +02:00
Monty
47010ccffa MDEV-23842 Atomic RENAME TABLE
- Major rewrite of ddl_log.cc and ddl_log.h
  - ddl_log.cc described in the beginning how the recovery works.
  - ddl_log.log has unique signature and is dynamic. It's easy to
    add more information to the header and other ddl blocks while still
    being able to execute old ddl entries.
  - IO_SIZE for ddl blocks is now dynamic. Can be changed without affecting
    recovery of old logs.
  - Code is more modular and is now usable outside of partition handling.
  - Renamed log file to dll_recovery.log and added option --log-ddl-recovery
    to allow one to specify the path & filename.
- Added ddl_log_entry_phase[], number of phases for each DDL action,
  which allowed me to greatly simply set_global_from_ddl_log_entry()
- Changed how strings are stored in log entries, which allows us to
  store much more information in a log entry.
- ddl log is now always created at start and deleted on normal shutdown.
  This simplices things notable.
- Added probes debug_crash_here() and debug_simulate_error() to simply
  crash testing and allow crash after a given number of times a probe
  is executed. See comments in debug_sync.cc and rename_table.test for
  how this can be used.
- Reverting failed table and view renames is done trough the ddl log.
  This ensures that the ddl log is tested also outside of recovery.
- Added helper function 'handler::needs_lower_case_filenames()'
- Extend binary log with Q_XID events. ddl log handling is using this
  to check if a ddl log entry was logged to the binary log (if yes,
  it will be deleted from the log during ddl_log_close_binlogged_events()
- If a DDL entry fails 3 time, disable it. This is to ensure that if
  we have a crash in ddl recovery code the server will not get stuck
  in a forever crash-restart-crash loop.

mysqltest.cc changes:
- --die will now replace $variables with their values
- $error will contain the error of the last failed statement

storage engine changes:
- maria_rename() was changed to be more robust against crashes during
  rename.
2021-05-19 22:54:12 +02:00
Monty
188b0b99cf Rename all external ddl_log function to start with ddl_log_ prefix
Rename deactivate_ddl_log_entry to ddl_log_increment_phase
2021-05-19 22:54:11 +02:00
Monty
02b6cef45e Move all ddl log code to ddl_log.cc and ddl_log.h
Part of prepration for: MDEV-17567 Atomic DDL

No notable code changes except moving code around
2021-05-19 22:54:11 +02:00