In any test that uses wait_all_purged.inc, ensure that InnoDB tables
will be created without persistent statistics.
This is a follow-up to commit cd04673a17
after a similar failure was observed in the innodb_zip.blob test.
The motivation of introducing the parameter
innodb_purge_rseg_truncate_frequency in
mysql/mysql-server@28bbd66ea5 and
mysql/mysql-server@8fc2120fed
seems to have been to avoid stalls due to freeing undo log pages
or truncating undo log tablespaces. In MariaDB Server,
innodb_undo_log_truncate=ON should be a much lighter operation
than in MySQL, because it will not involve any log checkpoint.
Another source of performance stalls should be
trx_purge_truncate_rseg_history(), which is shrinking the history list
by freeing the undo log pages whose undo records have been purged.
To alleviate that, we will introduce a purge_truncation_task that will
offload this from the purge_coordinator_task. In that way, the next
innodb_purge_batch_size pages may be parsed and purged while the pages
from the previous batch are being freed and the history list being shrunk.
The processing of innodb_undo_log_truncate=ON will still remain the
responsibility of the purge_coordinator_task.
purge_coordinator_state::count: Remove. We will ignore
innodb_purge_rseg_truncate_frequency, and act as if it had been
set to 1 (the maximum shrinking frequency).
purge_coordinator_state::do_purge(): Invoke an asynchronous task
purge_truncation_callback() to free the undo log pages.
purge_sys_t::iterator::free_history(): Free those undo log pages
that have been processed. This used to be a part of
trx_purge_truncate_history().
purge_sys_t::clone_end_view(): Take a new value of purge_sys.head
as a parameter, so that it will be updated while holding exclusive
purge_sys.latch. This is needed for race-free access to the field
in purge_truncation_callback().
Reviewed by: Vladislav Lesin
purge_sys_t::sees(): Wrapper for view.sees().
trx_purge_truncate_history(): Invoke purge_sys.sees() instead of
comparing to head.trx_no, to determine if undo pages can be safely freed.
The test innodb.cursor-restore-locking was adjusted by Vladislav Lesin,
as was the the debug instrumentation in row_purge_del_mark().
Reviewed by: Vladislav Lesin
It is not safe to invoke trx_purge_free_segment() or execute
innodb_undo_log_truncate=ON before all undo log records in
the rollback segment has been processed.
A prominent failure that would occur due to premature freeing of
undo log pages is that trx_undo_get_undo_rec() would crash when
trying to copy an undo log record to fetch the previous version
of a record.
If trx_undo_get_undo_rec() was not invoked in the unlucky time frame,
then the symptom would be that some committed transaction history is
never removed. This would be detected by CHECK TABLE...EXTENDED that
was impleented in commit ab0190101b.
Such a garbage collection leak should be possible even when using
innodb_undo_log_truncate=OFF, just involving trx_purge_free_segment().
trx_rseg_t::needs_purge: Change the type from Boolean to a transaction
identifier, noting the most recent non-purged transaction, or 0 if
everything has been purged. On transaction start, we initialize this
to 1 more than the transaction start ID. On recovery, the field may be
adjusted to the transaction end ID (TRX_UNDO_TRX_NO) if it is larger.
The field TRX_UNDO_NEEDS_PURGE becomes write-only; only some debug
assertions that would validate the value. The field reflects the old
inaccurate Boolean field trx_rseg_t::needs_purge.
trx_undo_mem_create_at_db_start(), trx_undo_lists_init(),
trx_rseg_mem_restore(): Remove the parameter max_trx_id.
Instead, store the maximum in trx_rseg_t::needs_purge,
where trx_rseg_array_init() will find it.
trx_purge_free_segment(): Contiguously hold a lock on
trx_rseg_t to prevent any concurrent allocation of undo log.
trx_purge_truncate_rseg_history(): Only invoke trx_purge_free_segment()
if the rollback segment is empty and there are no pending transactions
associated with it.
trx_purge_truncate_history(): Only proceed with innodb_undo_log_truncate=ON
if trx_rseg_t::needs_purge indicates that all history has been purged.
Tested by: Matthias Leich
btr_pcur_store_position(): Assert that the 'default row' record never
is the only record in a page. (If that would happen, an empty
root page would be re-created in the non-instant format, not containing
the special record.) When the cursor is positioned on the page infimum,
never use the 'default row' as the BTR_PCUR_BEFORE reference.
(This is additional cleanup, not fixing the bug.)
rec_copy_prefix_to_buf(): When converting a record prefix to
the non-instant-add format, copy the original number of null flags.
Rename the variable instant_len to instant_omit, and introduce a
few more variables to make the code easiser to read.
Note: In purge, rec_copy_prefix_to_buf() is also used for storing the
persistent cursor position on a 'default row' record. The stored record
reference will be garbage, but row_search_on_row_ref() will do special
handling to reposition the cursor on the 'default row', based on
ref->info_bits.
innodb.dml_purge: Also cover the 'default row'.
The rollback of the modification of a pre-existing record
should involve a purge-like operation. Before MDEV-12288
the only purge-like operation was the removal of a
delete-marked record.
After MDEV-12288, any rollback of updating an existing record
must reset the DB_TRX_ID column when it is no longer visible
in the purge read view.
row_vers_must_preserve_del_marked(): Remove. It is cleaner to
perform the check directly in row0umod.cc.
row_trx_id_offset(): Auxiliary function to retrieve the byte
offset of DB_TRX_ID in a clustered index leaf page record.
row_undo_mod_must_purge(): Determine if a record should be purged.
row_undo_mod_clust(): For temporary tables, skip the purge checks.
When rolling back an update so that the original record was not
delete-marked, reset DB_TRX_ID if the history is no longer visible.
srv_purge_should_exit(): Remove the parameter n_purged.
If we happened to have n_purged==0 while some transaction was still
active, and then that transaction was added to the history list,
we were prematurely stopping the purge. It is more appropriate to
first check for trx_sys.any_active_transactions() == 0
(this count can only decrease during shutdown) and then for
trx_sys.history_size() == 0 (that count typically decreases, but
can increase when any remaining active transactions are committed
or rolled back).
innodb.dml_purge: Remove a server restart, and explicitly wait for
purge, and use FLUSH TABLE FOR EXPORT to read the file contents.
This will make the test run faster, easier to debug, and also
allow it to run with --embedded. This might also help repeat
MDEV-11802 better. The issue MDEV-13603 remains will remain tested
by innodb.table_flags.
In the test, SET GLOBAL innodb_purge_rseg_truncate_frequency = 1
in order to ensure that everything is purged in the slow shutdown.
This could be viewed as a flaw of the slow shutdown.
This should have been part of MDEV-12288.
trx_undo_t::del_marks: Remove.
Purge needs to process all undo log records in order to
reset the DB_TRX_ID. Before MDEV-12288, it sufficed to only delete
the purgeable delete-marked records, and it ignore other undo log.
trx_rseg_t::needs_purge: Renamed from trx_rseg_t::last_del_marks.
Indicates whether a rollback segment needs to be processed by purge.
TRX_UNDO_NEEDS_PURGE: Renamed from TRX_UNDO_DEL_MARKS.
Indicates whether a rollback segment needs to be processed by purge.
This will be 1 until trx_purge_free_segment() has been invoked.
row_purge_record_func(): Set the is_insert flag for TRX_UNDO_INSERT_REC,
so that the DB_ROLL_PTR will match in row_purge_reset_trx_id().
trx_purge_fetch_next_rec(): Add a comment about row_purge_record_func()
going to set the is_insert flag.
trx_purge_read_undo_rec(): Always attempt to read the undo log record.
trx_purge_get_next_rec(): Do not skip any undo log records.
Even when no clustered index record is going to be removed,
we may want to reset some DB_TRX_ID,DB_ROLL_PTR.
trx_undo_rec_get_cmpl_info(), trx_undo_rec_get_extern_storage(): Remove.
trx_purge_add_undo_to_history(): Set the TRX_UNDO_NEEDS_PURGE flag
so that the resetting will work on undo logs that were originally
created before MDEV-12288 (MariaDB 10.3.1).
trx_undo_roll_ptr_is_insert(), trx_purge_free_segment(): Cleanup
(should be no functional change).
