Commit graph

25038 commits

Author SHA1 Message Date
Marko Mäkelä
a635588b56 MDEV-25236 Online log apply fails for ROW_FORMAT=REDUNDANT tables
In other ROW_FORMAT than REDUNDANT, the InnoDB record header
size calculation depends on dict_index_t::n_core_null_bytes.

In ROW_FORMAT=REDUNDANT, the record header always is 6 bytes
plus n_fields or 2*n_fields bytes, depending on the maximum
record size. But, during online ALTER TABLE, the log records
in the temporary file always use a format similar to
ROW_FORMAT=DYNAMIC, even omitting the 5-byte fixed-length part
of the header.

While creating a temporary file record for a ROW_FORMAT=REDUNDANT
table, InnoDB must refer to dict_index_t::n_nullable.
The field dict_index_t::n_core_null_bytes is only valid for
other than ROW_FORMAT=REDUNDANT tables.

The bug does not affect MariaDB 10.3, because only
commit 7a27db778e (MDEV-15563)
allowed an ALGORITHM=INSTANT change of a NOT NULL column to
NULL in a ROW_FORMAT=REDUNDANT table.

The fix was developed by Thirunarayanan Balathandayuthapani
and tested by Matthias Leich. The test case was simplified by me.
2021-07-02 16:11:01 +03:00
Marko Mäkelä
15dcb8bd3e Merge 10.4 into 10.5 2021-07-02 13:02:26 +03:00
Marko Mäkelä
315380a4d1 MDEV-26052 Assertion prebuilt->trx_id < table->def_trx_id failed
ha_innobase::truncate(): If the operation fails, preserve
also dict_table_t::def_trx_id.

This fixes a regression that had been introduced in
commit 1bd681c8b3 (MDEV-25506).
2021-07-01 19:06:53 +03:00
Marko Mäkelä
ed6b230744 MDEV-25919 preparation: Remove trx_t::internal
With commit 1bd681c8b3 (MDEV-25506)
it no longer is necessary to run DDL and DML operations in
separate transactions. Let us remove the flag trx_t::internal.
Dictionary transactions will be distinguished by trx_t::dict_operation.
2021-07-01 17:51:55 +03:00
Marko Mäkelä
0a67b15a9d Cleanup: Remove pointer indirection for trx_t::xid
The trx_t::xid is always allocated, so we might as well allocate it
directly in the trx_t object to improve the locality of reference.
2021-07-01 16:38:24 +03:00
Marko Mäkelä
83234719f1 MDEV-24671 fixup: Fix an off-by-one error
In commit e71e613353 we
accidentally made innodb_lock_wait_timeout=100000000
a "literal" value, not the smallest special value that
would mean "infinite" timeout.
2021-07-01 16:37:01 +03:00
Marko Mäkelä
8c5c3a4594 MDEV-26067 innodb_lock_wait_timeout values above 100,000,000 are useless
The practical maximum value of the parameter innodb_lock_wait_timeout
is 100,000,000. Any value larger than that specifies an infinite timeout.

Therefore, we should make 100,000,000 the maximum value of the parameter.
2021-07-01 10:31:08 +03:00
Sergei Petrunia
eebe2090c8 Merge 10.3 -> 10.4 2021-06-30 18:41:46 +03:00
Sergei Petrunia
586870f9ef Merge 10.2->10.3 2021-06-30 15:06:54 +03:00
Sergei Golubchik
6fab256bc8 disable spider/bugfix.wait_timeout 2021-06-30 09:34:26 +02:00
Sergei Golubchik
fa5c314377 fix spider tests for --ps in 10.6
see also c3a1ba0fd9, 068246c006, 690ae1de45
2021-06-30 09:34:26 +02:00
Marko Mäkelä
ff9150f3c5 MDEV-25942: Assertion failure in trx_t::drop_table()
trx_t::drop_table(): Relax also another assertion that would fail
due to an AUTO_INCREMENT lock that is being held by the current
test case. This should have been part of
commit 63e9a05440.
2021-06-30 09:00:52 +03:00
Marko Mäkelä
0237e9bb65 MDEV-26041 Recovery failure due to delete-marked SYS_FIELDS record
trx_t::drop_table(): Delete-mark the SYS_TABLES and SYS_INDEXES
record before delete-marking any SYS_COLUMNS or SYS_FIELDS records.
Otherwise, dict_load_indexes() could fail on recovery. This fixes up
commit 1bd681c8b3 (MDEV-25506).
2021-06-29 15:20:33 +03:00
Marko Mäkelä
e04bbf73dc MDEV-25496 Assertion 'trx->bulk_insert' failed on INSERT
row_get_prebuilt_insert_row(): Remove some fallback code that had been
added in commit 8ea923f55b (MDEV-24818).
It seems that after all, statement boundaries are being reliably
indicated by ha_innobase::start_stmt() or
(for partitioned tables) ha_innobase::external_lock().
2021-06-29 15:20:33 +03:00
Marko Mäkelä
30edd5549d MDEV-26029: Sparse files are inefficient on thinly provisioned storage
The MariaDB implementation of page_compressed tables for InnoDB used
sparse files. In the worst case, in the data file, every data page
will consist of some data followed by a hole. This may be extremely
inefficient in some file systems.

If the underlying storage device is thinly provisioned (can compress
data on the fly), it would be good to write regular files (with sequences
of NUL bytes at the end of each page_compressed block) and let the
storage device take care of compressing the data.

For reads, sparse file regions and regions containing NUL bytes will be
indistinguishable.

my_test_if_disable_punch_hole(): A new predicate for detecting thinly
provisioned storage. (Not implemented yet.)

innodb_atomic_writes: Correct the comment.

buf_flush_page(): Support all values of fil_node_t::punch_hole.
On a thinly provisioned storage device, we will always write
NUL-padded innodb_page_size bytes also for page_compressed tables.

buf_flush_freed_pages(): Remove a redundant condition.

fil_space_t::atomic_write_supported: Remove. (This was duplicating
fil_node_t::atomic_write.)

fil_space_t::punch_hole: Remove. (Duplicated fil_node_t::punch_hole.)

fil_node_t: Remove magic_n, and consolidate flags into bitfields.
For punch_hole we introduce a third value that indicates a
thinly provisioned storage device.

fil_node_t::find_metadata(): Detect all attributes of the file.
2021-06-29 15:18:22 +03:00
Marko Mäkelä
58252fff15 MDEV-26040 os_file_set_size() may not work on O_DIRECT files
os_file_set_size(): Trim the current size down to the file system
block size, to obey the constraints for unbuffered I/O.
2021-06-29 14:28:23 +03:00
Marko Mäkelä
63e9a05440 MDEV-25942: Assertion !table.n_waiting_or_granted_auto_inc_locks
trx_t::drop_table(): Remove a bogus debug assertion.
The current transaction may hold an AUTO_INCREMENT
lock on the table while
CREATE TABLE t2 (pk INT AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB SELECT...
is being rolled back due to lock wait timeout.
Remaining debug assertions will check that only this transaction
is holding locks on the table, and that one of them is an exclusive lock.
2021-06-28 15:37:29 +03:00
Marko Mäkelä
891a927e80 Merge 10.5 into 10.6 2021-06-26 11:53:28 +03:00
Marko Mäkelä
fc2ff46469 MDEV-26017: Assertion stat.flush_list_bytes <= curr_pool_size
buf_flush_relocate_on_flush_list(): If we are removing the block from
buf_pool.flush_list, subtract its size from buf_pool.stat.flush_list_bytes.
This fixes a regression that was introduced in
commit 22b62edaed (MDEV-25113).
2021-06-26 11:52:25 +03:00
Marko Mäkelä
aa95c42360 Cleanup: Remove unused mtr_block_dirtied 2021-06-26 11:17:05 +03:00
Marko Mäkelä
759deaa0a2 MDEV-26010 fixup: Use acquire/release memory order
In commit 5f22511e35 we depend on
Total Store Ordering. For correct operation on ISAs that implement
weaker memory ordering, we must explicitly use release/acquire stores
and loads on buf_page_t::oldest_modification_ to prevent a race condition
when buf_page_t::list does not happen to be on the same cache line.

buf_page_t::clear_oldest_modification(): Assert that the block is
not in buf_pool.flush_list, and use std::memory_order_release.

buf_page_t::oldest_modification_acquire(): Read oldest_modification_
with std::memory_order_acquire. In this way, if the return value is 0,
the caller may safely assume that it will not observe the buf_page_t
as being in buf_pool.flush_list, even if it is not holding
buf_pool.flush_list_mutex.

buf_flush_relocate_on_flush_list(), buf_LRU_free_page():
Invoke buf_page_t::oldest_modification_acquire().
2021-06-26 11:16:40 +03:00
Olivier Bertrand
1c4b917f0e Fix clang compile error in value.cpp 2021-06-25 18:35:26 +02:00
Olivier Bertrand
7863554e42 Fix clang compile error in tabjson and typo in tabbson.cpp 2021-06-25 12:54:34 +02:00
Olivier Bertrand
af7a0895ac Fix clang compile error in tabjson.cpp and tabbson.cpp 2021-06-25 12:30:10 +02:00
Olivier Bertrand
6239e2a4ec Fix clang compile error in tabjson.cpp and tabbson.cpp 2021-06-25 11:38:16 +02:00
Olivier Bertrand
5c5d60b897 Fix compile error in value.h and remove unused variables in ha_connect.cc 2021-06-25 01:15:51 +02:00
Olivier Bertrand
330b32ebd9 Fix clang compile error 2021-06-24 23:56:13 +02:00
Marko Mäkelä
a8350cfb5e Merge 10.5 into 10.6 2021-06-24 21:56:44 +03:00
Marko Mäkelä
5f22511e35 MDEV-26010: Assertion lsn > 2 failed in buf_pool_t::get_oldest_modification
In commit 22b62edaed (MDEV-25113)
we introduced a race condition. buf_LRU_free_page() would read
buf_page_t::oldest_modification() as 0 and assume that
buf_page_t::list can be used (for attaching the block to the
buf_pool.free list). In the observed race condition,
buf_pool_t::delete_from_flush_list() had cleared the field,
and buf_pool_t::delete_from_flush_list_low() was executing
concurrently with buf_LRU_block_free_non_file_page(),
which resulted in buf_pool.flush_list.end becoming corrupted.

buf_pool_t::delete_from_flush_list(), buf_flush_relocate_on_flush_list():
First remove the block from buf_pool.flush_list, and only then
invoke buf_page_t::clear_oldest_modification(), to ensure that
reading oldest_modification()==0 really implies that the block
no longer is in buf_pool.flush_list.
2021-06-24 21:55:10 +03:00
Marko Mäkelä
e329dc8d86 MDEV-25948 fixup: Demote a warning to a note
buf_dblwr_t::recover(): Issue a note, not a warning, about
pages whose FIL_PAGE_LSN is in the future. This was supposed to be
part of commit 762bcb81b5 (MDEV-25948)
but had been accidentally omitted.
2021-06-24 18:51:05 +03:00
Marko Mäkelä
82fe83a34c MDEV-26012 InnoDB purge and shutdown hangs after failed ALTER TABLE
ha_innobase::commit_inplace_alter_table(): Invoke
purge_sys.resume_FTS() on all error handling paths
if purge_sys.stop_FTS() had been called.

This fixes a regression that had been introduced in
commit 1bd681c8b3 (MDEV-25506).
2021-06-24 16:07:27 +03:00
Marko Mäkelä
033e29b6a1 MDEV-26007 Rollback unnecessarily initiates redo log write
trx_t::commit_in_memory(): Do not initiate a redo log write if
the transaction has no visible effect. If anything for this
transaction had been made durable, crash recovery will roll back
the transaction just fine even if the end of ROLLBACK is not
durably written.

Rollbacks of transactions that are associated with XA identifiers
(possibly internally via the binlog) will always be persisted.
The test rpl.rpl_gtid_crash covers this.
2021-06-24 15:00:34 +03:00
Marko Mäkelä
b4c9cd201b Merge 10.5 into 10.6 2021-06-24 12:39:34 +03:00
Marko Mäkelä
60ed479711 MDEV-26004 Excessive wait times in buf_LRU_get_free_block()
buf_LRU_get_free_block(): Initially wait for a single block to be
freed, signaled by buf_pool.done_free. Only if that fails and no
LRU eviction flushing batch is already running, we initiate a
flushing batch that should serve all threads that are currently
waiting in buf_LRU_get_free_block().

Note: In an extreme case, this may introduce a performance regression
at larger numbers of connections. We observed this in sysbench
oltp_update_index with 512MiB buffer pool, 4GiB of data on fast NVMe,
and 1000 concurrent connections, on a 20-thread CPU. The contention point
appears to be buf_pool.mutex, and the improvement would turn into a
regression somewhere beyond 32 concurrent connections.

On slower storage, such regression was not observed; instead, the
throughput was improving and maximum latency was reduced.

The excessive waits were pointed out by Vladislav Vaintroub.
2021-06-24 11:01:18 +03:00
Olivier Bertrand
ed70f76cf7 - Make function strz return null when LEX_STRING is null
modified:   storage/connect/ha_connect.cc

- Use NOTE instead of WARNING in connect_assisted_discovery
  This because MariaDB raise an error when doing so
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/tabrest.cpp

- Make MONGO tables recognize STRINGIFY and JsonAllPath
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/mongo.h
  modified:   storage/connect/tabcmg.h
  modified:   storage/connect/tabjmg.cpp
  modified:   storage/connect/tabcmg.cpp
  modified:   storage/connect/tabjmg.h

- Fix OBJECT option for Pretty != 2 JSN and BSON tables
  Accept all syntaxes  for the OBJECT option
  modified:   storage/connect/tabbson.cpp
  modified:   storage/connect/tabjson.cpp

- Use my_snprintf in function xcurl (by vuvova)
  modified:   storage/connect/tabrest.cpp

- Format dates entered as integer when formatted
  modified:   storage/connect/value.cpp
  modified:   storage/connect/value.h
2021-06-24 00:46:12 +02:00
Marko Mäkelä
101da87228 Merge 10.5 into 10.6 2021-06-23 19:36:45 +03:00
Marko Mäkelä
6441bc614a MDEV-25113: Introduce a page cleaner mode before 'furious flush'
MDEV-23855 changed the way how the page cleaner is signaled by
user threads. If a threshold is exceeded, a mini-transaction commit
would invoke buf_flush_ahead() in order to initiate page flushing
before all writers would eventually grind to halt in
log_free_check(), waiting for the checkpoint age to reduce.

However, buf_flush_ahead() would always initiate 'furious flushing',
making the buf_flush_page_cleaner thread write innodb_io_capacity_max
pages per batch, and sleeping no time between batches, until the
limit LSN is reached. Because this could saturate the I/O subsystem,
system throughput could significantly reduce during these
'furious flushing' spikes.

With this change, we introduce a gentler version of flush-ahead,
which would write innodb_io_capacity_max pages per second until
the 'soft limit' is reached.

buf_flush_ahead(): Add a parameter to specify whether furious flushing
is requested.

buf_flush_async_lsn: Similar to buf_flush_sync_lsn, a limit for
the less intrusive flushing.

buf_flush_page_cleaner(): Keep working until buf_flush_async_lsn
has been reached.

log_close(): Suppress a warning message in the event that a new log
is being created during startup, when old logs did not exist.
Return what type of page cleaning will be needed.

mtr_t::finish_write(): Also when m_log.is_small(), invoke log_close().
Return what type of page cleaning will be needed.

mtr_t::commit(): Invoke buf_flush_ahead() based on the return value of
mtr_t::finish_write().
2021-06-23 19:06:52 +03:00
Marko Mäkelä
22b62edaed MDEV-25113: Make page flushing faster
buf_page_write_complete(): Reduce the buf_pool.mutex hold time,
and do not acquire buf_pool.flush_list_mutex at all.
Instead, mark blocks clean by setting oldest_modification to 1.
Dirty pages of temporary tables will be identified by the special
value 2 instead of the previous special value 1.
(By design of the ib_logfile0 format, actual LSN values smaller
than 2048 are not possible.)

buf_LRU_free_page(), buf_pool_t::get_oldest_modification()
and many other functions will remove the garbage (clean blocks)
from buf_pool.flush_list while holding buf_pool.flush_list_mutex.

buf_pool_t::n_flush_LRU, buf_pool_t::n_flush_list:
Replaced with non-atomic variables, protected by buf_pool.mutex,
to avoid unnecessary synchronization when modifying the counts.

export_vars: Remove unnecessary indirection for
innodb_pages_created, innodb_pages_read, innodb_pages_written.
2021-06-23 19:06:52 +03:00
Marko Mäkelä
8af538979b MDEV-25801: buf_flush_dirty_pages() is very slow
In commit 7cffb5f6e8 (MDEV-23399)
the implementation of buf_flush_dirty_pages() was replaced with
a slow one, which would perform excessive scans of the
buf_pool.flush_list and make little progress.

buf_flush_list(), buf_flush_LRU(): Split from buf_flush_lists().
Vladislav Vaintroub noticed that we will not need to invoke
log_flush_task.wait() for the LRU eviction flushing.

buf_flush_list_space(): Replaces buf_flush_dirty_pages().
This is like buf_flush_list(), but operating on a single
tablespace at a time. Writes at most innodb_io_capacity
pages. Returns whether some of the tablespace might remain
in the buffer pool.
2021-06-23 19:06:52 +03:00
Marko Mäkelä
762bcb81b5 MDEV-25948 Remove log_flush_task
Vladislav Vaintroub suggested that invoking log_flush_up_to()
for every page could perform better than invoking a log write
between buf_pool.flush_list batches, like we started doing in
commit 3a9a3be1c6 (MDEV-23855).
This could depend on the sequence in which pages are being
modified. The buf_pool.flush_list is ordered by
oldest_modification, while the FIL_PAGE_LSN of the pages is
theoretically independent of that. In the pathological case,
we will wait for a log write before writing each individual page.

It turns out that we can defer the call to log_flush_up_to()
until just before submitting the page write. If the doublewrite
buffer is being used, we can submit a write batch of "future" pages
to the doublewrite buffer, and only wait for the log write right
before we are writing an already doublewritten page.
The next doublewrite batch will not be initiated before the last
page write from the current batch has completed.

When a future version introduces asynchronous writes if the log,
we could initiate a write at the start of a flushing batch, to
reduce waiting further.
2021-06-23 19:06:52 +03:00
Marko Mäkelä
6dfd44c828 MDEV-25954: Trim os_aio_wait_until_no_pending_writes()
It turns out that we had some unnecessary waits for no outstanding
write requests to exist. They were basically working around a
bug that was fixed in MDEV-25953.

On write completion callback, blocks will be marked clean.
So, it is sufficient to consult buf_pool.flush_list to determine
which writes have not been completed yet.

On FLUSH TABLES...FOR EXPORT we must still wait for all pending
asynchronous writes to complete, because buf_flush_file_space()
would merely guarantee that writes will have been initiated.
2021-06-23 19:06:49 +03:00
Marko Mäkelä
6e12ebd4a7 MDEV-25062: Reduce trx_rseg_t::mutex contention
redo_rseg_mutex, noredo_rseg_mutex: Remove the PERFORMANCE_SCHEMA keys.
The rollback segment mutex will be uninstrumented.

trx_sys_t: Remove pointer indirection for rseg_array, temp_rseg.
Align each element to the cache line.

trx_sys_t::rseg_id(): Replaces trx_rseg_t::id.

trx_rseg_t::ref: Replaces needs_purge, trx_ref_count, skip_allocation
in a single std::atomic<uint32_t>.

trx_rseg_t::latch: Replaces trx_rseg_t::mutex.

trx_rseg_t::history_size: Replaces trx_sys_t::rseg_history_len

trx_sys_t::history_size_approx(): Replaces trx_sys.rseg_history_len
in those places where the exact count does not matter. We must not
acquire any trx_rseg_t::latch while holding index page latches, because
normally the trx_rseg_t::latch is acquired before any page latches.

trx_sys_t::history_exists(): Replaces trx_sys.rseg_history_len!=0
with an approximation.

We remove some unnecessary trx_rseg_t::latch acquisition around
trx_undo_set_state_at_prepare() and trx_undo_set_state_at_finish().
Those operations will only access fields that remain constant
after trx_rseg_t::init().
2021-06-23 13:42:11 +03:00
Marko Mäkelä
b3e8788009 MDEV-25967: Correctly extend deferred-recovery files
recv_sys_t::recover_deferred(): Set the file size to match the number
of pages. Mariabackup might copy the file while it was being extended.
2021-06-23 13:37:11 +03:00
Marko Mäkelä
592a925c0c MDEV-25996 sux_lock::s_lock(): Assertion !have_s() failed on startup
dict_check_sys_tables(): Correctly advance the cursor position.
This fixes a regression that was caused by
commit 49e2c8f0a6 (MDEV-25743).
2021-06-23 13:36:04 +03:00
Marko Mäkelä
3a566de22d Merge 10.5 into 10.6 2021-06-23 09:24:32 +03:00
Marko Mäkelä
344e59904d Merge 10.4 into 10.5 2021-06-23 08:17:49 +03:00
Marko Mäkelä
09b03ff31b Merge 10.3 into 10.4 2021-06-23 08:05:27 +03:00
Marko Mäkelä
9fc67c6bf3 MDEV-25950 Ignoring strange row from mysql.innodb_index_stats after DDL
commit_try_rebuild(): Invoke trx_t::drop_table_statistics()
with the correct (original) name of the table.
This fixes a regression that was introduced in
commit 1bd681c8b3 (MDEV-25506 part 3).
2021-06-22 17:52:54 +03:00
Marko Mäkelä
e1c953ef23 MDEV-25989 Crash (or hang) on startup after restoring backup
If an ALTER TABLE that affected a table with FULLTEXT INDEX was
in progress while a backup was made, InnoDB would crash or hang
during the first startup after restoring the backup,
while trying to drop the #sql-alter- table for the DDL operation.

drop_garbage_tables_after_restore(): Invoke dict_sys.unlock()
before invoking the FTS functions. Also, invoke purge_sys.stop_FTS()
in debug builds to silence debug assertions. (Purge is not yet
running at this point.)
2021-06-22 15:25:58 +03:00
Marko Mäkelä
35a9aaebe2 MDEV-25981 InnoDB upgrade fails
trx_undo_mem_create_at_db_start(): Relax too strict upgrade checks
that were introduced in
commit e46f76c974 (MDEV-15912).
On commit, pages will typically be set to TRX_UNDO_CACHED state.
Having the type TRX_UNDO_INSERT in such pages is common and
unproblematic; the type would be reset in trx_undo_reuse_cached().

trx_rseg_array_init(): On failure, clean up the rollback segments
that were initialized so far, to avoid an assertion failure later
during shutdown.
2021-06-22 09:30:25 +03:00