Commit graph

129 commits

Author SHA1 Message Date
Eugene Kosov
562c037b48 MDEV-18115 Remove dummy tablespace for the redo log
Redo log subsystem was decoupled from tablespace subsystem. It now manages file
descriptors for redo log files by itself.

FIL_TYPE_LOG: removed, code in various places was simplified

SRV_LOG_SPACE_FIRST_ID: renamed to SRV_SPACE_ID_UPPER_BOUND
  to better match its purpose. Code in various places was simplified

fil_n_log_flushes: replaced with log_sys::flushes
fil_n_pending_log_flushes: replaced with log_sys::pending_flushes

log_t::files::files: redo log file descriptors
log_t::files::file_names: redo log file names

log_t::files::set_file_names(): set file names without opening them
log_t::files::open_files(): opens redo log files
log_t::files::read(): treats several files as one big
log_t::files::write(): treats several files as one big
log_t::files::fsync(): flushes page cache to disk
log_t::files::close_files(): closes redo log files

fil_open_log_and_system_tablespace_files(): renamed to
  fil_open_system_tablespace_files()
  and obviously it now doesn't open redo log files

global files[1000]: removed. Why it was needed at all?
2020-01-01 22:09:51 +08:00
Vladislav Vaintroub
a808c18b73 Make .clang-format work with clang-8
Remove keywords that are too new.
2019-11-15 18:09:30 +01:00
Vladislav Vaintroub
5e62b6a5e0 MDEV-16264 Use threadpool for Innodb background work.
Almost all threads have gone
- the "ticking" threads, that sleep a while then do some work)
(srv_monitor_thread, srv_error_monitor_thread, srv_master_thread)
were replaced with timers. Some timers are periodic,
e.g the "master" timer.

- The btr_defragment_thread is also replaced by a timer , which
reschedules it self when current defragment "item" needs throttling

- the buf_resize_thread and buf_dump_threads are substitutes with tasks
Ditto with page cleaner workers.

- purge workers threads are not tasks as well, and purge cleaner
coordinator is a combination of a task and timer.

- All AIO is outsourced to tpool, Innodb just calls thread_pool::submit_io()
and provides the callback.

- The srv_slot_t was removed, and innodb_debug_sync used in purge
is currently not working, and needs reimplementation.
2019-11-15 18:09:30 +01:00
Oleksandr Byelkin
c07325f932 Merge branch '10.3' into 10.4 2019-05-19 20:55:37 +02:00
Marko Mäkelä
be85d3e61b Merge 10.2 into 10.3 2019-05-14 17:18:46 +03:00
Marko Mäkelä
26a14ee130 Merge 10.1 into 10.2 2019-05-13 17:54:04 +03:00
Vicențiu Ciorbaru
c0ac0b8860 Update FSF address 2019-05-11 19:25:02 +03:00
Marko Mäkelä
5c3ff5cb93 Merge 10.3 into 10.4 2019-04-02 11:04:54 +03:00
Sergei Golubchik
4e1d3f83b7 Merge branch '10.2' into 10.3 2019-03-29 19:41:41 +01:00
Sergei Golubchik
cc71e7501c post-merge: -Werror fixes in 10.2 2019-03-29 10:58:25 +01:00
Marko Mäkelä
0a1c3477bf MDEV-18493 Remove page_size_t
MySQL 5.7 introduced the class page_size_t and increased the size of
buffer pool page descriptors by introducing this object to them.

Maybe the intention of this exercise was to prepare for a future
where the buffer pool could accommodate multiple page sizes.
But that future never arrived, not even in MySQL 8.0. It is much
easier to manage a pool of a single page size, and typically all
storage devices of an InnoDB instance benefit from using the same
page size.

Let us remove page_size_t from MariaDB Server. This will make it
easier to remove support for ROW_FORMAT=COMPRESSED (or make it a
compile-time option) in the future, just by removing various
occurrences of zip_size.
2019-02-07 12:21:35 +02:00
Brave Galera Crew
36a2a185fe Galera4 2019-01-23 15:30:00 +04:00
Marko Mäkelä
0abd2766b1 Merge 10.2 into 10.3
Also, related to MDEV-15522, MDEV-17304, MDEV-17835,
remove the Galera xtrabackup tests, because xtrabackup never worked
with MariaDB Server 10.3 due to InnoDB redo log format changes.
2018-11-30 09:38:56 +02:00
Marko Mäkelä
447e493179 Remove some unnecessary InnoDB #include 2018-11-29 12:53:44 +02:00
Marko Mäkelä
7830fb7f45 Merge 10.2 into 10.3 2018-08-28 12:22:56 +03:00
Marko Mäkelä
9258097fa3 Merge 10.1 into 10.2 2018-08-21 15:20:34 +03:00
Marko Mäkelä
45dbd47026 MDEV-17003 service_manager_extend_timeout() being called too often
buf_dump(): Only generate the output when shutdown is in progress.

log_write_up_to(): Only generate the output before actually writing
to the redo log files.

srv_purge_should_exit(): Rate-limit the output, and instead of
displaying the work done, indicate the work that remains to be done
until the completion of the slow shutdown.
2018-08-21 12:33:40 +03:00
Marko Mäkelä
05459706f2 Merge 10.2 into 10.3 2018-08-03 15:57:23 +03:00
Marko Mäkelä
ef3070e997 Merge 10.1 into 10.2 2018-08-02 08:19:57 +03:00
Oleksandr Byelkin
865e807125 Merge branch '10.0' into 10.1 2018-07-31 11:58:29 +02:00
Marko Mäkelä
d17e9a02c4 Fix InnoDB/XtraDB warnings by GCC 8.2.0 2018-07-30 14:05:24 +03:00
Marko Mäkelä
de942c9f61 MDEV-15983 Reduce fil_system.mutex contention further
fil_space_t::n_pending_ops, n_pending_ios: Use a combination of
fil_system.mutex and atomic memory access for protection.

fil_space_t::release(): Replaces fil_space_release().
Does not acquire fil_system.mutex.

fil_space_t::release_for_io(): Replaces fil_space_release_for_io().
Does not acquire fil_system.mutex.
2018-04-23 13:15:54 +03:00
Vicențiu Ciorbaru
65eefcdc60 Merge remote-tracking branch '10.2' into 10.3 2018-04-12 12:41:19 +03:00
Vicențiu Ciorbaru
45e6d0aebf Merge branch '10.1' into 10.2 2018-04-10 17:43:18 +03:00
Daniel Black
1479273cdb MDEV-14705: slow innodb startup/shutdown can exceed systemd timeout
Use systemd EXTEND_TIMEOUT_USEC to advise systemd of progress

Move towards progress measures rather than pure time based measures.

Progress reporting at numberious shutdown/startup locations incuding:
* For innodb_fast_shutdown=0 trx_roll_must_shutdown() for rolling back incomplete transactions.
* For merging the change buffer (in srv_shutdown(bool ibuf_merge))
* For purging history, srv_do_purge

Thanks Marko for feedback and suggestions.
2018-04-06 09:58:14 +03:00
Marko Mäkelä
980dd09be6 Merge 10.2 into 10.3 2018-03-29 17:03:34 +03:00
Marko Mäkelä
6cccef21a6 MDEV-15720 ib_buffer_pool unnecessarily includes the temporary tablespace
The purpose of the InnoDB buffer pool dump is to allow InnoDB to be
restarted with the same persistent data pages in the buffer pool.

The InnoDB temporary tablespace that was introduced in MariaDB 10.2.2
is always reinitialized on restart. Therefore, it does not make sense
to attempt to dump or restore any pages of the temporary tablespace.
2018-03-29 13:22:16 +03:00
Vicențiu Ciorbaru
82aeb6b596 Merge branch '10.1' into 10.2 2018-03-21 10:36:49 +02:00
Vicențiu Ciorbaru
24b353162f Merge branch '10.0-galera' into 10.1 2018-03-19 15:21:01 +02:00
Daniel Black
8b54c31486 MDEV-8743: where O_CLOEXEC is available, use for innodb buf_dump
As this is the only moderately critical fopened for writing file,
create an alternate path to use open and fdopen for non-glibc platforms
that support O_CLOEXEC (BSDs).

Tested on Linux (by modifing the GLIBC defination) to take this
alternate path:

$ cd /proc/23874
$ more fdinfo/71
pos:    0
flags:  02100001
mnt_id: 24
$ ls -la fd/71
l-wx------. 1 dan dan 64 Mar 14 13:30 fd/71 -> /dev/shm/var_auto_i7rl/mysqld.1/data/ib_buffer_pool.incomplete
2018-03-15 12:07:43 +02:00
Daniel Black
fafec1a9cc MDEV-8743: where O_CLOEXEC is available, use for innodb buf_dump
As this is the only moderately critical fopened for writing file,
create an alternate path to use open and fdopen for non-glibc platforms
that support O_CLOEXEC (BSDs).

Tested on Linux (by modifing the GLIBC defination) to take this
alternate path:

$ cd /proc/23874
$ more fdinfo/71
pos:    0
flags:  02100001
mnt_id: 24
$ ls -la fd/71
l-wx------. 1 dan dan 64 Mar 14 13:30 fd/71 -> /dev/shm/var_auto_i7rl/mysqld.1/data/ib_buffer_pool.incomplete
2018-03-14 13:31:31 +11:00
Daniel Black
0a63c91ab0 MDEV-8743: fopen mode e (glibc only) to prevent galera SST scripts accessing server files
For RemoteDatafile::read_link_file and buffer poool dumps

Note: STR_O_CLOEXEC needs to be at the end of the fopen otherwise
fopen will return EINVAL.
2018-03-05 12:00:09 +11:00
Daniel Black
9b8d0d9ff9 MDEV-11455: test case for status variable innodb_buffer_pool_load_incomplete
Add innodb debug system variable, innodb_buffer_pool_load_pages_abort, to test
the behaviour of innodb_buffer_pool_load_incomplete.
(innodb_buufer_pool_dump_abort_loads.test)
2018-02-22 15:50:50 +11:00
Daniel Black
8440e8fa98 MDEV-11455: create status variable innodb_buffer_pool_load_incomplete
This status variable indicates that an innodb buffer pool load never
completed and dumping at shutdown would result in an incomplete dump file.

This status variable is set to 1 once a buffer pool loads. Upon a successful
load this status variable returns to 0.

With this status variable set, the system variable
innodb_buffer_pool_dump_at_shutdown==1 will have no effect as dumping after
an incomplete load will generate a less complete dump file than the current
one.

If a user aborts a buffer pool load by changing the system variable
innodb_buffer_pool_load_abort=1 will cause the the status variable
innodb_buffer_pool_load_incomplete to remain set to 1.

A shutdown that occurs while innodb is loading the buffer pool will
not save the buffer pool on shutdown.

A user may indirectly set innodb_buffer_pool_load_incomplete
to 0 by:
* Forcing a load, by setting innodb_buffer_pool_load_now=ON, or
* Forcing a dump, by setting innodb_buffer_pool_dump_now=ON

This will enable the next dump on shutdown to complete.

Signed-off-by: Daniel Black <daniel.black@au.ibm.com>
2018-02-22 15:50:47 +11:00
Sergei Golubchik
d4df7bc9b1 Merge branch 'github/10.0' into 10.1 2018-02-02 10:09:44 +01:00
Jan Lindström
c7e5feb259 Merge tag 'mariadb-10.0.34' into 10.0-galera
Conflicts:
	storage/innobase/lock/lock0lock.cc
	storage/xtradb/lock/lock0lock.cc
	storage/xtradb/lock/lock0wait.cc
	support-files/mysql.server.sh
2018-02-01 14:09:48 +02:00
Vicențiu Ciorbaru
b20f821e07 Fix Innodb ASAN error on init
Backport 7c03edf2fe from xtradb to innodb
2018-01-24 15:18:36 +02:00
Marko Mäkelä
b2dd5232d4 InnoDB: Remove ut_vsnprintf() and the use of my_vsnprintf(); use vsnprintf() 2017-11-13 04:32:56 +02:00
Marko Mäkelä
c19ef508b8 InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf() 2017-11-13 02:11:48 +02:00
Marko Mäkelä
17bd6ed29a Remove STATUS_VERBOSE (there is no visible output) 2017-11-12 17:27:36 +02:00
Jan Lindström
eba0120d8f Fix test failures on embedded server.
Problem was incorrect definition of wsrep_recovery,
trx_sys_update_wsrep_checkpoint and
trx_sys_read_wsrep_checkpoint functions causing
innodb_plugin not to load as there was undefined symbols.
2017-08-31 14:04:02 +03:00
Marko Mäkelä
a36c369bda Merge 10.1 into 10.2
For running the Galera tests, the variable my_disable_leak_check
was set to true in order to avoid assertions due to memory leaks
at shutdown.

Some adjustments due to MDEV-13625 (merge InnoDB tests from MySQL 5.6)
were performed. The most notable behaviour changes from 10.0 and 10.1
are the following:

* innodb.innodb-table-online: adjustments for the DROP COLUMN
behaviour change (MDEV-11114, MDEV-13613)

* innodb.innodb-index-online-fk: the removal of a (1,NULL) record
from the result; originally removed in MySQL 5.7 in the
Oracle Bug  fix
377774689b

* innodb.create-index-debug: disabled due to MDEV-13680
(the MySQL Bug  fix was not merged from 5.6 to 5.7.10)

* innodb.innodb-alter-autoinc: MariaDB 10.2 behaves like MySQL 5.6/5.7,
while MariaDB 10.0 and 10.1 assign different values when
auto_increment_increment or auto_increment_offset are used.
Also MySQL 5.6/5.7 exhibit different behaviour between
LGORITHM=INPLACE and ALGORITHM=COPY, so something needs to be tested
and fixed in both MariaDB 10.0 and 10.2.

* innodb.innodb-wl5980-alter: disabled because it would trigger an
InnoDB assertion failure (MDEV-13668 may need additional effort in 10.2)
2017-08-31 09:30:40 +03:00
Jan Lindström
b29f26d774 Fix test failures on embedded server.
Problem was incorrect definition of wsrep_recovery,
trx_sys_update_wsrep_checkpoint and
trx_sys_read_wsrep_checkpoint functions causing
innodb_plugin not to load as there was undefined symbols.
2017-08-31 08:38:26 +03:00
Jan Lindström
c23efc7d50 Merge remote-tracking branch 'origin/10.0-galera' into 10.1 2017-08-21 13:35:00 +03:00
Daniele Sciascia
3ef3c467ad MW-365 Do not load/dump innodb buffer pool with wsrep_recover 2017-08-11 14:15:27 +03:00
Marko Mäkelä
6e0e5eefbc Fix some printf format type mismatch 2017-06-06 12:04:29 +03:00
Marko Mäkelä
65e1399e64 Merge 10.0 into 10.1
Significantly reduce the amount of InnoDB, XtraDB and Mariabackup
code changes by defining pfs_os_file_t as something that is
transparently compatible with os_file_t.
2017-05-20 08:41:20 +03:00
Vicențiu Ciorbaru
b87873b221 Merge branch 'merge-innodb-5.6' into bb-10.0-vicentiu
This merge reverts commit 6ca4f693c1ce472e2b1bf7392607c2d1124b4293
from current 5.6.36 innodb.

Bug 	OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE
                       INDEX APPLIED BY UNCOMMITTED ROW
Problem:
========
row_search_for_mysql() does whole table traversal for range query
even though the end range is passed. Whole table traversal happens
when the record is not with in transaction read view.

Solution:
=========

Convert the innodb last record of page to mysql format and compare
with end range if the traversal of row_search_mvcc() exceeds 100,
no ICP involved. If it is out of range then InnoDB can avoid the
whole table traversal. Need to refactor the code little bit to
make it compile.

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
Reviewed-by: Knut Hatlen <knut.hatlen@oracle.com>
Reviewed-by: Dmitry Shulga <dmitry.shulga@oracle.com>
RB: 14660
2017-05-17 14:53:28 +03:00
Vicențiu Ciorbaru
0af9818240 5.6.36 2017-05-15 17:17:16 +03:00
Thirunarayanan Balathandayuthapani
88a84f49b3 Bug CRASH WHEN ASSIGNING MY_ERRNO - MISSING MY_THREAD_INIT IN BACKGROUND THREAD
Description:
===========
Add my_thread_init() and my_thread_exit() for background threads which
initializes and frees the st_my_thread_var structure.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 15003
2017-04-26 23:03:32 +03:00