Problem:- When setting max_binlog_stmt_cache_size=18446744073709547520
from either command line or .cnf file, server fails to start.
Solution:- Added one more function eval_num_suffix_ull , which uses
strtoull to get unsigned ulonglong from string. And getopt_ull calls this
function instead of eval_num_suffix. Also changed previous eval_num_suffix to
eval_num_suffix_ll to remain consistent.
==== Description ====
Flashback can rollback the instances/databases/tables to an old snapshot.
It's implement on Server-Level by full image format binary logs (--binlog-row-image=FULL), so it supports all engines.
Currently, it’s a feature inside mysqlbinlog tool (with --flashback arguments).
Because the flashback binlog events will store in the memory, you should check if there is enough memory in your machine.
==== New Arguments to mysqlbinlog ====
--flashback (-B)
It will let mysqlbinlog to work on FLASHBACK mode.
==== New Arguments to mysqld ====
--flashback
Setup the server to use flashback. This enables binary log in row mode
and will enable extra logging for DDL's needed by flashback feature
==== Example ====
I have a table "t" in database "test", we can compare the output with "--flashback" and without.
#client/mysqlbinlog /data/mysqldata_10.0/binlog/mysql-bin.000001 -vv -d test -T t --start-datetime="2013-03-27 14:54:00" > /tmp/1.sql
#client/mysqlbinlog /data/mysqldata_10.0/binlog/mysql-bin.000001 -vv -d test -T t --start-datetime="2013-03-27 14:54:00" -B > /tmp/2.sql
Then, importing the output flashback file (/tmp/2.log), it can flashback your database/table to the special time (--start-datetime).
And if you know the exact postion, "--start-postion" is also works, mysqlbinlog will output the flashback logs that can flashback to "--start-postion" position.
==== Implement ====
1. As we know, if binlog_format is ROW (binlog-row-image=FULL in 10.1 and later), all columns value are store in the row event, so we can get the data before mis-operation.
2. Just do following things:
2.1 Change Event Type, INSERT->DELETE, DELETE->INSERT.
For example:
INSERT INTO t VALUES (...) ---> DELETE FROM t WHERE ...
DELETE FROM t ... ---> INSERT INTO t VALUES (...)
2.2 For Update_Event, swapping the SET part and WHERE part.
For example:
UPDATE t SET cols1 = vals1 WHERE cols2 = vals2
--->
UPDATE t SET cols2 = vals2 WHERE cols1 = vals1
2.3 For Multi-Rows Event, reverse the rows sequence, from the last row to the first row.
For example:
DELETE FROM t WHERE id=1; DELETE FROM t WHERE id=2; ...; DELETE FROM t WHERE id=n;
--->
DELETE FROM t WHERE id=n; ...; DELETE FROM t WHERE id=2; DELETE FROM t WHERE id=1;
2.4 Output those events from the last one to the first one which mis-operation happened.
For example:
Change default to zlib, this has effect only if user has
explicitly requested page compression and then user
naturally expects that pages are really compressed
if they can be compressed.
Most notably, this includes MDEV-11623, which includes a fix and
an upgrade procedure for the InnoDB file format incompatibility
that is present in MariaDB Server 10.1.0 through 10.1.20.
In other words, this merge should address
MDEV-11202 InnoDB 10.1 -> 10.2 migration does not work
restarting server with encryption and read-only
buf0buf.cc: Temporary slots used in encryption was calculated
by read_threads * write_threads. However, in read-only mode
write_threads is zero. Correct way is to calculate
(read_threads + write_threads) * max pending IO requests.
When a query containing a WITH clause is printed by EXPLAIN
EXTENDED command there should not be any data expansion in
the query specifications of the WITH elements of this WITH
clause.
This is the final preparation for the merge of MDEV-11623
from MariaDB Server 10.1 (correcting the assignment of
FSP_SPACE_FLAGS in MariaDB Server 10.1).
We must avoid reading FSP_SPACE_FLAGS directly from page 0,
because the flags that affect the uncompressed page size
cannot be trusted if we are upgrading from MariaDB 10.1.
We will pass fil_space_t* instead of a numeric tablespace ID
in many low-level functions, so that fil_space_t::flags
will be available. This will also reduce the amount o
tablespace ID lookups and the pressure on fil_system->mutex.
fil_space_is_being_truncated(), fil_space_is_redo_skipped(),
PageCallback::set_page_size(), fsp_header_get_page_size(): Remove.
fil_node_open_file(), fil_space_get_crypt_data(): Use space->flags.
fsp_free_extent(), fseg_get_first_extent(), fsp_get_space_header(),
xdes_get_descriptor_with_space_hdr(), xdes_get_descriptor(),
xdes_lst_get_descriptor(), fsp_space_modify_check(),
fsp_init_file_page(), fsp_alloc_free_extent(), fsp_page_create(),
fsp_alloc_free_page(), fsp_free_page(), fsp_alloc_seg_inode_page(),
fsp_alloc_seg_inode(), fsp_free_seg_inode(), fseg_fill_free_list(),
fseg_mark_page_used(), fseg_free_page_low(), fseg_free_extent():
Take fil_space_t* as a parameter, instead of taking a numeric ID.
InnoDB keeps track of buffer-fixed buf_block_t or acquired rw_lock_t
within a mini-transaction. There are some memo_contains assertions
in the code that document when certain blocks or rw_locks must be held.
But, these assertions only check the mini-transaction memo, not the fact
whether the rw_lock_t are actually being held by the caller.
btr_pcur_store_position(): Remove #ifdef, and assert that the block
is always buffer-fixed.
rtr_pcur_getnext_from_path(), rtr_pcur_open_low(),
ibuf_rec_get_page_no_func(), ibuf_rec_get_space_func(),
ibuf_rec_get_info_func(), ibuf_rec_get_op_type_func(),
ibuf_build_entry_from_ibuf_rec_func(), ibuf_rec_get_volume_func(),
ibuf_get_merge_page_nos_func(), ibuf_get_volume_buffered_count_func()
ibuf_get_entry_counter_low_func(), page_set_ssn_id(),
row_vers_old_has_index_entry(), row_vers_build_for_consistent_read(),
row_vers_build_for_semi_consistent_read(),
trx_undo_prev_version_build():
Make use of mtr_memo_contains_page_flagged().
mtr_t::memo_contains(): Take a const memo. Assert rw_lock_own().
FindPage, FlaggedCheck: Assert rw_lock_own_flagged().
innodb_file_format=Barracuda is the default in MariaDB 10.2.
Do not set it, because the option will be removed in MariaDB 10.3.
Also, do not set innodb_file_per_table=1 because it is the default.
Note that MDEV-11828 should fix the test innodb.innodb-64k
already in 10.1.
When MySQL 5.7.9 (and MariaDB Server 10.2) introduced
innodb_default_row_format and made ROW_FORMAT=DYNAMIC the default,
it became possible to create any ROW_FORMAT tables in the InnoDB
system tablespace, except ROW_FORMAT=COMPRESSED.
In MySQL 5.7, it is possible to create ROW_FORMAT=DYNAMIC
tables when TABLESPACE=innodb_system is explicitly specified.
Because MariaDB Server 10.2 does not support the MySQL 5.7
TABLESPACE=innodb_system attribute for tables, we should allow
ROW_FORMAT=DYNAMIC when innodb_file_per_table=0.
Also, remove the test innodb_zip.innodb-create-options, which was
an outdated copy of innodb_zip.create_options.
MySQL 5.7 allows temporary tables to be created in ROW_FORMAT=COMPRESSED.
The usefulness of this is questionable. WL#7899 in MySQL 8.0.0
prevents the creation of such compressed tables, so that all InnoDB
temporary tables will be located inside the predefined
InnoDB temporary tablespace.
Pick up and adjust some tests from MySQL 5.7 and 8.0.
dict_tf_to_fsp_flags(): Remove the parameter is_temp.
fsp_flags_init(): Remove the parameter is_temporary.
row_mysql_drop_temp_tables(): Remove. There cannot be any temporary
tables in InnoDB. (This never removed #sql* tables in the datadir
which were created by DDL.)
dict_table_t::dir_path_of_temp_table: Remove.
create_table_info_t::m_temp_path: Remove.
create_table_info_t::create_options_are_invalid(): Do not allow
ROW_FORMAT=COMPRESSED or KEY_BLOCK_SIZE for temporary tables.
create_table_info_t::innobase_table_flags(): Do not unnecessarily
prevent CREATE TEMPORARY TABLE with SPATIAL INDEX.
(MySQL 5.7 does allow this.)
fil_space_belongs_in_lru(): The only FIL_TYPE_TEMPORARY tablespace
is never subjected to closing least-recently-used files.
MySQL 5.7 introduced partial support for user-created shared tablespaces
(for example, import and export are not supported).
MariaDB Server does not support tablespaces at this point of time.
Let us remove most InnoDB code and data structures that is related
to shared tablespaces.
The MariaDB 10.1 page_compression is incompatible with the Oracle
implementation that was introduced in MySQL 5.7 later.
Remove the Oracle implementation. Also remove the remaining traces of
MYSQL_ENCRYPTION.
This will also remove traces of PUNCH_HOLE until it is implemented
better. The only effective call to os_file_punch_hole() was in
fil_node_create_low() to test if the operation is supported for the file.
In other words, it looks like page_compression is not working in
MariaDB 10.2, because no code equivalent to the 10.1 os_file_trim()
is enabled.
MariaDB will likely never support MySQL-style encryption for
InnoDB, because we cannot link with the Oracle encryption plugin.
This is preparation for merging MDEV-11623.
in Item_partition_func_safe_string(THD *thd, const char *name_arg,
uint length, CHARSET_INFO *cs= NULL), the 'name_arg' is the value
of the string constant and 'length' is the length of this constant,
so length == strlen(name_arg).
check_that_all_fields_are_given_values() relied on write_set,
but was run too early, before triggers updated write_set.
also, when triggers are present, fields might get values conditionally,
so we need to check that all fields are given values for every row.
In Debian, the default install is made more secure by omitting the anonymous
user and by making the root account authenticate by unix socket
authentication instead of the default password-less root. However, Debian
hard-codes this change in mysql_install_db, which breaks that program for
other users.
This commit instead implements new general options for mysql_install_db that
can be used by anyone to similarly perform a more secure install:
--skip-auth-anonymous-user: omits the anonymous user.
--auth-root-authentication-method=normal: Keeps the existing behaviour
with a password-less root account. Currently on by default.
--auth-root-socket-user=USER
--auth-root-authentication-method=socket: creates the MariaDB root user
with the name USER (defaults to 'root') and using unix socket
authentication. This way, only that user has MariaDB root access
after install.
The idea with --auth-root-authentication-method=normal is that
applications that need this behaviour can give that option explicitly.
Then eventually we could make --auth-root-authentication-method=socket
the default, giving a more secure default installation.
Note that it is perfectly possible to do a secure install with
--auth-root-authentication-method=normal. For example, installing a
private server just for local access by a single OS-level user, by
using --skip-networking and putting the connection socket in a
location without public access. So it is important to preserve this
API for backwards compatibility.
Gtid_list_log_event::do_apply_event() did not free_root(thd->mem_root).
It can allocate on this in record_gtid(), and in some scenarios there is
nothing else that does free_root(), leading to temporary memory leak until
stop of SQL thread. One scenario is in circular replication with only one
master active. The active master receives only its own events on the slave,
all of which are ignored. But whenever the SQL thread catches up with the IO
thread, a Gtid_list_log_event is applied, leading to the leak.
The INFORMATION_SCHEMA view INNODB_TEMP_TABLE_INFO was added to
MySQL 5.7 as part of the work to implement temporary tables
without any redo logging.
The only use case of this view was SELECT COUNT(*) in some tests,
to see how many temporary tables exist in InnoDB. The columns do
not report much useful information. For example, the table name
would not be the user-specified table name, but a generated #sql
name. Also, the session that created the table is not identified.
* Remove duplicate lines from tests
* Use thd instead of current_thd
* Remove extra wsrep_binlog_format_names
* Correctly merge union patch from 5.5 wrt duplicate rows.
* Correctly merge SELinux changes into 10.1
The function fsp_flags_try_adjust(), which is called on startup,
is incrementing the Innodb_pages0_read counts for every affected file.
Adjust the result of encryption.innodb_lotoftables accordingly.
innodb.doublewrite: Similar to what was done to innodb.101_compatibility,
add an explicit $_ parameter to the Perl unpack function.
Also, fix some diagnostic messages in the Perl code.
innodb.innodb-wl5522-debug: Adjust for the changed error codes and
messages on fault injection.
Apparently, in older versions of Perl unpack does not have a logic
for using $_ as a default value for the second argument. Fixed by
specifying it explicitly
MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K
The storage format of FSP_SPACE_FLAGS was accidentally broken
already in MariaDB 10.1.0. This fix is bringing the format in
line with other MySQL and MariaDB release series.
Please refer to the comments that were added to fsp0fsp.h
for details.
This is an INCOMPATIBLE CHANGE that affects users of
page_compression and non-default innodb_page_size. Upgrading
to this release will correct the flags in the data files.
If you want to downgrade to earlier MariaDB 10.1.x, please refer
to the test innodb.101_compatibility how to reset the
FSP_SPACE_FLAGS in the files.
NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret
uncompressed data files with innodb_page_size=4k or 64k as
compressed innodb_page_size=16k files, and then probably fail
when trying to access the pages. See the comments in the
function fsp_flags_convert_from_101() for detailed analysis.
Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16.
In this way, compressed innodb_page_size=16k tablespaces will not
be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20.
Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the
dict_table_t::flags when the table is available, in
fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace().
During crash recovery, fil_load_single_table_tablespace() will use
innodb_compression_level for the PAGE_COMPRESSION_LEVEL.
FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags
that are not to be written to FSP_SPACE_FLAGS. Currently, these will
include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR.
Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support
one innodb_page_size for the whole instance.
When creating a dummy tablespace for the redo log, use
fil_space_t::flags=0. The flags are never written to the redo log files.
Remove many FSP_FLAGS_SET_ macros.
dict_tf_verify_flags(): Remove. This is basically only duplicating
the logic of dict_tf_to_fsp_flags(), used in a debug assertion.
fil_space_t::mark: Remove. This flag was not used for anything.
fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter
mark_space, and add a parameter for table flags. Check that
fil_space_t::flags match the table flags, and adjust the (memory-only)
flags based on the table flags.
fil_node_open_file(): Remove some redundant or unreachable conditions,
do not use stderr for output, and avoid unnecessary server aborts.
fil_user_tablespace_restore_page(): Convert the flags, so that the
correct page_size will be used when restoring a page from the
doublewrite buffer.
fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove.
It suffices to have fil_space_is_page_compressed().
FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL,
FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not
exist in the FSP_SPACE_FLAGS but only in memory.
fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS
in page 0. Called by fil_open_single_table_tablespace(),
fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql()
except if --innodb-read-only is active.
fsp_flags_is_valid(ulint): Reimplement from the scratch, with
accurate comments. Do not display any details of detected
inconsistencies, because the output could be confusing when
dealing with MariaDB 10.1.x data files.
fsp_flags_convert_from_101(ulint): Convert flags from buggy
MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags
cannot be in MariaDB 10.1.x format.
fsp_flags_match(): Check the flags when probing files.
Implemented based on fsp_flags_is_valid()
and fsp_flags_convert_from_101().
dict_check_tablespaces_and_store_max_id(): Do not access the
page after committing the mini-transaction.
IMPORT TABLESPACE fixes:
AbstractCallback::init(): Convert the flags.
FetchIndexRootPages::operator(): Check that the tablespace flags match the
table flags. Do not attempt to convert tablespace flags to table flags,
because the conversion would necessarily be lossy.
PageConverter::update_header(): Write back the correct flags.
This takes care of the flags in IMPORT TABLESPACE.
contains a bad and a good copy
Clean up the InnoDB doublewrite buffer code.
buf_dblwr_init_or_load_pages(): Do not add empty pages to the buffer.
buf_dblwr_process(): Do consider changes to pages that are all zero.
Do not abort when finding a corrupted copy of a page in the doublewrite
buffer, because there could be multiple copies in the doublewrite buffer,
and only one of them needs to be good.
