Commit graph

269 commits

Author SHA1 Message Date
Marko Mäkelä
604fea1ad6 MDEV-12266: Remove dict_index_t::space
We can rely on the dict_table_t::space. All indexes of a table object
are always in the same tablespace. (For fulltext indexes, the data is
located in auxiliary tables, and these will continue to have their own
table objects, separate from the main table.)
2018-03-29 20:47:37 +03:00
Marko Mäkelä
e7980f9cee MDEV-12266: Add dict_index_t::set_modified(mtr)
This is a non-functional change (pure refactoring).
2018-03-29 20:47:36 +03:00
Sergei Golubchik
b1818dccf7 Merge branch '10.2' into 10.3 2018-03-28 17:31:57 +02:00
Marko Mäkelä
bd7ed1b923 MDEV-13935 INSERT stuck at state Unlocking tables
Revert the dead code for MySQL 5.7 multi-master replication (GCS),
also known as
WL#6835: InnoDB: GCS Replication: Deterministic Deadlock Handling
(High Prio Transactions in InnoDB).

Also, make innodb_lock_schedule_algorithm=vats skip SPATIAL INDEX,
because the code does not seem to be compatible with them.

Add FIXME comments to some SPATIAL INDEX locking code. It looks
like Galera write-set replication might not work with SPATIAL INDEX.
2018-03-16 15:50:04 +02:00
Daniel Black
b600f30786 MDEV-10814: Innodb large allocations - madvise - Don't dump
Note: Linux only

Core dumps of large buffer pool pages take time and space
and pose potential data expose in scenarios where data-at-rest
encryption is deployed.

Here we use madvise(MADV_DONT_DUMP) on large memory allocations
used by the innodb buffer pool, log_sys and recv_sys. The effect
of this system call is that these memory areas will not appear in
a core dump. Data from these buffers is rarely useful in fault
diagnosis.

log_sys and recv_sys structures now use large memory allocations
for their large buffer.

Debug builds don't include the madvise syscall and as such will
include full core dumps.

A function, buf_madvise_do_dump, is added but never called. It
is there to be called from a debugger to re-enable the core
dumping of all of these pages if for some reason the entire
contents of these buffers are needed.

Idea thanks to Hartmut Holzgraefe
2018-02-17 20:00:56 +11:00
Marko Mäkelä
b006d2ead4 Merge bb-10.2-ext into 10.3 2018-02-15 10:22:03 +02:00
Marko Mäkelä
609d0a9194 MDEV-14407 Assertion failure during rollback
Rollback attempted to dereference DB_ROLL_PTR=0, which cannot possibly
be a valid undo log pointer. A safer canonical value would be
roll_ptr_t(1) << ROLL_PTR_INSERT_FLAG_POS
which is what was chosen in MDEV-12288, corresponding to reset_trx_id.

No deterministic test case for the bug was found. The simplest test
cases may be related to MDEV-11415, which suppresses undo logging for
ALGORITHM=COPY operations. In those operations, in the spirit of
MDEV-12288, we should actually have written reset_trx_id instead of
using the transaction identifier of the current transaction
(and a bogus value of DB_ROLL_PTR=0). However, thanks to MySQL Bug#28432
which I had fixed in MySQL 5.6.8 as part of WL#6255, access to the
rebuilt table by earlier-started transactions should actually have been
refused with ER_TABLE_DEF_CHANGED.

reset_trx_id: Move the definition to data0type.cc and the declaration
to data0type.h.

btr_cur_ins_lock_and_undo(): When undo logging is disabled, use the
safe value that corresponds to reset_trx_id.

btr_cur_optimistic_insert(): Validate the DB_TRX_ID,DB_ROLL_PTR before
inserting into a clustered index leaf page.

ins_node_t::sys_buf[]: Replaces row_id_buf and trx_id_buf and some
heap usage.

row_ins_alloc_sys_fields(): Init ins_node_t::sys_buf[] to reset_trx_id.

row_ins_buf(): Only if undo logging is enabled, copy trx->id
to node->sys_buf. Otherwise, rely on the initialization in
row_ins_alloc_sys_fields().

row_purge_reset_trx_id(): Invoke mlog_write_string() with reset_trx_id
directly. (No functional change.)

trx_undo_page_report_modify(): Assert that the DB_ROLL_PTR is not 0.

trx_undo_get_undo_rec_low(): Assert that the roll_ptr is valid before
trying to dereference it.

dict_index_t::is_primary(): Check if the index is the primary key.

PageConverter::adjust_cluster_record(): Fix
MDEV-15249 Crash in MVCC read after IMPORT TABLESPACE
by resetting the system fields to reset_trx_id instead of writing
the current transaction ID (which will be committed at the
end of the IMPORT TABLESPACE) and DB_ROLL_PTR=0.
This can partially be viewed as a follow-up fix of MDEV-12288,
because IMPORT should already then have written
DB_TRX_ID=0 and DB_ROLL_PTR=1<<55 to prevent unnecessary
DB_TRX_ID lookups in subsequent accesses to the table.
2018-02-08 12:14:34 +02:00
Sergei Golubchik
4771ae4b22 Merge branch 'github/10.1' into 10.2 2018-02-06 14:50:50 +01:00
Sergei Golubchik
d4df7bc9b1 Merge branch 'github/10.0' into 10.1 2018-02-02 10:09:44 +01:00
Marko Mäkelä
78716fffce Silence a bogus warning about uninitialized m_size
If crypt_block != NULL the entire object crypt_pfx should be
guaranteed to be initialized, including m_size, which will have been
initialized either in allocate_large(), either directly or via
allocate_trace().
2018-02-01 18:53:41 +02:00
Monty
a7e352b54d Changed database, tablename and alias to be LEX_CSTRING
This was done in, among other things:
- thd->db and thd->db_length
- TABLE_LIST tablename, db, alias and schema_name
- Audit plugin database name
- lex->db
- All db and table names in Alter_table_ctx
- st_select_lex db

Other things:
- Changed a lot of functions to take const LEX_CSTRING* as argument
  for db, table_name and alias. See init_one_table() as an example.
- Changed some function arguments from LEX_CSTRING to const LEX_CSTRING
- Changed some lists from LEX_STRING to LEX_CSTRING
- threads_mysql.result changed because process list_db wasn't always
  correctly updated
- New append_identifier() function that takes LEX_CSTRING* as arguments
- Added new element tmp_buff to Alter_table_ctx to separate temp name
  handling from temporary space
- Ensure we store the length after my_casedn_str() of table/db names
- Removed not used version of rename_table_in_stat_tables()
- Changed Natural_join_column::table_name and db_name() to never return
  NULL (used for print)
- thd->get_db() now returns db as a printable string (thd->db.str or "")
2018-01-30 21:33:55 +02:00
Marko Mäkelä
706ed8552d Revert "MDEV-6928: Add trx pointer to struct mtr_t"
This reverts commit 3486135bb5.

The commit comment ended in the words: "This is needed later."
Apparently the "later" never arrived.
2018-01-29 11:05:17 +02:00
Marko Mäkelä
041a32abcd Remove trx_mod_tables_t::vers_by_trx
Only invoke set_versioned() on trx_id versioned tables.

dict_table_t::versioned_by_id(): New accessor, to determine if
a table is system versioned by transaction ID.
2018-01-28 22:21:48 +02:00
Marko Mäkelä
9875d5c3e1 Merge bb-10.2-ext into 10.3 2018-01-24 14:00:33 +02:00
Vicențiu Ciorbaru
d833bb65d5 Merge remote-tracking branch '5.5' into 10.0 2018-01-24 12:29:31 +02:00
Sergey Vojtovich
4dc30f3c17 MDEV-15019 - InnoDB: store ReadView on trx
This will allow us to reduce critical section protected by
trx_sys.mutex:
- no need to maintain global m_free list
- eliminate if (trx->read_view == NULL) condition.

On x86_64 sizeof(Readview) is 144 mostly due to padding, sizeof(trx_t)
with ReadView is 1200.

Also don't close ReadView for read-write transactions, just mark it
closed similarly to read-only.

Clean-up: removed n_prepared_recovered_trx and n_prepared_trx, which
accidentally re-appeared after some rebase.
2018-01-22 16:23:15 +04:00
Monty
f67b8273c0 Fixed wrong arguments to printf in InnoDB 2018-01-21 20:22:00 +02:00
Eugene Kosov
acdfacee75
IB: TRT is not updated on ADD SYSTEM VERSIONING [fixes #413] 2017-12-20 22:46:28 +03:00
Aleksey Midenkov
b55a149194
Timestamp-based versioning for InnoDB [closes #209]
* Removed integer_fields check
* Reworked Vers_parse_info::check_sys_fields()
* Misc renames
* versioned as vers_sys_type_t

* Removed versioned_by_sql(), versioned_by_engine()

versioned() works as before;
versioned(VERS_TIMESTAMP) is versioned_by_sql();
versioned(VERS_TRX_ID) is versioned_by_engine().

* create_tmp_table() fix
* Foreign constraints for timestamp-based
* Range auto-specifier fix
* SQL: 1-row partition rotation fix [fixes #260]
* Fix 'drop system versioning, algorithm=inplace'
2017-12-18 19:03:51 +03:00
Aleksey Midenkov
79dd77e6ae System Versioning 1.0 pre3
Merge branch '10.3' into trunk
2017-12-11 15:43:41 +03:00
Eugene Kosov
2968543f8d IB: NULL instead of autodecrement [closes #373] 2017-12-08 19:37:24 +03:00
Sergey Vojtovich
5b624f00fc MDEV-14529 - InnoDB rw-locks: optimize memory barriers
Remove volatile modifier from waiters: it's not supposed for inter-thread
communication, use appropriate atomic operations instead.

Changed waiters to int32_t, my_atomic friendly type.
2017-12-08 17:55:41 +04:00
Marko Mäkelä
7cb3520c06 Merge bb-10.2-ext into 10.3 2017-11-30 08:16:37 +02:00
Marko Mäkelä
0b89a42ffc Remove the flag vers_update_trt
THD::vers_update_trt, trx_t::vers_update_trt, trx_savept_t::vers_update_trt:
Remove. Instead, determine from trx_t::mod_tables whether versioned
columns were affected by the transaction.

handlerton::prepare_commit_versioned: Replaces vers_get_trt_data.
Return the transaction start ID and also the commit ID, in case
the transaction modified any system-versioned columns (0 if not).

TR_table::store_data(): Remove (merge with update() below).

TR_table::update(): Add the parameters start_id, end_id.

ha_commit_trans(): Remove a condition on SQLCOM_ALTER_TABLE.
If we need something special for ALTER TABLE...ALGORITHM=INPLACE,
that can be done inside InnoDB by modifying trx_t::mod_tables.

innodb_prepare_commit_versioned(): Renamed from innodb_get_trt_data().
Check trx_t::mod_tables to see if any changes to versioned columns
are present.

trx_mod_table_time_t: A pair of logical timestamps, replacing the
undo_no_t in trx_mod_tables_t. Keep track of not only the first
modification to a persistent table in each transaction, but also
the first modification of a versioned column in a table.

dtype_t, dict_col_t: Add the accessor is_any_versioned(), to check
if the type refers to a system-versioned user or system column.

upd_t::affects_versioned(): Check if an update affects a versioned
column.

trx_undo_report_row_operation(): If a versioned column is affected
by the update, invoke trx_mod_table_time_t::set_versioned().

trx_rollback_to_savepoint_low(): If all changes to versioned columns
were rolled back, invoke trx_mod_table_time_t::rollback_versioned(),
so that trx_mod_table_time_t::is_versioned() will no longer hold.
2017-11-27 15:07:33 +03:00
Marko Mäkelä
03fbfeef66 Identify system-versioned columns in the InnoDB dictionary
Both bits DATA_VERSIONED will be set in prtype if the column
is system-versioned. The bits will be 0 for normal unversioned
columns. For the special columns identifying the logical start
and end times of versions, only one bit will be set:
DATA_VERS_START or DATA_VERS_END.

create_table_info_t::create_table_def(),
prepare_inplace_alter_table_dict(): Set prtype |= DATA_VERSIONED
for system-versioned columns.

dfield_t::is_version_historical_end(): Determine if a data tuple
field is_version_end() and contains a timestamp in the past
(not TRX_ID_MAX).

dtype_t, dict_col_t: Add the accessors
is_versioned(), is_version_start(), is_version_end().

trx_id_max_bytes[]: The bit pattern of TRX_ID_MAX, for use with
memcmp().
2017-11-27 15:07:33 +03:00
Eugene Kosov
0cdc1164dc SQL, IB: various refactoring [#337] 2017-11-27 15:07:32 +03:00
Aleksey Midenkov
cbe93291e4 SQL,IB: add auto_inc copy mode fix [#347]
Related to #212.

Extra update of sys_trx_start removed [#87].
2017-11-23 18:31:55 +03:00
Marko Mäkelä
f233c9778e Adjust the MySQL 5.7 tests for MariaDB 10.2 2017-11-20 16:05:41 +02:00
Aleksey Midenkov
33085349e9 IB, SQL: removed VTQ, added TRT on SQL layer [closes #305] 2017-11-15 00:22:10 +03:00
Eugene Kosov
fa79f6ac86 IB: style changes [closes #306] 2017-11-15 00:22:09 +03:00
Aleksey Midenkov
8972291ac1 IB: misc fixes [#305]
* System fields renamed
* Removed row0ins.ic
* Removed row_update_for_mysql() wrapper
2017-11-15 00:22:09 +03:00
Eugene Kosov
e674806282 IB: DICT_TF2_VERSIONED flag removed
Reverted DICT_TF2_BITS (midenok)
2017-11-15 00:18:41 +03:00
Aleksey Midenkov
d8d7251019 System Versioning pre0.12
Merge remote-tracking branch 'origin/archive/2017-10-17' into 10.3
2017-11-07 00:37:49 +03:00
Alexander Barkov
835cbbcc7b Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3
TODO: enable MDEV-13049 optimization for 10.3
2017-10-30 20:47:39 +04:00
Sergei Golubchik
9d2e2d7533 Merge branch '10.0' into 10.1 2017-10-22 13:03:41 +02:00
Sergei Golubchik
607d8f9e97 MDEV-14081 ALTER TABLE CHANGE COLUMN Corrupts Index Leading to Crashes in 10.2
remove remnants of 10.0 bugfix, incorrectly merged into 10.2

Using col_names[i] was obviously, wrong, must've been col_names[ifield->col_no].
incorrect column name resulted in innodb having index unique_id2(id1),
while the server thought it's unique_id2(id4).
But col_names[ifield->col_no] is wrong too, because `table` has non-renamed
columns, so the correct column name is always dict_table_get_col_name(table, ifield->col_no)
2017-10-18 22:15:27 +02:00
Jan Lindström
fc9ff69578 MDEV-13838: Wrong result after altering a partitioned table
Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly
also related bugs:

MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and adding index
MDEV-9469: 'Incorrect key file' on ALTER TABLE
MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect key file for table"
MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash
MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column fails with "Incorrect key file for table..."

Root cause for all these bugs is the fact that MariaDB .frm file
can contain virtual columns but InnoDB dictionary does not and
previous fixes were incorrect or unnecessarily forced table
rebuilt. In index creation key_part->fieldnr can be bigger than
number of columns in InnoDB data dictionary. We need to skip not
stored fields when calculating correct column number for InnoDB
data dictionary.

dict_table_get_col_name_for_mysql
        Remove

innobase_match_index_columns
        Revert incorrect change done on MDEV-7367

innobase_need_rebuild
        Remove unnecessary rebuild force when column is renamed.

innobase_create_index_field_def
        Calculate InnoDB column number correctly and remove
        unnecessary column name set.

innobase_create_index_def, innobase_create_key_defs
        Remove unneeded fields parameter. Revert unneeded memset.

prepare_inplace_alter_table_dict
        Remove unneeded col_names parameter

index_field_t
        Remove unneeded col_name member.

row_merge_create_index
        Remove unneeded col_names parameter and resolution.

Effected tests:
         innodb-alter-table : Add test case for MDEV-13668
         innodb-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
                        and restore original tests
         innodb-wl5980-alter : Remove MDEV-13668,  MDEV-9469 FIXMEs
                        and restore original tests
2017-10-10 17:03:40 +03:00
Marko Mäkelä
a4948dafcd MDEV-11369 Instant ADD COLUMN for InnoDB
For InnoDB tables, adding, dropping and reordering columns has
required a rebuild of the table and all its indexes. Since MySQL 5.6
(and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing
concurrent modification of the tables.

This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT
and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously,
with only minor changes performed to the table structure. The counter
innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS
is incremented whenever a table rebuild operation is converted into
an instant ADD COLUMN operation.

ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN.

Some usability limitations will be addressed in subsequent work:

MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY
and ALGORITHM=INSTANT
MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE

The format of the clustered index (PRIMARY KEY) is changed as follows:

(1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT,
and a new field PAGE_INSTANT will contain the original number of fields
in the clustered index ('core' fields).
If instant ADD COLUMN has not been used or the table becomes empty,
or the very first instant ADD COLUMN operation is rolled back,
the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset
to 0 and FIL_PAGE_INDEX.

(2) A special 'default row' record is inserted into the leftmost leaf,
between the page infimum and the first user record. This record is
distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the
same format as records that contain values for the instantly added
columns. This 'default row' always has the same number of fields as
the clustered index according to the table definition. The values of
'core' fields are to be ignored. For other fields, the 'default row'
will contain the default values as they were during the ALTER TABLE
statement. (If the column default values are changed later, those
values will only be stored in the .frm file. The 'default row' will
contain the original evaluated values, which must be the same for
every row.) The 'default row' must be completely hidden from
higher-level access routines. Assertions have been added to ensure
that no 'default row' is ever present in the adaptive hash index
or in locked records. The 'default row' is never delete-marked.

(3) In clustered index leaf page records, the number of fields must
reside between the number of 'core' fields (dict_index_t::n_core_fields
introduced in this work) and dict_index_t::n_fields. If the number
of fields is less than dict_index_t::n_fields, the missing fields
are replaced with the column value of the 'default row'.
Note: The number of fields in the record may shrink if some of the
last instantly added columns are updated to the value that is
in the 'default row'. The function btr_cur_trim() implements this
'compression' on update and rollback; dtuple::trim() implements it
on insert.

(4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new
status value REC_STATUS_COLUMNS_ADDED will indicate the presence of
a new record header that will encode n_fields-n_core_fields-1 in
1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header
always explicitly encodes the number of fields.)

We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for
covering the insert of the 'default row' record when instant ADD COLUMN
is used for the first time. Subsequent instant ADD COLUMN can use
TRX_UNDO_UPD_EXIST_REC.

This is joint work with Vin Chen (陈福荣) from Tencent. The design
that was discussed in April 2017 would not have allowed import or
export of data files, because instead of the 'default row' it would
have introduced a data dictionary table. The test
rpl.rpl_alter_instant is exactly as contributed in pull request #408.
The test innodb.instant_alter is based on a contributed test.

The redo log record format changes for ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPACT are as contributed. (With this change present,
crash recovery from MariaDB 10.3.1 will fail in spectacular ways!)
Also the semantics of higher-level redo log records that modify the
PAGE_INSTANT field is changed. The redo log format version identifier
was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1.

Everything else has been rewritten by me. Thanks to Elena Stepanova,
the code has been tested extensively.

When rolling back an instant ADD COLUMN operation, we must empty the
PAGE_FREE list after deleting or shortening the 'default row' record,
by calling either btr_page_empty() or btr_page_reorganize(). We must
know the size of each entry in the PAGE_FREE list. If rollback left a
freed copy of the 'default row' in the PAGE_FREE list, we would be
unable to determine its size (if it is in ROW_FORMAT=COMPACT or
ROW_FORMAT=DYNAMIC) because it would contain more fields than the
rolled-back definition of the clustered index.

UNIV_SQL_DEFAULT: A new special constant that designates an instantly
added column that is not present in the clustered index record.

len_is_stored(): Check if a length is an actual length. There are
two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL.

dict_col_t::def_val: The 'default row' value of the column.  If the
column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT.

dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(),
instant_value().

dict_col_t::remove_instant(): Remove the 'instant ADD' status of
a column.

dict_col_t::name(const dict_table_t& table): Replaces
dict_table_get_col_name().

dict_index_t::n_core_fields: The original number of fields.
For secondary indexes and if instant ADD COLUMN has not been used,
this will be equal to dict_index_t::n_fields.

dict_index_t::n_core_null_bytes: Number of bytes needed to
represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable).

dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that
n_core_null_bytes was not initialized yet from the clustered index
root page.

dict_index_t: Add the accessors is_instant(), is_clust(),
get_n_nullable(), instant_field_value().

dict_index_t::instant_add_field(): Adjust clustered index metadata
for instant ADD COLUMN.

dict_index_t::remove_instant(): Remove the 'instant ADD' status
of a clustered index when the table becomes empty, or the very first
instant ADD COLUMN operation is rolled back.

dict_table_t: Add the accessors is_instant(), is_temporary(),
supports_instant().

dict_table_t::instant_add_column(): Adjust metadata for
instant ADD COLUMN.

dict_table_t::rollback_instant(): Adjust metadata on the rollback
of instant ADD COLUMN.

prepare_inplace_alter_table_dict(): First create the ctx->new_table,
and only then decide if the table really needs to be rebuilt.
We must split the creation of table or index metadata from the
creation of the dictionary table records and the creation of
the data. In this way, we can transform a table-rebuilding operation
into an instant ADD COLUMN operation. Dictionary objects will only
be added to cache when table rebuilding or index creation is needed.
The ctx->instant_table will never be added to cache.

dict_table_t::add_to_cache(): Modified and renamed from
dict_table_add_to_cache(). Do not modify the table metadata.
Let the callers invoke dict_table_add_system_columns() and if needed,
set can_be_evicted.

dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the
system columns (which will now exist in the dict_table_t object
already at this point).

dict_create_table_step(): Expect the callers to invoke
dict_table_add_system_columns().

pars_create_table(): Before creating the table creation execution
graph, invoke dict_table_add_system_columns().

row_create_table_for_mysql(): Expect all callers to invoke
dict_table_add_system_columns().

create_index_dict(): Replaces row_merge_create_index_graph().

innodb_update_n_cols(): Renamed from innobase_update_n_virtual().
Call my_error() if an error occurs.

btr_cur_instant_init(), btr_cur_instant_init_low(),
btr_cur_instant_root_init():
Load additional metadata from the clustered index and set
dict_index_t::n_core_null_bytes. This is invoked
when table metadata is first loaded into the data dictionary.

dict_boot(): Initialize n_core_null_bytes for the four hard-coded
dictionary tables.

dict_create_index_step(): Initialize n_core_null_bytes. This is
executed as part of CREATE TABLE.

dict_index_build_internal_clust(): Initialize n_core_null_bytes to
NO_CORE_NULL_BYTES if table->supports_instant().

row_create_index_for_mysql(): Initialize n_core_null_bytes for
CREATE TEMPORARY TABLE.

commit_cache_norebuild(): Call the code to rename or enlarge columns
in the cache only if instant ADD COLUMN is not being used.
(Instant ADD COLUMN would copy all column metadata from
instant_table to old_table, including the names and lengths.)

PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields.
This is repurposing the 16-bit field PAGE_DIRECTION, of which only the
least significant 3 bits were used. The original byte containing
PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B.

page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT.

page_ptr_get_direction(), page_get_direction(),
page_ptr_set_direction(): Accessors for PAGE_DIRECTION.

page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION.

page_direction_increment(): Increment PAGE_N_DIRECTION
and set PAGE_DIRECTION.

rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes,
and assume that heap_no is always set.
Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records,
even if the record contains fewer fields.

rec_offs_make_valid(): Add the parameter 'leaf'.

rec_copy_prefix_to_dtuple(): Assert that the tuple is only built
on the core fields. Instant ADD COLUMN only applies to the
clustered index, and we should never build a search key that has
more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR.
All these columns are always present.

dict_index_build_data_tuple(): Remove assertions that would be
duplicated in rec_copy_prefix_to_dtuple().

rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose
number of fields is between n_core_fields and n_fields.

cmp_rec_rec_with_match(): Implement the comparison between two
MIN_REC_FLAG records.

trx_t::in_rollback: Make the field available in non-debug builds.

trx_start_for_ddl_low(): Remove dangerous error-tolerance.
A dictionary transaction must be flagged as such before it has generated
any undo log records. This is because trx_undo_assign_undo() will mark
the transaction as a dictionary transaction in the undo log header
right before the very first undo log record is being written.

btr_index_rec_validate(): Account for instant ADD COLUMN

row_undo_ins_remove_clust_rec(): On the rollback of an insert into
SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the
last column from the table and the clustered index.

row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(),
trx_undo_update_rec_get_update(): Handle the 'default row'
as a special case.

dtuple_t::trim(index): Omit a redundant suffix of an index tuple right
before insert or update. After instant ADD COLUMN, if the last fields
of a clustered index tuple match the 'default row', there is no
need to store them. While trimming the entry, we must hold a page latch,
so that the table cannot be emptied and the 'default row' be deleted.

btr_cur_optimistic_update(), btr_cur_pessimistic_update(),
row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low():
Invoke dtuple_t::trim() if needed.

row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling
row_ins_clust_index_entry_low().

rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number
of fields to be between n_core_fields and n_fields. Do not support
infimum,supremum. They are never supposed to be stored in dtuple_t,
because page creation nowadays uses a lower-level method for initializing
them.

rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the
number of fields.

btr_cur_trim(): In an update, trim the index entry as needed. For the
'default row', handle rollback specially. For user records, omit
fields that match the 'default row'.

btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
Skip locking and adaptive hash index for the 'default row'.

row_log_table_apply_convert_mrec(): Replace 'default row' values if needed.
In the temporary file that is applied by row_log_table_apply(),
we must identify whether the records contain the extra header for
instantly added columns. For now, we will allocate an additional byte
for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table
has been subject to instant ADD COLUMN. The ROW_T_DELETE records are
fine, as they will be converted and will only contain 'core' columns
(PRIMARY KEY and some system columns) that are converted from dtuple_t.

rec_get_converted_size_temp(), rec_init_offsets_temp(),
rec_convert_dtuple_to_temp(): Add the parameter 'status'.

REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG | REC_STATUS_COLUMNS_ADDED:
An info_bits constant for distinguishing the 'default row' record.

rec_comp_status_t: An enum of the status bit values.

rec_leaf_format: An enum that replaces the bool parameter of
rec_init_offsets_comp_ordinary().
2017-10-06 09:50:10 +03:00
Marko Mäkelä
2c1067166d Merge bb-10.2-ext into 10.3 2017-10-04 08:24:06 +03:00
Marko Mäkelä
3c4cff3357 Merge 10.1 into 10.2 2017-10-02 11:16:53 +03:00
Vladislav Vaintroub
7354dc6773 MDEV-13384 - misc Windows warnings fixed 2017-09-28 17:20:46 +00:00
Jan Lindström
4d01dd79a1 MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to temporary file
After review cleanup.
2017-09-28 12:38:51 +03:00
Marko Mäkelä
d4b2dfa967 Replace dict_table_get_n_sys_cols(table) with DATA_N_SYS_COLS
This could have been done as part of MDEV-11487, which removed the
support for InnoDB internal temporary tables.
2017-09-21 15:07:31 +03:00
Marko Mäkelä
e3d44f5d62 Merge bb-10.2-ext into 10.3 2017-09-21 08:12:19 +03:00
Marko Mäkelä
48192f963a Add the parameter bool leaf to rec_get_offsets()
This should affect debug builds only. Debug builds will check that
the status bits of ROW_FORMAT!=REDUNDANT records match the is_leaf
parameter.

The only observable change to non-debug should be the addition of
the is_leaf parameter to the function rec_copy_prefix_to_dtuple(),
and the removal of some calls to update the adaptive hash index
(it is only built for the leaf pages).

This change should have been made in MySQL 5.0.3, instead of
introducing the status flags in the ROW_FORMAT=COMPACT record header.
2017-09-20 16:53:34 +03:00
Marko Mäkelä
e17a282da9 Merge bb-10.2-ext into 10.3 2017-09-18 11:38:07 +03:00
Marko Mäkelä
d9277732d7 Merge 10.1 into 10.2
This should also fix the MariaDB 10.2.2 bug
MDEV-13826 CREATE FULLTEXT INDEX on encrypted table fails.

MDEV-12634 FIXME: Modify innodb-index-online, innodb-table-online
so that they will write and read merge sort files. InnoDB 5.7
introduced some optimizations to avoid using the files for small tables.

Many collation test results have been adjusted for MDEV-10191.
2017-09-17 11:05:33 +03:00
Marko Mäkelä
836d4e74d9 Write proper tests for MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes
Introduce innodb_encrypt_log.combinations and prove that
the encryption and decryption take place during both
online ADD INDEX (WL#5266) and online table-rebuilding ALTER (WL#6625).
2017-09-16 21:15:38 +03:00
Jan Lindström
fa2701c6f7 MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to tem…
…porary file

Fixed by removing writing key version to start of every block that
was encrypted. Instead we will use single key version from log_sys
crypt info.

After this MDEV also blocks writen to row log are encrypted and blocks
read from row log aren decrypted if encryption is configured for the
table.

innodb_status_variables[], struct srv_stats_t
	Added status variables for merge block and row log block
	encryption and decryption amounts.

Removed ROW_MERGE_RESERVE_SIZE define.

row_merge_fts_doc_tokenize
	Remove ROW_MERGE_RESERVE_SIZE

row_log_t
	Add index, crypt_tail, crypt_head to be used in case of
	encryption.

row_log_online_op, row_log_table_close_func
	Before writing a block encrypt it if encryption is enabled

row_log_table_apply_ops, row_log_apply_ops
	After reading a block decrypt it if encryption is enabled

row_log_allocate
	Allocate temporary buffers crypt_head and crypt_tail
	if needed.

row_log_free
	Free temporary buffers crypt_head and crypt_tail if they
	exist.

row_merge_encrypt_buf, row_merge_decrypt_buf
	Removed.

row_merge_buf_create, row_merge_buf_write
	Remove ROW_MERGE_RESERVE_SIZE

row_merge_build_indexes
	Allocate temporary buffer used in decryption and encryption
	if needed.

log_tmp_blocks_crypt, log_tmp_block_encrypt, log_temp_block_decrypt
	New functions used in block encryption and decryption

log_tmp_is_encrypted
	New function to check is encryption enabled.

Added test case innodb-rowlog to force creating a row log and
verify that operations are done using introduced status
variables.
2017-09-14 09:23:20 +03:00
Marko Mäkelä
348eaf4252 Merge bb-10.2-ext into 10.3 2017-09-14 09:12:47 +03:00