Commit graph

16854 commits

Author SHA1 Message Date
Sergei Golubchik
432b78c903 just like tempfiles: use key id 2 for temp Aria tables
introduce ENCRYPTION_KEY_SYSTEM_DATA and
ENCRYPTION_KEY_TEMPORARY_DATA constants; use them everywhere.
2015-06-02 18:53:37 +02:00
Sergei Golubchik
d9340d6c8e MDEV-8126 encryption for temp files
IO_CACHE tempfiles encryption
2015-06-02 18:53:37 +02:00
Sergei Golubchik
318c826e93 always use my_b_pread() instead of mysql_file_pread()
when working with IO_CACHE's, don't access IO_CACHE::file directly
2015-06-02 18:53:37 +02:00
Sergei Golubchik
196e852983 misc IO_CACHE cleanups
* remove unused (and not implemented) WRITE_NET type
* remove cast in my_b_write() macro. my_b_* macros are
  function-like, casts are responsibility of the caller
* replace hackish _my_b_write(info,0,0) with the explicit
  my_b_flush_io_cache() in my_b_write_byte()
* remove unused my_b_fill_cache()
* replace pbool -> my_bool
* make internal IO_CACHE functions static
* reformat comments, correct typos, remove obsolete comments (ISAM)
* assert valid cache type in init_functions()
* use IO_ROUND_DN() macro where appropriate
* remove unused DBUG_EXECUTE_IF in _my_b_cache_write()
* remove unnecessary __attribute__((unused))
* fix goto error in parse_file.cc
* remove redundant reinit_io_cache() in uniques.cc
* don't do reinit_io_cache() if the cache was not initialized
  in ma_check.c
* extract duplicate functionality from various _my_b_*_read
  functions into a common wrapper. Same for _my_b_*_write
* create _my_b_cache_write_r instead of having if's in
  _my_b_cache_write (similar to existing _my_b_cache_read and
  _my_b_cache_read_r)
* don't call mysql_file_write() from my_b_flush_io_cache(),
  call info->write_function() instead
2015-06-02 18:53:37 +02:00
Sergei Golubchik
1841557e40 myisam/aria: don't mess with IO_CACHE::file
that is, don't create a "temporary" IO_CACHE with file==-1
and manually swap it later to the valid file descriptor.
2015-06-02 18:53:36 +02:00
Sergei Golubchik
91dab5ddb6 fix cmake dependencies
perfschema after 5.6 merge depends on sql_yacc.h
2015-06-02 18:09:53 +02:00
Sergei Golubchik
72d01f2426 remove few #ifdefs in innodb/xtradb 2015-06-02 17:46:05 +02:00
Olivier Bertrand
af26c366e0 Fix handling of NULL values when reading from tables.
modified:   storage/connect/tabodbc.cpp
  modified:   storage/connect/value.h
2015-06-02 12:28:42 +02:00
Sergei Golubchik
0d54cb12eb compilation failure on windows 2015-06-01 21:35:02 +02:00
Sergei Golubchik
d602574542 Merge remote-tracking branch 'github/10.1' into 10.1 2015-06-01 16:01:42 +02:00
Sergei Golubchik
5091a4ba75 Merge tag 'mariadb-10.0.19' into 10.1 2015-06-01 15:51:25 +02:00
Jan Lindström
13235a549d Fix compiler warning. 2015-05-31 07:53:20 +03:00
Jan Lindström
59815a268b MDEV-7484: Log Time not consistent with InnoDB errors nor with MySQL error log time format 2015-05-30 20:45:17 +03:00
Olivier Bertrand
514a7d8462 Add unicode ODBC types to the types recognized by CONNECT.
Was added in function TranslateSQLType.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/odbconn.cpp
  modified:   storage/connect/value.h

Add some trace in particular in indexing routines.
  modified:   storage/connect/block.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/plugutil.c
  modified:   storage/connect/xindex.cpp
  modified:   storage/connect/xindex.h
2015-05-30 10:59:34 +02:00
Oleksandr Byelkin
0f01bf2676 MDEV-8241: Debug build on Windows is broken: error LNK2019: unresolved external symbol pthread_detach referenced in function ma_checkpoint_init
pthread_detach() replaced with pthread_detach_this_thread()

pthread_detach_this_thread() definition fixed
2015-05-29 23:07:49 +02:00
Sergei Golubchik
34e01f80e4 restore innodb_encrypt_tables validation function
that was apparently lost in 20c23048:

  commit 20c23048c1
  Author: Jan Lindström <jan.lindstrom@mariadb.com>
  Date:   Sun May 17 14:14:16 2015 +0300

      MDEV-8164: Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing

This also reverts 8635c4b4:

  commit 8635c4b4e6
  Author: Jan Lindström <jan.lindstrom@mariadb.com>
  Date:   Thu May 21 11:02:03 2015 +0300

      Fix test failure.
2015-05-28 14:41:48 +02:00
Olivier Bertrand
b6a56370d6 Change all preprocessor compiler directives to use __WIN__ as the mean of
specifying Windows or not Windows compile. This is what MariaDB does.
  modified:   storage/connect/array.cpp
  modified:   storage/connect/blkfil.cpp
  modified:   storage/connect/block.h
  modified:   storage/connect/colblk.cpp
  modified:   storage/connect/domdoc.cpp
  modified:   storage/connect/filamap.cpp
  modified:   storage/connect/filamdbf.cpp
  modified:   storage/connect/filamfix.cpp
  modified:   storage/connect/filamtxt.cpp
  modified:   storage/connect/filamvct.cpp
  modified:   storage/connect/filamzip.cpp
  modified:   storage/connect/filter.cpp
  modified:   storage/connect/filter.h
  modified:   storage/connect/fmdlex.c
  modified:   storage/connect/global.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/json.cpp
  modified:   storage/connect/macutil.cpp
  modified:   storage/connect/macutil.h
  modified:   storage/connect/maputil.cpp
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/myconn.cpp
  modified:   storage/connect/myconn.h
  modified:   storage/connect/myutil.cpp
  modified:   storage/connect/odbconn.cpp
  modified:   storage/connect/odbconn.h
  modified:   storage/connect/os.h
  modified:   storage/connect/osutil.c
  modified:   storage/connect/plgdbsem.h
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/plugutil.c
  modified:   storage/connect/rcmsg.c
  modified:   storage/connect/reldef.cpp
  modified:   storage/connect/reldef.h
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfmt.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabmac.cpp
  modified:   storage/connect/tabmac.h
  modified:   storage/connect/tabmul.cpp
  modified:   storage/connect/tabmul.h
  modified:   storage/connect/tabmysql.cpp
  modified:   storage/connect/taboccur.cpp
  modified:   storage/connect/tabodbc.cpp
  modified:   storage/connect/tabpivot.cpp
  modified:   storage/connect/tabsys.cpp
  modified:   storage/connect/tabtbl.cpp
  modified:   storage/connect/tabutil.cpp
  modified:   storage/connect/tabvct.cpp
  modified:   storage/connect/tabwmi.cpp
  modified:   storage/connect/tabxcl.cpp
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/valblk.cpp
  modified:   storage/connect/value.cpp
  modified:   storage/connect/xindex.cpp
  modified:   storage/connect/xindex.h
2015-05-27 16:23:38 +02:00
Jan Lindström
a25ccd4f83 MDEV-8238: Tables with encryption=yes using file_key_management plugin are not encrypted
Analysis: Problem was that encryption was skipped.

Fixed by making sure that tables with ENCRYPTED=YES are encrypted.
2015-05-27 16:52:36 +03:00
Jan Lindström
2bea4bd9ed MDEV-8233: InnoDB: Assertion failure in fil_page_decompress with encrypted tables
Analysis: Problem was that used compression method needs to be stored
to the page.

Fixed by storing compression method after key_version to the page.
2015-05-27 15:34:10 +03:00
Sergey
6bd76f8b7e Merge pull request #73 from akopytov/MDEV-7658-5.5
Fixes MDEV-7658: MDEV-7026 fix reintroduces MDEV-6615 on AArch64
2015-05-27 10:27:18 +04:00
Jan Lindström
a99efc00a6 Merge pull request #74 from akopytov/MDEV-7658-10.0
Mdev 7658 10.0
2015-05-27 09:16:24 +03:00
Alexey Kopytov
7f7cee8700 Merge branch 'MDEV-7658-5.5' into MDEV-7658-10.0 2015-05-26 23:58:51 +03:00
Alexey Kopytov
70bc0a3ef4 Fixes MDEV-7658: MDEV-7026 fix reintroduces MDEV-6615 on AArch64
This is an addendum to the fix for MDEV-7026. The ARM memory model is
similar to that of PowerPC and thus needs the same semantics with
respect to memory barriers. That is, os_atomic_test_and_set_*_release()
must be a store with a release barrier followed by a full
barrier. Unlike x86 using __sync_lock_test_and_set() which is
implemented as “exclusive load with acquire barriers + exclusive store”
is insufficient in contexts where os_atomic_test_and_set_*_release()
macros are used.
2015-05-26 23:56:00 +03:00
Jan Lindström
8c0ea281bf MDEV-8213: encryption.encryption_force, encryption.encrypt_and_grep fail with valgrind warnings (Invalid read)
Analysis: Problem was that code used old pointer.

Fixed by using correct pointer.
2015-05-26 16:09:36 +03:00
Olivier Bertrand
fb98632930 JSONColumns and XMLColumns revisited. They can retrieve their parameters directly
from the PTOS argument. For this to work, finding the table options is now split
in HA_CONNECT functions and exported functions available from out of ha_connect.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/libdoc.cpp
  modified:   storage/connect/mycat.h
  modified:   storage/connect/plgdbsem.h
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/tabxml.h

The BIN table formats have been changed to handle the case of floating point values
when used with Big Endian or Little Endian machines.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/mysql-test/connect/r/bin.result
  modified:   storage/connect/mysql-test/connect/t/bin.test
  modified:   storage/connect/reldef.cpp
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfix.
h
2015-05-26 01:02:33 +02:00
Jan Lindström
536112dd30 MDEV-8195: InnoDB: Error: trying to access tablespace 11262 page no. 7, InnoDB: but the tablespace does not exist or is just being dropped.
Analysis: Problem was that we did try to read from tablespace
that was being dropped.

Fixed by introducing a new function to find a tablespace only
if it is not being dropped currently and adding this check
before trying to read pages from tablespace.
2015-05-21 15:32:49 +03:00
Jan Lindström
137ba7d6f9 Fix compiler error. 2015-05-21 08:14:35 +03:00
Jan Lindström
925b64124a MDEV-8182: Failing assertion: 1 == UT_LIST_GET_LEN(space->chain)
Analysis: At fil_spage_get_space there is small change that space
is found from tablespace list but we have not yet created node
for it (and added it to space->chain) and as we hold fil_system
mutex here fil_node_create can't continue.

Fixed by allowing UT_LIST_GET_LEN(space->chain) == 0|| 1 and
introducint two new functions that access filespace list
and before returning space check that node is also created.
2015-05-20 20:32:10 +03:00
Jan Lindström
3e55ef26d4 MDEV-8173: InnoDB; Failing assertion: crypt_data->type == 1
Make sure that when we publish the crypt_data we access the
memory cache of the tablespace crypt_data. Make sure that
crypt_data is stored whenever it is really needed.

All this is not yet enough in my opinion because:

sql/encryption.cc has DBUG_ASSERT(scheme->type == 1) i.e.
crypt_data->type == CRYPT_SCHEME_1

However, for InnoDB point of view we have global crypt_data
for every tablespace. When we change variables on crypt_data
we take mutex. However, when we use crypt_data for
encryption/decryption we use pointer to this global
structure and no mutex to protect against changes on
crypt_data.

Tablespace encryption starts in fil_crypt_start_encrypting_space
from crypt_data that has crypt_data->type = CRYPT_SCHEME_UNENCRYPTED
and later we write page 0 CRYPT_SCHEME_1 and finally whe publish
that to memory cache.
2015-05-20 14:10:07 +03:00
Oleksandr Byelkin
44cd6f22d4 MDEV-7921: main.sp_sync fails in buildbot with valgrind
Part 2: detauch service thread.
2015-05-20 11:43:44 +02:00
Olivier Bertrand
37840d5313 Security: EOM modules must now be loaded from the plugin directory.
modified:   storage/connect/mycat.cc
  modified:   storage/connect/reldef.cpp

Json array index (position) always defaults to 0
  modified:   storage/connect/tabjson.cpp
2015-05-20 11:19:44 +02:00
Jan Lindström
20c23048c1 MDEV-8164: Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing or alike
Analysis: Problem was that tablespaces not encrypted might not have
crypt_data stored on disk.

Fixed by always creating crypt_data to memory cache of the tablespace.

MDEV-8138: strange results from encrypt-and-grep test

Analysis: crypt_data->type is not updated correctly on memory
cache. This caused problem with state tranfer on
encrypted => unencrypted => encrypted.

Fixed by updating memory cache of crypt_data->type correctly based on
current srv_encrypt_tables value to either CRYPT_SCHEME_1 or
CRYPT_SCHEME_UNENCRYPTED.
2015-05-18 13:28:13 +03:00
Olivier Bertrand
db33294fdc Json array index (position) was badly set for default array setting
modified:   storage/connect/tabjson.cpp
2015-05-17 19:55:48 +02:00
Olivier Bertrand
a82171c9aa In BIN table date_format now imply by default field_format='C'.
modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/reldef.cpp

Json array index (position) is now 0 based by default. This corresponds
to what all json applications and functions do. Also fix ROWNUM calculation.
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/mysql-test/connect/r/json.result
  modified:   storage/connect/mysql-test/connect/t/json.test
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h
2015-05-17 15:22:42 +02:00
Sergei Golubchik
476dfb1603 update big tokudb test results after dd8f93195
that is, after

  commit dd8f931957
  Author: Sergei Golubchik <serg@mariadb.org>
  Date:   Fri Apr 10 02:36:54 2015 +0200

    be less annoying about sysvar-based table attributes
    do not *always* add them to the create table definition,
    but only when a sysvar value is different from a default.
    also, when adding them - don't quote numbers
2015-05-16 16:31:46 +02:00
Sergei Golubchik
54672a4f1e MDEV-8043 innodb tablespace encryption
"use after free" bug, when a thread replaces space->crypt_data
and frees the old crypt_data object while it's being used
by another thread.
2015-05-15 18:12:03 +02:00
Sergei Golubchik
e5989d52a7 InnoDB: simplify innobase_compression_algorithm_validate() 2015-05-15 18:12:03 +02:00
Sergei Golubchik
8815fb3d45 MDEV-8158 InnoDB: Failing assertion: new_state->key_version != ENCRYPTION_KEY_VERSION_INVALID on dynamic change of encryption variables
don't allow to enable srv_encrypt_tables if no
encryption plugin is loaded
2015-05-15 18:12:02 +02:00
Sergei Golubchik
8258a34800 InnoDB: check if scrubbing is enabled before scanning the tablespace 2015-05-15 18:12:02 +02:00
Sergei Golubchik
a94cabd503 MDEV-8159 InnoDB: Failing assertion: key_state->key_id
to simplify various checks, set crypt_data->key_id even
for not encrypted tablespaces
2015-05-15 18:12:02 +02:00
Sergei Golubchik
2300fe2e0e Identical key derivation code in XtraDB/InnoDB/Aria
* Extract it into the "encryption_scheme" service.
* Make these engines to use the service, remove duplicate code.
* Change MY_AES_xxx error codes, to return them safely
  from encryption_scheme_encrypt/decrypt without conflicting
  with ENCRYPTION_SCHEME_KEY_INVALID error
2015-05-15 18:12:01 +02:00
Olivier Bertrand
b9c910909c Fix a bug in BIN buffer initialisation (in FIXFAM::AllocateBuffer)
modified:   storage/connect/filamfix.cpp

Second version of BIN table new field format (1st one was buggy)
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfix.h

Make bin.test not to fail in big-endian machines
  modified:   storage/connect/mysql-test/connect/r/bin.result
  modified:   storage/connect/mysql-test/connect/t/bin.test

Fix a bug causing wrong default offset to be generated when virtual
or special columns were placed beetween standard columns. Also
calculate the good offset for BIN columns with new field format.
  modified:   storage/connect/reldef.cpp
2015-05-15 11:56:29 +02:00
Jan Lindström
f8cacd03a7 MDEV-8143: InnoDB: Database page corruption on disk or a failed file read
Analysis: Problem was that we did create crypt data for encrypted table but
this new crypt data was not written to page 0. Instead a default crypt data
was written to page 0 at table creation.

Fixed by explicitly writing new crypt data to page 0 after successfull
table creation.
2015-05-14 11:32:24 +03:00
Olivier Bertrand
e6b60ee5af Make BIN table files more flexible with new column format.
In particular enable to set length and endian setting.
This should solve all problems on IBM390s machines.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfix.h

Make DBF tables to be usable in big-endian machines (test version)
  modified:   storage/connect/filamdbf.cpp

Ignore git files on storage/connect
  modified:   .gitignore
2015-05-13 19:58:21 +02:00
Sergei Golubchik
16b6ec2e37 MDEV-8130 Wrong error code/message while encrypting a partitioned InnoDB table
when checking for a flag, use & not ==
2015-05-13 14:27:18 +02:00
Sergei Golubchik
def48e623a MDEV-8141 InnoDB: background encryption thread uses FIL_DEFAULT_ENCRYPTION_KEY
* check key version per key id (that is, per tablespace).
* wake encryption thread when a tablespace needs re-encryption
2015-05-13 14:27:17 +02:00
Sergei Golubchik
a583976e78 MDEV-8015 InnoDB: Failing assertion: new_state->key_version != ENCRYPTION_KEY_VERSION_INVALID
InnoDB: refuse to start if encryption is requested but no
encryption plugin is available
2015-05-13 14:27:17 +02:00
Sergei Golubchik
b4777bfc11 cleanup, use encryption_key_id_exists() where appropriate 2015-05-13 14:27:17 +02:00
Sergei Golubchik
a35b538bde MDEV-8040 make aria encryption use real keys
applied upstream patch
2015-05-13 14:27:16 +02:00
Jan Lindström
58e8db2eb3 MDEV-7942: InnoDB: abuse of UNIV_LIKELY()/UNIV_UNLIKELY()
UNIV_LIKELY()/UNIV_UNLIKELY() hints are supposed to improve branch prediction.
Currently, they're expected to work only if cond evaluates to TRUE or FALSE.

However there're a few conditions that may evaluate to different values, e.g.:

page/page0zip.cc:		if (UNIV_LIKELY(c_stream->avail_in)) {
page/page0zip.cc:			if (UNIV_LIKELY(c_stream->avail_in)) {
dict/dict0mem.cc:		if (UNIV_LIKELY(i) && UNIV_UNLIKELY(!table->col_names)) {

Fixed these conditions so that they evaluate TRUE/FALSE.
2015-05-11 14:31:42 +03:00