Commit graph

19395 commits

Author SHA1 Message Date
Vladislav Vaintroub
f48d56c459 MDEV-14192 Apply marko's patch 2017-11-13 14:58:18 +00:00
Sergei Golubchik
8b18a44fa7 mroonga after-merge CMakeLists.txt fixes
1. remove erroneously committed *.orig
2. fix LZ4 detection on Mac OS X and FreeBSD. Cannot do

   pkg_check_modules(LIBLZ4 liblz4)
   find_library(LIBLZ4_LIBS ... )

because find_library(X) does not do anything if X is defined (documented),
and pkg_check_modules(Y) sets Y_LIBS to "" (undocumented!)
2017-11-13 13:11:53 +08:00
Marko Mäkelä
b2dd5232d4 InnoDB: Remove ut_vsnprintf() and the use of my_vsnprintf(); use vsnprintf() 2017-11-13 04:32:56 +02:00
Marko Mäkelä
c19ef508b8 InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf() 2017-11-13 02:11:48 +02:00
Marko Mäkelä
17bd6ed29a Remove STATUS_VERBOSE (there is no visible output) 2017-11-12 17:27:36 +02:00
Marko Mäkelä
386e5d476e Merge 10.2 into bb-10.2-ext 2017-11-10 16:07:01 +02:00
Marko Mäkelä
fa00fedaac MDEV-14100 Assertion `!is_user_rec || leaf || ...
rec_get_offsets_func(): Relax a bogus debug assertion.
It would fail when we are operating on a copied prefix
of a node pointer record.
2017-11-10 15:58:52 +02:00
Marko Mäkelä
9618c04e3f Follow-up fix of MDEV-13795/MDEV-14332
row_log_table_apply_op(): Remove references to dict_table_t::n_vcols.
Virtual column information is no longer being written to the log.

row_log_t: Remove the unused fields n_old_col, n_old_vcol.
2017-11-10 15:58:52 +02:00
Marko Mäkelä
1ca72a0c0d Merge 10.2 into bb-10.2-ext 2017-11-10 08:27:09 +02:00
Sergei Golubchik
2a4e4335c4 Merge branch 'github/10.0-galera' into 10.1 2017-11-10 01:38:03 +01:00
Sergei Golubchik
d79fd5ba7f Merge branch 'ks-10.1-mroonga' into 10.1 2017-11-10 01:14:55 +01:00
Marko Mäkelä
5d142f9958 MDEV-13795/MDEV-14332 Corruption during online table-rebuilding ALTER when VIRTUAL columns exist
When MySQL 5.7 introduced indexed virtual columns, it introduced
several bugs into the online table-rebuilding ALTER, that is,
the row_log_table_apply() family of functions.

The online_log format that was introduced for online table-rebuilding
ALTER in MySQL 5.6 should be sufficient. Ideally, any indexed virtual
column values would be evaluated based on the log records in the temporary
file. There is no need to log virtual column values.

(For ADD INDEX, that is row_log_apply(), we always must log the values of
the keys, no matter if the columns are virtual.)

Because omitting the virtual column values removes any chance of
row_log_table_apply() working with indexed virtual columns, we
will for now refuse LOCK=NONE in table-rebuilding ALTER operations
when indexes on virtual columns exist. This restriction would be
lifted in MDEV-14341.

innobase_indexed_virtual_exist(): New predicate, to determine if
indexed virtual columns exist in a table definition.

ha_innobase::check_if_supported_inplace_alter(): Refuse online rebuild
if indexed virtual columns exist.

rec_get_converted_size_temp_v(), rec_convert_dtuple_to_temp_v(): Remove.

row_log_table_delete(), row_log_table_update(, row_log_table_insert():
Remove parameters for virtual columns.

trx_undo_read_v_rows(): Remove the col_map parameter.

row_log_table_apply(): Do not deal with virtual columns.
2017-11-09 23:39:12 +02:00
Monty
0bb0d52221 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext
Conflicts:
	mysql-test/r/cte_recursive.result
	mysql-test/r/derived_cond_pushdown.result
	mysql-test/t/cte_recursive.test
	mysql-test/t/derived_cond_pushdown.test
	sql/datadict.cc
	sql/handler.cc
2017-11-09 23:21:41 +02:00
Sergei Petrunia
e2376e8137 MDEV-14334: Update test results for rocksdb.bulk_load_rev_data 2017-11-09 19:52:55 +03:00
Monty
d40c23570f Cleanup up after failed alter table in add_index_inplace_crash 2017-11-09 14:58:26 +02:00
Marko Mäkelä
761cf49265 Merge 10.1 into 10.2 2017-11-09 14:45:39 +02:00
Marko Mäkelä
d2ffafe00f MDEV-14333 Mariabackup --apply-log-only crashes if incomplete transactions with update_undo logs are present
trx_undo_free_prepared(): Relax the assertion for
mariabackup --apply-log-only.
2017-11-09 14:37:03 +02:00
sjaakola
f4f2e8fa2a MW-402 cascading FK issues
* created tests focusing in multi-master conflicts during cascading foreign key
  processing
* in row0upd.cc, calling wsrep_row_ups_check_foreign_constraints only when
  running in cluster
* in row0ins.cc fixed regression from MW-369, which caused crash with MW-402.test
2017-11-08 14:15:41 +02:00
Marko Mäkelä
7c85a8d936 Merge 10.1 into 10.2 2017-11-08 13:12:11 +02:00
Marko Mäkelä
644ffdeb92 Fix integer type mismatch in WSREP debug output 2017-11-08 09:26:46 +02:00
Marko Mäkelä
843e4508c0 Merge 10.1 into 10.2 2017-11-07 23:02:39 +02:00
Marko Mäkelä
a4feb04ace MDEV-14310 ALTER TABLE…ADD INDEX may corrupt the InnoDB system tablespace
FlushObserver::flush(): Never discard unwritten changes.
We do not want to risk corrupting the system tablespace
or .ibd files that are not part of a table-rebuilding ALTER.
2017-11-07 23:00:51 +02:00
Marko Mäkelä
d04c4b3905 MDEV-14304 Unnecessary conditions in buf_page_get_gen()
Ever since MDEV-10813 cleaned up InnoDB use of atomic memory operations
and made buf_block_fix() an atomic operation, some conditions around
buf_block_fix() have been unnecessary.
2017-11-07 22:45:41 +02:00
Marko Mäkelä
f830314fd5 Remove dead code for non-debug builds 2017-11-06 22:35:03 +02:00
Vladislav Vaintroub
40bae98c3d MDEV-12108 Fix backup for Innodb tables with DATA DIRECTORY 2017-11-06 19:21:23 +00:00
Marko Mäkelä
5691109689 Merge 10.0 into 10.1 2017-11-06 18:10:23 +02:00
Marko Mäkelä
51b4366bfb MDEV-13328 ALTER TABLE…DISCARD TABLESPACE takes a lot of time
With a big buffer pool that contains many data pages,
DISCARD TABLESPACE took a long time, because it would scan the
entire buffer pool to remove any pages that belong to the tablespace.
With a large buffer pool, this would take a lot of time, especially
when the table-to-discard is empty.

The minimum amount of work that DISCARD TABLESPACE must do is to
remove the pages of the to-be-discarded table from the
buf_pool->flush_list because any writes to the data file must be
prevented before the file is deleted.

If DISCARD TABLESPACE does not evict the pages from the buffer pool,
then IMPORT TABLESPACE must do it, because we must prevent pre-DISCARD,
not-yet-evicted pages from being mistaken for pages of the imported
tablespace.

It would not be a useful fix to simply move the buffer pool scan to
the IMPORT TABLESPACE step. What we can do is to actively evict those
pages that could be mistaken for imported pages. In this way, when
importing a small table into a big buffer pool, the import should
still run relatively fast.

Import is bypassing the buffer pool when reading pages for the
adjustment phase. In the adjustment phase, if a page exists in
the buffer pool, we could replace it with the page from the imported
file. Unfortunately I did not get this to work properly, so instead
we will simply evict any matching page from the buffer pool.

buf_page_get_gen(): Implement BUF_EVICT_IF_IN_POOL, a new mode
where the requested page will be evicted if it is found. There
must be no unwritten changes for the page.

buf_remove_t: Remove. Instead, use trx!=NULL to signify that a write
to file is desired, and use a separate parameter bool drop_ahi.

buf_LRU_flush_or_remove_pages(), fil_delete_tablespace():
Replace buf_remove_t.

buf_LRU_remove_pages(), buf_LRU_remove_all_pages(): Remove.

PageConverter::m_mtr: A dummy mini-transaction buffer

PageConverter::PageConverter(): Complete the member initialization list.

PageConverter::operator()(): Evict any 'shadow' pages from the
buffer pool so that pre-existing (garbage) pages cannot be mistaken
for pages that exist in the being-imported file.

row_discard_tablespace(): Remove a bogus comment that seems to
refer to IMPORT TABLESPACE, not DISCARD TABLESPACE.
2017-11-06 18:08:33 +02:00
Marko Mäkelä
57ba66b9ab Remove redundant function parameters
buf_flush_or_remove_pages(), buf_flush_dirty_pages(): Remove the
redundant parameter flush=(trx!=NULL).
2017-11-06 18:08:33 +02:00
Marko Mäkelä
6a524fcfdd MDEV-14140 IMPORT TABLESPACE must not go beyond FSP_FREE_LIMIT
ibuf_check_bitmap_on_import(): Only access the pages that
are below FSP_FREE_LIMIT. It is possible that especially with
ROW_FORMAT=COMPRESSED, the FSP_SIZE will be much bigger than
the FSP_FREE_LIMIT, and the bitmap pages (page_size*N, 1+page_size*N)
are filled with zero bytes.

buf_page_is_corrupted(), buf_page_io_complete(): Make the
fault injection compatible with MariaDB 10.2.

Backport the IMPORT tests from 10.2.
2017-11-06 14:55:34 +02:00
Marko Mäkelä
51679e5c38 MDEV-14132 InnoDB page corruption
On some old GNU/Linux systems, invoking posix_fallocate() with
offset=0 would sometimes cause already allocated bytes in the
data file to be overwritten.

Fix a correctness regression that was introduced in
commit 420798a81a
by invoking posix_fallocate() in a safer way.
A similar change was made in MDEV-5746 earlier.

os_file_get_size(): Avoid changing the state of the file handle,
by invoking fstat() instead of lseek().

os_file_set_size(): Determine the current size of the file
by os_file_get_size(), and then extend the file from that point
onwards.
2017-11-06 08:53:51 +02:00
Marko Mäkelä
30a8764b92 MDEV-14244 MariaDB fails to run with O_DIRECT
os_file_set_size(): If posix_fallocate() returns EINVAL, fall back
to writing zero bytes to the file. Also, remove some error log output,
and make it possible for a server shutdown to interrupt the fall-back
code.

MariaDB used to ignore any possible return value from posix_fallocate()
ever since innodb_use_fallocate was introduced in MDEV-4338. If EINVAL
was returned, the file would not be extended.

Starting with MDEV-11520, MariaDB would treat EINVAL as a hard error.

Why is the EINVAL returned? The GNU posix_fallocate() function
would first try the fallocate() system call, which would return
-EOPNOTSUPP for many file systems (notably, not ext4). Then, it
would fall back to extending the file one block at a time by invoking
pwrite(fd, "", 1, offset) where offset is 1 less than a multiple of
the file block size. This would fail with EINVAL if the file is in
O_DIRECT mode, because O_DIRECT requires aligned operation.
2017-11-06 08:53:50 +02:00
Monty
c9f612dbde Add more execution stages (commit, rollback, etc)
This was done to get more information about where time is spent.
Now we can get proper timing for time spent in commit, rollback,
binlog write etc.

Following stages was added:
- Commit
- Commit_implicit
- Rollback
- Rollback implicit
- Binlog write
- Init for update
  - This is used instead of "Init" for insert, update and delete.
- Staring cleanup

Following stages where changed:
- "Unlocking tables" stage reset stage to previous stage at end
- "binlog write" stage resets stage to previous stage at end
- "end" -> "end of update loop"
- "cleaning up" -> "Reset for next command"
- Added stage_searching_rows_for_update when searching for rows
  to be deleted.

Other things:
- Renamed all stages to start with big letter (before there was no
  consitency)
- Increased performance_schema_max_stage_classes from 150 to 160.
- Most of the test changes in performance schema comes from renaming of
  stages.
- Removed duplicate output of variables and inital state in a lot of
  performance schema tests.
  This was done to make it easier to change a default value for a
  performance variable without affecting all tests.
- Added start_server_variables.test to check configuration
- Removed some duplicate "closing tables" stages
- Updated position for "stage_init_update" and "stage_updating" for
  delete, insert and update to be just before update loop (for more
  exact timing).
- Don't set "Checking permissions" twice in a row.
- Remove stage_end stage from creating views (not done for create table
  either).
- Updated default performance history size from 10 to 20 because of new
  stages
- Ensure that ps_enabled is correct (to be used in a later patch)
2017-11-05 22:23:31 +02:00
Monty
61706e61fe Disable some test that fails constantly 2017-11-05 22:23:31 +02:00
Sergei Petrunia
8f2e8cf0cb Merge branch '10.2' of github.com:MariaDB/server into bb-10.2-mariarocks 2017-11-04 12:38:17 +03:00
Olivier Bertrand
b6563d773a - Fix MDEV-13860 CONNECT engine does not build with JDBC without ODBC.
By including Sergei's patch in connect_assisted_discovery.
  modified:   storage/connect/ha_connect.cc
2017-11-03 11:29:02 +01:00
Sachin Setiya
3cecb1bab3 Merge tag 'mariadb-10.0.33' into bb-10.0-galera 2017-11-03 12:34:05 +05:30
Alexander Barkov
5fc79476b5 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-03 09:29:21 +04:00
Monty
3ab112eb39 Fixed compiler warning and unitialized memory warning
- The valgrind warning came from JOIN::optimize() (sql_select.cc:1123)
2017-11-02 20:37:26 +02:00
Monty
c127a1ba14 Reduce memory used by maria_open
- Allocate only enough memory that we need for key status
  - Saves ~100K temp usage on maria_open
2017-11-02 20:37:26 +02:00
Vladislav Vaintroub
b0cfb16867 Fix a warning. 2017-11-02 17:48:50 +00:00
Sergei Petrunia
80d61515ac Make rocksdb.read_only_tx pass and enable it
- FB/MySQL 5.6' MyRocks has START TRANSACTION WITH CONSISTENT
  ROCKSDB SNAPSHOT, which returns binlog position.

- MariaDB has a cross-engine START TRANSACTION WITH CONSISTENT
  SNAPSHOT. It can be used for the same purpose. Binlog position
  can be obtained from Binlog_snapshot_file/position status vars.
2017-11-02 16:15:37 +00:00
Marko Mäkelä
19733efa7b MDEV-14244 MariaDB 10.2.10 fails to run on Debian Stretch with ext3 and O_DIRECT
os_file_set_size(): If posix_fallocate() returns EINVAL, fall back
to writing zero bytes to the file. Also, remove some error log output,
and make it possible for a server shutdown to interrupt the fall-back
code.

MariaDB 10.2 used to handle the EINVAL return value from posix_fallocate()
before commit b731a5bcf2
which refactored os_file_set_size() to try posix_fallocate().

Why is the EINVAL returned? The GNU posix_fallocate() function
would first try the fallocate() system call, which would return
-EOPNOTSUPP for many file systems (notably, not ext4). Then, it
would fall back to extending the file one block at a time by invoking
pwrite(fd, "", 1, offset) where offset is 1 less than a multiple of
the file block size. This would fail with EINVAL if the file is in
O_DIRECT mode, because O_DIRECT requires aligned operation.
2017-11-02 16:18:41 +02:00
Monty
0f4e005541 Fixed compiler warning and warning from valgrind
The failing test was main.gis-json
2017-11-02 15:40:27 +02:00
Sergei Golubchik
c4c48e9740 MDEV-11965 -Werror should not appear in released tarballs 2017-11-02 06:32:20 +00:00
Marko Mäkelä
2ec7b87053 Merge 10.2 into bb-10.2-ext 2017-11-01 14:37:31 +02:00
Marko Mäkelä
6692b5f74a Merge 10.1 into 10.2 2017-11-01 09:55:00 +02:00
Olivier Bertrand
332d9f7b02 - Change inihandl from c to c++.
Because it now includes global.h that contains a bool function definition
  that make compile to fail on Linux.
  modified:   storage/connect/CMakeLists.txt
  removed:    storage/connect/inihandl.c
  added:      storage/connect/inihandl.cpp
2017-10-31 18:53:15 +01:00
Olivier Bertrand
646ecb8970 Change inihandl from c to cpp (to accept bool) 2017-10-31 17:51:25 +01:00
Olivier Bertrand
3db76c99cc - Fix MDEV-13925: Actually this fixes SELECT queries when
the WHERE clause have single quote.
  modified:   storage/connect/ha_connect.cc

- Use Windows VirtualAlloc and VirtualFree for the Sarea workspace
  modified:   storage/connect/global.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/plugutil.cpp
  modified:   storage/connect/user_connect.cc
2017-10-31 14:44:50 +01:00
Marko Mäkelä
892cf2de13 Merge 10.0 into 10.1 2017-10-31 09:11:31 +02:00