This patch corrects the fix of the patch for mdev-19421 that resolved
the problem of parsing some embedded join expressions such as
t1 join t2 left join t3 on t2.a=t3.a on t1.a=t2.a.
Yet the patch contained a bug that prevented proper context analysis
of the queries where such expressions were used together with comma
separated table references in from clauses.
MemorySanitizer is a compile-time instrumentation layer in clang and GCC.
Together with AddressSanitizer mostly makes the run-time instrumentation
of Valgrind redundant. It is a little more tricky to set up, because
running with uninstrumented libraries will lead into false positives.
You will need an instrumented libc++, and you should use
-stdlib=libc++ instead of the default libstdc++. To build the
instrumented library, you can refer to
https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo
or you can adapt these steps that worked for me, for clang-8 version 8.0.1:
cd /mariadb
sudo apt source libc++-8-dev
cd llvm-toolchain-8-8.0.1
mkdir libc++msan; cd libc++msan
cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory \
-DCMAKE_C_COMPILER=clang-8 -DCMAKE_CXX_COMPILER=clang++-8
Then, in your MariaDB build directory, you have to compile with
libc++ and bundled libraries, such as WITH_SSL=bundled, WITH_ZLIB=bundled.
For uninstrumented system libraries, you will get false positives for
uninitialized values. Like this:
cmake -DWITH_MSAN=ON -DWITH_SSL=bundled -DWITH_ZLIB=bundled \
-DCMAKE_CXX_FLAGS='-stdlib=libc++' ..
Note: you should also add -O2 to the compiler options, or you may
get crashes due to stack overflow.
Finally, to run tests, you must replace libc++ with the instrumented one:
LD_LIBRARY_PATH=/mariadb/llvm-toolchain-8-8.0.1/libc++msan/lib \
MSAN_OPTIONS=abort_on_error=1 \
./mtr --big-test --parallel=auto --force --retry=0
Failure to do so will report numerous false positives related to
operations on std::string and the like.
This is work in progress. Some issues will still have to be fixed
for WITH_MSAN to be usable. See MDEV-20377 for details.
===================
- Modify tracing to use htrc to be compatible with old versions
when this code is used to make an EOM module.
modified: storage/connect/restget.cpp
modified: storage/connect/tabrest.cpp
- Path apparently not needed for the cpprest lib on Linux
modified: storage/connect/CMakeLists.txt
In addition to files and Mongo collections, JSON as well as XML and CSV data can be retrieved
from the net as answers from REST queries. Because it uses and external package (cpprestsdk)
this is currently available only to MariaDB servers compiled from source.
-- Add compile flags needed on Windows /MD or /MDd (debug)
-- Also include some changes needed on Linux
modified: storage/connect/CMakeLists.txt
- Add the xtrc tracing function
modified: storage/connect/global.h
modified: storage/connect/plugutil.cpp
- Modify tracing to use xtrc and some typo
modified: storage/connect/array.cpp
modified: storage/connect/block.h
modified: storage/connect/restget.cpp
- Fix compilation error when ZIP is not supported
modified: storage/connect/ha_connect.cc
modified: storage/connect/tabfmt.cpp
- Add some tracing + typo
modified: storage/connect/mycat.cc
modified: storage/connect/tabjson.cpp
- Add conditional code based on MARIADB
This to be able to use the same code in CONNECT and EOM modules
modified: storage/connect/osutil.h
modified: storage/connect/tabrest.cpp
- Replace PlugSetPath by some concat (crashed on Fedora) + typo
modified: storage/connect/reldef.cpp
- Try to fix test failures
modified: zlib/CMakeLists.txt
========
During ibd file creation, InnoDB flushes the page0 without crypt
information. During recovery, InnoDB encounters encrypted page read
before initialising the crypt data of the tablespace. So it leads t
corruption of page and doesn't allow innodb to start.
Solution:
=========
Write crypt_data information in page0 while creating .ibd file creation.
During recovery, crypt_data will be initialised while processing
MLOG_FILE_NAME redo log record.
Optimize the test by dropping the table early and by using only
one undo log thread, so that purge will be doing more useful work
and less busy work of suspending and resuming the worker threads.
The test used to cause shutdown timeout on 10.4 on buildbot, and
for me locally when using --mysqld=--innodb-sync-debug.
With these tweaks, it passes for me with --mysqld=--innodb-sync-debug.
It looks like the merge of MySQL 5.7.9 to MariaDB 10.2.2 conflicted with
earlier changes that were made in MDEV-8588.
row_search_mvcc(): If the page is corrupted, avoid invoking
btr_cur_store_position(). The caller should not try to fetch
the next record after a hard error.
There were two problems:
(1) If user wanted same time zone information on all nodes in the Galera
cluster all updates were not replicated as time zone information was
stored on MyISAM tables. This is fixed on Galera by altering time zone
tables to InnoDB while they are modified.
(2) If user wanted different time zone information to nodes in the Galera
cluster TRUNCATE TABLE for time zone tables was replicated by Galera
destroying time zone information from other nodes. This is fixed
on Galera by introducing new option for mysql_tzinfo_to_sql_symlink
tool --skip-write-binlog to disable Galera replication while
time zone tables are modified.
Changes to be committed:
modified: mysql-test/r/mysql_tzinfo_to_sql_symlink.result
modified: mysql-test/suite/wsrep/r/mysql_tzinfo_to_sql_symlink.result
new file: mysql-test/suite/wsrep/r/mysql_tzinfo_to_sql_symlink_skip.result
new file: mysql-test/suite/wsrep/t/mysql_tzinfo_to_sql_symlink_skip.test
modified: sql/tztime.cc
When discounting selectivity of ref access, don't discount the
selectivity we've already discounted for range access.
The 10.1 version of the fix. Will need to adjust condition filtering
test results in 10.4
Problem:
=======
DROP TABLE IF EXISTS was killed. The table still exists on
the master but the DDL was still logged.
Analysis:
=========
During the execution of DROP TABLE command "ha_delete_table" call is invoked
to delete the table. If the query is killed at this point, the kill command
is not handled within the code. This results in two issues.
1) The table which is not dropped also gets written into the binary log.
2) The code continues further upon receiving 'KILL QUERY'.
Fix:
===
Upon receiving the KILL command the query should stop its current execution.
Tables which were successfully dropped prior to KILL command should be
included in the binary log.
If there're multiple row versions in InnoDB, reading one row from PK
may have O(N) complexity and reading from secondary keys may have
O(N^2) complexity.
The problem occurs when there are many pending versions of the same
row, meaning that the primary key is the same, but a secondary key is
different. The slowdown occurs when the secondary index is
traversed. This patch creates a helper class for the function
row_sel_get_clust_rec_for_mysql() which can remember and re-use
cached_clust_rec & cached_old_vers so that rec_get_offsets() does not
need to be called over and over for the clustered record.
Corrections by Kevin Lewis <kevin.lewis@oracle.com>
MDEV-20341 Unstable innodb.innodb_bug14704286
Removed test that tested the ability of interrupting long query which
is not long anymore.
A histogram size that is odd in size with DOUBLE precision will leave the last
byte unwritten. When collecting histograms, this causes the last byte to
be uninitialized in the record. memset the buffer to 0 first to make
sure this does not happen.
- Include the valgrind suppressions from the FB upstream
- Use HAVE_Valgrind, not HAVE_Purify (like the rest of MariaDB code does)
The call to DisownData() is now actually disabled under Valgrind
Use DEBUG_SYNC to hang the execution at the interesting point,
and then kill and restart the server externally. This will work
also with Valgrind. DBUG_SUICIDE() causes Valgrind to hang,
and it could also cause uninteresting reports about memory leaks.
While we are at it, let us clean up innodb.innodb_bulk_create_index_debug
so that it will actually test the desired functionality also in future
versions (with instant ADD COLUMN and DROP COLUMN) and avoid
some unnecessary restarts.
We are adding two DEBUG_SYNC points for ALTER TABLE, because there were
none that would be executed right before ha_commit_trans().
Galera threads were not registered to performance schema and
used pthread_create when mysql_thread_create should have been
used.
Added test case to verify current galera performance schema
instrumentation does work.
Skip the test on big-endian systems.
In MariaDB Server 10.0 and 10.1 (as well as MySQL 5.6),
the implementation of innodb_checksum_algorithm=crc32
wrongly assumes little-endian byte order.
MDEV-17614 flags INSERT…ON DUPLICATE KEY UPDATE unsafe for statement-based
replication when there are multiple unique indexes. This correctly fixes
something whose attempted fix in MySQL 5.7
in mysql/mysql-server@c93b0d9a97
caused lock conflicts. That change was reverted in MySQL 5.7.26
in mysql/mysql-server@066b6fdd43
(with a substantial amount of other changes).
In MDEV-17073 we already disabled the unfortunate MySQL change when
statement-based replication was not being used. Now, thanks to MDEV-17614,
we can actually remove the change altogether.
This reverts commit 8a346f31b9 (MDEV-17073)
and mysql/mysql-server@c93b0d9a97 while
keeping the test cases.
- mysqltest didn't free read_command_buf
- wait_for_slave_param did write different things to the log if valgrind
was used.
- Table open cache should not write the initial variable value as it
can depend on the configuration or if valgrind is used
- A variable in GetResult was used uninitalized
... produces "bytes lost" warnings
When rocksdb_validate_update_cf_options() returns an error,
the update won't happen.
Free the copy of the string in this case.
Problem:-
When mysql executes INSERT ON DUPLICATE KEY INSERT, the storage engine checks
if the inserted row would generate a duplicate key error. If yes, it returns
the existing row to mysql, mysql updates it and sends it back to the storage
engine.When the table has more than one unique or primary key, this statement
is sensitive to the order in which the storage engines checks the keys.
Depending on this order, the storage engine may determine different rows
to mysql, and hence mysql can update different rows.The order that the
storage engine checks keys is not deterministic. For example, InnoDB checks
keys in an order that depends on the order in which indexes were added to
the table. The first added index is checked first. So if master and slave
have added indexes in different orders, then slave may go out of sync.
Solution:-
Make INSERT...ON DUPLICATE KEY UPDATE unsafe while using stmt or mixed format
When there is more then one unique key.
Although there is two exception.
1. Auto Increment key is not counted because Innodb will get gap lock for
failed Insert and concurrent insert will get a next increment value. But if
user supplies auto inc value it can be unsafe.
2. Count only unique keys for which insertion is performed.
So this patch also addresses the bug id #72921
MDEV-17717
Assertion `!table->pos_in_locked_tables' failed in tc_release_table on
flushing RocksDB table under SERIALIZABLE
MDEV-17998
Deadlock and eventual Assertion `!table->pos_in_locked_tables' failed
in tc_release_table on KILL_TIMEOUT
MDEV-19591
Assertion `!table->pos_in_locked_tables' failed in tc_release_table upon
altering table into S3 under lock.
The problem was that thd->open_tables->pos_in_locked_tables was not reset
when alter table failed to reopen a locked table.
- pcretest.c could use macro with side effect
- maria_chk could access freed memory
- Initialized some variables that could be accessed uninitalized
- Fixed compiler warning in my_atomic-t.c
The general reason why innodb redo log file is limited by 512G is that
log_block_convert_lsn_to_no() returns value limited by 1G. But there is no
need to have unique log block numbers in log group. The fix removes 512G
limit and limits log group size by
(uint32_t maximum value) * (minimum page size), which, in turns, can be
removed if fil_io() is no longer used for innodb redo log io.
- The commit ab6dd77408 wrongly sets the
condition inside innobase_srv_conc_enter_innodb(). Problem is that
InnoDB makes the thread to sleep indefinitely if it is a replication
slave thread.
Thanks to Sujatha Sivakumar for contributing the replication test case.
Non-owning reference to elements.
Use it as function argument instead of pointer+size pair or instead of
const std::vector<T>.
Do not use it for strings!
More info is here http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines
Or just google about it.
Problem:
=======
Failed CREATE OR REPLACE TEMPORARY TABLE statement which dropped the table but
failed at a later stage of creation of temporary table is not written to
binarylog in row based replication. This causes the slave to diverge.
Analysis:
========
CREATE OR REPLACE statements work as shown below.
CREATE OR REPLACE TABLE table_name (a int);
is basically the same as:
DROP TABLE IF EXISTS table_name;
CREATE TABLE table_name (a int);
Hence every CREATE OR REPLACE TABLE command which dropped the table should be
written to binary log, even when following CREATE TABLE part fails. In order
to achieve this, during the execution of CREATE OR REPLACE command, when a
table is dropped 'thd->log_current_statement' flag is set. When table creation
results in an error within 'mysql_create_table' code, the error handling part
looks for this flag. If it is set the failed CREATE OR REPLACE statement is
written into the binary log inspite of error. This ensure that slave doesn't
diverge from the master. In case of row based replication the error handling
code returns very early, if the table is of type temporary. This is done based
on the assumption that temporary tables are not replicated in row based
replication.
It fails to handle the cases where a temporary table was created as part of
statement based replication at an earlier stage and the binary log format was
changed to row because of an unsafe statement. In this case when a CREATE OR
REPLACE statement is executed on this temporary table it will dropped but the
query will not be written to binary log. Hence slave diverges.
Fix:
===
In error handling code check the return status of create table operation. If
it is successful and replication mode is row based and table is of type
temporary then return. Other wise proceed further to the code which checks for
thd->log_current_statement flag and does appropriate logging.
