Close MYSQL (and destroy THD) in the same thread where it was used,
because THD embeds MDL_context, that owns some LF_PINS, that remember
a pointer to my_thread_var->stack_ends_here.
simplify. move common code inside, specify common flags inside,
rewrite dead code (`if (mode & O_TEMPORARY)` on Linux, where
`O_TEMPORARY` is always 0) to actually do something.
The functionality of the socket system variable is extended
here such that a preciding '@' indicates that the socket
will be an abstract socket. Thie socket name wil be
the remainder of the name after the '@'. This is consistent
with the approached used by systemd in socket activation.
Thanks to Sergey Vojtovich:
On OS X sockaddr_un is defined as:
struct sockaddr_un
{
u_char sun_len;
u_char sun_family;
char sun_path[104];
};
There is a comment in man 7 unix (on linux):
"
On Linux, the above offsetof() expression equates to the same value as sizeof(sa_family_t),
but some other implementations include other fields before sun_path, so the offsetof()
expression more portably describes the size of the address structure.
"
As such, use the offsetof for Linux and use the previous sizeof(UNIXaddr)
for non-unix platforms as that's what worked before and they don't
support abstract sockets so there's no compatibility problem..
strace -fe trace=networking mysqld --skip-networking --socket @abc ...
...
[pid 10578] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 22
[pid 10578] setsockopt(22, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
[pid 10578] bind(22, {sa_family=AF_UNIX, sun_path=@"abc"}, 6) = 0
[pid 10578] listen(22, 80) = 0
...
Version: '10.3.6-MariaDB-log' socket: '@abc' port: 0 Source distribution
$ lsof -p 10578
mysqld 10578 dan 22u unix 0x00000000087e688c 0t0 4787815 @abc type=STREAM
MariaDB adjustments.
mysqltest.cc : Allow 12 error codes at --error
wait_until_connected_again.inc: Replace numeric error codes with symbols
mysqltest.test: Add error codes to test that tests too many errorcodes
The merge only covered 10.1 up to
commit 4d248974e0.
Actually merge the changes up to
commit 0a534348c7.
Also, remove the unused InnoDB field trx_t::abort_type.
Handle string length as size_t, consistently (almost always:))
Change function prototypes to accept size_t, where in the past
ulong or uint were used. change local/member variables to size_t
when appropriate.
This fix excludes rocksdb, spider,spider, sphinx and connect for now.
mysqltest cannot free all memory on exit by design,
so there's no need to check.
mysql frees memory in mysql_end(), so enable memory-leak-on-exit check
only after it was initialized enough to use mysql_end() - early exits
use my_end().
This will make it easier to how memory allocation is done when debugging
with either DBUG or gdb.
Will especially help when debugging stored procedures
Main change is a name argument as second argument to init_alloc_root()
init_sql_alloc()
Other things:
- Added DBUG_ENTER/EXIT to some Virtual_tmp_table functions
Main problem was that no log-event print function checked for disk
full error on the IO_CACHE.
All changes in this patch only affects mysqlbinlog, not the server!
- Changed all log-event print functions to return 1 on error
- Fixed memory usage when not using --flashback.
- Added printing of number of rows in row events. Can be disabled with
--print-row-count=0
- Print annotated rows when using mysqlbinlog --short-form
- Fixed that mysqlbinlog --debug works
- Fixed create_drop_binlog.test test failure
- Reorganized fields in PRINT_EVENT_INFO to be according to size to
optimize storage
- Don't change print_row_event_position or print_row_counts if set by user
- Remove some testing of argument to my_free is 0
- base64-output=never is now supported and works in all context
- Updated help information for --base64-output and --short-form
- print_row_count is now on by default. Reset automatically if --short-form
is used
- Removed obsolote warning for mysql 5.6.0
- More DBUG_PRINT for mysqltest.cc
- my_b_write_byte() now checks for flush failures. This fixed a memory
overrun on disk full
- my_b_printf() now returns 1 on failure, 0 on ok. This simplifies code
and no old code was using the old return value of my_b_printf().
- my_b_Write_backtick_quote() now returns 1 on failure and 0 on ok
- Fixed some error conditions in log printing that was not previously
handled.
- Slave_rows_error_report() can now handle longlong positions
- Write_on_release_cache() rewritten so that we can detect errors
on flush. Not depending on automatic release anymore.
- Changed types for Pos and End_log_pos to 64 bit in SHOW BINLOG EVENTS
- Fixed that copy_event_cache_to_string_and_reinit() works with strings
longer than 4G (Changed to use LEX_STRING instead of String)
- Restricted binlog_rows_event_max_size to UINT32_MAX-1 as String's are
anyway restricted to UINT32_MAX
- Fixed bug in rpl_binlog_state::write_to_iocache() which hide write
failures (duplicate variable name)
- Fixed bug in String::append if original string was not allocated
- Stop mysqlbinlog output at once if there is an error.
- Before printing error message, flush result file. This ensures that
the error message is printed last. (Easier to find)
find_type_or_exit() client helper did exit(1) on error, exit(1) moved to
clients.
mysql_read_default_options() did exit(1) on error, error is passed through and
handled now.
my_str_malloc_default() did exit(1) on error, replaced my_str_ allocator
functions with normal my_malloc()/my_realloc()/my_free().
sql_connect.cc did many exit(1) on hash initialisation failure. Removed error
check since my_hash_init() never fails.
my_malloc() did exit(1) on error. Replaced with abort().
my_load_defaults() did exit(1) on error, replaced with return 2.
my_load_defaults() still does exit(0) when invoked with --print-defaults.
- Fix win64 pointer truncation warnings
(usually coming from misusing 0x%lx and long cast in DBUG)
- Also fix printf-format warnings
Make the above mentioned warnings fatal.
- fix pthread_join on Windows to set return value.
Update.rdiff fle
Also, introduce my_popen()/my_fgets() wrapper function for popen()/fgets()
in mysqltest to workaround a popen() bug in Windows C runtime,
mentioned in MDEV-9409
This workaround was used previously for "exec". From now on, it is also
used are used also for "perl" snippets.
ATTRIBUTE_NORETURN is supported on all platforms (MSVS and GCC-like).
It declares that a function will not return; instead, the thread or
the whole process will terminate.
ATTRIBUTE_COLD is supported starting with GCC 4.3. It declares that
a function is supposed to be executed rarely. Rarely used error-handling
functions and functions that emit messages to the error log should be
tagged such.
For running the Galera tests, the variable my_disable_leak_check
was set to true in order to avoid assertions due to memory leaks
at shutdown.
Some adjustments due to MDEV-13625 (merge InnoDB tests from MySQL 5.6)
were performed. The most notable behaviour changes from 10.0 and 10.1
are the following:
* innodb.innodb-table-online: adjustments for the DROP COLUMN
behaviour change (MDEV-11114, MDEV-13613)
* innodb.innodb-index-online-fk: the removal of a (1,NULL) record
from the result; originally removed in MySQL 5.7 in the
Oracle Bug #16244691 fix
377774689b
* innodb.create-index-debug: disabled due to MDEV-13680
(the MySQL Bug #77497 fix was not merged from 5.6 to 5.7.10)
* innodb.innodb-alter-autoinc: MariaDB 10.2 behaves like MySQL 5.6/5.7,
while MariaDB 10.0 and 10.1 assign different values when
auto_increment_increment or auto_increment_offset are used.
Also MySQL 5.6/5.7 exhibit different behaviour between
LGORITHM=INPLACE and ALGORITHM=COPY, so something needs to be tested
and fixed in both MariaDB 10.0 and 10.2.
* innodb.innodb-wl5980-alter: disabled because it would trigger an
InnoDB assertion failure (MDEV-13668 may need additional effort in 10.2)
If this variable is set, skip actual AWS calls, and fake/mock
both generation and encryption of the keys.
The advantage of having a mock mode is that more aws_key_management tests
can be enabled on buildbot.
If compiling a non DBUG binary with
-DDBUG_ASSERT_AS_PRINTF asserts will be
changed to printf + stack trace (of stack
trace are enabled).
- Changed #ifndef DBUG_OFF to
#ifdef DBUG_ASSERT_EXISTS
for those DBUG_OFF that was just used to enable
assert
- Assert checking that could greatly impact
performance where changed to DBUG_ASSERT_SLOW which
is not affected by DBUG_ASSERT_AS_PRINTF
- Added one extra option to my_print_stacktrace() to
get more silent in case of stack trace printing as
part of assert.
Also, include fixes by Vladislav Vaintroub to the
aws_key_management plugin. The AWS C++ SDK specifically depends on
OPENSSL_LIBRARIES, not generic SSL_LIBRARIES (such as YaSSL).
- Do not throw output of exec command, if disable_result_log is set
save and dump it if exec fails. Need tha to meaningfully analyze
errors from mariabackup.
- rmdir now removes the entire tree. need that because xtrabackup tests
clean the whole directory.
- all filesystem modifying commands now require the argument to
be under MYSQLTEST_VARDIR or MYSQL_TMP_DIR.
* remove redundant casts
* fix fix_win_paths() not to pretend that it takes const char* string
because it changes it. Fix its callers not to pass const strings
into it.
* use _WIN32 not __WIN__
MYSQL_OPT_SSL_MODE option introduced.
It is set in case of --ssl-mode=REQUIRED and permits only SSL connection.
(cherry picked from commit 3b2d28578c526f347f5cfe763681eff365731f99)
- MDEV-11621 rpl.rpl_gtid_stop_start fails sporadically in buildbot
- MDEV-11620 rpl.rpl_upgrade_master_info fails sporadically in buildbot
The issue above was probably that the build machine was overworked and the
shutdown took longer than 30 resp 10 seconds, which caused MyISAM tables
to be marked as crashed.
Fixed by flushing myisam tables before doing a forced shutdown/kill.
I also increased timeout for forced shutdown from 10 seconds to 60 seconds
to fix other possible issues on slow machines.
Fixed also some compiler warnings
Backport the fix to 5.5, because it fails there too
The patch fixes two test failures:
- on slow builders, sometimes a connection attempt which should
fail due to the exceeded number of thread_pool_max_threads
actually succeeds;
- on even slow builders, MTR sometimes cannot establish the
initial connection, and check-testcase fails prior to the
test start
The problem with check-testcase was caused by connect-timeout=2
which was set for all clients in the test config file. On slow
builders it might be not enough.
There is no way to override it for the pre-test check, so it needed
to be substantially increased or removed.
The other problem was caused by a race condition between sleeps
that the test performs in existing connections and the connect
timeout for the connection attempt which was expected to fail.
If sleeps finished before the connect-timeout was exceeded, it
would allow the connection to succeed.
To solve each problem without making the other one worse,
connect-timeout should be configured dynamically during the test.
Due to the nature of the test (all connections must be busy
at the moment when we need to change the timeout, and cannot execute
SET GLOBAL ...), it needs to be done independently from the server.
The solution:
- recognize 'connect_timeout' as a connection option in mysqltest's
"connect" command;
- remove connect-timeout from the test configuration file;
- use the new connect_timeout option for those connections which
are expected to fail;
- re-arrange the test flow to allow running a huge SLEEP
without affecting the test execution time (because it would be
interrupted after the main test flow is finished).
The test is still subject to false negatives, e.g. if the connection
fails due to timeout rather than due to the exceeded number of
allowed threads, or if the connection on extra port succeeds due
to a race condition and not because the special logic for the extra
port. But those false negatives have always been possible there
on slow builders, they should not be critical because faster builders
should catch such failures if they appear.
Conflicts:
client/mysqltest.cc
mysql-test/r/pool_of_threads.result
mysql-test/t/pool_of_threads.test
The patch fixes two test failures:
- on slow builders, sometimes a connection attempt which should
fail due to the exceeded number of thread_pool_max_threads
actually succeeds;
- on even slow builders, MTR sometimes cannot establish the
initial connection, and check-testcase fails prior to the
test start
The problem with check-testcase was caused by connect-timeout=2
which was set for all clients in the test config file. On slow
builders it might be not enough.
There is no way to override it for the pre-test check, so it needed
to be substantially increased or removed.
The other problem was caused by a race condition between sleeps
that the test performs in existing connections and the connect
timeout for the connection attempt which was expected to fail.
If sleeps finished before the connect-timeout was exceeded, it
would allow the connection to succeed.
To solve each problem without making the other one worse,
connect-timeout should be configured dynamically during the test.
Due to the nature of the test (all connections must be busy
at the moment when we need to change the timeout, and cannot execute
SET GLOBAL ...), it needs to be done independently from the server.
The solution:
- recognize 'connect_timeout' as a connection option in mysqltest's
"connect" command;
- remove connect-timeout from the test configuration file;
- use the new connect_timeout option for those connections which
are expected to fail;
- re-arrange the test flow to allow running a huge SLEEP
without affecting the test execution time (because it would be
interrupted after the main test flow is finished).
The test is still subject to false negatives, e.g. if the connection
fails due to timeout rather than due to the exceeded number of
allowed threads, or if the connection on extra port succeeds due
to a race condition and not because the special logic for the extra
port. But those false negatives have always been possible there
on slow builders, they should not be critical because faster builders
should catch such failures if they appear.
mysqldump/mysql_install_db.exe fail
The bug is described in
https://connect.microsoft.com/VisualStudio/Feedback/Details/1902345
When reading from a pipe in text mode, using CRT function such as fread(),
some newlines may be lost. Workaround is to use binary mode on reading side
and if necessary, replace \r\n with \n.
The issue was that when running with valgrind the wait for master_pos_Wait()
was not long enough.
This patch also fixes two other failures that could affect rpl_mdev6020:
- check_if_conflicting_replication_locks() didn't properly check domains
- 'did_mark_start_commit' was after signals to other threads was sent which could
get the variable read too early.
Fix test whether process is alive in mysqltest.
Also fix SHUT_RD definition on Windows to be SD_RECEIVE.
SD_BOTH was used instead prior to this patch, and this would
occasionally make mysql_shutdown() fail - when the socket for the current connection
is not able send the COM_SHUTDOWN response anymore.
- Removed some QQ markers
- Removed some rows not compatible with valgrind 3.9.0
- Made mysql_install_db.sh more silent by default. --verbose now gives more information
- Added assert that auto-increment doesn't generate 0 (safety)
- Removed thd->set_time() in some places as it's set in init_for_queries()
- Fixed some --big tests in tokudb
- Fixed a bug in mysql_client_test.cc where sql_mode was not properly reset
cherry-pick f1daf9ce from 10.0 branch
-------------------------------------
Fix build failures caused by new C runtime library
- isnan, snprintf, struct timespec are now defined, attempt to
redefine them leads
- P_tmpdir, tzname are no more defined
- lfind() and lsearch() in lf_hash.c had to be renamed, declaration
conflicts with some C runtime functions with the same name declared in
a header included by stdlib.h
Also fix couple of annoying warnings :
- remove #define NOMINMAX from config.h to avoid "redefined" compiler
warnings(NOMINMAX is already in compile flags)
- disable incremental linker in Debug as well (feature not used much
and compiler crashes often)
Also simplify package building with Wix, require Wix 3.9 or later
(VS2015 is not compatible with old Wix 3.5/3.6)
Fix test whether process is alive in mysqltest.
Also fix SHUT_RD definition on Windows to be SD_RECEIVE.
SD_BOTH was used instead prior to this patch, and this would
occasionally make mysql_shutdown() fail - when the socket for the current connection
is not able send the COM_SHUTDOWN response anymore.
Fix build failures caused by new C runtime library
- isnan, snprintf, struct timespec are now defined, attempt to
redefine them leads
- P_tmpdir, tzname are no more defined
- lfind() and lsearch() in lf_hash.c had to be renamed, declaration
conflicts with some C runtime functions with the same name declared in
a header included by stdlib.h
Also fix couple of annoying warnings :
- remove #define NOMINMAX from config.h to avoid "redefined" compiler
warnings(NOMINMAX is already in compile flags)
- disable incremental linker in Debug as well (feature not used much
and compiler crashes often)
Also simplify package building with Wix, require Wix 3.9 or later
(VS2015 is not compatible with old Wix 3.5/3.6)
- If run with valgrind, mysqltest will now wait longer when syncronizing slave with master
- Ensure that we wait with cleanup() until slave thread has stopped.
- Added signal_thd_deleted() to signal close_connections() that all THD's has been freed.
- Check in handle_fatal_signal() that we don't use variables that has been freed.
- Increased some timeouts when run with --valgrind
Other things:
- Fixed wrong test in one_thread_per_connection_end() if galera is used.
- Removed not needed calls to THD_CHECK_SENTRY() when we are calling 'delete thd'.
As the MF_THREAD_SPECIFIC was introduced to the alloc_root's and
the prealloc added to the statement::mem_root and statement::result.alloc, we have to adjust
the embedded server to it. The preallocation was removed for the embedded server as it
makes no sence for it. The msyqltest should free the statement inside the proper thead to
make the memory statistics happy.
Part 1: first 2 cases of valgrind complain. context_analysis_only can be used on non-started LEX (opening tables)
obviouse fixes in DBUG and is_lex_started assignment.
in when using vio->read_timeout (and write_timeout) to set
timeouts of a new vio, as in:
vio_timeout(vio, 0, old_vio.read_timeout)
vio_timeout(vio, 0, old_vio.write_timeout)
remember that timeouts are stored in ms, but vio_timeout()'s
argument is in seconds.
The problem was that my_hash_sort didn't properly delete end-space characters properly, so strings that should compare
identically was seen as different strings. (Space was handled correctly, but not NBSP)
This caused duplicate key errors when a heap table was converted to Aria as part of overflow in group by.
Fixed by removing all characters that compares as end space when creating a hash.
Other things:
- Fixed that --sorted_results also works for errors in mysqltest.
- Speed up hash by not comparing strings that has different hash.
- Speed up many my_hash_sort functions by using registers to calculate hash instead of pointers.
This was previously done for some functions, but not for all.
- Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.
client/mysqltest.cc:
Fixed that --sorted_results also works for error messages.
mysql-test/r/ctype_partitions.result:
New test to ensure that partitions on hash works
mysql-test/suite/multi_source/gtid.result:
Updated result
mysql-test/suite/multi_source/gtid.test:
Test that --sorted_result works for error messages
mysql-test/suite/multi_source/gtid_ignore_duplicates.result:
Updated result
mysql-test/suite/multi_source/gtid_ignore_duplicates.test:
Updated result
mysql-test/suite/multi_source/load_data.result:
Updated result
mysql-test/suite/multi_source/load_data.test:
Updated result
mysql-test/t/ctype_partitions.test:
New test to ensure that partitions on hash works
storage/heap/hp_write.c:
Speed up hash by not comparing strings that has different hash.
storage/maria/ma_check.c:
Extra debug
strings/ctype-bin.c:
Use macro for hash function
strings/ctype-latin1.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-mb.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-simple.c:
Use macro for hash function
Use same variable names as in other my_hash_sort functions.
Update my_hash_sort_simple() to properly remove end space (patch by Bar)
strings/ctype-uca.c:
Ignore duplicated space inside strings and end space in my_hash_sort_uca(). This fixed MDEV-6255
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-ucs2.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-utf8.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/strings_def.h:
Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.
MySQL 5.6 implemented WL#344, which is about a MASTER_DELAY option to CHANGE
MASTER. But as part of this worklog, the format of the realy-log.info file was
changed. The new format is not understood by earlier versions, and nor by
MariaDB 10.0, so changing server to those versions would cause the slave to
abort with an error due to reading incorrect data out of relay-log.info.
Fix this by backporting from the WL#344 patch just the code that understands
the new relay-log.info format. We still write out the old format, and none of
the MASTER_DELAY feature is backported with this commit.
The bug was in the mysqltest.c. It didn't check the send_query result
properly, so when that fails and the thread gets disconnected,
it goes unnoticed, so the consequitive read_result call crashes.
Fixed by checking the sernd_query return.
Reduced number of my_hash_sort_bin() calls from 4 to 1 per query.
Reduced number of memory accesses done by my_hash_sort_bin().
Details:
- let MDL subsystem use pre-calculated hash value for hash
inserts and deletes
- let table cache use pre-calculated MDL hash value
- MDL namespace is excluded from hash value calculation, so that
hash value can be used by table cache as is
- hash value for MDL is calculated as resulting hash value + MDL
namespace
- extended hash implementation to accept user defined hash function
mysqltest in the 'embedded-server' mode runs queries in a separate thread,
but it didn't do so for the prepared statements - they were run in the main thread.
That leads to inconsistencies.
When a test sets SESSION 'dbug' variable like
SET SESSION debug_dbug="+d,warn_during_ha_commit_trans";
it is run as a plain query in that separate thread, so the main thread remains unaffected.
After that the prepared statement run in the main thread doesn't produce expected 'dbug' errors,
so the test fails.
To fix that I made prepared statement to be run in that special thread along with the plain queries.
That makes the environment consistent.
