Instead of encrypt(src, dst, key, iv) that encrypts all
data in one go, now we have encrypt_init(key,iv),
encrypt_update(src,dst), and encrypt_finish(dst).
This also causes collateral changes in the internal my_crypt.cc
encryption functions and in the encryption service.
There are wrappers to provide the old all-at-once encryption
functionality. But binlog events are often written piecewise,
they'll need the new api.
* remove unused (and not implemented) WRITE_NET type
* remove cast in my_b_write() macro. my_b_* macros are
function-like, casts are responsibility of the caller
* replace hackish _my_b_write(info,0,0) with the explicit
my_b_flush_io_cache() in my_b_write_byte()
* remove unused my_b_fill_cache()
* replace pbool -> my_bool
* make internal IO_CACHE functions static
* reformat comments, correct typos, remove obsolete comments (ISAM)
* assert valid cache type in init_functions()
* use IO_ROUND_DN() macro where appropriate
* remove unused DBUG_EXECUTE_IF in _my_b_cache_write()
* remove unnecessary __attribute__((unused))
* fix goto error in parse_file.cc
* remove redundant reinit_io_cache() in uniques.cc
* don't do reinit_io_cache() if the cache was not initialized
in ma_check.c
* extract duplicate functionality from various _my_b_*_read
functions into a common wrapper. Same for _my_b_*_write
* create _my_b_cache_write_r instead of having if's in
_my_b_cache_write (similar to existing _my_b_cache_read and
_my_b_cache_read_r)
* don't call mysql_file_write() from my_b_flush_io_cache(),
call info->write_function() instead
remove some 14-year old code that added support for
LOAD DATA replication to IO_CACHE:
* three callbacks, of which only two were actually used and that
were only needed for LOAD DATA replication but were
tested in every IO_CACHE instance
* an additional opaque void * argument in IO_CACHE, also only
used for LOAD DATA replication, but present everywhere
* the code to close IO_CACHE prematurely in LOAD DATA to have
these callbacks called in the correct order and a long
comment explaining what will happen if IO_CACHE is not
closed prematurely
* a variable to track whether IO_CACHE was closed prematurely
(to avoid double-closing it)
Clean up and improve the parallel implementation code, mainly related to
scheduling of work to threads and handling of stop and errors.
Fix a lot of bugs in various corner cases that could lead to crashes or
corruption.
Fix that a single replication domain could easily grab all worker threads and
stall all other domains; now a configuration variable
--slave-domain-parallel-threads allows to limit the number of
workers.
Allow next event group to start as soon as previous group begins the commit
phase (as opposed to when it ends it); this allows multiple event groups on
the slave to participate in group commit, even when no other opportunities for
parallelism are available.
Various fixes:
- Fix some races in the rpl.rpl_parallel test case.
- Fix an old incorrect assertion in Log_event iocache read.
- Fix repeated malloc/free of wait_for_commit and rpl_group_info objects.
- Simplify wait_for_commit wakeup logic.
- Fix one case in queue_for_group_commit() where killing one thread would
fail to correctly signal the error to the next, causing loss of the
transaction after slave restart.
- Fix leaking of pthreads (and their allocated stack) due to missing
PTHREAD_CREATE_DETACHED attribute.
- Fix how one batch of group-committed transactions wait for the previous
batch before starting to execute themselves. The old code had a very
complex scheduling where the first transaction was handled differently,
with subtle bugs in corner cases. Now each event group is always scheduled
for a new worker (in a round-robin fashion amongst available workers).
Keep a count of how many transactions have started to commit, and wait for
that counter to reach the appropriate value.
- Fix slave stop to wait for all workers to actually complete processing;
before, the wait was for update of last_committed_sub_id, which happens a
bit earlier, and could leave worker threads potentially accessing bits of
the replication state that is no longer valid after slave stop.
- Fix a couple of places where the test suite would kill a thread waiting
inside enter_cond() in connection with debug_sync; debug_sync + kill can
crash in rare cases due to a race with mysys_var_current_mutex in this
case.
- Fix some corner cases where we had enter_cond() but no exit_cond().
- Fix that we could get failure in wait_for_prior_commit() but forget to flag
the error with my_error().
- Fix slave stop (both for normal stop and stop due to error). Now, at stop
we pick a specific safe point (in terms of event groups executed) and make
sure that all event groups before that point are executed to completion,
and that no event group after start executing; this ensures a safe place to
restart replication, even for non-transactional stuff/DDL. In error stop,
make sure that all prior event groups are allowed to execute to completion,
and that any later event groups that have started are rolled back, if
possible. The old code could leave eg. T1 and T3 committed but T2 not, or
it could even leave half a transaction not rolled back in some random
worker, which would cause big problems when that worker was later reused
after slave restart.
- Fix the accounting of amount of events queued for one worker. Before, the
amount was reduced immediately as soon as the events were dequeued (which
happens all at once); this allowed twice the amount of events to be queued
in memory for each single worker, which is not what users would expect.
- Fix that an error set during execution of one event was sometimes not
cleared before executing the next, causing problems with the error
reporting.
- Fix incorrect handling of thd->killed in worker threads.
This is port of fix for MySQL BUG#17647863.
revno: 5572
revision-id: jon.hauglid@oracle.com-20131030232243-b0pw98oy72uka2sj
committer: Jon Olav Hauglid <jon.hauglid@oracle.com>
timestamp: Thu 2013-10-31 00:22:43 +0100
message:
Bug#17647863: MYSQL DOES NOT COMPILE ON OSX 10.9 GM
Rename test() macro to MY_TEST() to avoid conflict with libc++.
'MAX_BINLOG_CACHE_SIZE' ERROR
Problem:
=======
MySQL returns following error in win64.
"ERROR 1197 (HY000): Multi-statement transaction required more than
'max_binlog_cache_size' bytes of storage; increase this mysqld variable
and try again" when user tries to load >4G file even if
max_binlog_cache_size set to maximum value. On Linux everything
works fine.
Analysis:
========
The `max_binlog_cache_size' variable is of type `ulonglong'. This
value is set to `ULONGLONG_MAX' at the time of server start up. The
above value is stored in an intermediate variable named
`saved_max_binlog_cache_size' which is of type `ulong'. In visual
c++ complier the `ulong' type is of 4bytes in size and hence the value
is getting truncated to '4GB' and the cache is not able to grow beyond
4GB size. The same limitation is observed with
"max_binlog_stmt_cache_size" as well. Similar fix has been applied.
Fix:
===
As part of fix the type "ulong" is replaced with "my_off_t" which is of
type "ulonglong".
mysys/mf_iocache.c:
Added debug statement to simulate a scenario where the cache
file's current position is set to >4GB
sql/log.cc:
Replaced the type of `saved_max_binlog_cache_size' from "ulong" to
"my_off_t", which is a type def for "ulonglong".
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
Give more information when finding an error in a MyISAM table.
When killing system thread, use KILL_SYSTEM_THREAD instead of KILL_CONNECTION to make it easier to ignore the signal in sensitive context (like auto-repair)
Added new kill level: KILL_SERVER that will in the future to be used to signal killed by shutdown.
Add more warnings about killed connections when warning level > 3
include/myisamchk.h:
Added counting of printed info/notes
mysys/mf_iocache.c:
Remove duplicate assignment
sql/handler.cc:
Added test of KILL_SERVER
sql/log.cc:
Ignore new 'kill' error ER_NEW_ABORTING_CONNECTION when requesting query error code.
sql/mysqld.cc:
Add more warnings for killed connections when warning level > 3
sql/scheduler.cc:
Added checks for new kill signals
sql/slave.cc:
Ignore new kill signal ER_NEW_ABORTING_CONNECTION
sql/sp_head.cc:
Fixed assignment to bool
Added testing of new kill signals
sql/sql_base.cc:
Use KILL_SYSTEM_THREAD to auto-kill system threads
sql/sql_class.cc:
Add more warnings for killed connections when warning level > 3
thd_killed() now ignores KILL_BAD_DATA and THD::KILL_SYSTEM_THREAD as these should not abort sensitive operations.
sql/sql_class.h:
Added KILL_SYSTEM_THREAD and KILL_SERVER
sql/sql_connect.cc:
Added handling of KILL_SERVER
sql/sql_insert.cc:
Use KILL_SYSTEM_THREAD to auto-kill system threads
Added handling of KILL_SERVER
sql/sql_parse.cc:
Add more warnings for killed connections when warning level > 3
Added checking that thd->abort_on_warning is reset at end of query.
sql/sql_show.cc:
Update condition for when a query is 'killed'
storage/myisam/ha_myisam.cc:
Added counting of info/notes printed
storage/myisam/mi_check.c:
Always print an an error if we find data errors when checking/repairing a MyISAM table.
When a repair was killed, don't retry repair.
Added assert if sort_get_next_record() returned an error without an error message.
Removed nonsence check "if (sort_param->read_cache.error < 0)" in repair.
storage/myisam/myisamchk.c:
Added counting of notes printed
storage/pbxt/src/thread_xt.cc:
Better error message.
Added more printing of errors to myisamchk.
mysys/mf_iocache.c:
Write error message if failed seek.
sql/table.cc:
Fixed buffer overflow bug:
- It's not enough to check for mysql_version to to detect partion indicator as the version may have been updated by mysql_upgrade.
storage/myisam/ha_myisam.cc:
Don't log same error twice.
Don't reset log_all_errors if it's set
storage/myisam/mi_check.c:
Fixed bug that caused repair() to not report error if called twice (as when doing retry)
More printing of errors.
storage/myisam/sort.c:
Set my_errno in case of out of memory errors.
The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with
the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open
file is stored in the my_file_info struct, along with a flag for append mode
(because the Windows API does not support opening files in append mode in all cases)
The default max open files has been increased to 16384 and can be increased further
by setting --max-open-files=<value> during the server start.
Noteworthy benefit of this patch is that it removes limits from the table_cache size -
allowing for more simultaneus users
and 'THREAD_SAFE_CLIENT'.
As of MySQL 5.5, we no longer support non-threaded
builds. This patch removes all references to the
obsolete THREAD and THREAD_SAFE_CLIENT preprocessor
symbols. These were used to distinguish between
threaded and non-threaded builds.
Before this fix, file io for the binary log file was not accounted properly,
and showed no io at all.
This bug was due to the following issues:
1) file io for the binlog was instrumented:
- sometime as "wait/io/file/sql/binlog"
- sometime as "wait/io/file/sql/MYSQL_LOG"
leading to inconsistent event_names.
2) the binlog file itself was using an IO_CACHE,
but the IO_CACHE implementation in mysys/mf_iocache.c was
not instrumented to make performance schema calls to record file io.
3) The "wait/io/file/sql/MYSQL_LOG" instrumentation was used
for several log files, such as:
- the binary log
- the slow log
- the query log
which caused file io in these different log files to be accounted
against the same instrument.
The instrumentation needs to have a finer grain and report io
in different event_names, because each file really serves a different purpose.
With this fix:
- the IO_CACHE implementation is now instrumented
- the "wait/io/file/sql/MYSQL_LOG" instrument has been removed
- binlog io is now always instrumented with "wait/io/file/sql/binlog"
- the slow log is instrumented with a new name, "wait/io/file/sql/slow_log"
- the query log is instrumented with a new name, "wait/io/file/sql/query_log"
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
Fix warnings flagged by the new warning option -Wunused-but-set-variable
that was added to GCC 4.6 and that is enabled by -Wunused and -Wall. The
option causes a warning whenever a local variable is assigned to but is
later unused. It also warns about meaningless pointer dereferences.
client/mysql.cc:
Meaningless pointer dereferences.
client/mysql_upgrade.c:
Check whether reading from the file succeeded.
extra/comp_err.c:
Unused.
extra/yassl/src/yassl_imp.cpp:
Skip instead of reading data that is discarded.
include/my_pthread.h:
Variable is only used in debug builds.
include/mysys_err.h:
Add new error messages.
mysys/errors.c:
Add new error message for permission related functions.
mysys/mf_iocache.c:
Variable is only checked under THREAD.
mysys/my_copy.c:
Raise a error if chmod or chown fails.
mysys/my_redel.c:
Raise a error if chmod or chown fails.
regex/engine.c:
Use a equivalent variable for the assert.
server-tools/instance-manager/instance_options.cc:
Unused.
sql/field.cc:
Unused.
sql/item.cc:
Unused.
sql/log.cc:
Do not ignore the return value of freopen: only set buffer if
reopening succeeds.
Adjust doxygen comment to the right function.
Pass message lenght to log function.
sql/mysqld.cc:
Do not ignore the return value of freopen: only set buffer if
reopening succeeds.
sql/partition_info.cc:
Unused.
sql/slave.cc:
No need to set pointer to the address of '\0'.
sql/spatial.cc:
Unused. Left for historical purposes.
sql/sql_acl.cc:
Unused.
sql/sql_base.cc:
Pointers are always set to the same variables.
sql/sql_parse.cc:
End statement if reading fails.
Store the buffer after it has actually been updated.
sql/sql_repl.cc:
No need to set pointer to the address of '\0'.
sql/sql_show.cc:
Put variable under the same ifdef block.
sql/udf_example.c:
Set null pointer flag appropriately.
storage/csv/ha_tina.cc:
Meaningless dereferences.
storage/example/ha_example.cc:
Return the error since it's available.
storage/myisam/mi_locking.c:
Remove unused and dead code.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
decrease for INSERTs
Bulk inserts (multiple row, CREATE ... SELECT, INSERT ... SELECT) into
MyISAM tables were performed inefficiently. This was mainly affecting
use cases where read_buffer_size was considerably large (>256K) and low
number of rows was inserted (e.g. 30-100).
The problem was that during I/O cache initialization (this happens
before each bulk insert) allocated I/O buffer was unnecessarily
initialized to '\0'.
This was happening because of mess in flag values. MyISAM informs I/O
cache to wait for free space (if out of disk space) by passing
MY_WAIT_IF_FULL flag. Since MY_WAIT_IF_FULL and MY_ZEROFILL have the
same values, memory allocator was initializing memory to '\0'.
The performance gain provided with this patch may only be visible with
non-debug binaries, since safemalloc always initializes allocated memory
to 0xA5A5...
mysys/mf_iocache.c:
Remove MY_WAIT_IF_FULL from myflags before calling my_malloc
to prevent conflict with MY_ZEROFILL.
Fixed type that caused windows builds to fail
include/my_alloc.h:
Use size_t for memory areas instead of uint
mysys/mf_iocache.c:
Fixed compiler warnings by adding casts
mysys/my_compress.c:
Fixed compiler warnings by adding casts
mysys/my_getsystime.c:
Fixed typo
mysys/my_static.h:
Use size_t for memory areas
mysys/safemalloc.c:
Use size_t for memory areas
storage/maria/ma_bitmap.c:
Fixed compiler warnings by adding casts
storage/maria/ma_blockrec.c:
Fixed compiler warnings by adding casts
storage/maria/ma_cache.c:
Use size_t for memory areas
storage/maria/ma_info.c:
Use size_t for memory areas
storage/maria/ma_key.c:
Fixed compiler warnings by adding casts
storage/maria/ma_locking.c:
Fixed compiler warnings by adding casts
storage/maria/ma_open.c:
Fixed compiler warnings by adding casts
storage/maria/ma_packrec.c:
Fixed compiler warnings by fixing type for variable
storage/maria/ma_statrec.c:
Fixed compiler warnings by adding casts
storage/maria/ma_write.c:
Fixed compiler warnings by adding casts
storage/maria/maria_def.h:
Use size_t for memory areas
storage/myisam/mi_search.c:
Fixed compiler warnings by adding casts
Fixed bugs in undo logging
Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read)
Reserved place for reference-transid on key pages (for packing of transids)
ALTER TABLE and INSERT ... SELECT now uses fast creation of index
Known bugs:
ma_test_recovery fails because of a bug in redo handling when log is cut directly after a redo (Guilhem knows how to fix)
ma_test_recovery.excepted is not totally correct, because of the above bug
mysqld sometimes fails to restart; Fails with error "end_of_redo_phase: Assertion `long_trid != 0' failed"; Guilhem to investigate
include/maria.h:
Prototype changes
Added current_filepos to st_maria_sort_info
mysql-test/r/maria.result:
Updated results that changes as alter table and insert ... select now uses fast creation of index
mysys/mf_iocache.c:
Reset variable to gurard against double invocation
storage/maria/ma_bitmap.c:
Added _ma_bitmap_reset_cache() (needed for repair)
storage/maria/ma_blockrec.c:
Simplify code
More initial allocations
Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read)
storage/maria/ma_blockrec.h:
Moved TRANSID_SIZE to maria_def.h
Added prototype for new functions
storage/maria/ma_check.c:
Simplicy code
Fixed repair_by_sort to work with BLOCK_RECORD
- When using BLOCK_RECORD or UNPACK create new Maria handle
- Use common initializer function
- Align code with maria_repair()
Made some changes to maria_repair_parallel() to use common initializer function
Removed ASK_MONTY section by fixing noted problem
storage/maria/ma_close.c:
Moved check for readonly to _ma_state_info_write()
storage/maria/ma_key_recover.c:
Use different log entries if key root changes or not.
This fixed some bugs when tree grows
storage/maria/ma_key_recover.h:
Added keynr to st_msg_to_write_hook_for_undo_key
storage/maria/ma_loghandler.c:
Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT
storage/maria/ma_loghandler.h:
Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT
storage/maria/ma_open.c:
Added TRANSID to all key pages (for future compressing of trans id's)
For compressed records, alloc a bit bigger buffer to avoid valgrind warnings
If table is opened readonly, don't update state
storage/maria/ma_packrec.c:
Allocate bigger array for bit unpacking to avoid valgrind errors
storage/maria/ma_recovery.c:
Added UNDO_KEY_INSERT_WITH_ROOT & UNDO_KEY_DELETE_WITH_ROOT
storage/maria/ma_sort.c:
More logging
storage/maria/ma_test_all.sh:
More tests
storage/maria/ma_test_recovery.expected:
Update results
Note that this is not complete becasue of a bug in recovery
storage/maria/ma_test_recovery:
Removed recreation of index (not needed when we have redo for index pages)
storage/maria/maria_chk.c:
When using flag --read-only, don't update status for files
When using --unpack, don't use REPAIR_BY_SORT if other repair option is given
Enable repair_by_sort for BLOCK records
Removed not needed newline at start of --describe
storage/maria/maria_def.h:
Support for TRANSID_SIZE to key pages
storage/maria/maria_read_log.c:
renamed --only-display to --display-only
