Adds deprecation warning for the mysqlbinlog options
--base64-output=always and --base64-output.
A warning is printed when the flags are used,
and also when running mysqlbinlog --help.
The fix is to:
- introduce ORACLE_WELCOME_COPYRIGHT_NOTICE define to have a single place
to specify copyright notice;
- replace custom copyright notices with ORACLE_WELCOME_COPYRIGHT_NOTICE
in programs.
Although the C standard mandates that sprintf return the number
of bytes written, some very ancient systems (i.e. SunOS 4)
returned a pointer to the buffer instead. Since these systems
are not supported anymore and are hopefully long dead by now,
simply remove the portability wrapper that dealt with this
discrepancy. The autoconf check was causing trouble with GCC.
The server was not cleaning up some dbug allocated memory
before exiting. This is not a real problem, as this memory
would be deallocated anyway. Nonetheless, we improve the
mysqlbinlog exit procedure, wrt to memory book-keeping, when
no parameter is given.
To fix this, we deploy a call to my_end() before the
thread exits.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
The server was not cleaning up dbug allocated memory before
exiting. This is not a real problem, as this memory would be
deallocated anyway. Nonetheless, we improve the mysqlbinlog exit
procedure, wrt to memory book-keeping, when no parameter is
given.
To fix this, we deploy a call to my_thread_end() before the
thread exits, which will also free pending dbug related allocated
blocks.
strict aliasing violations.
Essentially, the problem is that large parts of the server were
developed in simpler times (last decades, pre C99 standard) when
strict aliasing and compilers supporting such optimizations were
rare to non-existent. Thus, when compiling the server with a modern
compiler that uses strict aliasing rules to perform optimizations,
there are several places in the code that might trigger undefined
behavior.
As evinced by some recent bugs, GCC does a somewhat good of job
misoptimizing such code, but on the other hand also gives warnings
about suspicious code. One problem is that the warnings aren't
always accurate, yet we can't afford to just shut them off as we
might miss real cases. False-positive cases are aggravated mostly
by casts that are likely to trigger undefined behavior.
The solution is to start a cleanup process focused on fixing and
reducing the amount of strict-aliasing related warnings produced
by GCC and others compilers. A good deal of noise reduction can
be achieved by just removing useless casts that are product of
historical cruft and are likely to trigger undefined behavior if
dereferenced.
Conflicts:
Text conflict in mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result
Text conflict in sql/log.cc
Text conflict in sql/set_var.cc
Text conflict in sql/sql_class.cc
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
When mysqlbinlog was given the --database=X flag, it always printed
'ROLLBACK TO', but the corresponding 'SAVEPOINT' statement was not
printed. The replicated filter(replicated-do/ignore-db) and binlog
filter (binlog-do/ignore-db) has the same problem. They are solved
in this patch together.
After this patch, We always check whether the query is 'SAVEPOINT'
statement or not. Because this is a literal check, 'SAVEPOINT' and
'ROLLBACK TO' statements are also binlogged in uppercase with no
any comments.
The binlog before this patch can be handled correctly except one case
that any comments are in front of the keywords. for example:
/* bla bla */ SAVEPOINT a;
/* bla bla */ ROLLBACK TO a;
I found three issues during the analysis:
1. Memory leak caused by temp_buf not being freed;
2. Memory leak caused when handling argv;
3. Conditional jump that depended on unitialized values.
Issue #1
--------
DESCRIPTION: when mysqlbinlog is reading from a remote location
the event temp_buf references the incoming stream (in NET
object), which is not freed by mysqlbinlog explicitly. On the
other hand, when it is reading local binary log, it points to a
temporary buffer that needs to be explicitly freed. For both
cases, the temp_buf was not freed by mysqlbinlog, instead was
set to 0. This clearly disregards the free required in the
second case, thence creating a memory leak.
FIX: we make temp_buf to be conditionally freed depending on
the value of remote_opt. Found out that similar fix is already
in most recent codebases.
Issue #2
--------
DESCRIPTION: load_defaults is called by parse_args, and it
reads default options from configuration files and put them
BEFORE the arguments that are already in argc and argv. This is
done resorting to MEM_ROOT. However, parse_args calls
handle_options immediately after which changes argv. Later when
freeing the defaults, pointers to MEM_ROOT won't match, causing
the memory not to be freed:
void free_defaults(char **argv)
{
MEM_ROOT ptr
memcpy_fixed((char*) &ptr,(char *) argv - sizeof(ptr), sizeof(ptr));
free_root(&ptr,MYF(0));
}
FIX: we remove load_defaults from parse_args and call it
before. Then we save argv with defaults in defaults_argv BEFORE
calling parse_args (which inside can then call handle_options
at will). Actually, found out that this is in fact kind of a
backport for BUG#38468 into 5.1, so I merged in the test case
as well and added error check for load_defaults call.
Fix based on:
revid:zhenxing.he@sun.com-20091002081840-uv26f0flw4uvo33y
Issue #3
--------
DESCRIPTION: the structure st_print_event_info constructor
would not initialize the sql_mode member, although it did for
sql_mode_inited (set to false). This would later raise the
warning in valgrind when printing the sql_mode in the event
header, as this print out is protected by a check against
sql_mode_inited and sql_mode variables. Given that sql_mode was
not initialized valgrind would output the warning.
FIX: we add initialization of sql_mode to the
st_print_event_info constructor.
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/suite/rpl/r/rpl_slow_query_log.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Conflict adding files to server-tools. Created directory.
Conflict because server-tools is not versioned, but has versioned children. Versioned directory.
Conflict adding files to server-tools/instance-manager. Created directory.
Conflict because server-tools/instance-manager is not versioned, but has versioned children. Versioned directory.
Contents conflict in server-tools/instance-manager/options.cc
Text conflict in sql/mysqld.cc
Several items said to be deprecated in the 4.1 manual
have never been removed. This worklog adds deprecation
warnings when these items are used, and warns the user
that the items will be removed in MySQL 5.6.
A couple of previously deprecation decision have been
reversed (see single file comments)
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/extra/rpl_tests/rpl_loaddata.test
Text conflict in mysql-test/r/mysqlbinlog2.result
Text conflict in mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
Text conflict in mysql-test/suite/binlog/r/binlog_unsafe.result
Text conflict in mysql-test/suite/rpl/r/rpl_insert_id.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result
Text conflict in mysql-test/suite/rpl/r/rpl_udf.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Text conflict in sql/field.h
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysql_priv.h
Text conflict in sql/share/errmsg.txt
Text conflict in sql/sp.cc
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_base.cc
Text conflict in sql/sql_class.h
Text conflict in sql/sql_db.cc
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_insert.cc
Text conflict in sql/sql_lex.cc
Text conflict in sql/sql_lex.h
Text conflict in sql/sql_load.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_update.cc
Text conflict in sql/sql_view.cc
Conflict adding files to storage/innobase. Created directory.
Conflict because storage/innobase is not versioned, but has versioned children. Versioned directory.
Conflict adding file storage/innobase. Moved existing file to storage/innobase.moved.
Conflict adding files to storage/innobase/handler. Created directory.
Conflict because storage/innobase/handler is not versioned, but has versioned children. Versioned directory.
Contents conflict in storage/innobase/handler/ha_innodb.cc
Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.
This patch implement type conversions between similar fields
on the master and slave.
The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').
Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.
If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.
Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.
Text conflict in mysql-test/collections/default.experimental
Text conflict in mysql-test/r/show_check.result
Text conflict in mysql-test/r/sp-code.result
Text conflict in mysql-test/suite/binlog/r/binlog_tmp_table.result
Text conflict in mysql-test/suite/rpl/t/disabled.def
Text conflict in mysql-test/t/show_check.test
Text conflict in mysys/my_delete.c
Text conflict in sql/item.h
Text conflict in sql/item_cmpfunc.h
Text conflict in sql/log.cc
Text conflict in sql/mysqld.cc
Text conflict in sql/repl_failsafe.cc
Text conflict in sql/slave.cc
Text conflict in sql/sql_parse.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_yacc.yy
Text conflict in storage/myisam/ha_myisam.cc
Corrected results for
stm_auto_increment_bug33029.reject 2009-12-01
20:01:49.000000000 +0300
<andrei> @@ -42,9 +42,6 @@
<andrei> RETURN i;
<andrei> END//
<andrei> CALL p1();
<andrei> -Warnings:
<andrei> -Note 1592 Statement may not be safe to log in statement
format.
<andrei> -Note 1592 Statement may not be safe to log in statement
format.
There should be indeed no Note present because there is in fact autoincrement
top-level query in sp() that triggers inserting in yet another auto-inc table.
(todo: alert DaoGang to improve the test).
The mentioned on the bug report set of bugs fixes have not be pushed to the main trees.
Fixed with extracting commits done to 6.0-rpl tree and applying them to the main 5.1.
Notes.
1. part of changes - the mtr's specific - were packported to the main 5.0 tree for mtr v1
as http://lists.mysql.com/commits/46562
However, there is no that fix anymore in the mtr v2. (This fact was mailed to mtr maintaining
people).
2. Bug@36929 crash in kill_zombie_dump_threads-> THD::awake() with replication tests
is not backported because the base code of the patch is libevent and that was removed
from the main trees due to its instability.
The reason for the bug is that mysqtest as well as other client tools
running in test suite (mysqlbinlog, mysqldump) will first try to connect
whatever database has created shared memory with default base name
"MySQL" and use this. (Same effect could be seen on Unix if mtr would
not care to calculate "port" and "socket" parameter).
The fix ensures that all client tools and running in mtr use unique
per-database shared memory base parameters, so there is no possibility
to clash with already installed one. We use socket name for shared memory
base (it's known to be unique). This shared-memory-base is written to the
MTR config file to the [client] and [mysqld] sections. Fix made also made
sure all client tools understand and correctly handle --shared-memory-base.
Prior to this patch it was not the case for mysqltest, mysqlbinlog and
mysql_client_test.
All new connections done from mtr scripts via connect() will by default
set shared-memory-base. And finally, there is a possibility to force
shared memory or pipe connection and overwrite shared memory/pipe base name
from within mtr scripts via optional PIPE or SHM modifier. This functionality
was manually backported from 6.0
(original patch http://lists.mysql.com/commits/74749)
Conflicts
=========
Text conflict in .bzr-mysql/default.conf
Text conflict in libmysqld/CMakeLists.txt
Text conflict in libmysqld/Makefile.am
Text conflict in mysql-test/collections/default.experimental
Text conflict in mysql-test/extra/rpl_tests/rpl_row_sp006.test
Text conflict in mysql-test/suite/binlog/r/binlog_tmp_table.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata_fatal.result
Text conflict in mysql-test/suite/rpl/r/rpl_row_create_table.result
Text conflict in mysql-test/suite/rpl/r/rpl_row_sp006_InnoDB.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_log.result
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_circular_simplex.result
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_sp006.result
Text conflict in mysql-test/t/mysqlbinlog.test
Text conflict in sql/CMakeLists.txt
Text conflict in sql/Makefile.am
Text conflict in sql/log_event_old.cc
Text conflict in sql/rpl_rli.cc
Text conflict in sql/slave.cc
Text conflict in sql/sql_binlog.cc
Text conflict in sql/sql_lex.h
21 conflicts encountered.
NOTE
====
mysql-5.1-rpl-merge has been made a mirror of mysql-next-mr:
- "mysql-5.1-rpl-merge$ bzr pull ../mysql-next-mr"
This is the first cset (merge/...) committed after pulling
from mysql-next-mr.
There are some problems about help text:
- It is stated that "auto" is the default twice. It need be stated only once.
- It is stated that --base64-output is short for --base64-output=always. But that sounds
like the default is "always", not "auto".
Make the help text clear as following:
Determine when the output statements should be base64-encoded BINLOG
statements: 'never' disables it and works only for binlogs without
row-based events; 'auto' prints base64 only when necessary (i.e.,
for row-based events and format description events); 'always' prints
base64 whenever possible. 'always' is for debugging only and should
not be used in a production system. If this argument is not given,
the default is 'auto'; if it is given with no argument, 'always' is used.
added code to exit a tool if the forced config file wasn't found
per-file comments:
client/mysql.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysql_upgrade.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqladmin.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlcheck.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqldump.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlimport.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlshow.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlslap.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
mysql-test/t/mysql.test
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
test added
sql/mysqld.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
storage/myisam/myisamchk.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
storage/myisam/myisampack.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
There were two memory leaks in mysqlbinlog command, one was already
fixed by previous patches, another one was that defaults_argv was
set to the value of argv after parse_args, in which called
handle_options after calling load_defaults and changed the value
of argv, and caused the memory allocated for defaults arguments
not freed.
Fixed the problem by setting defaults_argv right after calling
load_defaults.
The 'BEGIN/COMMIT/ROLLBACK' log event could be filtered out if the
database is not selected by --database option of mysqlbinlog command.
This can result in problem if there are some statements in the
transaction are not filtered out.
To fix the problem, mysqlbinlog will output 'BEGIN/ROLLBACK/COMMIT'
in regardless of the database filtering rules.
The 'BEGIN/COMMIT/ROLLBACK' log event could be filtered out if the
database is not selected by --database option of mysqlbinlog command.
This can result in problem if there are some statements in the
transaction are not filtered out.
To fix the problem, mysqlbinlog will output 'BEGIN/ROLLBACK/COMMIT'
in regardless of the database filtering rules.
Bug#24509 - 2048 file descriptor limit on windows needs increasing, also
WL#3049 - improved Windows I/O
The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with
the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open
file is stored in the my_file_info struct, along with a flag for append mode
because the Windows API does not support opening files in append mode in all cases)
The default max open files has been increased to 16384 and can be increased further
by setting --max-open-files=<value> during the server start.
Another major change in this patch that almost all Windows specific file IO code
has been moved to a new file my_winfile.c, greatly reducing the amount of code
in #ifdef blocks within mysys, thus improving readability.
Minor enhancements:
- my_(f)stat() is changed to use __stati64 structure with 64 file size
and timestamps. It will return correct file size now (C runtime implementation
used to report outdated information)
- my_lock on Windows is prepared to handle additional timeout parameter
- after review : changed __WIN__ to _WIN32 in the new and changed code.
