Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
strict aliasing violations.
Essentially, the problem is that large parts of the server were
developed in simpler times (last decades, pre C99 standard) when
strict aliasing and compilers supporting such optimizations were
rare to non-existent. Thus, when compiling the server with a modern
compiler that uses strict aliasing rules to perform optimizations,
there are several places in the code that might trigger undefined
behavior.
As evinced by some recent bugs, GCC does a somewhat good of job
misoptimizing such code, but on the other hand also gives warnings
about suspicious code. One problem is that the warnings aren't
always accurate, yet we can't afford to just shut them off as we
might miss real cases. False-positive cases are aggravated mostly
by casts that are likely to trigger undefined behavior.
The solution is to start a cleanup process focused on fixing and
reducing the amount of strict-aliasing related warnings produced
by GCC and others compilers. A good deal of noise reduction can
be achieved by just removing useless casts that are product of
historical cruft and are likely to trigger undefined behavior if
dereferenced.
client/mysql.cc:
Remove now-unnecessary casts.
Break up large strings.
client/mysql_upgrade.c:
Remove now-unnecessary casts.
client/mysqladmin.cc:
Remove now-unnecessary casts.
Break up large strings.
client/mysqlbinlog.cc:
Remove now-unnecessary casts.
client/mysqlcheck.c:
Remove now-unnecessary casts.
client/mysqldump.c:
Remove now-unnecessary casts.
client/mysqlimport.c:
Remove now-unnecessary casts.
client/mysqlshow.c:
Remove now-unnecessary casts.
client/mysqlslap.c:
Remove now-unnecessary casts.
client/mysqltest.cc:
Remove now-unnecessary casts.
extra/comp_err.c:
Remove now-unnecessary casts.
extra/my_print_defaults.c:
Remove now-unnecessary casts.
Break up large strings.
extra/mysql_waitpid.c:
Remove now-unnecessary casts.
extra/perror.c:
Remove now-unnecessary casts.
extra/resolve_stack_dump.c:
Remove now-unnecessary casts.
extra/resolveip.c:
Remove now-unnecessary casts.
include/my_getopt.h:
Use a void pointer type as the opaque type to avoid problems with type
incompatibility -- GCC issues warnings when the type name is not type
compatible with a operand. As a side bonus, a explicit cast won't be
necessary anymore.
include/sslopt-longopts.h:
Remove now-unnecessary casts.
Break up large strings.
mysys/my_getopt.c:
Update opaque type and introduce a type definition for the
argument to my_getopt_register_get_addr.
server-tools/instance-manager/options.cc:
Remove now-unnecessary casts.
sql/mysqld.cc:
Remove now-unnecessary casts.
Break up large strings.
Update mysql_getopt_value prototype (the old prototype
was different from the definition anyway).
sql/sql_plugin.cc:
The type of a pointer to a function must be compatible with the
pointed-to function type, otherwise the behavior is undefined.
sql/table.cc:
The variable buf pointer to pointer to pointer to constant char
could improperly alias a incompatible type in call to fix_type_
pointers. Since this was actually dead code, it is simply removed.
sql/unireg.cc:
Remove call to get_form_pos. The code creates a new FRM file which
is always truncated and writes the form position as 0. Hence, no
need to retrieve it, we now for sure it is 0.
storage/archive/archive_reader.c:
Remove now-unnecessary casts.
storage/myisam/ft_nlq_search.c:
Read weight directly from the buffer.
storage/myisam/fulltext.h:
Add explanation about the type duality of a key buffer.
Add accessor macro to retrieve a FT float value.
storage/myisam/mi_test1.c:
Remove now-unnecessary casts.
storage/myisam/myisam_ftdump.c:
Read weight directly from the buffer.
storage/myisam/myisamchk.c:
Remove now-unnecessary casts.
storage/myisam/myisamlog.c:
A pointer to char was used to alias a pointer to pointer to
unsigned char, thus violating strict aliasing rules.
storage/myisam/myisampack.c:
Remove now-unnecessary casts.
strings/decimal.c:
Remove aliasing violation, printing the value is enough for
debugging purposes.
tests/mysql_client_test.c:
Remove now-unnecessary casts.
strict aliasing violations.
Essentially, the problem is that large parts of the server were
developed in simpler times (last decades, pre C99 standard) when
strict aliasing and compilers supporting such optimizations were
rare to non-existent. Thus, when compiling the server with a modern
compiler that uses strict aliasing rules to perform optimizations,
there are several places in the code that might trigger undefined
behavior.
As evinced by some recent bugs, GCC does a somewhat good of job
misoptimizing such code, but on the other hand also gives warnings
about suspicious code. One problem is that the warnings aren't
always accurate, yet we can't afford to just shut them off as we
might miss real cases. False-positive cases are aggravated mostly
by casts that are likely to trigger undefined behavior.
The solution is to start a cleanup process focused on fixing and
reducing the amount of strict-aliasing related warnings produced
by GCC and others compilers. A good deal of noise reduction can
be achieved by just removing useless casts that are product of
historical cruft and are likely to trigger undefined behavior if
dereferenced.
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/suite/rpl/r/rpl_slow_query_log.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Conflict adding files to server-tools. Created directory.
Conflict because server-tools is not versioned, but has versioned children. Versioned directory.
Conflict adding files to server-tools/instance-manager. Created directory.
Conflict because server-tools/instance-manager is not versioned, but has versioned children. Versioned directory.
Contents conflict in server-tools/instance-manager/options.cc
Text conflict in sql/mysqld.cc
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/suite/rpl/r/rpl_slow_query_log.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Conflict adding files to server-tools. Created directory.
Conflict because server-tools is not versioned, but has versioned children. Versioned directory.
Conflict adding files to server-tools/instance-manager. Created directory.
Conflict because server-tools/instance-manager is not versioned, but has versioned children. Versioned directory.
Contents conflict in server-tools/instance-manager/options.cc
Text conflict in sql/mysqld.cc
Fix for Bug#31173: mysqlslap.exe crashes if called without any parameters
.bzrignore:
Fixed .bzrignore rules. Many were simply not ignoring what they were meant to.
client/mysqlslap.c:
Fixed bug for Bug#31173: mysqlslap.exe crashes if called without any parameters
The original patch could cause memory leaks and odd problems depending on how connection was made.
This code ensures that all mysql_options() are set for each mysql_real_connect().
(This patch by Monty)
mysys/my_thr_init.c:
Fixed multiply-initialized critical section on Windows, due to code incorrectly
checking the wrong field in an attempt to prevent multiple-initialization.
sql-common/client.c:
Don't use shared memory if it's not set (for example after failed mysql_real_connect).
Ensure that mysql_close() resets all resources so that it's safe to call it twice.
(Patch by monty, related to Bug#31173: mysqlslap.exe crashes if called without any parameters)
sql/CMakeLists.txt:
Added page fault counters for SHOW PROFILE on Windows.
sql/mysqld.cc:
Fixed attempt to set a NULL event. The code now only sets the event if appropriate (i.e. shared memory is being used)
sql/sql_profile.cc:
Added page fault counters for SHOW PROFILE on Windows.
sql/sql_profile.h:
Added page fault counters for SHOW PROFILE on Windows.
sql/udf_example.def:
Some cleanup functions were not exported from udf_example.dll, causing them to
never be executed, and as a result multiple-initialization of kernel objects
occurred and resources were not being freed correctly.
storage/maria/ma_close.c:
Condition variable share->key_del_cond was never being destroyed, while its
containing heap block was being freed in maria_close(), leaking kernel
resources.
Author: Antony Curtis
License: BSD
client/mysql.cc:
Reset variable if CTRL-C was used to kill running query, so that the user can do it again
client/mysqlcheck.c:
Indentation fix
client/mysqlslap.c:
Indentation fixes
client/mysqltest.cc:
Make testing of commands safer by also testing length
Removed not used variable
Fixed indentation to be as it was before last patch
mysql-test/lib/mtr_report.pm:
Fixed typo
mysql-test/mysql-test-run.pl:
Merge (Align code with default mysql-tes-run.pl)
mysql-test/suite/federated/disabled.def:
Removed test case
storage/federatedx/ha_federatedx.cc:
Removed my changes and applied Antony's instead
----------------------------------------------------------------------
ChangeSet@1.2571, 2008-04-08 12:30:06+02:00, vvaintroub@wva. +122 -0
Bug#32082 : definition of VOID in my_global.h conflicts with Windows
SDK headers
VOID macro is now removed. Its usage is replaced with void cast.
In some cases, where cast does not make much sense (pthread_*, printf,
hash_delete, my_seek), cast is ommited.
- Moved some code from innodb_plugin to xtradb, to ensure that all tests runs
- Did changes in pbxt and maria storage engines becasue of changes in thd->query
- Reverted wrong code in sql_table.cc for how ROW_FORMAT is used.
This is a re-commit of Monty's merge to eliminate an extra commit from
MySQL-5.1.42 that was accidentally included in the merge.
This is a merge of the MySQL 5.1.41 clone-off (clone-5.1.41-build). In
case there are any extra changes done before final MySQL 5.1.41
release, these will need to be merged later before MariaDB 5.1.41
release.
- Moved some code from innodb_plugin to xtradb, to ensure that all tests runs
- Did changes in pbxt and maria storage engines becasue of changes in thd->query
- Reverted wrong code in sql_table.cc for how ROW_FORMAT is used.
Todo before joining with main 5.1 tree:
- Join test fails (Igor to investigate)
- mysql-test-run shows warnings from tests; Some suppression rule is not working (Kristian to investiage)
- Run through all buildbots
sql/sql_table.cc:
Reverted code for ROW_FORMAT is used. We must set the HA_CREATE_USED_ROW_FORMAT flag in alter table
to signal the handler that it should not change row_type in update_create_info() (as happens for SHOW CREATE).
storage/maria/ha_maria.cc:
Update for change in defintion of thd->query
storage/myisam/mi_check.c:
Simplify code
storage/pbxt/src/discover_xt.cc:
Update for change in defintion of thd->query
storage/xtradb/dict/dict0dict.c:
Update for change in defintion of thd->query
storage/xtradb/handler/ha_innodb.cc:
Copy some critical changes from innodb_plugin to get tests to pass
storage/xtradb/handler/ha_innodb.h:
Copy some critical changes from innodb_plugin to get tests to pass
storage/xtradb/handler/handler0alter.cc:
Copy some critical changes from innodb_plugin to get tests to pass
- Added checking of return value for system(), freopen(), fgets() and chown()
- Ensure that calls that require a format strings gets a format string
- Other trivial things
Updated test suite results (especially for pbxt and embedded server)
Removed warning for "Invalid (old?) table or database name 'mysqld.1'" from pbxt tests
Speed up some pbxt tests by inserting begin ; commit; around "while loops with inserts"
Added mysqld startup option '--debug-flush'
Create maria_recovery.trace in data directory instead of current directory
client/mysql.cc:
Check return value from system()
client/mysql_upgrade.c:
Check return value from fgets()
client/mysqladmin.cc:
Check return value from fgets()
client/mysqlslap.c:
Check return value from system() (but ignore it, as it's not critical)
extra/yassl/src/crypto_wrapper.cpp:
Check return value from fgets() (but ignore it, as it's internal file)
extra/yassl/taocrypt/src/aes.cpp:
Added extra {} to remove compiler warning
extra/yassl/taocrypt/src/blowfish.cpp:
Added extra {} to remove compiler warning
extra/yassl/taocrypt/src/misc.cpp:
Ifdef not used code
include/mysys_err.h:
Added error message for failing chown()
mysql-test/mysql-test-run.pl:
Don't give warning for skipping ndbcluster (never enabled in MariaDB)
mysql-test/suite/funcs_1/r/is_columns_is_embedded.result:
Update with new information schema information
mysql-test/suite/funcs_1/r/is_tables_is_embedded.result:
New test
mysql-test/suite/funcs_1/r/is_tables_myisam_embedded.result:
Update test results (has not been tested for a long time)
mysql-test/suite/funcs_1/r/is_tables_mysql_embedded.result:
Update test results (has not been tested for a long time)
mysql-test/suite/funcs_1/t/is_tables_is.test:
Don't run with embedded server (as results differ)
I added a new test for embedded server
mysql-test/suite/funcs_1/t/is_tables_is_embedded.test:
New test
mysql-test/suite/pbxt/my.cnf:
Allow one to run pbxt tests without having to specify --mysqld=--default-storage-engine=pbxt
mysql-test/suite/pbxt/t/count_distinct3.test:
Speed up test by inserting begin; ... commit;
mysql-test/suite/pbxt/t/subselect.test:
Speed up test by inserting begin; ... commit;
mysys/errors.c:
Added error message for failing chown()
mysys/my_copy.c:
Added error message for failing chown()
mysys/my_redel.c:
Added error message for failing chown()
mysys/safemalloc.c:
Added cast to get rid of compiler warning
sql/ha_partition.cc:
Fixed wrong argument to sql_print_error() (it requires a format string)
sql/log.cc:
Test return value of freopen()
sql/mysqld.cc:
Test return value of freopen()
Added startup option '--debug-flush' to be used when one gets a core dump (easy to explain to people on IRC)
sql/rpl_rli.cc:
Fixed wrong argument to sql_print_error() (it requires a format string)
sql/set_var.cc:
Added {} to get rid of compiler warnings
sql/slave.cc:
Fixed wrong argument to mi->report() and sql_print...() (they require a format string)
sql/sql_cache.cc:
Fixed wrong argument to sql_printinformation() (it requires a format string)
sql/sql_parse.cc:
Test return value of fgets()
sql/sql_plugin.cc:
Fixed wrong argument to sql_print_error() (it requires a format string)
sql/sql_select.cc:
Use unique table name for internal temp tables instead of full path
(Simple speed & space optimization)
sql/udf_example.c:
Removed compiler warning about not used variable
storage/maria/ha_maria.cc:
Fixed wrong argument to sql_print_error() and ma_check_print_error() (they require a format string)
storage/maria/ma_recovery.c:
Create maria_recovery.trace in data directory instead of current directory
storage/maria/unittest/ma_test_loghandler-t.c:
Fixed wrong argument to ok(); Requires a format string
storage/pbxt/src/strutil_xt.cc:
Detect temporary tables by checking if that path for the table is in the mysql data directory.
The database for temporary tables is after this patch, from PBXT point of view, ""
This is needed to stop PBXT from calling filename_to_tablename() with the base directory as an argument, which caused
ERROR: Invalid (old?) table or database name 'mysqld.1'" in the log when running the test suite.
tests/mysql_client_test.c:
Fixed compiler warnings
unittest/mysys/base64-t.c:
Fixed wrong argument to diag() (it requires a format string)
Added a comment that the current 'print' of differing buffers doesn't print the right thing,
but didn't fix this as it's not important (unless we find a bug in the real code)
added code to exit a tool if the forced config file wasn't found
per-file comments:
client/mysql.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysql_upgrade.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqladmin.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlcheck.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqldump.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlimport.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlshow.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlslap.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
mysql-test/t/mysql.test
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
test added
sql/mysqld.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
storage/myisam/myisamchk.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
storage/myisam/myisampack.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
added code to exit a tool if the forced config file wasn't found
per-file comments:
client/mysql.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysql_upgrade.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqladmin.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlcheck.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqldump.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlimport.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlshow.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
client/mysqlslap.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
mysql-test/t/mysql.test
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
test added
sql/mysqld.cc
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
storage/myisam/myisamchk.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
storage/myisam/myisampack.c
Bug#47216 programs should quit if the file specified by --defaults-file option isn't found
added code to exit a tool if the forced config file wasn't found
mysqlslap: setting --engine does not get replicated
http://bugs.mysql.com/bug.php?id=46967
and
mysqlslap: specifying --engine and --create does not
work with --engine=<storage_engine>:<option>
https://bugs.launchpad.net/maria/+bug/429773
Problems were that an --engine=<storage_engine> was translated
to a "set storage_engine = <storage_engine>", wich is _not_
replicated. A --engine=<storage_engine>:<option> was not always
properly parsed and in some cases crashed.
Fixed by eliminating "set storage_engine = ..." and adding
proper DDL generation. Initialized an unitialized buffer to
prevent crashes and fixed to use proper pointer for in
case of --engine=<storage_engine>:<option> being the last
element in list of --engines.
Also cleaned up code for better readability.
Q: Should MySQL's replication actually replicate a
"set storage_engine = ..." command or not?
A: No it should not. It is documented here:
http://dev.mysql.com/doc/refman/5.1/en/replication-features-variables.html
...
"The storage_engine system variable is not replicated, which is a
good thing for replication between different storage engines." ...
Before the patch, mysqlslap was behaving this way:
+-------------------------------+--------+-------------+
| | single | replication |
+-------------------------------+--------+-------------+
| Before patch |
+-------------------------------+--------+-------------+
| --engine[1] |
+-----+-------------------------+--------+-------------+
| 1.1 | eng1 | OK | Not OK |
| 1.2 | eng1,eng2 | OK | Not OK |
| 1.3 | eng1,eng2,eng3 | OK | Not OK |
| 1.4 | memory:option | OK | Not OK |
| 1.5 | memory:option,eng1 | OK | Not OK |
| 1.6 | eng1,memory:option | Not OK | Not OK |
| 1.7 | memory:option,eng1,eng2 | Crash | Not OK |
| 1.8 | eng1,memory:option,eng2 | OK | Not OK |
| 1.9 | eng1,eng2,memory:option | Not OK | Not OK |
+-----+-------------------------+--------+-------------+
+-------------------------------+--------+-------------+
| --create --engine[2] |
+-----+-------------------------+--------+-------------+
| 2.1 | eng1 | OK | Not OK |
| 2.2 | eng1,eng2 | OK | Not OK |
| 2.3 | eng1,eng2,eng3 | OK | Not OK |
| 2.4 | memory:option | Not OK | Not OK |
| 2.5 | memory:option,eng1 | Not OK | Not OK |
| 2.6 | eng1,memory:option | Not OK | Not OK |
| 2.7 | memory:option,eng1,eng2 | Crash | Not OK |
| 2.8 | eng1,memory:option,eng2 | Not OK | Not OK |
| 2.9 | eng1,eng2,memory:option | Not OK | Not OK |
+-----+-------------------------+--------+-------------+
After my final patch, mysqlslap now runs like this:
+-------------------------------+--------+-------------+
| | single | replication |
+-------------------------------+--------+-------------+
| After third patch |
+-------------------------------+--------+-------------+
| --engine[1] |
+-----+-------------------------+--------+-------------+
| 1.1 | eng1 | OK | OK |
| 1.2 | eng1,eng2 | OK | OK |
| 1.3 | eng1,eng2,eng3 | OK | OK |
| 1.4 | memory:option | OK | OK |
| 1.5 | memory:option,eng1 | OK | OK |
| 1.6 | eng1,memory:option | OK | OK |
| 1.7 | memory:option,eng1,eng2 | OK | OK |
| 1.8 | eng1,memory:option,eng2 | OK | OK |
| 1.9 | eng1,eng2,memory:option | OK | OK |
+-----+-------------------------+--------+-------------+
+-------------------------------+--------+-------------+
| --create --engine[2] |
+-----+-------------------------+--------+-------------+
| 2.1 | eng1 | OK | OK |
| 2.2 | eng1,eng2 | OK | OK |
| 2.3 | eng1,eng2,eng3 | OK | OK |
| 2.4 | memory:option | OK | OK |
| 2.5 | memory:option,eng1 | OK | OK |
| 2.6 | eng1,memory:option | OK | OK |
| 2.7 | memory:option,eng1,eng2 | OK | OK |
| 2.8 | eng1,memory:option,eng2 | OK | OK |
| 2.9 | eng1,eng2,memory:option | OK | OK |
+-----+-------------------------+--------+-------------+
Option "--without-server" still not working in 5.1
The general approach is to make sure that source files
which require thread support are only compiled if the build
really needs thread support,
which means when the server is built or a thread-safe client
library.
This required several changes:
- Make sure the subdirectories "storage/" and "plugin/" are
only processed if the server is built, not ifclient-only.
- Make the compilation of some modules which inherently
require threading depend on thread supportin the build.
- Separate the handling of threading in "configure.in" from
that of server issues, threading is also needed in a
non-server build of a thread-safe client library.
Also, "libdbug" must get built even in a client-only build,
so "dbug/" must be in the list of client directories.
In addition, calls to thread functions in source files which
can be built without thread support must use the wrapper
functions which handle the non-threaded build.
So the modules "client/mysqlimport.c" and "client/mysqlslap.c"
must call "my_thread_end()" only via "mysql_thread_end()".
Makefile.am:
The directories "storage/" and "plugin/" contain files
which are needed for the server only, so their contents
is to be built only if a server is built.
They must not be named unconditionally, because building
their contents will fail unless threads are enabled.
These directories are now listed in the "configure"
variable "sql_server_dirs" which becomes part of
"sql_union_dirs" if the server is to be built.
client/mysqlimport.c:
Use the wrapper function "mysql_thread_end()" which
correctly handles the case of a non-threaded build.
client/mysqlslap.c:
Use the wrapper function "mysql_thread_end()" which
correctly handles the case of a non-threaded build.
configure.in:
Various changes to support builds "--without-server":
1) For the unit tests, we need "libdbug".
2) Separate the treatment of the server from that of the
thread-safe client library.
3) Introduce an "automake conditional" "NEED_THREAD"
which can be checked in some "Makefile.am".
4) Include "storage/" and "plugin/" in the list of
"sql_server_dirs" so that they are handled in the
top "Makefile.am" only if the server is to be built
(see the change in that file).
mysys/Makefile.am:
The code of "mf_keycache.c" in 5.1 is no longer safe
to be built without thread support.
(In 5.0, this was possible.)
Rather than fix these issues, which is tedious and risky,
avoid the need to ever build it without thread support:
It is needed in the server only, which needs thread support.
The only case where we build a "libmysys" without thread
support is for a non-threaded client, where "mf_keycache"
is not neded.
So its inclusion in the list of source files can depend
on the new conditional "NEED_THREAD".
unittest/mysys/Makefile.am:
Test program "my_atomic-t" is to verify the correct handling
of threads only, it cannot be built without thread support
and is not needed in such cases either.
Let its build depend on the new conditional "NEED_THREAD".
Option "--without-server" still not working in 5.1
The general approach is to make sure that source files
which require thread support are only compiled if the build
really needs thread support,
which means when the server is built or a thread-safe client
library.
This required several changes:
- Make sure the subdirectories "storage/" and "plugin/" are
only processed if the server is built, not ifclient-only.
- Make the compilation of some modules which inherently
require threading depend on thread supportin the build.
- Separate the handling of threading in "configure.in" from
that of server issues, threading is also needed in a
non-server build of a thread-safe client library.
Also, "libdbug" must get built even in a client-only build,
so "dbug/" must be in the list of client directories.
In addition, calls to thread functions in source files which
can be built without thread support must use the wrapper
functions which handle the non-threaded build.
So the modules "client/mysqlimport.c" and "client/mysqlslap.c"
must call "my_thread_end()" only via "mysql_thread_end()".
Bug#24509 - 2048 file descriptor limit on windows needs increasing, also
WL#3049 - improved Windows I/O
The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with
the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open
file is stored in the my_file_info struct, along with a flag for append mode
because the Windows API does not support opening files in append mode in all cases)
The default max open files has been increased to 16384 and can be increased further
by setting --max-open-files=<value> during the server start.
Another major change in this patch that almost all Windows specific file IO code
has been moved to a new file my_winfile.c, greatly reducing the amount of code
in #ifdef blocks within mysys, thus improving readability.
Minor enhancements:
- my_(f)stat() is changed to use __stati64 structure with 64 file size
and timestamps. It will return correct file size now (C runtime implementation
used to report outdated information)
- my_lock on Windows is prepared to handle additional timeout parameter
- after review : changed __WIN__ to _WIN32 in the new and changed code.
client/mysqlbinlog.cc:
fileno -> my_fileno
client/readline.cc:
fileno -> my_fileno
include/config-win.h:
Increase OS_FILE_LIMIT for Windows.
Remove O_SHARE - Windows does not support it. Its definition conflicts with
O_SHORT_LIVED, that has different semantics.
include/my_dir.h:
Use stat64 for stat() family of functions on Windows, because of 64 bit file size.
include/my_global.h:
Increased default value for open file limit to 16K
include/my_sys.h:
- my_file_info got new structure members - file handle and open flags
- 2 new Windows-only mysys functions : my_get_osfhandle and my_osmaperr,
modelled after Windows C runtime functions _get_osfhandle and _dosmaperr
libmysql/CMakeLists.txt:
new files my_winfile.c and my_winerr.c
mysql-test/suite/large_tests/r/lock_tables_big.result:
test for more then 2048 open file descriptors on Windows
mysql-test/suite/large_tests/t/lock_tables_big.test:
test for more then 2048 open file descriptors on Windows
mysys/CMakeLists.txt:
new files my_winfile.c and my_winerr.c
mysys/Makefile.am:
new files my_winfile.c and my_winerr.c
mysys/default_modify.c:
fileno -> my_fileno
mysys/my_chsize.c:
implementation of chsize on Windows now moved to my_winfile.c
mysys/my_create.c:
- my_sopen->my_win_open
- close open file before removing (won't generally work on Windows otherwise)
mysys/my_file.c:
On Windows, files returned by my_open will not start with 0, but 2048
(making it simple to detect incompatible mix of CRT and mysys io functions)
mysys/my_fopen.c:
fileno->my_win_fileno , fclose->my_win_fclose, fdopen->my_win_fdopen
Check for legal filename is done by my_win_[f]open functions
mysys/my_fstream.c:
fileno->my_fileno
mysys/my_lib.c:
Windows stat() functions are moved to my_winfile.c
mysys/my_lock.c:
Move Windows code under #ifdef to a separate function win_lock().
Add a parameter for lock wait timeout
mysys/my_mmap.c:
_get_osfhandle->my_get_osfhandle
mysys/my_open.c:
my_sopen->my_win_open (simpler interface)
mysys/my_pread.c:
moved most windows specific code to my_win_file.c
Use my_win_pread
mysys/my_quick.c:
Use my_win_read/my_win_write
mysys/my_read.c:
Moved most of windows specific code to my_win_file.c
Use my_win_read()
mysys/my_seek.c:
On Windows, use my_win_lseek() in my_seek()/my_tell()
Removed dead code (synchronization of lseeks)
Improved DBUG tracing (file position is ulonglong, not ulong)
mysys/my_static.c:
Removed array initialization. my_file_info_default is global variable
thus it is initialized with all zeros anyway
mysys/my_sync.c:
_commit->my_win_fsync
mysys/my_winerr.c:
New file my_winerr.c
Exports my_osmaperr modelled after undocumented C runtime
function _dosmaperr(). The problem with _dosmaperr() used previously is that
1) it is nowhere documented and thus code relying on it is not guaranteed to work
in subsequent releases on the C runtime
2) it is present only in static C runtime (mysqld does not link if compiled with
/MD)
mysys/my_winfile.c:
New file my_winfile.c
Implements ANSI/Posix file IO routines, when possible using native Windows IO, without
C runtime (C runtime dropped because of the 2048 file descriptor limit).
mysys/my_write.c:
write->my_win_write
mysys/mysys_priv.h:
Declaration of Windows Posix functions (private to mysys, shall not be visible
outside)
storage/innobase/handler/ha_innodb.cc:
mysys native Windows IO : correct innodb tmp file handling
mysql_tmpfile does not return valid CRT file descriptor, thus
it is not possible to dup() it. Instead, the native file handle has
to be duplicated and then converted to CRT descriptor.
storage/myisam/mi_locking.c:
_commit->my_sync
Bug#24509 - 2048 file descriptor limit on windows needs increasing, also
WL#3049 - improved Windows I/O
The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with
the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open
file is stored in the my_file_info struct, along with a flag for append mode
because the Windows API does not support opening files in append mode in all cases)
The default max open files has been increased to 16384 and can be increased further
by setting --max-open-files=<value> during the server start.
Another major change in this patch that almost all Windows specific file IO code
has been moved to a new file my_winfile.c, greatly reducing the amount of code
in #ifdef blocks within mysys, thus improving readability.
Minor enhancements:
- my_(f)stat() is changed to use __stati64 structure with 64 file size
and timestamps. It will return correct file size now (C runtime implementation
used to report outdated information)
- my_lock on Windows is prepared to handle additional timeout parameter
- after review : changed __WIN__ to _WIN32 in the new and changed code.
* Finished Monty and Jani's merge
* Some InnoDB tests still fail (because it's old xtradb code run against
newer testsuite). They are expected to go after mergning with the latest
xtradb.
