The idea of the patch
is that every cursor gets its own lock id for table level locking.
Thus cursors are protected from updates performed within the same
connection. Additionally a list of transient (must be closed at
commit) cursors is maintained and all transient cursors are closed
when necessary. Lastly, this patch adds support for deadlock
timeouts to TLL locking when using cursors.
+ post-review fixes.
include/thr_lock.h:
- add a notion of lock owner to table level locking. When using
cursors, lock owner can not be identified by a thread id any more,
as we must protect cursors from updates issued within the same
connection (thread). So, each cursor has its own lock identifier to
use with table level locking.
- extend return values of thr_lock and thr_multi_lock with
THR_LOCK_TIMEOUT and THR_LOCK_DEADLOCK, since these conditions
are now possible (see comments to thr_lock.c)
mysys/thr_lock.c:
Better support for cursors:
- use THR_LOCK_OWNER * as lock identifier, not pthread_t.
- check and return an error for a trivial deadlock case, when an
update statement is issued to a table locked by a cursor which has
been previously opened in the same connection.
- add support for locking timeouts: with use of cursors, trivial
deadlocks can occur. For now the only remedy is the lock wait timeout,
which is initialized from a new global variable 'table_lock_wait_timeout'
Example of a deadlock (assuming the storage engine does not downgrade
locks):
con1: open cursor for select * from t1;
con2: open cursor for select * from t2;
con1: update t2 set id=id*2; -- blocked
con2: update t1 set id=id*2; -- deadlock
Lock timeouts are active only if a connection is using cursors.
- the check in the wait_for_lock loop has been changed from
data->cond != cond to data->cond != 0. data->cond is zeroed
in every place it's changed.
- added comments
sql/examples/ha_archive.cc:
- extend the handlerton with the info about cursor behaviour at commit.
sql/examples/ha_archive.h:
- ctor moved to .cc to make use of archive handlerton
sql/examples/ha_example.cc:
- add handlerton instance, init handler::ht with it
sql/examples/ha_example.h:
- ctor moved to .cc to make use of ha_example handlerton
sql/examples/ha_tina.cc:
- add handlerton instance, init handler::ht with it
sql/examples/ha_tina.h:
- ctor moved to .cc to make use of CSV handlerton
sql/ha_berkeley.cc:
- init handlerton::flags and handler::ht
sql/ha_berkeley.h:
- ctor moved to .cc to make use of BerkeleyDB handlerton
sql/ha_blackhole.cc:
- add handlerton instance, init handler::ht with it
sql/ha_blackhole.h:
- ctor moved to .cc to make use of blackhole handlerton
sql/ha_federated.cc:
- add handlerton instance, init handler::ht with it
sql/ha_federated.h:
- ctor moved to .cc to make use of federated handlerton
sql/ha_heap.cc:
- add handlerton instance, init handler::ht with it
sql/ha_heap.h:
- ctor moved to .cc to make use of ha_heap handlerton
sql/ha_innodb.cc:
- init handlerton::flags and handler::ht of innobase storage engine
sql/ha_innodb.h:
- ctor moved to .cc to make use of archive handlerton
sql/ha_myisam.cc:
- add handlerton instance, init handler::ht with it
sql/ha_myisam.h:
- ctor moved to .cc to make use of MyISAM handlerton
sql/ha_myisammrg.cc:
- init handler::ht in the ctor
sql/ha_myisammrg.h:
- ctor moved to .cc to make use of MyISAM MERGE handlerton
sql/ha_ndbcluster.cc:
- init handlerton::flags and handler::ht
sql/handler.cc:
- drop support for ISAM storage engine, which was removed from 5.0
- close all "transient" cursors at COMMIT/ROLLBACK. A "transient"
SQL level cursor is a cursor that uses tables that have a transaction-
specific state.
sql/handler.h:
- extend struct handlerton with flags, add handlerton *ht to every
handler instance.
sql/lock.cc:
- extend mysql_lock_tables to send error to the client if
thr_multi_lock returns a timeout or a deadlock error.
sql/mysqld.cc:
- add server option --table_lock_wait_timeout (in seconds)
sql/set_var.cc:
- add new global variable 'table_lock_wait_timeout' to specify
a wait timeout for table-level locks of MySQL (in seconds). The default
timeout is 50 seconds. The timeout is active only if the connection
has open cursors.
sql/sql_class.cc:
- implement Statement_map::close_transient_cursors
- safety suggests that we need an assert ensuring
llock_info->n_cursors is functioning properly, adjust destruction of
the Statement_map to allow such assert in THD::~THD
sql/sql_class.h:
- add support for Cursors registry to Statement map.
sql/sql_prepare.cc:
- maintain a list of cursors that must be closed at commit/rollback.
sql/sql_select.cc:
- extend class Cursor to support specific at-COMMIT/ROLLBACK behavior.
If a cursor uses tables of a storage engine that
invalidates all open tables at COMMIT/ROLLBACK, it must be closed
before COMMIT/ROLLBACK is executed.
sql/sql_select.h:
- add an own lock_id and commit/rollback status flag to class Cursor
tests/mysql_client_test.c:
A test case for Bug#10760 and complementary issues: test a simple
deadlock case too.
mysql-test/var:
New BitKeeper file ``mysql-test/var''
as well as changes to the test.
mysql-test/r/federated.result:
New test results
mysql-test/t/federated.test:
Removed explains per brian's suggestion since on his 64-bit amd, you get
different information, something to be ironed out later, as well as it
having nothing to do with federated.
sql/ha_federated.cc:
- Eric Herman's implementation of repair and optimize!!!
- Eric's changes to 1EQ0 to FALSE
sql/ha_federated.h:
Eric's addition of LEN for each SQL string tokens.
WHERE statement"
sql/ha_federated.cc:
Changes per Jimw's review.
notes about some suggestions:
1. Tried to replace my_sprintf with strxmov, but had problems. Couldn't
get the test suite to run at all, and could not find any particular
error. Since this is something that is not critical, I'll leave it
as is for now.
2. I would like to use my_error to create all of my error messages, but
cannot figure out how to get it to take more than one argument to be
passed to the error message (I tried this once but had errors, and
cannot find an example of how to get it to work despite seeing messages
with obviously more than one arg
3. Replaced all string building from literals to defined SQL string pieces
such as "SELECT ", "FROM " ...
4. Format fixes, spacing fixes completed
5. Removed records_in_range records calculation functionality
sql/ha_federated.h:
Formatting changes, new defines for SQL query string tokens
sql/share/errmsg.txt:
changed 'SRC' to 'SOURCE' per Jimw's review
deallocation cleanup
mysql-test/r/federated.result:
new test results
mysql-test/t/federated.test:
New tests, and added explains per Timour's suggestion. This will keep
track of whether changes have been made to the optimiser that might affect
Federated.
Also changed the error codes to federated error codes in the error tests
sql/ha_federated.cc:
1. Not using MAX_REMOTE_SIZE, so removed
2. Changed all defines that were IO_SIZE to FEDERATED_QUERY_BUFFER_SIZE
3. Now using Federated Error Codes
4. Implemented ::info (gets valid 'records' number)
5. Implemented ::read_range_first - now uses indexes in range operations
6. Better allocation in get_share
7. Cleaned up memory bug in get_share
8. Better initial query "SELECT * FROM t1 WHERE 1=0" for check_foreign_data_src
9. emit_key_part_name and emit_key_part_element for read_range_first to not
have redundancy
10. Made sure all 'append's pass length, and use append("") to ensure proper
null termination of strings being built.
11. Better message creation. If ER_QUERY_ON_FOREIGN_DATA_SRC, I also pass
the actual error code and message from the foreign data src.
12. Cleaned up how the result set and share->scheme is freed.
sql/share/errmsg.txt:
New Federated error messages that are more tailored to the type of information
we want to see when federated has an error.
mysql-test/include/federated.inc:
New bitkeeper file, include file for separating setup from main test suite
mysql-test/include/federated_cleanup.inc:
New BitKeeper file ``mysql-test/include/federated_cleanup.inc''
sql/ha_federated.h:
Added comments where needed, one was already commented about
This is the first changeset of suggested changes recommended in Kostja's
review of my patch, 1.1846, which includes only functionality changes.
Style changes/Documentation patch to follow.
include/mysql.h:
removed declaration of cli_fetch_lengths per Kostja's suggestion
libmysql/libmysql.c:
moved mysql_fetch_lengths to client.c (for server to access) per Kostja's
suggestion
sql-common/client.c:
added back 'static' to function definition, added mysql_fetch_lengths
sql/ha_federated.cc:
changed to use defines as opposed to hardcoded values
sql/ha_federated.h:
took out duplicate table_flag, fixed a resolve mistake
Testing on hp3750 shows these fixes fix the compile problems on HPUX, but I have
a problem where when I run the tests, the test shows that the tables default to MyISAM!
include/mysql.h:
HPUX's compiler does not like 'ulong' - the compile fails due to this.
sql/ha_federated.h:
Serg's recommendation to fix bug #8599 (which is a good fix since it's obviously
from when I added the method via cut and paste ;). Also caught some comment style issues.
for bugs 8033, 8065, 8535, 8582
This particular changeset contains style changes per code review suggestions
and does not contain bug fixes in of itself.
ha_federated.h:
standardised code style to conform to internals.texi
ha_federated.cc:
more code standardisation to conform to internals.texi.
- casts
- declarations
- comments
- 80 char width
also, append using string1.append(string2) and not
string1.append(string2.c_ptr_quick())
sql/ha_federated.cc:
more code standardisation to conform to internals.texi.
- casts
- declarations
- comments
- 80 char width
also, append using string1.append(string2) and not
string1.append(string2.c_ptr_quick())
sql/ha_federated.h:
standardised code style to conform to internals.texi
This changeset/patch is on top of changesets 1.1814 and 1.1846
(for bugs 8033 and 8065) and now fixes bug 8535.
These changes have been built and tested successfully on build.mysql.com
handler.cc:
Added hooks for federated_db_init() and federated_db_end(),
as done with ha_archive_db does, per suggestion by Ingo in
code review of patch 1.1846.
ha_federated.h:
declaration of federated_db_init() and federated_db_end()
ha_federated.cc:
- Fixed some indentation problems from indent-ex (mainly to do with
cases where "variablename += value"
- Added federated_db_init() and federated_db_end(), as done with
archive, which also handler more elegantly one of the memory leaks
from bug 8033 where the federated_mutex was not freed
- Removed extrenous debug messages in parse_url()
- Fixed bug 8535, caused by NULL being quoted in write_row. This used to
work (incorrectly) but a recent change was made in the server that
exposed this
sql/ha_federated.cc:
- Fixed some indentation problems from indent-ex (mainly to do with
cases where "variablename += value"
- Added federated_db_init() and federated_db_end(), as done with
archive, which also handler more elegantly one of the memory leaks
from bug 8033 where the federated_mutex was not freed
- Removed extrenous debug messages in parse_url()
sql/ha_federated.h:
declaration of federated_db_init() and federated_db_end()
sql/handler.cc:
Added hooks for federated_db_init() and federated_db_end(), as done with ha_archive_db does.
This patch contains all that my previous patch (1.1814) contained, with the addition of using cli_fetch_lengths for
handling binary data (Bar noted this on the review of 1.1814, Guilhem suggested using cli_fetch_lenghts by
making available via removal of static in method definition and declaration in mysql.h, but
Konstantin had some reservations, but he said to commit the patch using this anyway,
and I suppose this can be discussed. I abandoned 1.1814 because Monty made a couple
fixes to my code as well as formatting changes, and I thought it would just be easier
to hand-edit my changes into a fresh clone and then make a patch.
The reason for using cli_fetch_lengths is so that I can correctly get the length of
the field I am setting into the field. I was previously using 'strlen' but Bar pointed out this
won't correctly get the length of binary data and is also less effecient. Upon testing,
it was in fact verified that binary data in a blob table was being inserted correctly,
but not being retrieved correctly, all due to not having the correct value for the
field:
(*field)->store(row[x], strlen(row[x]), &my_charset_bin);
was changed to:
(*field)->store(row[x], lengths[x], &my_charset_bin);
lengths being a unsigned long pointer to the values of the field lengths from a
MYSQL_ROW.
Since the server doesn't have the function "mysql_fetch_lengths" available, I tried
to use "result->lengths", but this isn't set, so I finally successfully used
cli_fetch_lenghts, which does give the correct lengths, and now the binary data gets
retrieved correctly.
I've also run the code through indent-ex and am using Brian's vimrc to ensure correct formatting!
This code passes the entire test suite, without any errors or warning on both my
workstation and build.mysql.com
include/mysql.h:
added cli_fetch_lengths to mysql.h in order to use this function in the federated handler
mysql-test/r/federated.result:
- Moved countries to be created and inserted prior to federated test table
- Added a test of inserting binary values into a blob table
mysql-test/t/federated.test:
- Moved order of countries table creation to prior to test table creation
- Test insertion of binary values in a blob table
sql-common/client.c:
removed 'static' to allow cli_fetch_lengths to be used in the federated handler
sql/ha_federated.cc:
1. share->scheme that was created in parse_url was not being freed
2. HASH federated_open_tables was being deleted, but not freed
3. 'result' from mysql_store_result was not being free in several instances
4. Fixed the problem where a table scan was being performed after
index_read_idx, which didn't cause a problem because the result set from
idx_read_idx was not being freed, but once the result set was properly freed,
it broke update_row. Now, I'm using the bool 'scan' to determine if I need to
perform a table scan, which it magically is false when the query is an update
with an index.
5. Changed all stings containing the query to perform in mysql_real_query
calls from string.c_ptr_quick() to string.ptr() per Monty's suggestion
(better performance)
6. Fixed various cast/type/truth compile warnings.
7. Removed 'load_conn_info' and just let 'parse_url' handle it.
8. Added the use of cli_fetch_lengths, needed to fix binary values being retrieved
from the database in rnd_next/convert_row_to_internal_format
9. Formatting changes by using indent-ex!
sql/ha_federated.h:
added scan flag, setting defaults for result and scan_flag
-New tests and results
logging_ok:
Logging to logging@openlogging.org accepted
ha_federated.h:
removed quote_data and type_quote (now in the Field class)
ha_federated.cc:
moved quote_data and type_quote to field class
field.h:
new methods quote_data and needs_quotes declared
field.cc:
new field class methods quote_data and needs_quotes (per Monty's request)
federated.test:
more tests, joins, index tests
have_federated_db.require:
new name of federated system var
federated.result:
new test results for federated handler
have_federated_db.inc:
changed name of variable in test due to change in vars
sql_analyse.cc:
over-ridden append_escaped to take (String *, char *, uint) per requirements of 'create_where_from_key' method in federated handler.
mysql_priv.h:
define over-ridden append_escaped to take arguments from 'create_where_from_key' method in federated handler
ha_federated.cc:
implemented "create_where_from_key" to deal properly with two-byte prefix and multi keys. Initial testing shows it works, but I still need to move quoting to field class and also look at changes per Segei's suggestions.
sql/mysql_priv.h:
define over-ridden append_escaped to take arguments from 'create_where_from_key' method in federated handler
sql/sql_analyse.cc:
over-ridden append_escaped to take (String *, char *, uint) per requirements of 'create_where_from_key' method in federated handler.
mysql-test/include/have_federated_db.inc:
changed name of variable in test due to change in vars
mysql-test/r/federated.result:
new test results for federated handler
mysql-test/r/have_federated_db.require:
new name of federated system var
mysql-test/t/federated.test:
more tests, joins, index tests
sql/field.cc:
new field class methods quote_data and needs_quotes (per Monty's request)
sql/field.h:
new methods quote_data and needs_quotes declared
sql/ha_federated.cc:
moved quote_data and type_quote to field class
sql/ha_federated.h:
removed quote_data and type_quote (now in the Field class)
BitKeeper/etc/logging_ok:
Logging to logging@openlogging.org accepted
sql/ha_federated.cc:
added code to handle multi-keys (per monty). This code still fails to compile because sql_analyse.cc's method, append_escaped only accepts two string pointers. I don't know if append_escaped will be overloaded to accept string pointer, byte/char pointer, and length, as is being passed in this example. Need to talk to devs/monty about this.
sql/ha_federated.h:
added method declaration of create_where_from_key
mysql-test/r/federated.result:
new test results
mysql-test/t/federated.test:
added order by, group by
sql/ha_federated.cc:
- added 'scheme' to URL
- added proper escaping
- made sure &my_charset_bin is being used throughout handler
- made sure create_table catches improper URL in comment upon table creation
sql/ha_federated.h:
added scheme to share
configure.in:
inclusion of federated handler autoheader macro
mysql-test/mysql-test-run.sh:
allow usage of replication tests for federated handler
sql/Makefile.am:
inclusion of federated header and source file
sql/field.h:
overloaded method val_str() to work with fields in 'old_data' in 'update_row()'
sql/handler.cc:
added code to include federated handler
sql/handler.h:
add db type for federated
sql/mysql_priv.h:
added code for federated handler
sql/mysqld.cc:
added code for federated handler
sql/set_var.cc:
added code for federated handler
