Solve 3 way deadlock between plugin_initialiaze(), THD::init() and
mysql_sys_var_char().
The deadlock exists because of the lock order inversion between
LOCK_global_system_variables mutex and LOCK_system_variables_hash
read-write lock-
In this case, it is enough to change LOCK_system_variables_hash to prefer
reads to fix the deadlock, i.e change it to mysql_prlock_t
Other things, mainly to get
create_mysqld_error_find_printf_error tool to work:
- Added protection to not include mysqld_error.h twice
- Include "unireg.h" instead of "mysqld_error.h" in server
- Added protection if ER_XX messages are already defined
- Removed wrong calls to my_error(ER_OUTOFMEMORY) as
my_malloc() and my_alloc will do this automatically
- Added missing %s to ER_DUP_QUERY_NAME
- Removed old and wrong calls to my_strerror() when using
MY_ERROR_ON_RENAME (wrong merge)
- Fixed deadlock error message from Galera. Before the extra
information given to ER_LOCK_DEADLOCK was missing because
ER_LOCK_DEADLOCK doesn't provide any extra information.
I kept #ifdef mysqld_error_find_printf_error_used in sql_acl.h
to make it easy to do this kind of check again in the future
Part2: make MyRocks add its directory into @@ignore_db_dirs when starting.
This is necessary because apparently not everybody are using plugin's my.cnf
So load ha_rocksdb.{so,dll} manually and then hit MDEV-12451, MDEV-14461
etc.
LOCK_thd_data was used to protect both THD data and
ensure that the THD is not deleted while it was in use
This patch moves the THD delete protection to LOCK_thd_kill,
which already protects the THD for kill.
The benefits are:
- More well defined what LOCK_thd_data protects
- LOCK_thd_data usage is now much simpler and easier to verify
- Less chance of deadlocks in SHOW PROCESS LIST as there is less
chance of interactions between mutexes
- Remove not needed LOCK_thread_count from
thd_get_error_context_description()
- Fewer mutex taken for thd->awake()
Other things:
- Don't take mysys->var mutex in show processlist to check if thread
is kill marked
- thd->awake() now automatically takes the LOCK_thd_kill mutex
(Simplifies code)
- Apc uses LOCK_thd_kill instead of LOCK_thd_data
This will allow show processlist to continue, without blocking
all new connections, if some threads gets stuck while holding
LOCK_thd_data or mysys_var->mutex
Connections that has mutex 'stuck' are marked as 'Busy' in 'Command'
Todo:
Make F_BACKOFF to do 'pause' instead of just (1)
The background is that one user had a lot of views and using some complex
queries on information schema temporary memory of more than 2G was used.
- Added new element 'total_alloc' to MEM_ROOT for easier debugging.
- Added MAX_MEMORY_USED to information_schema.processlist.
- Added new status variable "Memory_used_initial" that shows how much MariaDB
uses at startup. This gives the base value for "Memory_used".
- Reuse memory continuously for information schema queries instead of
only freeing memory at query end.
Other things
- Removed some not needed set_notnull() calls for not null columns.
Fixing mysql_execute_command() and store_schema_proc() not to use
stored object type codes TYPE_ENUM_PROCEDURE and TYPE_ENUM_FUNCTION.
Using pointers to Sp_handler instead, to make the code symmetric across
existing (function,procedure) and future (e.g package) SP object types.
in buildbot
Removed plugin_array_version: it is being checked without mutex protection and
thus is prone to data race and race conditions. In effect plugins are not
protected from concurrent destruction.
Removed state_mask inversion: doesn't seem to make any sense.
When collecting local plugins list, only add plugins that match state_mask.
Use plugin ref counting to protect against concurrent plugin destruction.
Optimizations:
* avoid faster ext_table_discovery_simple() *only* when there are
engines with discover_table_names() method *and* they look at
file names. P_S implements discover_table_names(), but it's not
a reason to use slower extension_based_table_discovery().
* don't pre-sort table names if ORDER BY or GROUP BY was specified
* starting from 10.3 only sort table names for SHOW commands
Some innobase/xtrabackup changes around from 10.1 are null merged
, in partucular using os_set_file_size to extend tablespaces in server
or mariabackup.
They require non-trivial amount of additional work in 10.2, due to
innobase differences between 10.1 and 10.2
Don't write to a temporary file, use String.
Remove strange one-liner "helpers", use String methods.
Don't use current_thd, don't allocate memory for 1-byte strings, etc.
The problems were in the code of sql_show.cc. There the tables
could be opened in such a way that mysql_derived_init() never
worked for CTE tables. As a result they were not marked as
derived and mysql_handle_derived() were not called for derived
tables used in their specifications.
The problem lies in not checking role privileges as well during SHOW
DATABASES command. This problem is also apparent for SHOW CREATE
DATABASE command.
Other SHOW COMMANDS make use of check_access, which in turn makes use of
acl_get for both priv_user and priv_role parts, which allows them to
function correctly.
- Changed names of SEQUENCE table columns to be more close to ANSI
- Fixed error message for SHOW SEQUENCE non_existing_sequence
- Allow syntax CACHE +1
- Fixed ALTER TABLE for TEMPORARY sequences.
Do not silence uncertain cases, or fix any bugs.
The only functional change should be that ha_federated::extra()
is not calling DBUG_PRINT to report an unhandled case for
HA_EXTRA_PREPARE_FOR_DROP.
Do not silence uncertain cases, or fix any bugs.
The only functional change should be that ha_federated::extra()
is not calling DBUG_PRINT to report an unhandled case for
HA_EXTRA_PREPARE_FOR_DROP.
- Adding Type_handler::make_table_field() and moving pieces of the code
from Item::tmp_table_field_from_field_type() to virtual implementations
for various type handlers.
- Adding a new Type_all_attributes, to access to Item's extended
attributes, such as decimal_precision() and geometry_type().
- Adding a new class Record_addr, to pass record related information
to Type_handler methods (ptr, null_ptr and null_bit) as a single structure.
Note, later it will possibly be extended for BIT-alike field purposes,
by adding new members (bit_ptr_arg, bit_ofs_arg).
- Moving the code from Field_new_decimal::create_from_item()
to Type_handler_newdecimal::make_table_field().
- Removing Field_new_decimal() and Field_geom() helper constructor
variants that were used for temporary field creation.
- Adding Item_field::type_handler(), Field::type_handler() and
Field_blob::type_handler() to return correct type handlers for
blob variants, according to Field_blob::packlength.
- Adding Type_handler_blob_common, as a common parent for
Type_handler_tiny_blob, Type_handler_blob, Type_handler_medium_blob
and Type_handler_long_blob.
- Implementing Type_handler_blob_common::Item_hybrid_func_fix_attributes().
It's needed for cases when TEXT variants of different character sets are mixed
in LEAST, GREATEST, CASE and its abreviations (IF, IFNULL, COALESCE), e.g.:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT COALESCE(a,b) FROM t1;
Type handler aggregation returns TINYTEXT as a common data type
for the two columns. But as conversion from latin1 to utf8
happens for "a", the maximum possible length of "a" grows from 255 to 255*3.
Type_handler_blob_common::Item_hybrid_func_fix_attributes() makes sure
to update the blob type handler according to max_length.
- Adding Type_handler::blob_type_handler(uint max_octet_length).
- Adding a few m_type_aggregator_for_result.add() pairs, because
now Item_xxx::type_handler() can return pointers to type_handler_tiny_blob,
type_handler_blob, type_handler_medium_blob, type_handler_long_blob.
Before the patch only type_handler_blob was possible result of type_handler().
- Making type_handler_tiny_blob, type_handler_blob, type_handler_medium_blob,
type_handler_long_blob public.
- Removing the condition in Item_sum_avg::create_tmp_field()
checking Item_sum_avg::result_type() against DECIMAL_RESULT.
Now both REAL_RESULT and DECIMAL_RESULT are symmetrically handled
by tmp_table_field_from_field_type().
- Removing Item_geometry_func::create_field_for_create_select(),
as the inherited version perfectly works.
- Fixing Item_func_as_wkb::field_type() to return MYSQL_TYPE_LONG_BLOB
rather than MYSQL_TYPE_BLOB. It's needed to make sure that
tmp_table_field_from_field_type() creates a LONGBLOB field for AsWKB().
- Fixing Item_func_as_wkt::fix_length_and_dec() to set max_length to
UINT32_MAX rather than MAX_BLOB_WIDTH, to make sure that
tmp_table_field_from_field_type() creates a LONGTEXT field for AsWKT().
- Removing Item_func_set_user_var::create_field_for_create_select(),
as the inherited version works fine.
- Adding Item_func_get_user_var::create_field_for_create_select() to
make sure that "CREATE TABLE t1 AS SELECT @string_user variable"
always creates a field of LONGTEXT/LONGBLOB type.
- Item_func_ifnull::create_field_for_create_select()
behavior has changed. Before the patch it passed set_blob_packflag=false,
which meant to create LONGBLOB for all blob variants.
Now it takes into account max_length, which gives better column
data types for:
CREATE TABLE t2 AS SELECT IFNULL(blob_column1, blob_column2) FROM t1;
- Fixing Item_func_nullif::fix_length_and_dec() to use
set_handler(args[2]->type_handler()) instead of
set_handler_by_field_type(args[2]->field_type()).
This is needed to distinguish between BLOB variants.
- Implementing Item_blob::type_handler(), to make sure to create
proper BLOB field variant, according to max_length, for queries like:
CREATE TABLE t1 AS
SELECT some_blob_field FROM INFORMATION_SCHEMA.SOME_TABLE;
- Fixing Item_field::real_type_handler() to make sure that
the code aggregating fields for UNION gets a proper BLOB
variant type handler from fields.
- Adding a special code into Item_type_holder::make_field_by_type(),
to make sure that after aggregating field types it also properly
takes into account max_length when mixing TEXT variants of different
character sets and chooses a proper TEXT variant:
CREATE TABLE t1 (
a TINYTEXT CHARACTER SET latin1,
b TINYTEXT CHARACTER SET utf8
);
CREATE TABLE t2 AS SELECT a FROM t1 UNION SELECT b FROM t1;
- Adding tests, for better coverage of IFNULL, NULLIF, UNION.
- The fact that tmp_table_field_from_field_type() now takes
into account BLOB variants (instead of always creating LONGBLOB),
tests results for WEIGHT_STRING() and NULLIF() and UNION
have become more precise.
Benefits of this patch:
- Removed a lot of calls to strlen(), especially for field_string
- Strings generated by parser are now const strings, less chance of
accidently changing a string
- Removed a lot of calls with LEX_STRING as parameter (changed to pointer)
- More uniform code
- Item::name_length was not kept up to date. Now fixed
- Several bugs found and fixed (Access to null pointers,
access of freed memory, wrong arguments to printf like functions)
- Removed a lot of casts from (const char*) to (char*)
Changes:
- This caused some ABI changes
- lex_string_set now uses LEX_CSTRING
- Some fucntions are now taking const char* instead of char*
- Create_field::change and after changed to LEX_CSTRING
- handler::connect_string, comment and engine_name() changed to LEX_CSTRING
- Checked printf() related calls to find bugs. Found and fixed several
errors in old code.
- A lot of changes from LEX_STRING to LEX_CSTRING, especially related to
parsing and events.
- Some changes from LEX_STRING and LEX_STRING & to LEX_CSTRING*
- Some changes for char* to const char*
- Added printf argument checking for my_snprintf()
- Introduced null_clex_str, star_clex_string, temp_lex_str to simplify
code
- Added item_empty_name and item_used_name to be able to distingush between
items that was given an empty name and items that was not given a name
This is used in sql_yacc.yy to know when to give an item a name.
- select table_name."*' is not anymore same as table_name.*
- removed not used function Item::rename()
- Added comparision of item->name_length before some calls to
my_strcasecmp() to speed up comparison
- Moved Item_sp_variable::make_field() from item.h to item.cc
- Some minimal code changes to avoid copying to const char *
- Fixed wrong error message in wsrep_mysql_parse()
- Fixed wrong code in find_field_in_natural_join() where real_item() was
set when it shouldn't
- ER_ERROR_ON_RENAME was used with extra arguments.
- Removed some (wrong) ER_OUTOFMEMORY, as alloc_root will already
give the error.
TODO:
- Check possible unsafe casts in plugin/auth_examples/qa_auth_interface.c
- Change code to not modify LEX_CSTRING for database name
(as part of lower_case_table_names)
Working features:
CREATE OR REPLACE [TEMPORARY] SEQUENCE [IF NOT EXISTS] name
[ INCREMENT [ BY | = ] increment ]
[ MINVALUE [=] minvalue | NO MINVALUE ]
[ MAXVALUE [=] maxvalue | NO MAXVALUE ]
[ START [ WITH | = ] start ] [ CACHE [=] cache ] [ [ NO ] CYCLE ]
ENGINE=xxx COMMENT=".."
SELECT NEXT VALUE FOR sequence_name;
SELECT NEXTVAL(sequence_name);
SELECT PREVIOUS VALUE FOR sequence_name;
SELECT LASTVAL(sequence_name);
SHOW CREATE SEQUENCE sequence_name;
SHOW CREATE TABLE sequence_name;
CREATE TABLE sequence-structure ... SEQUENCE=1
ALTER TABLE sequence RENAME TO sequence2;
RENAME TABLE sequence TO sequence2;
DROP [TEMPORARY] SEQUENCE [IF EXISTS] sequence_names
Missing features
- SETVAL(value,sequence_name), to be used with replication.
- Check replication, including checking that sequence tables are marked
not transactional.
- Check that a commit happens for NEXT VALUE that changes table data (may
already work)
- ALTER SEQUENCE. ANSI SQL version of setval.
- Share identical sequence entries to not add things twice to table list.
- testing insert/delete/update/truncate/load data
- Run and fix Alibaba sequence tests (part of mysql-test/suite/sql_sequence)
- Write documentation for NEXT VALUE / PREVIOUS_VALUE
- NEXTVAL in DEFAULT
- Ensure that NEXTVAL in DEFAULT uses database from base table
- Two NEXTVAL for same row should give same answer.
- Oracle syntax sequence_table.nextval, without any FOR or FROM.
- Sequence tables are treated as 'not read constant tables' by SELECT; Would
be better if we would have a separate list for sequence tables so that
select doesn't know about them, except if refereed to with FROM.
Other things done:
- Improved output for safemalloc backtrack
- frm_type_enum changed to Table_type
- Removed lex->is_view and replaced with lex->table_type. This allows
use to more easy check if item is view, sequence or table.
- Added table flag HA_CAN_TABLES_WITHOUT_ROLLBACK, needed for handlers
that want's to support sequences
- Added handler calls:
- engine_name(), to simplify getting engine name for partition and sequences
- update_first_row(), to be able to do efficient sequence implementations.
- Made binlog_log_row() global to be able to call it from ha_sequence.cc
- Added handler variable: row_already_logged, to be able to flag that the
changed row is already logging to replication log.
- Added CF_DB_CHANGE and CF_SCHEMA_CHANGE flags to simplify
deny_updates_if_read_only_option()
- Added sp_add_cfetch() to avoid new conflicts in sql_yacc.yy
- Moved code for add_table_options() out from sql_show.cc::show_create_table()
- Added String::append_longlong() and used it in sql_show.cc to simplify code.
- Added extra option to dd_frm_type() and ha_table_exists to indicate if
the table is a sequence. Needed by DROP SQUENCE to not drop a table.