Adding a new virtual function MY_CHARSET_HANDLER::copy_abort().
Moving character set specific code into the correspoding implementations
(for simple, multi-byte and mbmaxlen>1 character sets).
The problem was that my_hash_sort didn't properly delete end-space characters properly, so strings that should compare
identically was seen as different strings. (Space was handled correctly, but not NBSP)
This caused duplicate key errors when a heap table was converted to Aria as part of overflow in group by.
Fixed by removing all characters that compares as end space when creating a hash.
Other things:
- Fixed that --sorted_results also works for errors in mysqltest.
- Speed up hash by not comparing strings that has different hash.
- Speed up many my_hash_sort functions by using registers to calculate hash instead of pointers.
This was previously done for some functions, but not for all.
- Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.
client/mysqltest.cc:
Fixed that --sorted_results also works for error messages.
mysql-test/r/ctype_partitions.result:
New test to ensure that partitions on hash works
mysql-test/suite/multi_source/gtid.result:
Updated result
mysql-test/suite/multi_source/gtid.test:
Test that --sorted_result works for error messages
mysql-test/suite/multi_source/gtid_ignore_duplicates.result:
Updated result
mysql-test/suite/multi_source/gtid_ignore_duplicates.test:
Updated result
mysql-test/suite/multi_source/load_data.result:
Updated result
mysql-test/suite/multi_source/load_data.test:
Updated result
mysql-test/t/ctype_partitions.test:
New test to ensure that partitions on hash works
storage/heap/hp_write.c:
Speed up hash by not comparing strings that has different hash.
storage/maria/ma_check.c:
Extra debug
strings/ctype-bin.c:
Use macro for hash function
strings/ctype-latin1.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-mb.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-simple.c:
Use macro for hash function
Use same variable names as in other my_hash_sort functions.
Update my_hash_sort_simple() to properly remove end space (patch by Bar)
strings/ctype-uca.c:
Ignore duplicated space inside strings and end space in my_hash_sort_uca(). This fixed MDEV-6255
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-ucs2.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/ctype-utf8.c:
Use macro for hash function
Use registers to calculate hash (speedup)
strings/strings_def.h:
Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.
COLLATIONS ARE USED.
ISSUE :
-------
Code points of HALF WIDTH KATAKANA in SJIS/CP932 range from
A1 to DF. In function my_wildcmp_mb_bin_impl while comparing
such single byte code points, there is a code which compares
signed character with unsigned character. Because of this,
comparisons of two same code points representing a HALF
WIDTH KATAKANA character always fails.
Solution:
---------
A code point of HALF WIDTH KATAKANA at-least need 8 bits.
Promoting the variable from uchar to int will fix the issue.
mysql-test/t/ctype_cp932.test:
Tests which have conditions
LIKE 'STRING PATTERN WITH HALF WIDTH KATAKANA'.
strings/ctype-mb.c:
A code point of HALF WIDTH KATAKANA at-least need 8 bits.
Promoting the variable from uchar to int will fix the issue.
Description: A typo in create_tailoring() causes the "contraction_flags" to be written
into cs->contractions in the wrong place. This causes two problems:
(1) Anyone relying on `contraction_flags` to decide "could this character be
part of a contraction" is 100% broken.
(2) Anyone relying on `contractions` to determine the weight of a contraction
is mostly broken
Analysis: When we are preparing the contraction in create_tailoring(), we are corrupting the
cs->contractions memory location which is supposed to store the weights(8k) + contraction information(256 bytes). We started storing the contraction information after the 4k location. This is because of logic flaw in the code.
Fix: When we create the contractions, we need to calculate the contraction with (char*) (cs->contractions + 0x40*0x40) from ((char*) cs->contractions) + 0x40*0x40. This makes the "cs->contractions" to move to 8k bytes and stores the contraction information from there. Similarly when we are calculating it for like range queries we need to calculate it from the 8k bytes onwards, this can be done by changing the logic to (const char*) (cs->contractions + 0x40*0x40). And for ucs2 charsets we need to modify the my_cs_can_be_contraction_head() and my_cs_can_be_contraction_tail() to point to 8k+ locations.
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
Added strings_def.h into strings library to be able to have a DBUG_ASSERT() version without _db_flush() call (as strings.a should not depend on dbug.a)
Remove include of m_string.h in all string files (as it's included by string_def.h).
Fixed include order.
Changed "m_ctype.h" -> <m_ctype.h>
include/my_dbug.h:
Flush DBUG log in case of DBUG_ASSERT()
strings/bchange.c:
Include strings_def.h
strings/bcmp.c:
Include strings_def.h
strings/bfill.c:
Include strings_def.h
strings/bmove.c:
Include strings_def.h
strings/bmove512.c:
Include strings_def.h
strings/bmove_upp.c:
Include strings_def.h
strings/conf_to_src.c:
Include strings_def.h
Fixed copyright
strings/ctype-big5.c:
Include strings_def.h
strings/ctype-bin.c:
Include strings_def.h
strings/ctype-cp932.c:
Include strings_def.h
strings/ctype-czech.c:
Include strings_def.h
strings/ctype-euc_kr.c:
Include strings_def.h
strings/ctype-eucjpms.c:
Include strings_def.h
strings/ctype-extra.c:
Include strings_def.h
strings/ctype-gbk.c:
Include strings_def.h
strings/ctype-latin1.c:
Include strings_def.h
strings/ctype-mb.c:
Include strings_def.h
strings/ctype-simple.c:
Include strings_def.h
strings/ctype-sjis.c:
Include strings_def.h
strings/ctype-tis620.c:
Include strings_def.h
strings/ctype-uca.c:
Include strings_def.h
strings/ctype-ucs2.c:
Include strings_def.h
strings/ctype-ujis.c:
Include strings_def.h
strings/ctype-utf8.c:
Include strings_def.h
strings/ctype-win1250ch.c:
Include strings_def.h
strings/ctype.c:
Include strings_def.h
strings/decimal.c:
Include strings_def.h
strings/do_ctype.c:
Include strings_def.h
strings/int2str.c:
Include strings_def.h
strings/is_prefix.c:
Include strings_def.h
strings/llstr.c:
Include strings_def.h
strings/longlong2str.c:
Include strings_def.h
strings/longlong2str_asm.c:
Include strings_def.h
strings/my_strchr.c:
Include strings_def.h
strings/my_strtoll10.c:
Include strings_def.h
strings/my_vsnprintf.c:
Include strings_def.h
strings/r_strinstr.c:
Include strings_def.h
strings/str2int.c:
Include strings_def.h
strings/str_alloc.c:
Include strings_def.h
strings/str_test.c:
Include strings_def.h
Fixed compiler warnings
strings/strappend.c:
Include strings_def.h
strings/strcend.c:
Include strings_def.h
strings/strcont.c:
Include strings_def.h
strings/strend.c:
Include strings_def.h
strings/strfill.c:
Include strings_def.h
strings/strinstr.c:
Include strings_def.h
strings/strmake.c:
Include strings_def.h
strings/strmov.c:
Include strings_def.h
strings/strmov_overlapp.c:
Include strings_def.h
strings/strnlen.c:
Include strings_def.h
strings/strnmov.c:
Include strings_def.h
strings/strstr.c:
Include strings_def.h
strings/strto.c:
Include strings_def.h
strings/strtod.c:
Include strings_def.h
strings/strtol.c:
Include strings_def.h
strings/strtoll.c:
Include strings_def.h
strings/strtoul.c:
Include strings_def.h
strings/strtoull.c:
Include strings_def.h
strings/strxmov.c:
Include strings_def.h
strings/strxnmov.c:
Include strings_def.h
strings/uctypedump.c:
Include strings_def.h
Fixed compiler warnings
Removed double include of m_ctype.h
strings/udiv.c:
Include strings_def.h
strings/xml.c:
Include strings_def.h
Problem: LIKE over an indexed column optimized away good results,
because my_like_range_utf32/utf16 returned wrong ranges for contractions.
Contraction related code was missing in my_like_range_utf32/utf16,
but did exist in my_like_range_ucs2/utf8.
It was forgotten in utf32/utf16 versions (during mysql-6.0 push/revert mess).
Fix:
The patch removes individual functions my_like_range_ucs2,
my_like_range_utf16, my_like_range_utf32 and introduces a single function
my_like_range_generic() instead. The new function handles contractions
correctly. It can handle any character set with cs->min_sort_char and
cs->max_sort_char represented in Unicode code points.
added:
@ mysql-test/include/ctype_czech.inc
@ mysql-test/include/ctype_like_ignorable.inc
@ mysql-test/r/ctype_like_range.result
@ mysql-test/t/ctype_like_range.test
Adding tests
modified:
@ include/m_ctype.h
- Adding helper functions for contractions.
- Prototypes: removing ucs2,utf16,utf32 functions, adding generic function.
@ mysql-test/r/ctype_uca.result
@ mysql-test/r/ctype_utf16_uca.result
@ mysql-test/r/ctype_utf32_uca.result
@ mysql-test/t/ctype_uca.test
@ mysql-test/t/ctype_utf16_uca.test
@ mysql-test/t/ctype_utf32_uca.test
- Adding tests.
@ strings/ctype-mb.c
- Pad function did not put the last character.
- Implementing my_like_range_generic() - an universal replacement
for three separate functions
my_like_range_ucs2(), my_like_range_utf16() and my_like_range_utf32(),
with correct contraction handling.
@ strings/ctype-ucs2.c
- my_fill_mb2 did not put the high byte, as previously
it was used to put only characters in ASCII range.
Now it puts high byte as well
(needed to pupulate cs->max_sort_char correctly).
- Adding DBUG_ASSERT()
- Removing character set specific functions:
my_like_range_ucs2(), my_like_range_utf16() and my_like_range_utf32().
- Using my_like_range_generic() instead of the old functions.
@ strings/ctype-uca.c
- Using generic function instead of the old character set specific ones.
@ sql/item_create.cc
@ sql/item_strfunc.cc
@ sql/item_strfunc.h
- Adding SQL functions LIKE_RANGE_MIN and LIKE_RANGE_MAX,
available only in debug build to make sure like_range()
works correctly for all character sets and collations.
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
Problem: The functions my_like_range_xxx() returned
badly formed maximum strings for Asian character sets,
which made problems for storage engines.
Fix:
- Removed a number my_like_range_xxx() implementations,
which were in fact dumplicate code pieces.
- Using generic my_like_range_mb() instead.
- Setting max_sort_char member properly for Asian character sets
- Adding unittest/strings/strings-t.c,
to test that my_like_range_xxx() return well-formed
min and max strings.
Notes:
- No additional tests in mysql/t/ available.
Old tests cover the affected code well enough.
- Fixing crash on attempt to create a fulltext index with an utf8mb4 column
- fixing wrong border width for supplementary characters in mysql client:
mysql --default-character-set=utf8mb4 -e "select concat(_utf32 0x20000,'a')"
config/ac-macros/character_sets.m4:
- Adding configure definitions for utf8mb4, utf16, utf32
include/config-win.h:
- Enabling utf8mb4, utf16, utf32 in Windows build
include/m_ctype.h:
- Adding new flags
- Adding new shared functions prototypes
mysql-test/include/ctype_datetime.inc:
- Adding test to check that datetime functions
work with "real" multibyte character sets.
mysql-test/include/ctype_like.inc:
- Adding LIKE tests
mysql-test/include/have_utf16.inc:
New file
mysql-test/include/have_utf32.inc:
New file
mysql-test/include/have_utf8mb4.inc:
New file
mysql-test/r/ctype_ldml.result:
- Adding tests for utf8mb4, utf16, utf32
mysql-test/r/ctype_many.result:
- Adding tests to check superset/subset relations
between all Unicode character sets.
mysql-test/r/ctype_utf16.result:
New file
mysql-test/r/ctype_utf16_uca.result:
New file
mysql-test/r/ctype_utf32.result:
New file
mysql-test/r/ctype_utf32_uca.result:
New file
mysql-test/r/ctype_utf8.result:
- Adding tests for utf8mn3 alias
mysql-test/r/ctype_utf8mb4.result:
- Adding tests for utf8mb4
mysql-test/r/have_utf16.require:
New file
mysql-test/r/have_utf32.require:
New file
mysql-test/r/have_utf8mb4.require:
New file
mysql-test/std_data/Index.xml:
- Adding tests for loadable utf8m4, utf16, utf32 collations
mysql-test/suite/sys_vars/r/character_set_client_basic.result:
- Adding tests for utf16, utf32.
- Fixing new number of character sets
mysql-test/suite/sys_vars/r/character_set_connection_basic.result:
- Fixing new number of character sets
mysql-test/suite/sys_vars/r/character_set_database_basic.result:
- Fixing new number of character sets
mysql-test/suite/sys_vars/r/character_set_filesystem_basic.result:
- Fixing new number of character sets
mysql-test/suite/sys_vars/r/character_set_results_basic.result:
- Fixing new number of character sets
mysql-test/suite/sys_vars/t/character_set_client_basic.test:
- Adding tests for new character sets
mysql-test/suite/sys_vars/t/character_set_connection_basic.test:
- Adding dependency on utf8mb4, utf16, utf32
mysql-test/suite/sys_vars/t/character_set_database_basic.test:
- Adding dependency on utf8mb4, utf16, utf32
mysql-test/suite/sys_vars/t/character_set_filesystem_basic.test:
- Adding dependency on utf8mb4, utf16, utf32
mysql-test/suite/sys_vars/t/character_set_results_basic.test:
- Adding dependency on utf8mb4, utf16, utf32
mysql-test/t/ctype_ldml.test:
- Adding tests for dynamic utf8mb4, utf16, utf32 collations
mysql-test/t/ctype_many.test:
- Adding tests to check superset/subset relations
between all Unicode character sets
mysql-test/t/ctype_utf16.test:
New file
mysql-test/t/ctype_utf16_uca.test:
New file
mysql-test/t/ctype_utf32.test:
New file
mysql-test/t/ctype_utf32_uca.test:
New file
mysql-test/t/ctype_utf8.test:
- Adding tests for utf8mb4 alias
mysql-test/t/ctype_utf8mb4.test:
New file
mysys/charset-def.c:
- Adding initialization of utf8mb4, utf16, utf32 built-int collations
mysys/charset.c:
- Adding initialization of utf8mb4, utf16, utf32 dynamic collations
sql/field.cc:
- Fixing "truncated" error with datetime functions:
Force conversion in case of non-ascii character sets.
sql/item.cc:
- Adding superset/subset relation check for utf8mb4/utf8
sql/item_strfunc.cc:
- Fixing a problem with CHAR(x USING utf32)
sql/sql_string.cc:
- Fixing problems with zero padding for UTF32
sql/sql_table.cc:
- Fixing buffer size, to make utf32 comma fit.
strings/ctype-mb.c:
- Making handlers for multi-byte binary collations public
strings/ctype-uca.c:
- Adding definitions for utf8mb4, utf16, utf32 UCA collations
strings/ctype-ucs2.c:
- Adding functions which are shared between ucs2, utf16, utf32
- Ading utf16 implementation
- Adding utf32 implementation
strings/ctype-utf8.c:
- Adding functions shared between utf8 and utf8mb4
- Adding implementation of utf8mb4
Removed compiler warnings
extra/libevent/epoll.c:
Removed compiler warnings
extra/libevent/evbuffer.c:
Removed compiler warnings
extra/libevent/event.c:
Removed compiler warnings
extra/libevent/select.c:
Removed compiler warnings
extra/libevent/signal.c:
Removed compiler warnings
include/m_ctype.h:
Define CHARSET_INFO, MY_CHARSET_HANDLER, MY_COLLATION_HANDLER, MY_UNICASE_INFO, MY_UNI_CTYPE and MY_UNI_IDX as const structures.
Declare that pointers point to const data
include/m_string.h:
Declare that pointers point to const data
include/my_sys.h:
Redefine variables and function prototypes
include/mysql.h:
Declare charset as const
include/mysql.h.pp:
Declare charset as const
include/mysql/plugin.h:
Declare charset as const
include/mysql/plugin.h.pp:
Declare charset as const
mysys/charset-def.c:
Charset can't be of type CHARSET_INFO as they are changed when they are initialized.
mysys/charset.c:
Functions that change CHARSET_INFO must use 'struct charset_info_st'
Add temporary variables to not have to change all_charsets[] (Which now is const)
sql-common/client.c:
Added cast to const
sql/item_cmpfunc.h:
Added cast to avoid compiler error.
sql/sql_class.cc:
Added cast to const
sql/sql_lex.cc:
Added cast to const
storage/maria/ma_ft_boolean_search.c:
Added cast to avoid compiler error.
storage/maria/ma_ft_parser.c:
Added cast to avoid compiler error.
storage/maria/ma_search.c:
Added cast to const
storage/myisam/ft_boolean_search.c:
Added cast to avoid compiler error
storage/myisam/ft_parser.c:
Added cast to avoid compiler error
storage/myisam/mi_search.c:
Added cast to const
storage/pbxt/src/datadic_xt.cc:
Added cast to const
storage/pbxt/src/ha_pbxt.cc:
Added cast to const
Removed compiler warning by changing prototype of XTThreadPtr()
storage/pbxt/src/myxt_xt.h:
Character sets should be const
storage/pbxt/src/xt_defs.h:
Character sets should be const
storage/xtradb/btr/btr0cur.c:
Removed compiler warning
strings/conf_to_src.c:
Added const
Functions that change CHARSET_INFO must use 'struct charset_info_st'
strings/ctype-big5.c:
Made arrays const
strings/ctype-bin.c:
Made arrays const
strings/ctype-cp932.c:
Made arrays const
strings/ctype-czech.c:
Made arrays const
strings/ctype-euc_kr.c:
Made arrays const
strings/ctype-eucjpms.c:
Made arrays const
strings/ctype-extra.c:
Made arrays const
strings/ctype-gb2312.c:
Made arrays const
strings/ctype-gbk.c:
Made arrays const
strings/ctype-latin1.c:
Made arrays const
strings/ctype-mb.c:
Made arrays const
strings/ctype-simple.c:
Made arrays const
strings/ctype-sjis.c:
Made arrays const
strings/ctype-tis620.c:
Made arrays const
strings/ctype-uca.c:
Made arrays const
strings/ctype-ucs2.c:
Made arrays const
strings/ctype-ujis.c:
Made arrays const
strings/ctype-utf8.c:
Made arrays const
strings/ctype-win1250ch.c:
Made arrays const
strings/ctype.c:
Made arrays const
Added cast to const
Functions that change CHARSET_INFO must use 'struct charset_info_st'
strings/int2str.c:
Added cast to const
Added support for Croatian sorting orders utf8_croatian_ci and ucs2_croatian_ci.
Patch done by Alexander Barkov. See http://www.collation-charts.org/articles/croatian.htm
mysql-test/r/ctype_uca.result:
Added testing of Croatian sort order
mysql-test/t/ctype_uca.test:
Added testing of Croatian sort order
(From: gkodinov)
Use and int * where possible to scan for trailing space in a
string instead of always iterating char-by-char.
Using the attached benchmark file on a 32 bit Intel Core 2
Duo CPU I've got 43485 ms run with the fix compared to 44373
without it.
Backported to 5.6.0 (next-mr-runtime)
6.0-codebase revid: 2476.1362.1
include/m_string.h:
scan for space through ints
strings/ctype-bin.c:
scan for space through ints
strings/ctype-latin1.c:
scan for space through ints
strings/ctype-mb.c:
scan for space through ints
strings/ctype-simple.c:
scan for space through ints
- Remove bothersome warning messages. This change focuses on the warnings
that are covered by the ignore file: support-files/compiler_warnings.supp.
- Strings are guaranteed to be max uint in length
into mysql.com:/home/bar/mysql-work/mysql-5.1-new-rpl
strings/ctype-uca.c:
Auto merged
strings/ctype-mb.c:
After merge fix.
strings/ctype-ucs2.c:
After merge fix.
