Commit graph

115 commits

Author SHA1 Message Date
Alexander Barkov
0d17c540a5 MDEV-27277 Add a warning when max_sort_length is reached
Step#1: fixing the return type of strnxfrm() from size_t to this structure:

typedef struct
{
  size_t m_output_length;
  size_t m_source_length_used;
  uint m_warnings;
} my_strnxfrm_ret_t;
2024-10-22 21:42:53 +07:00
Alexander Barkov
fd247cc21f MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp()
This patch also fixes:
  MDEV-33050 Build-in schemas like oracle_schema are accent insensitive
  MDEV-33084 LASTVAL(t1) and LASTVAL(T1) do not work well with lower-case-table-names=0
  MDEV-33085 Tables T1 and t1 do not work well with ENGINE=CSV and lower-case-table-names=0
  MDEV-33086 SHOW OPEN TABLES IN DB1 -- is case insensitive with lower-case-table-names=0
  MDEV-33088 Cannot create triggers in the database `MYSQL`
  MDEV-33103 LOCK TABLE t1 AS t2 -- alias is not case sensitive with lower-case-table-names=0
  MDEV-33109 DROP DATABASE MYSQL -- does not drop SP with lower-case-table-names=0
  MDEV-33110 HANDLER commands are case insensitive with lower-case-table-names=0
  MDEV-33119 User is case insensitive in INFORMATION_SCHEMA.VIEWS
  MDEV-33120 System log table names are case insensitive with lower-cast-table-names=0

- Removing the virtual function strnncoll() from MY_COLLATION_HANDLER

- Adding a wrapper function CHARSET_INFO::streq(), to compare
  two strings for equality. For now it calls strnncoll() internally.
  In the future it will turn into a virtual function.

- Adding new accent sensitive case insensitive collations:
    - utf8mb4_general1400_as_ci
    - utf8mb3_general1400_as_ci
  They implement accent sensitive case insensitive comparison.
  The weight of a character is equal to the code point of its
  upper case variant. These collations use Unicode-14.0.0 casefolding data.

  The result of
     my_charset_utf8mb3_general1400_as_ci.strcoll()
  is very close to the former
     my_charset_utf8mb3_general_ci.strcasecmp()

  There is only a difference in a couple dozen rare characters, because:
    - the switch from "tolower" to "toupper" comparison, to make
      utf8mb3_general1400_as_ci closer to utf8mb3_general_ci
    - the switch from Unicode-3.0.0 to Unicode-14.0.0
  This difference should be tolarable. See the list of affected
  characters in the MDEV description.

  Note, utf8mb4_general1400_as_ci correctly handles non-BMP characters!
  Unlike utf8mb4_general_ci, it does not treat all BMP characters
  as equal.

- Adding classes representing names of the file based database objects:

    Lex_ident_db
    Lex_ident_table
    Lex_ident_trigger

  Their comparison collation depends on the underlying
  file system case sensitivity and on --lower-case-table-names
  and can be either my_charset_bin or my_charset_utf8mb3_general1400_as_ci.

- Adding classes representing names of other database objects,
  whose names have case insensitive comparison style,
  using my_charset_utf8mb3_general1400_as_ci:

  Lex_ident_column
  Lex_ident_sys_var
  Lex_ident_user_var
  Lex_ident_sp_var
  Lex_ident_ps
  Lex_ident_i_s_table
  Lex_ident_window
  Lex_ident_func
  Lex_ident_partition
  Lex_ident_with_element
  Lex_ident_rpl_filter
  Lex_ident_master_info
  Lex_ident_host
  Lex_ident_locale
  Lex_ident_plugin
  Lex_ident_engine
  Lex_ident_server
  Lex_ident_savepoint
  Lex_ident_charset
  engine_option_value::Name

- All the mentioned Lex_ident_xxx classes implement a method streq():

  if (ident1.streq(ident2))
     do_equal();

  This method works as a wrapper for CHARSET_INFO::streq().

- Changing a lot of "LEX_CSTRING name" to "Lex_ident_xxx name"
  in class members and in function/method parameters.

- Replacing all calls like
    system_charset_info->coll->strcasecmp(ident1, ident2)
  to
    ident1.streq(ident2)

- Taking advantage of the c++11 user defined literal operator
  for LEX_CSTRING (see m_strings.h) and Lex_ident_xxx (see lex_ident.h)
  data types. Use example:

  const Lex_ident_column primary_key_name= "PRIMARY"_Lex_ident_column;

  is now a shorter version of:

  const Lex_ident_column primary_key_name=
    Lex_ident_column({STRING_WITH_LEN("PRIMARY")});
2024-04-18 15:22:10 +04:00
Alexander Barkov
929c2e06aa MDEV-31531 Remove my_casedn_str() and my_caseup_str()
Under terms of MDEV 27490 we'll add support for non-BMP identifiers
and upgrade casefolding information to Unicode version 14.0.0.
In Unicode-14.0.0 conversion to lower and upper cases can increase octet length
of the string, so conversion won't be possible in-place any more.

This patch removes virtual functions performing in-place casefolding:
  - my_charset_handler_st::casedn_str()
  - my_charset_handler_st::caseup_str()
and fixes the code to use the non-inplace functions instead:
  - my_charset_handler_st::casedn()
  - my_charset_handler_st::caseup()
2024-02-28 22:20:29 +04:00
Alexander Barkov
6075f12c65 MDEV-31071 Refactor case folding data types in Unicode collations
This is a non-functional change. It changes the way how case folding data
and weight data (for simple Unicode collations) are stored:

- Removing data types MY_UNICASE_CHARACTER, MY_UNICASE_INFO
- Using data types MY_CASEFOLD_CHARACTER, MY_CASEFOLD_INFO instead.

This patch changes simple Unicode collations in a similar way
how MDEV-30695 previously changed Asian collations.

No new MTR tests are needed. The underlying code is thoroughly
covered by a number of ctype_*_ws.test and ctype_*_casefold.test
files, which were added recently as a preparation
for this change.

Old and new Unicode data layout
-------------------------------

Case folding data is now stored in separate tables
consisting of MY_CASEFOLD_CHARACTER elements with two members:

    typedef struct casefold_info_char_t
    {
      uint32 toupper;
      uint32 tolower;
    } MY_CASEFOLD_CHARACTER;

while weight data (for simple non-UCA collations xxx_general_ci
and xxx_general_mysql500_ci) is stored in separate arrays of
uint16 elements.

Before this change case folding data and simple weight data were
stored together, in tables of the following elements with three members:

    typedef struct unicase_info_char_st
    {
      uint32 toupper;
      uint32 tolower;
      uint32 sort;          /* weights for simple collations */
    } MY_UNICASE_CHARACTER;

This data format was redundant, because weights (the "sort" member) were
needed only for these two simple Unicode collations:
- xxx_general_ci
- xxx_general_mysql500_ci

Adding case folding information for Unicode-14.0.0 using the old
format would waste memory without purpose.

Detailed changes
----------------
- Changing the underlying data types as described above

- Including unidata-dump.c into the sources.
  This program was earlier used to dump UnicodeData.txt
  (e.g. https://www.unicode.org/Public/14.0.0/ucd/UnicodeData.txt)
  into MySQL / MariaDB source files.
  It was originally written in 2002, but has not been distributed yet
  together with MySQL / MariaDB sources.

- Removing the old format Unicode data earlier dumped from UnicodeData.txt
  (versions 3.0.0 and 5.2.0) from ctype-utf8.c.
  Adding Unicode data in the new format into separate header files,
  to maintain the code easier:

    - ctype-unicode300-casefold.h
    - ctype-unicode300-casefold-tr.h
    - ctype-unicode300-general_ci.h
    - ctype-unicode300-general_mysql500_ci.h
    - ctype-unicode520-casefold.h

- Adding a new file ctype-unidata.c as an aggregator for
  the header files listed above.
2023-04-18 11:29:25 +04:00
Alexander Barkov
33f8f92b74 MDEV-30695 Refactor case folding data types in Asian collations
This is a non-functional change and should not change the server behavior.

Casefolding information is now stored in items of a new data type MY_CASEFOLD_CHARACTER:

typedef struct casefold_info_char_t
{
  uint32 toupper;
  uint32 tolower;
} MY_CASEFOLD_CHARACTER;

Before this change, casefolding tables for Asian collations were stored in:

typedef struct unicase_info_char_st
{
  uint32 toupper;
  uint32 tolower;
  uint32 sort;
} MY_UNICASE_CHARACTER;

The "sort" member was not used in the code handling Asian collations,
it only wasted space.
(it's only used by Unicode _general_ci and _general_mysql500_ci collations).

Unicode collations (at least UCA and _bin) should also be refactored later,
but under terms of a separate task.
2023-02-21 14:10:25 +04:00
Alexander Barkov
7f6b648d7d MDEV-30661 UPPER() returns an empty string for U+0251 in uca1400 collations for utf8
String length growth during upper/lower conversion
in Unicode collations depends only on the underlying MY_UNICASE_INFO
used in the collation.

Maintaining a separate member CHARSET_INFO::caseup_multiply and
CHARSET_INFO::casedn_multiply duplicated this information
and caused bugs like this (when MY_UNICASE_INFO and case??_multiply
when out of sync because of incomplete CHARSET_INFO initialization).

Fix:

Changing CHARSET_INFO::caseup_multiply and CHARSET_INFO::casedn_multiply
from members to virtual functions.
The virtual functions in Unicode collations calculate case conversion
growth factors from the MY_UNICASE_INFO. This guarantees that the growth
factors are always in sync with the MY_UNICASE_INFO.
2023-02-17 17:33:27 +04:00
Marko Mäkelä
e71aca8200 Merge 10.9 into 10.10 2022-08-30 13:33:02 +03:00
anson1014
966d22b715
Ensure that source files contain only valid UTF8 encodings (#2188)
Modern software (including text editors, static analysis software,
and web-based code review interfaces) often requires source code files
to be interpretable via a consistent character encoding, with UTF-8 or
ASCII (a strict subset of UTF-8) as the default. Several of the MariaDB
source files contain bytes that are not valid in either the UTF-8 or
ASCII encodings, but instead represent strings encoded in the
ISO-8859-1/Latin-1 or ISO-8859-2/Latin-2 encodings.

These inconsistent encodings may prevent software from correctly
presenting or processing such files. Converting all source files to
valid UTF8 characters will ensure correct handling.

Comments written in Czech were replaced with lightly-corrected
translations from Google Translate. Additionally, comments describing
the proper handling of special characters were changed so that the
comments are now purely UTF8.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.

Co-authored-by: Andrew Hutchings <andrew@linuxjedi.co.uk>
2022-08-30 09:21:40 +01:00
Alexander Barkov
133446828c MDEV-27009 Add UCA-14.0.0 collations
- Added one neutral and 22 tailored (language specific) collations based on
  Unicode Collation Algorithm version 14.0.0.

  Collations were added for Unicode character sets
  utf8mb3, utf8mb4, ucs2, utf16, utf32.

  Every tailoring was added with four accent and case
  sensitivity flag combinations, e.g:

  * utf8mb4_uca1400_swedish_as_cs
  * utf8mb4_uca1400_swedish_as_ci
  * utf8mb4_uca1400_swedish_ai_cs
  * utf8mb4_uca1400_swedish_ai_ci

  and their _nopad_ variants:

  * utf8mb4_uca1400_swedish_nopad_as_cs
  * utf8mb4_uca1400_swedish_nopad_as_ci
  * utf8mb4_uca1400_swedish_nopad_ai_cs
  * utf8mb4_uca1400_swedish_nopad_ai_ci

- Introducing a conception of contextually typed named collations:

  CREATE DATABASE db1 CHARACTER SET utf8mb4;
  CREATE TABLE db1.t1 (a CHAR(10) COLLATE uca1400_as_ci);

  The idea is that there is no a need to specify the character set prefix
  in the new collation names. It's enough to type just the suffix
  "uca1400_as_ci". The character set is taken from the context.

  In the above example script the context character set is utf8mb4.
  So the CREATE TABLE will make a column with the collation
  utf8mb4_uca1400_as_ci.

  Short collations names can be used in any parts of the SQL syntax
  where the COLLATE clause is understood.

- New collations are displayed only one time
  (without character set combinations) by these statements:

     SELECT * FROM INFORMATION_SCHEMA.COLLATIONS;
     SHOW COLLATION;

  For example, all these collations:
  - utf8mb3_uca1400_swedish_as_ci
  - utf8mb4_uca1400_swedish_as_ci
  - ucs2_uca1400_swedish_as_ci
  - utf16_uca1400_swedish_as_ci
  - utf32_uca1400_swedish_as_ci
  have just one entry in INFORMATION_SCHEMA.COLLATIONS and SHOW COLLATION,
  with COLLATION_NAME equal to "uca1400_swedish_as_ci", which is the suffix
  without the character set name:

SELECT COLLATION_NAME FROM INFORMATION_SCHEMA.COLLATIONS
WHERE COLLATION_NAME LIKE '%uca1400_swedish_as_ci';

+-----------------------+
| COLLATION_NAME        |
+-----------------------+
| uca1400_swedish_as_ci |
+-----------------------+

  Note, the behaviour of old collations did not change.
  Non-unicode collations (e.g. latin1_swedish_ci) and
  old UCA-4.0.0 collations (e.g. utf8mb4_unicode_ci)
  are still displayed with the character set prefix, as before.

- The structure of the table INFORMATION_SCHEMA.COLLATIONS was changed.

  The NOT NULL constraint was removed from these columns:
  - CHARACTER_SET_NAME
  - ID
  - IS_DEFAULT
  and from the corresponding columns in SHOW COLLATION.

  For example:

SELECT COLLATION_NAME, CHARACTER_SET_NAME, ID, IS_DEFAULT
FROM INFORMATION_SCHEMA.COLLATIONS
WHERE COLLATION_NAME LIKE '%uca1400_swedish_as_ci';
+-----------------------+--------------------+------+------------+
| COLLATION_NAME        | CHARACTER_SET_NAME | ID   | IS_DEFAULT |
+-----------------------+--------------------+------+------------+
| uca1400_swedish_as_ci | NULL               | NULL | NULL       |
+-----------------------+--------------------+------+------------+

  The NULL value in these columns now means that the collation
  is applicable to multiple character sets.
  The behavioir of old collations did not change.
  Make sure your client programs can handle NULL values in these columns.

- The structure of the table
  INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY was changed.

  Three new NOT NULL columns were added:
  - FULL_COLLATION_NAME
  - ID
  - IS_DEFAULT

  New collations have multiple entries in COLLATION_CHARACTER_SET_APPLICABILITY.
  The column COLLATION_NAME contains the collation name without the character
  set prefix. The column FULL_COLLATION_NAME contains the collation name with
  the character set prefix.

  Old collations have full collation name in both FULL_COLLATION_NAME and
  COLLATION_NAME.

SELECT COLLATION_NAME, FULL_COLLATION_NAME, CHARACTER_SET_NAME, ID, IS_DEFAULT
FROM INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY
WHERE FULL_COLLATION_NAME RLIKE '^(utf8mb4|latin1).*swedish.*ci$';
+-----------------------------+-------------------------------------+--------------------+------+------------+
| COLLATION_NAME              | FULL_COLLATION_NAME                 | CHARACTER_SET_NAME | ID   | IS_DEFAULT |
+-----------------------------+-------------------------------------+--------------------+------+------------+
| latin1_swedish_ci           | latin1_swedish_ci                   | latin1             |    8 | Yes        |
| latin1_swedish_nopad_ci     | latin1_swedish_nopad_ci             | latin1             | 1032 |            |
| utf8mb4_swedish_ci          | utf8mb4_swedish_ci                  | utf8mb4            |  232 |            |
| uca1400_swedish_ai_ci       | utf8mb4_uca1400_swedish_ai_ci       | utf8mb4            | 2368 |            |
| uca1400_swedish_as_ci       | utf8mb4_uca1400_swedish_as_ci       | utf8mb4            | 2370 |            |
| uca1400_swedish_nopad_ai_ci | utf8mb4_uca1400_swedish_nopad_ai_ci | utf8mb4            | 2372 |            |
| uca1400_swedish_nopad_as_ci | utf8mb4_uca1400_swedish_nopad_as_ci | utf8mb4            | 2374 |            |
+-----------------------------+-------------------------------------+--------------------+------+------------+

- Other INFORMATION_SCHEMA queries:

  SELECT COLLATION_NAME FROM INFORMATION_SCHEMA.COLUMNS;
  SELECT COLLATION_NAME FROM INFORMATION_SCHEMA.PARAMETERS;
  SELECT TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES;
  SELECT DEFAULT_COLLATION_NAME FROM INFORMATION_SCHEMA.SCHEMATA;
  SELECT COLLATION_NAME FROM INFORMATION_SCHEMA.ROUTINES;
  SELECT COLLATION_CONNECTION FROM INFORMATION_SCHEMA.EVENTS;
  SELECT DATABASE_COLLATION FROM INFORMATION_SCHEMA.EVENTS;
  SELECT COLLATION_CONNECTION FROM INFORMATION_SCHEMA.ROUTINES;
  SELECT DATABASE_COLLATION FROM INFORMATION_SCHEMA.ROUTINES;
  SELECT COLLATION_CONNECTION FROM INFORMATION_SCHEMA.TRIGGERS;
  SELECT DATABASE_COLLATION FROM INFORMATION_SCHEMA.TRIGGERS;
  SELECT COLLATION_CONNECTION FROM INFORMATION_SCHEMA.VIEWS;

  display full collation names, including character sets prefix,
  for all collations, including new collations.

  Corresponding SHOW commands also display full collation names
  in collation related columns:

  SHOW CREATE TABLE t1;
  SHOW CREATE DATABASE db1;
  SHOW TABLE STATUS;
  SHOW CREATE FUNCTION f1;
  SHOW CREATE PROCEDURE p1;
  SHOW CREATE EVENT ev1;
  SHOW CREATE TRIGGER tr1;
  SHOW CREATE VIEW;

  These INFORMATION_SCHEMA queries and SHOW statements may change in
  the future, to display show collation names.
2022-08-10 15:04:24 +02:00
Oleksandr Byelkin
f5c5f8e41e Merge branch '10.5' into 10.6 2022-02-03 17:01:31 +01:00
Oleksandr Byelkin
cf63eecef4 Merge branch '10.4' into 10.5 2022-02-01 20:33:04 +01:00
Alexander Barkov
b915f79e4e MDEV-25904 New collation functions to compare InnoDB style trimmed NO PAD strings 2022-01-21 12:16:07 +04:00
Alexander Barkov
0d68b0a2d6 MDEV-26669 Add MY_COLLATION_HANDLER functions min_str() and max_str() 2021-09-27 17:10:22 +04:00
Monty
a206658b98 Change CHARSET_INFO character set and collaction names to LEX_CSTRING
This change removed 68 explict strlen() calls from the code.

The following renames was done to ensure we don't use the old names
when merging code from earlier releases, as using the new variables
for print function could result in crashes:
- charset->csname renamed to charset->cs_name
- charset->name renamed to charset->coll_name

Almost everything where mechanical changes except:
- Changed to use the new Protocol::store(LEX_CSTRING..) when possible
- Changed to use field->store(LEX_CSTRING*, CHARSET_INFO*) when possible
- Changed to use String->append(LEX_CSTRING&) when possible

Other things:
- There where compiler issues with ensuring that all character set names
  points to the same string: gcc doesn't allow one to use integer constants
  when defining global structures (constant char * pointers works fine).
  To get around this, I declared defines for each character set name
  length.
2021-05-19 22:54:07 +02:00
Monty
dbcd3384e0 MDEV-7947 strcmp() takes 0.37% in OLTP RO
This patch ensures that all identical character sets shares the same
cs->csname.
This allows us to replace strcmp() in my_charset_same() with comparisons
of pointers. This fixes a long standing performance issue that could cause
as strcmp() for every item sent trough the protocol class to the end user.

One consequence of this patch is that we don't allow one to add a character
definition in the Index.xml file that changes the csname of an existing
character set. This is by design as changing character set names of existing
ones is extremely dangerous, especially as some storage engines just records
character set numbers.

As we now have a hash over character set's csname, we can in the future
use that for faster access to a specific character set. This could be done
by changing the hash to non unique and use the hash to find the next
character set with same csname.
2020-07-23 10:54:33 +03:00
Alexander Barkov
cfe5ee90c8 MDEV-22043 Special character leads to assertion in my_wc_to_printable_generic on 10.5.2 (debug)
The code did not take into account that:
- U+005C (backslash) can occupy more than mbminlen characters (e.g. in sjis)
- Some character sets do not have a code for U+005C (e.g. swe7)

Adding a new function my_wc_to_printable into MY_CHARSET_HANDLER to
cover all special cases easier.
2020-05-09 16:01:30 +04:00
Marko Mäkelä
26a14ee130 Merge 10.1 into 10.2 2019-05-13 17:54:04 +03:00
Vicențiu Ciorbaru
cb248f8806 Merge branch '5.5' into 10.1 2019-05-11 22:19:05 +03:00
Vicențiu Ciorbaru
5543b75550 Update FSF Address
* Update wrong zip-code
2019-05-11 21:29:06 +03:00
Alexander Barkov
5058ced5df MDEV-7769 MY_CHARSET_INFO refactoring# On branch 10.2
Part 3 (final): removing MY_CHARSET_HANDLER::well_formed_len().
2016-10-10 14:36:09 +04:00
Alexander Barkov
ee19806b8e MDEV-9711 NO PAD collations
Based on the patch from Daniil Medvedev (a Google Summer of Code task)
2016-09-06 12:50:02 +04:00
Alexander Barkov
e7ff281d2e MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring 2016-05-17 15:27:10 +04:00
Alexander Barkov
1d73005bf3 MDEV-8360 Clean-up CHARSET_INFO: strnncollsp: diff_if_only_endspace_difference
- Removing the "diff_if_only_endspace_difference" argument from
  MY_COLLATION_HANDLER::strnncollsp(), my_strnncollsp_simple(),
  as well as in the function template MY_FUNCTION_NAME(strnncollsp)
  in strcoll.ic

- Removing the "diff_if_only_space_different" from ha_compare_text(),
  hp_rec_key_cmp().

- Adding a new function my_strnncollsp_padspace_bin() and reusing
  it instead of duplicate code pieces in my_strnncollsp_8bit_bin(),
  my_strnncollsp_latin1_de(), my_strnncollsp_tis620(),
  my_strnncollsp_utf8_cs().

- Adding more tests for better coverage of the trailing space handling.

- Removing the unused definition of HA_END_SPACE_ARE_EQUAL
2016-03-31 11:04:48 +04:00
Alexander Barkov
e09299511e MDEV-9665 Remove cs->cset->ismbchar()
Using a more powerfull cs->cset->charlen() instead.
2016-03-16 10:55:12 +04:00
Alexander Barkov
3ba2a958be MDEV-8694 Wrong result for SELECT..WHERE a NOT LIKE 'a ' AND a='a'
Note, the patch for MDEV-8661 unintentionally fixed MDEV-8694 as well,
as a side effect. Adding a real clear fix: implementing
Item_func_like::propagate_equal_fields() with comments.
2015-08-28 17:03:09 +04:00
Alexander Barkov
78b80cb6ba Adding MY_CHARSET_HANDLER::native_to_mb().
This is a pre-requisite patch for:
- MDEV-8433 Make field<'broken-string' use indexes
- MDEV-8625 Bad result set with ignorable characters when using a prefix key
- MDEV-8626 Bad result set with contractions when using a prefix key
2015-08-14 18:34:41 +04:00
Alexander Barkov
197afb413f MDEV-6566 Different INSERT behaviour on bad bytes with and without character set conversion 2015-03-13 16:51:36 +04:00
Alexander Barkov
b1b6101af2 A preparatory patch for MDEV-6566.
Adding a new virtual function MY_CHARSET_HANDLER::copy_abort().
Moving character set specific code into the correspoding implementations
(for simple, multi-byte and mbmaxlen>1 character sets).
2015-03-02 18:24:22 +04:00
Michael Widenius
c4f5326bb7 MDEV-6255 DUPLICATE KEY Errors on SELECT .. GROUP BY that uses temporary and filesort.
The problem was that my_hash_sort didn't properly delete end-space characters properly, so strings that should compare
identically was seen as different strings.  (Space was handled correctly, but not NBSP)
This caused duplicate key errors when a heap table was converted to Aria as part of overflow in group by.

Fixed by removing all characters that compares as end space when creating a hash.

Other things:
- Fixed that --sorted_results also works for errors in mysqltest.
- Speed up hash by not comparing strings that has different hash.
- Speed up many my_hash_sort functions by using registers to calculate hash instead of pointers.
  This was previously done for some functions, but not for all.
- Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.







client/mysqltest.cc:
  Fixed that --sorted_results also works for error messages.
mysql-test/r/ctype_partitions.result:
  New test to ensure that partitions on hash works
mysql-test/suite/multi_source/gtid.result:
  Updated result
mysql-test/suite/multi_source/gtid.test:
  Test that --sorted_result works for error messages
mysql-test/suite/multi_source/gtid_ignore_duplicates.result:
  Updated result
mysql-test/suite/multi_source/gtid_ignore_duplicates.test:
  Updated result
mysql-test/suite/multi_source/load_data.result:
  Updated result
mysql-test/suite/multi_source/load_data.test:
  Updated result
mysql-test/t/ctype_partitions.test:
  New test to ensure that partitions on hash works
storage/heap/hp_write.c:
  Speed up hash by not comparing strings that has different hash.
storage/maria/ma_check.c:
  Extra debug
strings/ctype-bin.c:
  Use macro for hash function
strings/ctype-latin1.c:
  Use macro for hash function
  Use registers to calculate hash (speedup)
strings/ctype-mb.c:
  Use macro for hash function
  Use registers to calculate hash (speedup)
strings/ctype-simple.c:
  Use macro for hash function
  Use same variable names as in other my_hash_sort functions.
  Update my_hash_sort_simple() to properly remove end space (patch by Bar)
strings/ctype-uca.c:
  Ignore duplicated space inside strings and end space in my_hash_sort_uca(). This fixed MDEV-6255
  Use macro for hash function
  Use registers to calculate hash (speedup)
strings/ctype-ucs2.c:
  Use macro for hash function
  Use registers to calculate hash (speedup)
strings/ctype-utf8.c:
  Use macro for hash function
  Use registers to calculate hash (speedup)
strings/strings_def.h:
  Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.
2014-09-11 22:42:35 +03:00
Alexander Barkov
426d246f5b MDEV-5163 Merge WEIGHT_STRING function from MySQL-5.6 2013-10-23 20:25:52 +04:00
Alexander Barkov
0b6c4bb34f MDEV-4928 Merge collation customization improvements
Merging the following MySQL-5.6 changes:
- WL#5624: Collation customization improvements
  http://dev.mysql.com/worklog/task/?id=5624

- WL#4013: Unicode german2 collation
  http://dev.mysql.com/worklog/task/?id=4013

- Bug#62429 XML: ExtractValue, UpdateXML max arg length 127 chars
  http://bugs.mysql.com/bug.php?id=62429
  (required by WL#5624)
2013-10-02 15:04:07 +04:00
Sergei Golubchik
a9035be5b7 10.0-base merge 2013-04-15 15:09:22 +02:00
Alexander Barkov
d1e162e011 Merging utf16le from MySQL-5.6
added:
  mysql-test/include/ctype_heap.inc
  mysql-test/include/ctype_strtoll10.inc
  mysql-test/r/ctype_utf16le.result
  mysql-test/t/ctype_utf16le.test
modified:
  cmake/character_sets.cmake
  include/m_ctype.h
  mysql-test/r/ctype_ucs.result
  mysql-test/r/ctype_utf16.result
  mysql-test/r/ctype_utf32.result
  mysql-test/suite/funcs_1/r/innodb_func_view.result
  mysql-test/suite/funcs_1/r/memory_func_view.result
  mysql-test/suite/funcs_1/r/myisam_func_view.result
  mysql-test/suite/sys_vars/r/character_set_client_basic.result
  mysql-test/suite/sys_vars/r/character_set_connection_basic.result
  mysql-test/suite/sys_vars/r/character_set_database_basic.result
  mysql-test/suite/sys_vars/r/character_set_filesystem_basic.result
  mysql-test/suite/sys_vars/r/character_set_results_basic.result
  mysql-test/t/ctype_ucs.test
  mysql-test/t/ctype_utf16.test
  mysql-test/t/ctype_utf32.test
  mysys/charset-def.c
  sql/item_func.cc
  sql/sys_vars.cc
  strings/ctype-latin1.c
  strings/ctype-ucs2.c
2013-03-28 17:19:09 +04:00
Alexander Barkov
f5c3c2855d Performance improvements in "from latin1" and "to utf8" conversion.
Mini-benchmarking demonstrates up to 10% improvement in latin1->utf8
conversion.

modified:
  @ strings/ctype-latin1.c
  redundant test in ctype-latin1.c removed

  @ strings/ctype-utf8.c
  my_uni_utf8 rewritten in a more efficient way
2013-03-12 18:33:19 +04:00
Sergei Golubchik
4f435bddfd 5.3 merge 2012-01-13 15:50:02 +01:00
Michael Widenius
6920457142 Merge with MariaDB 5.1 2011-11-24 18:48:58 +02:00
Michael Widenius
a8d03ab235 Initail merge with MySQL 5.1 (XtraDB still needs to be merged)
Fixed up copyright messages.
2011-11-21 19:13:14 +02:00
Sergei Golubchik
0e007344ea mysql-5.5.18 merge 2011-11-03 19:17:05 +01:00
Sergei Golubchik
76f0b94bb0 merge with 5.3
sql/sql_insert.cc:
  CREATE ... IF NOT EXISTS may do nothing, but
  it is still not a failure. don't forget to my_ok it.
  ******
  CREATE ... IF NOT EXISTS may do nothing, but
  it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
  small cleanup
  ******
  small cleanup
2011-10-19 21:45:18 +02:00
Sergei Golubchik
9809f05199 5.5-merge 2011-07-02 22:08:51 +02:00
Kent Boortz
68f00a5686 Updated/added copyright headers 2011-06-30 17:37:13 +02:00
Kent Boortz
02e07e3b51 Updated/added copyright headers 2011-06-30 17:46:53 +02:00
Michael Widenius
1be5462d59 Merge with MariaDB 5.1 2011-05-03 19:10:10 +03:00
Michael Widenius
e415ba0fb2 Merge with MySQL 5.1.57/58
Moved some BSD string functions from Unireg
2011-05-02 20:58:45 +03:00
Michael Widenius
3358cdd504 Merge with 5.1 to get in changes from MySQL 5.1.55 2011-02-28 19:39:30 +02:00
Michael Widenius
785695e7c3 Flush DBUG log in case of DBUG_ASSERT()
Added strings_def.h into strings library to be able to have a DBUG_ASSERT() version without _db_flush() call (as strings.a should not depend on dbug.a)
Remove include of m_string.h in all string files (as it's included by string_def.h).
Fixed include order.
Changed "m_ctype.h" -> <m_ctype.h>

 

include/my_dbug.h:
  Flush DBUG log in case of DBUG_ASSERT()
strings/bchange.c:
  Include strings_def.h
strings/bcmp.c:
  Include strings_def.h
strings/bfill.c:
  Include strings_def.h
strings/bmove.c:
  Include strings_def.h
strings/bmove512.c:
  Include strings_def.h
strings/bmove_upp.c:
  Include strings_def.h
strings/conf_to_src.c:
  Include strings_def.h
  Fixed copyright
strings/ctype-big5.c:
  Include strings_def.h
strings/ctype-bin.c:
  Include strings_def.h
strings/ctype-cp932.c:
  Include strings_def.h
strings/ctype-czech.c:
  Include strings_def.h
strings/ctype-euc_kr.c:
  Include strings_def.h
strings/ctype-eucjpms.c:
  Include strings_def.h
strings/ctype-extra.c:
  Include strings_def.h
strings/ctype-gbk.c:
  Include strings_def.h
strings/ctype-latin1.c:
  Include strings_def.h
strings/ctype-mb.c:
  Include strings_def.h
strings/ctype-simple.c:
  Include strings_def.h
strings/ctype-sjis.c:
  Include strings_def.h
strings/ctype-tis620.c:
  Include strings_def.h
strings/ctype-uca.c:
  Include strings_def.h
strings/ctype-ucs2.c:
  Include strings_def.h
strings/ctype-ujis.c:
  Include strings_def.h
strings/ctype-utf8.c:
  Include strings_def.h
strings/ctype-win1250ch.c:
  Include strings_def.h
strings/ctype.c:
  Include strings_def.h
strings/decimal.c:
  Include strings_def.h
strings/do_ctype.c:
  Include strings_def.h
strings/int2str.c:
  Include strings_def.h
strings/is_prefix.c:
  Include strings_def.h
strings/llstr.c:
  Include strings_def.h
strings/longlong2str.c:
  Include strings_def.h
strings/longlong2str_asm.c:
  Include strings_def.h
strings/my_strchr.c:
  Include strings_def.h
strings/my_strtoll10.c:
  Include strings_def.h
strings/my_vsnprintf.c:
  Include strings_def.h
strings/r_strinstr.c:
  Include strings_def.h
strings/str2int.c:
  Include strings_def.h
strings/str_alloc.c:
  Include strings_def.h
strings/str_test.c:
  Include strings_def.h
  Fixed compiler warnings
strings/strappend.c:
  Include strings_def.h
strings/strcend.c:
  Include strings_def.h
strings/strcont.c:
  Include strings_def.h
strings/strend.c:
  Include strings_def.h
strings/strfill.c:
  Include strings_def.h
strings/strinstr.c:
  Include strings_def.h
strings/strmake.c:
  Include strings_def.h
strings/strmov.c:
  Include strings_def.h
strings/strmov_overlapp.c:
  Include strings_def.h
strings/strnlen.c:
  Include strings_def.h
strings/strnmov.c:
  Include strings_def.h
strings/strstr.c:
  Include strings_def.h
strings/strto.c:
  Include strings_def.h
strings/strtod.c:
  Include strings_def.h
strings/strtol.c:
  Include strings_def.h
strings/strtoll.c:
  Include strings_def.h
strings/strtoul.c:
  Include strings_def.h
strings/strtoull.c:
  Include strings_def.h
strings/strxmov.c:
  Include strings_def.h
strings/strxnmov.c:
  Include strings_def.h
strings/uctypedump.c:
  Include strings_def.h
  Fixed compiler warnings
  Removed double include of m_ctype.h
strings/udiv.c:
  Include strings_def.h
strings/xml.c:
  Include strings_def.h
2011-01-30 12:41:44 +02:00
Alexander Barkov
435289acd4 Updating Copyright information 2011-01-19 16:17:52 +03:00
Alexander Barkov
dfb7930b33 Merging Copyright update from 5.1 2011-01-19 16:31:17 +03:00
Sergei Golubchik
65ca700def merge.
checkpoint.
does not compile.
2010-11-25 18:17:28 +01:00
Michael Widenius
f83113df07 Applied Antony T Curtis patch for declaring many CHARSET objects as const
Removed compiler warnings

extra/libevent/epoll.c:
  Removed compiler warnings
extra/libevent/evbuffer.c:
  Removed compiler warnings
extra/libevent/event.c:
  Removed compiler warnings
extra/libevent/select.c:
  Removed compiler warnings
extra/libevent/signal.c:
  Removed compiler warnings
include/m_ctype.h:
  Define CHARSET_INFO, MY_CHARSET_HANDLER, MY_COLLATION_HANDLER, MY_UNICASE_INFO, MY_UNI_CTYPE and MY_UNI_IDX as const structures.
  Declare that pointers point to const data
include/m_string.h:
  Declare that pointers point to const data
include/my_sys.h:
  Redefine variables and function prototypes
include/mysql.h:
  Declare charset as const
include/mysql.h.pp:
  Declare charset as const
include/mysql/plugin.h:
  Declare charset as const
include/mysql/plugin.h.pp:
  Declare charset as const
mysys/charset-def.c:
  Charset can't be of type CHARSET_INFO as they are changed when they are initialized.
mysys/charset.c:
  Functions that change CHARSET_INFO must use 'struct charset_info_st'
  Add temporary variables to not have to change all_charsets[] (Which now is const)
sql-common/client.c:
  Added cast to const
sql/item_cmpfunc.h:
  Added cast to avoid compiler error.
sql/sql_class.cc:
  Added cast to const
sql/sql_lex.cc:
  Added cast to const
storage/maria/ma_ft_boolean_search.c:
  Added cast to avoid compiler error.
storage/maria/ma_ft_parser.c:
  Added cast to avoid compiler error.
storage/maria/ma_search.c:
  Added cast to const
storage/myisam/ft_boolean_search.c:
  Added cast to avoid compiler error
storage/myisam/ft_parser.c:
  Added cast to avoid compiler error
storage/myisam/mi_search.c:
  Added cast to const
storage/pbxt/src/datadic_xt.cc:
  Added cast to const
storage/pbxt/src/ha_pbxt.cc:
  Added cast to const
  Removed compiler warning by changing prototype of XTThreadPtr()
storage/pbxt/src/myxt_xt.h:
  Character sets should be const
storage/pbxt/src/xt_defs.h:
  Character sets should be const
storage/xtradb/btr/btr0cur.c:
  Removed compiler warning
strings/conf_to_src.c:
  Added const
  Functions that change CHARSET_INFO must use 'struct charset_info_st'
strings/ctype-big5.c:
  Made arrays const
strings/ctype-bin.c:
  Made arrays const
strings/ctype-cp932.c:
  Made arrays const
strings/ctype-czech.c:
  Made arrays const
strings/ctype-euc_kr.c:
  Made arrays const
strings/ctype-eucjpms.c:
  Made arrays const
strings/ctype-extra.c:
  Made arrays const
strings/ctype-gb2312.c:
  Made arrays const
strings/ctype-gbk.c:
  Made arrays const
strings/ctype-latin1.c:
  Made arrays const
strings/ctype-mb.c:
  Made arrays const
strings/ctype-simple.c:
  Made arrays const
strings/ctype-sjis.c:
  Made arrays const
strings/ctype-tis620.c:
  Made arrays const
strings/ctype-uca.c:
  Made arrays const
strings/ctype-ucs2.c:
  Made arrays const
strings/ctype-ujis.c:
  Made arrays const
strings/ctype-utf8.c:
  Made arrays const
strings/ctype-win1250ch.c:
  Made arrays const
strings/ctype.c:
  Made arrays const
  Added cast to const
  Functions that change CHARSET_INFO must use 'struct charset_info_st'
strings/int2str.c:
  Added cast to const
2010-01-06 21:20:16 +02:00