mariadb/strings
Alexander Barkov 1748b02ec6 MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5)
Fixing the code adding MySQL _0900_ collations as _uca1400_ aliases
not to perform deep initialization of the corresponding _uca1400_
collations.

Only basic initialization is now performed which allows to watch
these collations (both _0900_ and _uca1400_) in queries to
INFORMATION_SCHEMA tables COLLATIONS and
COLLATION_CHARACTER_SET_APPLICABILITY,
as well as in SHOW COLLATION statements.

Deep initialization is now performed only when a collation
(either the _0900_ alias or the corresponding  _uca1400_ collation)
is used for the very first time after the server startup.

Refactoring was done to maintain the code easier:
- most of the _uca1400_ code was moved from ctype-uca.c
  to a new file ctype-uca1400.c
- most of the _0900_ code was moved from type-uca.c
  to a new file ctype-uca0900.c

Change details:

- The original function add_alias_for_collation() added by the patch for
   "MDEV-20912 Add support for utf8mb4_0900_* collations in MariaDB Server"
  was removed from mysys/charset.c, as it had two two problems:

  a. it forced deep initialization of the _uca1400_ collations
     when adding _0900_ aliases for them at the server startup
     (the main reported problem)

  b. it introduced cyclic dependency between /mysys and /strings
     - /mysys/charset-def.c depended on /strings/ctype-uca.c
     - /strings/ctype-uca.c depended on /mysys/charset.c

  The code from add_alias_for_collation() was split into separate functions.
  Cyclic dependency was removed. `#include <my_sys.h>` was removed
  from /strings/ctype-uca.c. Collations are now added using a callback
  function MY_CHARSET_LOADED::add_collation, like it is done for
  user collations defined in Index.xml. The code in /mysys sets
  MY_CHARSET_LOADED::add_collation to add_compiled_collation().

- The function compare_collations() was removed.
  A new virtual function was added into my_collation_handler_st instead:

    my_bool (*eq_collation)(CHARSET_INFO *self, CHARSET_INFO *other);

  because it is the collation handler who knows how to detect equal
  collations by comparing only some of CHARSET_INFO members without
  their deep initialization.

  Three implementations were added:
  - my_ci_eq_collation_uca() for UCA collations, it compares
    _0900_ collations as equal to their corresponding _uca1400_ collations.
  - my_ci_eq_collation_utf8mb4_bin(), it compares
    utf8mb4_nopad_bin and utf8mb4_0900_bin as equal.
  - my_ci_eq_collation_generic() - the default implementation,
    which compares all collations as not equal.

  A C++ wrapper CHARSET_INFO::eq_collations() was added.
  The code in /sql was changes to use the wrapper instead of
  the former calls for the removed function compare_collations().

- A part of add_alias_for_collation() was moved into a new function
  my_ci_alloc(). It allocates a memory for a new charset_info_st
  instance together with the collation name and the comment using a single
  MY_CHARSET_LOADER::once_alloc call, which normally points to my_once_alloc().

- A part of add_alias_for_collation() was moved into a new function
  my_ci_make_comment_for_alias(). It makes an "Alias for xxx" string,
  e.g. "Alias for utf8mb4_uca1400_swedish_ai_ci" in case of
  utf8mb4_sv_0900_ai_ci.

- A part of the code in create_tailoring() was moved to
  a new function my_uca1400_collation_get_initialized_shared_uca(),
  to reuse the code between _uca1400_ and _0900_ collations.

- A new function my_collation_id_is_mysql_uca0900() was added
  in addition to my_collation_id_is_mysql_uca1400().

- Functions to build collation names were added:
   my_uca0900_collation_build_name()
   my_uca1400_collation_build_name()

- A shared function function was added:

  my_bool
  my_uca1400_collation_alloc_and_init(MY_CHARSET_LOADER *loader,
                                      LEX_CSTRING name,
                                      LEX_CSTRING comment,
                                      const uca_collation_def_param_t *param,
                                      uint id)

  It's reused to add _uca1400_ and _0900_ collations, with basic
  initialization (without deep initialization).

- The function add_compiled_collation() changed its return type from
  void to int, to make it compatible with MY_CHARSET_LOADER::add_collation.

- Functions mysql_uca0900_collation_definition_add(),
  mysql_uca0900_utf8mb4_collation_definitions_add(),
  mysql_utf8mb4_0900_bin_add() were added into ctype-uca0900.c.
  They get MY_CHARSET_LOADER as a parameter.

- Functions my_uca1400_collation_definition_add(),
  my_uca1400_collation_definitions_add() were moved from
  charset-def.c to strings/ctype-uca1400.c.
  The latter now accepts MY_CHARSET_LOADER as the first parameter
  instead of initializing a MY_CHARSET_LOADER inside.

- init_compiled_charsets() now initializes a MY_CHARSET_LOADER
  variable and passes it to all functions adding collations:
  - mysql_utf8mb4_0900_collation_definitions_add()
  - mysql_uca0900_utf8mb4_collation_definitions_add()
  - mysql_utf8mb4_0900_bin_add()

- A new structure was added into ctype-uca.h:

  typedef struct uca_collation_def_param
  {
    my_cs_encoding_t cs_id;
    uint tailoring_id;
    uint nopad_flags;
    uint level_flags;
  } uca_collation_def_param_t;

  It simplifies reusing the code for _uca1400_ and _0900_ collations.

- The definition of MY_UCA1400_COLLATION_DEFINITION was
  moved from ctype-uca.c to ctype-uca1400.h, to reuse
  the code for _uca1400_ and _0900_ collations.

- The definitions of "MY_UCA_INFO my_uca_v1400" and
  "MY_UCA_INFO my_uca1400_info_tailored[][]" were moved from
  ctype-uca.c to ctype-uca1400.c.

- The definitions/declarations of:
  - mysql_0900_collation_start,
  - struct mysql_0900_to_mariadb_1400_mapping
  - mysql_0900_to_mariadb_1400_mapping
  - mysql_utf8mb4_0900_collation_definitions_add()
  were moved from ctype-uca.c to ctype-uca0900.c

- Functions
  my_uca1400_make_builtin_collation_id()
  my_uca1400_collation_definition_init()
  my_uca1400_collation_id_uca400_compat()
  my_ci_get_collation_name_uca1400_context()
  were moved from ctype-uca.c to ctype-uca1400.c and ctype-uca1400.h

- A part of my_uca1400_collation_definition_init()
  was moved into my_uca1400_collation_source(),
  to make functions smaller.
2025-03-31 18:17:26 +04:00
..
bchange.c
bmove_upp.c
CHARSET_INFO.txt MDEV-19897 Rename source code variable names from utf8 to utf8mb3 2019-06-28 12:37:04 +04:00
CMakeLists.txt MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
conf_to_src.c MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-ascii.h MDEV-26572 Improve simple multibyte collation performance on the ASCII range 2021-09-13 08:03:25 +04:00
ctype-big5.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-bin.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-cp932.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-czech.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-euc_kr.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-eucjpms.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-extra.c MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-gb2312.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-gbk.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-latin1.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-mb.c Merge 10.11 into 11.4 2025-03-28 13:55:21 +02:00
ctype-mb.h MDEV-26669 Add MY_COLLATION_HANDLER functions min_str() and max_str() 2021-09-27 17:10:22 +04:00
ctype-mb.inl Merge branch '10.4' into 10.5 2022-02-01 20:33:04 +01:00
ctype-simple.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-simple.h MDEV-26669 Add MY_COLLATION_HANDLER functions min_str() and max_str() 2021-09-27 17:10:22 +04:00
ctype-sjis.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-tis620.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca-scanner_next.inl A follow-up patch MDEV-27266 Improve UCA collation performance for utf8mb3 and utf8mb4 2022-09-02 13:23:24 +04:00
ctype-uca.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca.h MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca.inl MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca0900.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca0900.h MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca1400.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-uca1400.h MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-ucs2.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-ucs2.h
ctype-ujis.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-unicode300-casefold-tr.h MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-unicode300-casefold.h MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-unicode300-general_ci.h MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-unicode300-general_mysql500_ci.h MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-unicode520-casefold.h MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
ctype-unicode1400-casefold-tr.h MDEV-30577 Case folding for uca1400 collations is not up to date 2023-04-18 11:31:05 +04:00
ctype-unicode1400-casefold.h MDEV-30577 Case folding for uca1400 collations is not up to date 2023-04-18 11:31:05 +04:00
ctype-unidata.c MDEV-30577 Case folding for uca1400 collations is not up to date 2023-04-18 11:31:05 +04:00
ctype-unidata.h MDEV-30577 Case folding for uca1400 collations is not up to date 2023-04-18 11:31:05 +04:00
ctype-utf8.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype-utf8.h
ctype-utf16.h
ctype-utf32.h
ctype-win1250ch.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
ctype.c MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
decimal.c Merge 10.5 into 10.6 2022-03-29 16:16:21 +03:00
do_ctype.c Update FSF Address 2019-05-11 21:29:06 +03:00
dtoa.c MDEV-28345 ASAN: use-after-poison or unknown-crash in my_strtod_int from charset_info_st::strntod or test_if_number 2024-07-17 12:17:27 +04:00
dump_map.c Update FSF Address 2019-05-11 21:29:06 +03:00
int2str.c MDEV-30879 Add support for up to BASE 62 to CONV() 2024-01-17 15:24:26 +00:00
is_prefix.c
json_lib.c Merge branch '10.6' into 10.11 2024-02-01 18:36:14 +01:00
json_normalize.c Merge 10.6 into 10.11 2024-11-29 13:43:17 +02:00
latin2.def
llstr.c
longlong2str.c MDEV-30879 Add support for up to BASE 62 to CONV() 2024-01-17 15:24:26 +00:00
my_strchr.c MDEV-21581 Helper functions and methods for CHARSET_INFO 2020-01-28 12:29:23 +04:00
my_strtoll10.c MDEV-31221 UBSAN runtime error: negation of -9223372036854775808 cannot be represented in type 'long long int' in my_strtoll10_utf32 2024-09-20 13:04:57 +04:00
my_vsnprintf.c Merge branch '10.5' into '10.6' 2024-12-18 05:09:23 +01:00
README
str2int.c MDEV-30879 Add support for up to BASE 62 to CONV() 2024-01-17 15:24:26 +00:00
strappend.c
strcend.c
strcoll.inl Merge branch '10.6' into 10.11 2025-01-30 11:55:13 +01:00
strcont.c
strend.c
strfill.c
string.doc MDEV-30879 Add support for up to BASE 62 to CONV() 2024-01-17 15:24:26 +00:00
strings_def.h MDEV-36213 Doubled memory usage (11.4.4 <-> 11.4.5) 2025-03-31 18:17:26 +04:00
strmake.c Update FSF Address 2019-05-11 21:29:06 +03:00
strmov.c
strmov_overlapp.c Update FSF address 2019-05-10 20:52:00 +03:00
strnlen.c
strnmov.c
strxmov.c
strxnmov.c
t_ctype.h Update FSF Address 2019-05-11 21:29:06 +03:00
uca-dump.c Fixed some compiler issues appeared after d7ffb7c3dd 2022-08-16 09:26:27 -07:00
uctypedump.c Minimize unsafe C functions usage 2023-03-08 10:36:25 +00:00
unidata-dump.c MDEV-31071 Refactor case folding data types in Unicode collations 2023-04-18 11:29:25 +04:00
utr11-dump.c Update FSF Address 2019-05-11 21:29:06 +03:00
xml.c Merge remote-tracking branch '10.4' into 10.5 2023-03-31 21:32:41 +02:00

File   : README
Author : Richard A. O'Keefe.
Updated: 30 April 1984
Purpose: Explain the new strings package.

    The UNIX string libraries (described in the string(3) manual page)
differ from UNIX to UNIX (e.g. strtok is not in V7 or 4.1bsd).  Worse,
the sources are not in the public domain, so that if there is a string
routine which is nearly what you want but not quite you can't  take  a
copy  and  modify it.  And of course C programmers on non-UNIX systems
are at the mercy of their supplier.

    This package was designed to let me do reasonable things with  C's
strings  whatever UNIX (V7, PaNiX, UX63, 4.1bsd) I happen to be using.
Everything in the System III manual is here and does just what the  S3
manual  says  it does.  There are also lots of new goodies.  I'm sorry
about the names, but the routines do have to work  on  asphyxiated-at-
birth  systems  which  truncate identifiers.  The convention is that a
routine is called
 str [n] [c] <operation>
If there is an "n", it means that the function takes an (int) "length"
argument, which bounds the number of characters to be moved or  looked
at.  If the function has a "set" argument, a "c" in the name indicates
that  the complement of the set is used.  Functions or variables whose
names start with _ are support routines which aren't really meant  for
general  use.  I don't know what the "p" is doing in "strpbrk", but it
is there in the S3 manual so it's here too.  "istrtok" does not follow
this rule, but with 7 letters what can you do?

    I have included new versions of atoi(3) and atol(3) as well.  They
use a new primitive str2int, which takes a pair of bounds and a radix,
and does much more thorough checking than the normal atoi and atol do.
The result returned by atoi & atol is valid if and only if errno == 0.
There is also an output conversion routine int2str, with itoa and ltoa
as interface macros.  Only after writing int2str did I notice that the
str2int routine has no provision for unsigned numbers.  On reflection,
I don't greatly care.   I'm afraid that int2str may depend on your "C"
compiler in unexpected ways.  Do check the code with -S.

    Several of these routines have "asm" inclusions conditional on the
VaxAsm option.  These insertions can make the routines which have them
quite a bit faster, but there is a snag.  The VAX architects, for some
reason best known to themselves and their therapists, decided that all
"strings" were shorter than 2^16 bytes.  Even when the length operands
are in 32-bit registers, only 16 bits count.  So the "asm" versions do
not work for long strings.  If you can guarantee that all your strings
will be short, define VaxAsm in the makefile, but in general, and when
using other machines, do not define it.

    To use this library, you need the "strings.a" library file and the
"strings.h" and "ctypes.h" header files.  The other header  files  are
for compiling the library itself, though if you are hacking extensions
you  may  find  them useful.  General users really shouldn't see them.
I've defined a few macros I find useful in "strings.h"; if you have no
need for "index", "rindex", "streql", and "beql", just edit them  out.
On the 4.1bsd system I am using declaring all these functions 'extern'
does not mean that they will all be loaded; but only the ones you use.
When using lesser systems you may find it necessary to break strings.h
up, or you could get by with just adding "extern" declarations for the
functions you want as you need them.  Many of these functions have the
same names as functions in the "standard C library", by design as this
is a replacement/reimplementation of part of that library.  So you may
have to talk the loader into loading this library first.   Again, I've
found no problems on 4.1bsd.

    You may wonder at my failure to provide manual pages for this code.
For the things in V7, 4.?, or SIII, you should be able to use whichever
manual page came with that system,  and anything I might write would be
so like it as to raise suspicions of violating AT&T copyrights.  In the
sources you will find comments which provide far more documentation for
these routines than AT&T ever provided for their strings stuff,  I just
don't happen to have put it in nroff -man form.   Had I done so, the .3
files would have outbulked the .c files!

    These files are in the public domain.  This includes getopt.c, which
is the work of Henry Spencer, University of Toronto Zoology, who says of
it "None of this software is derived from Bell software. I had no access
to the source for Bell's versions at the time I wrote it.  This software
is hereby explicitly placed in the public domain.  It may  be  used  for
any purpose on any machine by anyone." I would greatly prefer it if *my*
material received no military use.