mirror of
https://github.com/MariaDB/server.git
synced 2025-01-15 19:42:28 +01:00
c4f5326bb7
The problem was that my_hash_sort didn't properly delete end-space characters properly, so strings that should compare identically was seen as different strings. (Space was handled correctly, but not NBSP) This caused duplicate key errors when a heap table was converted to Aria as part of overflow in group by. Fixed by removing all characters that compares as end space when creating a hash. Other things: - Fixed that --sorted_results also works for errors in mysqltest. - Speed up hash by not comparing strings that has different hash. - Speed up many my_hash_sort functions by using registers to calculate hash instead of pointers. This was previously done for some functions, but not for all. - Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions. client/mysqltest.cc: Fixed that --sorted_results also works for error messages. mysql-test/r/ctype_partitions.result: New test to ensure that partitions on hash works mysql-test/suite/multi_source/gtid.result: Updated result mysql-test/suite/multi_source/gtid.test: Test that --sorted_result works for error messages mysql-test/suite/multi_source/gtid_ignore_duplicates.result: Updated result mysql-test/suite/multi_source/gtid_ignore_duplicates.test: Updated result mysql-test/suite/multi_source/load_data.result: Updated result mysql-test/suite/multi_source/load_data.test: Updated result mysql-test/t/ctype_partitions.test: New test to ensure that partitions on hash works storage/heap/hp_write.c: Speed up hash by not comparing strings that has different hash. storage/maria/ma_check.c: Extra debug strings/ctype-bin.c: Use macro for hash function strings/ctype-latin1.c: Use macro for hash function Use registers to calculate hash (speedup) strings/ctype-mb.c: Use macro for hash function Use registers to calculate hash (speedup) strings/ctype-simple.c: Use macro for hash function Use same variable names as in other my_hash_sort functions. Update my_hash_sort_simple() to properly remove end space (patch by Bar) strings/ctype-uca.c: Ignore duplicated space inside strings and end space in my_hash_sort_uca(). This fixed MDEV-6255 Use macro for hash function Use registers to calculate hash (speedup) strings/ctype-ucs2.c: Use macro for hash function Use registers to calculate hash (speedup) strings/ctype-utf8.c: Use macro for hash function Use registers to calculate hash (speedup) strings/strings_def.h: Made a macro of the hash function, to simplify code and to be able to experiment with new hash functions.
119 lines
4.3 KiB
C
119 lines
4.3 KiB
C
#ifndef STRINGS_DEF_INCLUDED
|
|
#define STRINGS_DEF_INCLUDED
|
|
/* Copyright (C) 2011 Monty Program Ab
|
|
|
|
This program is free software; you can redistribute it and/or modify
|
|
it under the terms of the GNU General Public License as published by
|
|
the Free Software Foundation; version 2 of the License.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU General Public License for more details.
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
along with this program; if not, write to the Free Software
|
|
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
|
|
|
|
/* This file is to be include first in all files in the string directory */
|
|
|
|
#include <my_global.h> /* Define standar vars */
|
|
#include "m_string.h" /* Exernal defintions of string functions */
|
|
|
|
/*
|
|
We can't use the original DBUG_ASSERT() (which includes _db_flush())
|
|
in the strings library as libdbug is compiled after the the strings
|
|
library and we don't want to have strings depending on libdbug which
|
|
depends on mysys and strings.
|
|
*/
|
|
|
|
#if !defined(DBUG_OFF)
|
|
#undef DBUG_ASSERT
|
|
#define DBUG_ASSERT(A) assert(A)
|
|
#endif
|
|
|
|
/* SPACE_INT is a word that contains only spaces */
|
|
#if SIZEOF_INT == 4
|
|
#define SPACE_INT 0x20202020
|
|
#elif SIZEOF_INT == 8
|
|
#define SPACE_INT 0x2020202020202020
|
|
#else
|
|
#error define the appropriate constant for a word full of spaces
|
|
#endif
|
|
|
|
/**
|
|
Skip trailing space.
|
|
|
|
On most systems reading memory in larger chunks (ideally equal to the size of
|
|
the chinks that the machine physically reads from memory) causes fewer memory
|
|
access loops and hence increased performance.
|
|
This is why the 'int' type is used : it's closest to that (according to how
|
|
it's defined in C).
|
|
So when we determine the amount of whitespace at the end of a string we do
|
|
the following :
|
|
1. We divide the string into 3 zones :
|
|
a) from the start of the string (__start) to the first multiple
|
|
of sizeof(int) (__start_words)
|
|
b) from the end of the string (__end) to the last multiple of sizeof(int)
|
|
(__end_words)
|
|
c) a zone that is aligned to sizeof(int) and can be safely accessed
|
|
through an int *
|
|
2. We start comparing backwards from (c) char-by-char. If all we find is
|
|
space then we continue
|
|
3. If there are elements in zone (b) we compare them as unsigned ints to a
|
|
int mask (SPACE_INT) consisting of all spaces
|
|
4. Finally we compare the remaining part (a) of the string char by char.
|
|
This covers for the last non-space unsigned int from 3. (if any)
|
|
|
|
This algorithm works well for relatively larger strings, but it will slow
|
|
the things down for smaller strings (because of the additional calculations
|
|
and checks compared to the naive method). Thus the barrier of length 20
|
|
is added.
|
|
|
|
@param ptr pointer to the input string
|
|
@param len the length of the string
|
|
@return the last non-space character
|
|
*/
|
|
|
|
static inline const uchar *skip_trailing_space(const uchar *ptr,size_t len)
|
|
{
|
|
const uchar *end= ptr + len;
|
|
|
|
if (len > 20)
|
|
{
|
|
const uchar *end_words= (const uchar *)(intptr)
|
|
(((ulonglong)(intptr)end) / SIZEOF_INT * SIZEOF_INT);
|
|
const uchar *start_words= (const uchar *)(intptr)
|
|
((((ulonglong)(intptr)ptr) + SIZEOF_INT - 1) / SIZEOF_INT * SIZEOF_INT);
|
|
|
|
DBUG_ASSERT(((ulonglong)(intptr)ptr) >= SIZEOF_INT);
|
|
if (end_words > ptr)
|
|
{
|
|
while (end > end_words && end[-1] == 0x20)
|
|
end--;
|
|
if (end[-1] == 0x20 && start_words < end_words)
|
|
while (end > start_words && ((unsigned *)end)[-1] == SPACE_INT)
|
|
end -= SIZEOF_INT;
|
|
}
|
|
}
|
|
while (end > ptr && end[-1] == 0x20)
|
|
end--;
|
|
return (end);
|
|
}
|
|
|
|
/* Macros for hashing characters */
|
|
|
|
#define MY_HASH_ADD(A, B, value) \
|
|
do { A^= (((A & 63)+B)*((value)))+ (A << 8); B+=3; } while(0)
|
|
|
|
#define MY_HASH_ADD_16(A, B, value) \
|
|
do { MY_HASH_ADD(A, B, ((value) & 0xFF)) ; MY_HASH_ADD(A, B, ((value >>8 ))); } while(0)
|
|
|
|
/*
|
|
This one is needed to ensure we get the exact same hash as MariaDB 5.1
|
|
This is needed to ensure that old partitioned tables still work as before.
|
|
*/
|
|
#define MY_HASH_ADD_16_INV(A, B, value) \
|
|
do { MY_HASH_ADD(A, B, ((value >> 8))) ; MY_HASH_ADD(A, B, ((value & 0xFF ))); } while(0)
|
|
|
|
#endif
|