Commit graph

1494 commits

Author SHA1 Message Date
Oleksandr Byelkin
8e1e2856f2 Merge branch '10.4' into 10.5 2020-11-01 14:26:15 +01:00
Oleksandr Byelkin
80c951ce28 Merge branch '10.3' into 10.4 2020-10-31 21:06:49 +01:00
Oleksandr Byelkin
794f665139 Merge branch '10.2' into 10.3 2020-10-30 17:23:53 +01:00
Marko Mäkelä
898521e2dd Merge 10.4 into 10.5 2020-10-30 11:15:30 +02:00
Marko Mäkelä
7b2bb67113 Merge 10.3 into 10.4 2020-10-29 13:38:38 +02:00
Oleksandr Byelkin
65e26bc1ba Merge branch '10.1' into 10.2 2020-10-28 10:56:38 +01:00
Marko Mäkelä
a8de8f261d Merge 10.2 into 10.3 2020-10-28 10:01:50 +02:00
Oleksandr Byelkin
784473b986 Merge remote-tracking branch 'connect/10.2' into 10.2 2020-10-26 09:08:44 +01:00
Monty
21c15cc20b Fixed compiler warning in connect/filemazip.cpp 2020-10-21 03:09:29 +03:00
Olivier Bertrand
dc3a693b70 - Inline MakePtr and MakeOff with OFFSET as size_t
Also add a new member Saved_Size in the Global structure.
  modified:   storage/connect/global.h
  modified:   storage/connect/plugutil.cpp
  modified:   storage/connect/user_connect.cc
  modified:   storage/connect/jsonudf.cpp

- Add session variables json_all_path and default_depth
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabxml.cpp

- ADD column options JPATH and XPATH
  Work as FIELD_FORMAT but are more readable
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/ha_connect.h
  modified:   storage/connect/mysql-test/connect/r/json_java_2.result
  modified:   storage/connect/mysql-test/connect/r/json_java_3.result
  modified:   storage/connect/mysql-test/connect/r/json_mongo_c.result

- Handle negative numbes in the option list
  modified:   storage/connect/ha_connect.cc

- Fix Json parse that could crash the server.
  Was because it could use THROW out of the TRY block.
  Also handle all error by THROW.
  It is now done by a new class JSON.
  modified:   storage/connect/json.cpp
  modified:   storage/connect/json.h

- Add a new UDF function jfile_translate.
  It translate a Json file to pretty = 0.
  Fast because it does not a real parse of the file.
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/jsonudf.h

- Add a now options JSIZE and STRINGIFY to Json tables.
  STRINGIFY makes Objects or Arrays to be returned by their
  json representation instead of by their concatenated values.
  JSIZE allows to specify the LRECL (was 256) defaults to 1024.
  Also fix a bug about locating the sub-table by its path.
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h
2020-10-18 17:20:44 +02:00
Olivier Bertrand
5d2ddef26e Fix search for json subtable in tabjson.cpp 2020-10-06 12:50:12 +02:00
Olivier Bertrand
d4138e7eed Fix compile error in tabjson.cpp (ULONG -> ulong) 2020-10-05 15:21:58 +02:00
Olivier Bertrand
307258c8ee - Use BIN type when charset='binary'
modified:   storage/connect/ha_connect.cc

- Allow JSON columns to be "binary"
  By setting their type as VARBINAY(132)
  and their name begin with Jbin_
  modified:   storage/connect/json.h
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/value.cpp
  modified:   storage/connect/value.h

- CHARSET BINARY cannot be used for text columns
  modified:   storage/connect/mysql-test/connect/r/updelx.result
  modified:   storage/connect/mysql-test/connect/t/updelx.test
2020-10-05 12:29:51 +02:00
Olivier Bertrand
c6eb127ca8 - Fix allocating work space larger than 4GB
The variable connect_work_size is now ulong or ulonglong for 64bit machines.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/user_connect.cc
2020-10-03 19:06:05 +02:00
Olivier Bertrand
99ab562a92 - Make possible to allocate work space larger than 4GB
All variables handling sizes that were uint are now size_t.
  The variable connect_work_size is now ulong (was uint);
  Also make Json functiosn to allocate a larger memory (M=9 was 7)
  modified:   storage/connect/global.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/json.cpp
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/plugutil.cpp
  modified:   storage/connect/user_connect.cc

- Fix uninitialised variable (pretty) in Json_File.
  Make Jbin_file accept the same arguments as Json_File ones.
  modified:   storage/connect/jsonudf.cpp

- Change the Level option to Depth (the word currently used)
  (Level being still accepted)
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabxml.cpp

- Suppress 2nd argument default value for MYSQLtoPLG function
  modified:   storage/connect/myutil.h

- Allow REST tables to be create not specifying a file_name
  modified:   storage/connect/tabrest.cpp
2020-10-01 19:18:26 +02:00
Michael Widenius
58e759a939 Added 'final' to some classes to improve generated code
Final added to:
- All reasonable classes inhereted from Field
- All classes inhereted from Protocol
- Almost all Handler classes
- Some important Item classes

The stripped size of mariadbd is just 4K smaller, but several object files
showed notable improvements in common execution paths.
- Checked field.o and item_sum.o

Other things:
- Added 'override' to a few class functions touched by this patch.
- Removed 'virtual' from a new class functions that had/got 'override'
- Changed Protocol_discard to inherit from Protocol instad of Protocol_text
2020-08-04 17:27:32 +02:00
Oleksandr Byelkin
48b5777ebd Merge branch '10.4' into 10.5 2020-08-04 17:24:15 +02:00
Oleksandr Byelkin
57325e4706 Merge branch '10.3' into 10.4 2020-08-03 14:44:06 +02:00
Oleksandr Byelkin
c32f71af7e Merge branch '10.2' into 10.3 2020-08-03 13:41:29 +02:00
Oleksandr Byelkin
6d8af36bc7 Merge remote-tracking branch 'connect/10.2' into 10.2 2020-08-02 11:14:56 +02:00
Olivier Bertrand
671d9b6c61 - Fix MDEV-22571 and MDEV-22572. Allow multiple ZIP table
and enable using special column in them.
  modified:   storage/connect/tabzip.cpp
  modified:   storage/connect/tabzip.h

- Fix some compiler errors
  modified:   storage/connect/tabcmg.cpp
2020-07-16 16:30:54 +02:00
Olivier Bertrand
ad0d2424dd Fix compile error for KVM. Modified filamdbf.cpp 2020-07-13 20:02:51 +02:00
Olivier Bertrand
1e07df99f3 - Fix MDEV-22561 Unable to access DBF inside a ZIP archive
modified:   storage/connect/filamdbf.cpp
  modified:   storage/connect/filamdbf.h
  modified:   storage/connect/filamzip.cpp
  modified:   storage/connect/filamzip.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/plgxml.cpp
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabfix.h

- Add/Init Level class member
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/mongo.h
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h

- Typo
  modified:   storage/connect/connect.cc
2020-07-13 16:30:57 +02:00
Monty
5bcb1d6532 MDEV-11412 Ensure that table is truly dropped when using DROP TABLE
The used code is largely based on code from Tencent

The problem is that in some rare cases there may be a conflict between .frm
files and the files in the storage engine. In this case the DROP TABLE
was not able to properly drop the table.

Some MariaDB/MySQL forks has solved this by adding a FORCE option to
DROP TABLE. After some discussion among MariaDB developers, we concluded
that users expects that DROP TABLE should always work, even if the
table would not be consistent. There should not be a need to use a
separate keyword to ensure that the table is really deleted.

The used solution is:
- If a .frm table doesn't exists, try dropping the table from all storage
  engines.
- If the .frm table exists but the table does not exist in the engine
  try dropping the table from all storage engines.
- Update storage engines using many table files (.CVS, MyISAM, Aria) to
  succeed with the drop even if some of the files are missing.
- Add HTON_AUTOMATIC_DELETE_TABLE to handlerton's where delete_table()
  is not needed and always succeed. This is used by ha_delete_table_force()
  to know which handlers to ignore when trying to drop a table without
  a .frm file.

The disadvantage of this solution is that a DROP TABLE on a non existing
table will be a bit slower as we have to ask all active storage engines
if they know anything about the table.

Other things:
- Added a new flag MY_IGNORE_ENOENT to my_delete() to not give an error
  if the file doesn't exist. This simplifies some of the code.
- Don't clear thd->error in ha_delete_table() if there was an active
  error. This is a bug fix.
- handler::delete_table() will not abort if first file doesn't exists.
  This is bug fix to handle the case when a drop table was aborted in
  the middle.
- Cleaned up mysql_rm_table_no_locks() to ensure that if_exists uses
  same code path as when it's not used.
- Use non_existing_Table_error() to detect if table didn't exists.
  Old code used different errors tests in different position.
- Table_triggers_list::drop_all_triggers() now drops trigger file if
  it can't be parsed instead of leaving it hanging around (bug fix)
- InnoDB doesn't anymore print error about .frm file out of sync with
  InnoDB directory if .frm file does not exists. This change was required
  to be able to try to drop an InnoDB file when .frm doesn't exists.
- Fixed bug in mi_delete_table() where the .MYD file would not be dropped
  if the .MYI file didn't exists.
- Fixed memory leak in Mroonga when deleting non existing table
- Fixed memory leak in Connect when deleting non existing table

Bugs fixed introduced by the original version of this commit:
MDEV-22826 Presence of Spider prevents tables from being force-deleted from
           other engines
2020-06-14 19:39:42 +03:00
Marko Mäkelä
701efbb25b Merge 10.4 into 10.5 2020-06-03 09:45:39 +03:00
Marko Mäkelä
8059148154 Merge 10.3 into 10.4 2020-06-03 07:32:09 +03:00
Marko Mäkelä
8300f639a1 Merge 10.2 into 10.3 2020-06-02 10:25:11 +03:00
Alexander Barkov
6a6aa1c089 MDEV-21764 CONNECT table with INET6 field produces warnings upon SELECT 2020-06-01 11:46:30 +04:00
Sergei Golubchik
4832b751ad cmake: quieter 2020-05-31 11:29:43 +02:00
Monty
4ea171ffab Fixed compiler warnings from gcc and clang 5.0.1 2020-05-23 12:29:10 +03:00
Olivier Bertrand
2654ddd2d5 - Fix MDEV-22571 and MDEV-22572. Allow multiple ZIP table
and enable using special column in them.
  modified:   storage/connect/tabzip.cpp
  modified:   storage/connect/tabzip.h

- Fix some compiler errors
  modified:   storage/connect/tabcmg.cpp
2020-05-19 00:05:56 +02:00
Marko Mäkelä
7bcaa541aa Merge 10.4 into 10.5 2020-05-05 21:16:22 +03:00
Marko Mäkelä
2c3c851d2c Merge 10.3 into 10.4 2020-05-05 20:33:10 +03:00
Oleksandr Byelkin
7fb73ed143 Merge branch '10.2' into 10.3 2020-05-04 16:47:11 +02:00
Oleksandr Byelkin
ca091e6372 Merge branch '10.1' into 10.2 2020-05-02 08:44:17 +02:00
Marko Mäkelä
37c14690fc Merge 10.4 into 10.5 2020-03-30 19:07:25 +03:00
Marko Mäkelä
e2f1f88fa6 Merge 10.3 into 10.4 2020-03-30 14:50:23 +03:00
Marko Mäkelä
1a9b6c4c7f Merge 10.2 into 10.3 2020-03-30 11:12:56 +03:00
Monty
eb483c5181 Updated optimizer costs in multi_range_read_info_const() and sql_select.cc
- multi_range_read_info_const now uses the new records_in_range interface
- Added handler::avg_io_cost()
- Don't calculate avg_io_cost() in get_sweep_read_cost if avg_io_cost is
  not 1.0.  In this case we trust the avg_io_cost() from the handler.
- Changed test_quick_select to use TIME_FOR_COMPARE instead of
  TIME_FOR_COMPARE_IDX to align this with the rest of the code.
- Fixed bug when using test_if_cheaper_ordering where we didn't use
  keyread if index was changed
- Fixed a bug where we didn't use index only read when using order-by-index
- Added keyread_time() to HEAP.
  The default keyread_time() was optimized for blocks and not suitable for
  HEAP. The effect was the HEAP prefered table scans over ranges for btree
  indexes.
- Fixed get_sweep_read_cost() for HEAP tables
- Ensure that range and ref have same cost for simple ranges
  Added a small cost (MULTI_RANGE_READ_SETUP_COST) to ranges to ensure
  we favior ref for range for simple queries.
- Fixed that matching_candidates_in_table() uses same number of records
  as the rest of the optimizer
- Added avg_io_cost() to JT_EQ_REF cost. This helps calculate the cost for
  HEAP and temporary tables better. A few tests changed because of this.
- heap::read_time() and heap::keyread_time() adjusted to not add +1.
  This was to ensure that handler::keyread_time() doesn't give
  higher cost for heap tables than for normal tables. One effect of
  this is that heap and derived tables stored in heap will prefer
  key access as this is now regarded as cheap.
- Changed cost for index read in sql_select.cc to match
  multi_range_read_info_const(). All index cost calculation is now
  done trough one function.
- 'ref' will now use quick_cost for keys if it exists. This is done
  so that for '=' ranges, 'ref' is prefered over 'range'.
- scan_time() now takes avg_io_costs() into account
- get_delayed_table_estimates() uses block_size and avg_io_cost()
- Removed default argument to test_if_order_by_key(); simplifies code
2020-03-27 03:58:32 +02:00
Monty
f36ca142f7 Added page_range to records_in_range() to improve range statistics
Prototype change:
-  virtual ha_rows records_in_range(uint inx, key_range *min_key,
-                                   key_range *max_key)
+  virtual ha_rows records_in_range(uint inx, const key_range *min_key,
+                                   const key_range *max_key,
+                                   page_range *res)

The handler can ignore the page_range parameter. In the case the handler
updates the parameter, the optimizer can deduce the following:
- If previous range's last key is on the same block as next range's first
  key
- If the current key range is in one block
- We can also assume that the first and last block read are cached!
  This can be used for a better calculation of IO seeks when we
  estimate the cost of a range index scan.

The parameter is fully implemented for MyISAM, Aria and InnoDB.
A separate patch will update handler::multi_range_read_info_const() to
take the benefits of this change and also remove the double
records_in_range() calls that are not anymore needed.
2020-03-27 03:54:45 +02:00
Monty
736998cb75 Cleanups & indentation changes
- Only indentation changes in sql_rename.cc
- Ignore some WSREP error messages when there isn't a internet connection
- Force restart of stat_tables_part.test to make result stable
- Fixed compiler warnings in CONNECT
2020-03-24 21:00:02 +02:00
Marko Mäkelä
de9072ca19 Connect: Remove some unused variables 2020-03-21 12:52:07 +02:00
Monty
305cffebab merge 10.4 to 10.5 2020-03-18 12:00:38 +02:00
Monty
1242eb3d32 Removed double records_in_range calls from multi_range_read_info_const
This was to remove a performance regression between 10.3 and 10.4
In 10.5 we will have a better implementation of records_in_range
that will enable us to get more statistics.
This change was not done in 10.4 because the 10.5 will be part of
a larger change that is not suitable for the GA 10.4 version

Other things:
- Changed default handler block_size to 8192 to fix things statistics
  for engines that doesn't set the block size.
- Fixed a bug in spider when using multiple part const ranges
  (Patch from Kentoku)
2020-03-17 02:16:48 +02:00
Otto Kekäläinen
c8388de2fd Fix various spelling errors
e.g.
- dont -> don't
- occurence -> occurrence
- succesfully -> successfully
- easyly -> easily

Also remove trailing space in selected files.

These changes span:
- server core
- Connect and Innobase storage engine code
- OQgraph, Sphinx and TokuDB storage engines

Related to MDEV-21769.
2020-03-16 00:10:50 +02:00
Olivier Bertrand
41acc81f4d - Fix https://stackoverflow.com/questions/60625778/import-complex-xml-from-multiple-files-in-mariadb/60637429#60637429
Import complex XML from multiple files in MariaDB
  Some row results are missing and replaced by the last file one.
  Thats because Nx and Sx column members are not reset when changing file.
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/tabxml.h
2020-03-11 23:14:19 +01:00
Olivier Bertrand
f5f9659b15 - Fix https://stackoverflow.com/questions/60625778/import-complex-xml-from-multiple-files-in-mariadb/60637429#60637429
Import complex XML from multiple files in MariaDB
  Some row results are missing and replaced by the last file one.
  Thats because Nx and Sx column members are not reset when changing file.
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/tabxml.h
2020-03-11 14:52:20 +01:00
Sergei Golubchik
7c58e97bf6 perfschema memory related instrumentation changes 2020-03-10 19:24:22 +01:00
Marko Mäkelä
6b317c1cc3 Remove some redundant code flagged by clang or GCC 2020-03-05 07:31:52 +02:00
Marko Mäkelä
8b6cfda631 Merge 10.4 into 10.5 2020-02-07 08:51:20 +02:00