Commit graph

41 commits

Author SHA1 Message Date
Sergei Golubchik
3a2116241b Merge branch '10.4' into 10.5 2022-10-02 14:38:13 +02:00
Sergei Golubchik
d4f6d2f08f Merge branch '10.3' into 10.4 2022-10-01 23:07:26 +02:00
Andrew Hutchings
59c9e2f202
MDEV-29579 Fix CONNECT ASAN hits (#2277)
There are currently two things causing ASAN hits on CONNECT engine when
the plugin is used as a dynamic module. These are libxml2 and libodbc.

libxml2 has some quirks when not the first and last thing called in the
main thread of an application, some of the global memory isn't cleaned
up correctly. The same is assumed of libodbc but this does not have
explicit API for this.

This is being fixed in two ways.

First we are removing the libxml2 cleanup call. This is because the
current one is messy and whatever it fixed has gone away. But also
because if this is called and libxml2 is used again this can cause
issues. For example if two different plugins to MariaDB both happen to
use libxml2.

The second fix is a hack that exploits `STB_GNU_UNIQUE` so that when
compiled with ASAN the plugin will remain in memory after dlclose().
This allows libodbc to cleanup and has the added advatage that we will
get clean stacks from ASAN for CONNECT when the leak is detected at the
end of execution.

Details of the `STB_GNU_UNIQUE` method can be found here:

https://web.archive.org/web/20100730094324/http://www.redhat.com/archives/posix-c++-wg/2009-August/msg00002.html
2022-09-26 13:59:23 +01:00
Marko Mäkelä
098c0f2634 Merge 10.4 into 10.5 2022-07-27 17:17:24 +03:00
Oleksandr Byelkin
3bb36e9495 Merge branch '10.3' into 10.4 2022-07-27 11:02:57 +02:00
Mikhail Chalov
19af1890b5 Use memory safe snprintf() in Connect Engine
This commit replaces sprintf(buf, ...) with
snprintf(buf, sizeof(buf), ...),
specifically in the "easy" cases where buf is allocated with a size
known at compile time.

The changes make sure we are not write outside array/string bounds which
will lead to undefined behaviour. In case the code is trying to write
outside bounds - safe version of functions simply cut the string
messages so we process this gracefully.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license.  I am contributing on behalf of my employer Amazon Web Services,
Inc.

bsonudf.cpp warnings cleanup by Daniel Black

Reviewer: Daniel Black
2022-07-26 16:28:59 +10:00
Sergei Golubchik
00a313ecf3 Merge branch 'bb-10.3-release' into bb-10.4-release
Note, the fix for "MDEV-23328 Server hang due to Galera lock conflict resolution"
was null-merged. 10.4 version of the fix is coming up separately
2021-02-12 17:44:22 +01:00
Sergei Golubchik
a7b6943ee4 CONNECT: compiler warnings 2021-02-01 13:50:16 +01:00
Marko Mäkelä
8b6cfda631 Merge 10.4 into 10.5 2020-02-07 08:51:20 +02:00
Monty
4d61f1247a Fixed compiler warnings from gcc 7.4.1
- Fixed possible error in rocksdb/rdb_datadic.cc
2020-01-29 23:23:55 +02:00
Marko Mäkelä
780d2bb8a7 Merge 10.4 into 10.5 2019-09-06 14:25:20 +03:00
Olivier Bertrand
e4797a991f In CONNECT version 1.6.10 NOSQL facility is enhanced by a new way to retrieve NOSQL data.
In addition to files and Mongo collections, JSON as well as XML and CSV data can be retrieved
from the net as answers from REST queries. Because it uses and external package (cpprestsdk)
this is currently available only to MariaDB servers compiled from source.

-- Add the REST support when Microsoft Casablanca package (cpprestsdk) is installed.
-- Also include some changes specific to MariaDB 10.3.
  modified:   storage/connect/CMakeLists.txt

-- Add conditional REST support
-- Added string options HTTP and URI.
-- Added added internal table type TAB_REST.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/mycat.h
  modified:   storage/connect/plgdbsem.h

-- Fix MDEV-19648 Variable connect_conv_size doesn't change
-- Change Variable wrong block parameter from 8169 to 1.
-- Also change connect_conv_size default value to 1024.
  modified:   storage/connect/ha_connect.cc

-- Avoid possible buffer overflow
-- In particular by the function ShowValue.
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabfmt.cpp
  modified:   storage/connect/value.cpp
  modified:   storage/connect/value.h

-- Add some cast to avoid some compiler warnings
  modified:   storage/connect/filamdbf.cpp

-- Fix some C++ error
  modified:   storage/connect/javaconn.cpp
  modified:   storage/connect/jmgoconn.cpp
  modified:   storage/connect/plugutil.cpp

-- Miscellaneous Typo and warning suppressing changes
  modified:   storage/connect/connect.cpp
  modified:   storage/connect/connect.h
  modified:   storage/connect/filamvct.cpp
  modified:   storage/connect/inihandl.cpp
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/libdoc.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabtbl.cpp
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/user_connect.cc
  modified:   storage/connect/user_connect.h

-- Update failing test results and disbling
  modified:   storage/connect/mysql-test/connect/disabled.def
  modified:   storage/connect/mysql-test/connect/r/dir.result
  modified:   storage/connect/mysql-test/connect/r/grant.result
  modified:   storage/connect/mysql-test/connect/r/jdbc.result
  modified:   storage/connect/mysql-test/connect/r/jdbc_postgresql.result
  modified:   storage/connect/mysql-test/connect/r/xml.result
  modified:   storage/connect/mysql-test/connect/r/xml2.result
  modified:   storage/connect/mysql-test/connect/r/xml2_mult.result
  modified:   storage/connect/mysql-test/connect/r/xml_mult.result

-- Add an option
  modified:   storage/connect/mysql-test/connect/t/grant.test
2019-07-30 22:45:04 +02:00
Noel Kuntze
46553c2508 Fix compiler warnings GCC8 2019-07-30 09:58:17 +03:00
Alexander Barkov
3e7e87ddcc MDEV-19897 Rename source code variable names from utf8 to utf8mb3 2019-06-28 12:37:04 +04:00
Olivier Bertrand
79c1df4c23 - Change the connect_xtrace variable to from int to set
modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/libdoc.cpp
2018-01-31 01:08:52 +01:00
Olivier Bertrand
3e36eb230b Fix gcc compiler warnings reported by Sergei
modified:   storage/connect/array.cpp
  modified:   storage/connect/array.h
  modified:   storage/connect/blkfil.cpp
  modified:   storage/connect/blkfil.h
  modified:   storage/connect/block.h
  modified:   storage/connect/colblk.cpp
  modified:   storage/connect/colblk.h
  modified:   storage/connect/csort.h
  modified:   storage/connect/filamvct.cpp
  modified:   storage/connect/filter.cpp
  modified:   storage/connect/filter.h
  modified:   storage/connect/global.h
  modified:   storage/connect/json.h
  modified:   storage/connect/plgdbsem.h
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/tabcol.cpp
  modified:   storage/connect/tabcol.h
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/table.cpp
  modified:   storage/connect/tabodbc.cpp
  modified:   storage/connect/tabodbc.h
  modified:   storage/connect/tabsys.h
  modified:   storage/connect/tabxml.h
  modified:   storage/connect/value.cpp
  modified:   storage/connect/value.h
  modified:   storage/connect/xindex.cpp
  modified:   storage/connect/xindex.h
  modified:   storage/connect/xobject.cpp
  modified:   storage/connect/xobject.h
  modified:   storage/connect/xtable.h

Set values as nullable when retrieving catalog info
  modified:   storage/connect/jdbconn.cpp
  modified:   storage/connect/mysql-test/connect/r/odbc_oracle.result
  modified:   storage/connect/odbconn.cpp

Change format of Jpath
  modified:   storage/connect/json.cpp
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/mysql-test/connect/r/json.result
  modified:   storage/connect/mysql-test/connect/r/json_udf.result
  modified:   storage/connect/mysql-test/connect/r/json_udf_bin.result
  modified:   storage/connect/mysql-test/connect/r/zip.result
  modified:   storage/connect/mysql-test/connect/t/json.test
  modified:   storage/connect/mysql-test/connect/t/json_udf.test
  modified:   storage/connect/mysql-test/connect/t/json_udf_bin.test
  modified:   storage/connect/mysql-test/connect/t/zip.test
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h
  modified:   storage/connect/tabmgo.cpp

Change null representation from ??? to <null>
  modified:   storage/connect/json.cpp

Change the name of UDF that are equal to a native JSON function name
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/jsonudf.h
  modified:   storage/connect/mysql-test/connect/t/json_udf.inc
  modified:   storage/connect/mysql-test/connect/t/json_udf2.inc

Fix bug in making JSON project info
  modified:   storage/connect/mongofam.cpp

Fix COMPUTE when one argument is null
  modified:   storage/connect/value.cpp

Value is null only when nullable
  modified:   storage/connect/value.h
2017-05-23 19:35:50 +02:00
Olivier Bertrand
e6b563f8be Fix some XML table type bugs:
- in DOMNODELIST::DropItem
  if (Listp == NULL || Listp->length <= n)
    return true;
is wrong, should be:
  if (Listp == NULL || Listp->length < n)
    return true;
- Crash in discovery with libxml2 in XMLColumns because:
            if (!tdp->Usedom)    // nl was destroyed
              vp->nl = vp->pn->GetChildElements(g);
is executed with vp->pn uninitialized. Fixed by adding:
          vp->pn = node;
line 264.
-In discovery with libxml2 some columns are not found.
Because list was not recovered properly, nodes being modified and not reallocated.
Fixed lines 214 and 277.
  modified:   storage/connect/domdoc.cpp
  modified:   storage/connect/tabxml.cpp

Add support for zipped table files
  modified:   storage/connect/domdoc.cpp
  modified:   storage/connect/domdoc.h
  modified:   storage/connect/filamap.cpp
  modified:   storage/connect/filamap.h
  modified:   storage/connect/filamzip.cpp
  modified:   storage/connect/filamzip.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/libdoc.cpp
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/plgxml.cpp
  modified:   storage/connect/plgxml.h
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabfmt.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabxml.cpp
2016-12-23 16:58:32 +01:00
Olivier Bertrand
fb98632930 JSONColumns and XMLColumns revisited. They can retrieve their parameters directly
from the PTOS argument. For this to work, finding the table options is now split
in HA_CONNECT functions and exported functions available from out of ha_connect.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/libdoc.cpp
  modified:   storage/connect/mycat.h
  modified:   storage/connect/plgdbsem.h
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/tabxml.h

The BIN table formats have been changed to handle the case of floating point values
when used with Big Endian or Little Endian machines.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/mysql-test/connect/r/bin.result
  modified:   storage/connect/mysql-test/connect/t/bin.test
  modified:   storage/connect/reldef.cpp
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfix.
h
2015-05-26 01:02:33 +02:00
Olivier Bertrand
eae8318b19 - Fix Catalog JSON table crash when no Jpath
- Added JSON OBJECT specification for pretty != 2.
- Fix NULL values not recognized for nullable JSON columns
- Issue an error message when a JSON table is created without specifying LRECL if PRETTY != 2.
- Make JSONColumns use a TDBJSON class.
- Make JSON table using MAPFAM
modified:
  filamap.h
  filamtxt.h
  ha_connect.cc
  json.result
  tabjson.cpp
  tabjson.h
  table.cpp

- Implementing Discovery for the XML table type.
modified:
  domdoc.cpp
  domdoc.h
  ha_connect.cc
  libdoc.cpp
  plgxml.cpp
  plgxml.h
  reldef.cpp
  reldef.h
  tabxml.cpp
  tabxml.h

- Providing an error message when creating an ODBC table via discovery returns
  columns of more than one table.
modified:
  ha_connect.cc

- TableOptionStruct declaration moved from ha_connect.h to mycat.h
  To make it easier to use by other classes.
modified:
  ha_connect.cc
  ha_connect.h
  mycat.cc
  mycat.h
  reldef.cpp
  tabmysql.cpp
  taboccur.cpp
  tabpivot.cpp
  tabtbl.cpp
  tabutil.cpp
  tabxcl.cpp
2015-04-17 20:05:41 +02:00
Olivier Bertrand
2bb42803df This commit includes changes done in a previous (deleted) branch plus new ones.
From the previous branch:
commit eda4928ff122a0845baf5ade83b4aa29244a3a89
Author: Olivier Bertrand <bertrandop@gmail.com>
Date:   Mon Mar 9 22:34:56 2015 +0100

- Add discovery to JSON tables
  When columns are not defined, CONNECT analyses the json file to find column definitions.
  This wors only on table that are an array of objects. Pair keys are used to generate the
  column names and pair values are used for its definition. When the LEVEL option is defined
  as a not null integer, the eventual JPATH is scanned up to the LEVEL value.

From the current one:
- Fix MDEV-7521 when column names are utf8 encoded (not a general multi-charset fix)

- Adds more to JSON discovery processing and UDF's

- Use PlugDup everywhere it replaces PlugSubAlloc + strcpy.
2015-03-18 13:30:14 +01:00
Olivier Bertrand
56e2771321 1) Handling string memory allocation with a new STRING class. This is only
the beginning. Defining the STRING class and begining to use it (MYSQL)

2) Change the xtrace, use_tempfile and exact_info connect variables from
   GLOBAL to SESSION. Remaining GLOBAL variables have been made readonly.

3) Take care of LEX_STRING variables. The .str should not be regarded as
   allways being 0 terminated. This is handled by the Strz functions that
   make sure to return 0 terminated strings.

Bug fix:
- When inserting in MYSQL table with special column(s) a query such as:
insert into t2 values(0,4,'new04'),(0,5,'new05');
failed saying: column id (the special column) not found in t2.
It is now accepted but must be counted in values (these 0 are ignored)
- ROWID was returning row numbers based 0. Now it is from base 1.

modified:
  storage/connect/array.cpp
  storage/connect/blkfil.cpp
  storage/connect/colblk.cpp
  storage/connect/connect.cc
  storage/connect/filamap.cpp
  storage/connect/filamdbf.cpp
  storage/connect/filamfix.cpp
  storage/connect/filamtxt.cpp
  storage/connect/filamvct.cpp
  storage/connect/filamzip.cpp
  storage/connect/filamzip.h
  storage/connect/filter.cpp
  storage/connect/global.h
  storage/connect/ha_connect.cc
  storage/connect/ha_connect.h
  storage/connect/libdoc.cpp
  storage/connect/mycat.cc
  storage/connect/myconn.cpp
  storage/connect/odbconn.cpp
  storage/connect/plgdbutl.cpp
  storage/connect/plugutil.c
  storage/connect/reldef.cpp
  storage/connect/tabcol.cpp
  storage/connect/tabdos.cpp
  storage/connect/tabfix.cpp
  storage/connect/tabfmt.cpp
  storage/connect/table.cpp
  storage/connect/tabmul.cpp
  storage/connect/tabmysql.cpp
  storage/connect/tabmysql.h
  storage/connect/taboccur.cpp
  storage/connect/tabodbc.cpp
  storage/connect/tabpivot.cpp
  storage/connect/tabsys.cpp
  storage/connect/tabtbl.cpp
  storage/connect/tabutil.cpp
  storage/connect/tabvct.cpp
  storage/connect/tabwmi.cpp
  storage/connect/tabwmi.h
  storage/connect/tabxcl.cpp
  storage/connect/tabxml.cpp
  storage/connect/user_connect.cc
  storage/connect/valblk.cpp
  storage/connect/value.cpp
  storage/connect/value.h
  storage/connect/xindex.cpp
  storage/connect/xobject.cpp
  storage/connect/xobject.h
  storage/connect/xtable.h
2014-10-21 17:29:51 +02:00
Sergei Golubchik
1b75bed00f 5.5.40+ merge 2014-10-09 10:30:11 +02:00
Alexander Barkov
3a424fc119 ConnectSE: fixing memory leaks reported by valgrind 2013-10-29 13:44:05 +04:00
Olivier Bertrand
069edfe125 - Reset some variables when closing just in case the XML tre is re-used.
modified:
  storage/connect/libdoc.cpp
2013-09-25 18:41:20 +02:00
Olivier Bertrand
65b0e5455b - Fix several bugs causing memory leak or invalid access detected
by Valgrind. This concerns the XML libxml2 support.

modified:
  storage/connect/domdoc.cpp
  storage/connect/domdoc.h
  storage/connect/ha_connect.cc
  storage/connect/libdoc.cpp
  storage/connect/plgdbsem.h
  storage/connect/plgxml.h
  storage/connect/tabxml.cpp
  storage/connect/tabxml.h
2013-09-22 13:40:31 +02:00
Alexander Barkov
e46f8f3d0a Fixing warnings:
- no previous declaration for ‘void XmlCleanupParserLib()’
- no previous declaration for ‘void CloseXML2File’
- no previous declaration for ‘void XmlInitParserLib()’

modified:
  storage/connect/ha_connect.cc
  storage/connect/libdoc.cpp
  storage/connect/libdoc.h
  storage/connect/plgdbutl.cpp
2013-07-08 11:05:59 +04:00
Olivier Bertrand
ce1b9cb4ba - Try to fix a uninitialised valgrind warning
modified:
  storage/connect/ha_connect.cc
  storage/connect/ha_connect.h
  storage/connect/mycat.cc

- Fix a regression error on XML libdoc wrong Nlist freing

modified:
  storage/connect/domdoc.h
  storage/connect/libdoc.cpp
  storage/connect/libdoc.h
  storage/connect/plgxml.h
  storage/connect/tabxml.cpp
2013-07-05 13:13:45 +02:00
Olivier Bertrand
ffedc6b891 - Makes memory check conditionally
modified:
  storage/connect/libdoc.cpp
2013-07-03 23:58:22 +02:00
Olivier Bertrand
37fce46f66 - Fix memory leak in libdoc.cpp in LIBXMLDOC::GetNodeList replacing
xmlXPathFreeNodeSetList(Xop);         // Caused memory leak, by
  xmlXPathFreeObject(Xop);

modified:
  storage/connect/libdoc.cpp
2013-07-02 00:07:48 +02:00
Olivier Bertrand
5972b56a6d - Fix conversion bug for MS-DOM XML tables. The node content was written
and read as if the table DATA_CHARSET was ANSI instead of UTF-8. Warning
  are now provided when the read content of a node is truncated.

modified:
  storage/connect/domdoc.cpp
  storage/connect/domdoc.h
  storage/connect/libdoc.cpp
  storage/connect/libdoc.h
  storage/connect/plgxml.h
  storage/connect/tabxml.cpp

- Conditional compilation of pre_create depending on the MARIADB setting.

modified:
  storage/connect/ha_connect.cc
  storage/connect/ha_connect.h
2013-03-02 17:58:18 +01:00
Olivier Bertrand
9716acfcc3 - Re-install blank trimming to have the xml test pass.
Note that the problem if far more complex. To be revisited.

modified:
  storage/connect/libdoc.cpp
  storage/connect/tabxml.h
2013-02-20 23:23:35 +01:00
Olivier Bertrand
b63eb1d8b4 - Fix the elimination of control characters from node content
- Take care of XML special chars (<>& etc.)
- Remove Encode, Decode

modified:
  storage/connect/libdoc.cpp
  storage/connect/libdoc.h

- Neither libmysql.lib nor mysqlclient.lib needed for MYSQL table type

modified:
  storage/connect/CMakeLists.txt
2013-02-20 18:49:18 +01:00
Olivier Bertrand
c448839c4f - Fix crash on making an XML table with encoding=XXX
- Set parameters so libxml2 does not anymore add extra characters
  when retrieving several subnodes of a node.
- Make a CONNECT file header (was PlugDB)

modified:
  storage/connect/domdoc.cpp
  storage/connect/libdoc.cpp
  storage/connect/tabxml.cpp

- Change the version number

modified:
  storage/connect/ha_connect.cc

- Begin eliminate use of libmysql functions in MYSQL table type
  Not finished yet

modified:
  storage/connect/myconn.cpp
  storage/connect/myconn.h
2013-02-20 01:30:37 +01:00
Alexander Barkov
f41aadfaed Adding DATA_CHARSET table option. 2013-02-18 19:21:52 +04:00
Olivier Bertrand
a2cf90d9ce Suppress multiple blanks and control characters from the
node text retrieved by libxml2.

Modified:
libdoc.cpp
2013-02-15 15:53:27 +01:00
Alexander Barkov
30c4b0ebc2 - Fixing TAB to 2 spaces
- Fixing line endings from "\r\n" to "\n"
2013-02-07 13:34:27 +04:00
Alexander Barkov
22a8fb03db Introducing functions global_open() and global_fopen() for these purposes:
- Removing duplicate code to generate error message text
- In the future they will most likely check secure_file_priv directory.


modified:
  storage/connect/filamdbf.cpp
  storage/connect/filamfix.cpp
  storage/connect/filamtxt.cpp
  storage/connect/filamvct.cpp
  storage/connect/libdoc.cpp
  storage/connect/maputil.cpp
  storage/connect/plgdbsem.h
  storage/connect/plgdbutl.cpp
  storage/connect/tabfmt.cpp
  storage/connect/tabmul.cpp
  storage/connect/tabxml.cpp
  storage/connect/xindex.cpp
2013-02-06 15:17:34 +04:00
Alexander Barkov
dc6e20bda4 Fixing my_charset_utf8_bin to my_charset_utf8_general_ci.
The formed required strings.lib in Windows.
The latter is an exported symbol from mysqld and does not need strings.lib.

modified:
  storage/connect/ha_connect.cc
  storage/connect/libdoc.cpp
2013-02-04 11:37:35 +04:00
Olivier Bertrand
9ca150aa2c 2>libdoc.cpp
2>D:\CommonSource\mariadb-10.0\include\my_pthread.h(120) : warning C4005: '_REENTRANT' : redéfinition de macro
2>        D:\Libxml\include\libxml/xmlexports.h(77) : voir la définition précédente de '_REENTRANT'
2>.\libdoc.cpp(378) : error C2664: 'strlen' : impossible de convertir le paramètre 1 de 'xmlChar *' en 'const char *'
2>        Les types pointés n'ont aucun rapport entre eux ; conversion nécessitant reinterpret_cast, cast de style C ou cast de style fonction
2>.\libdoc.cpp(379) : error C2664: 'copy_and_convert' : impossible de convertir le paramètre 4 de 'xmlChar *' en 'const char *'
2>        Les types pointés n'ont aucun rapport entre eux ; conversion nécessitant reinterpret_cast, cast de style C ou cast de style fonction
2>

Fix for that.
2013-02-02 00:18:32 +01:00
Alexander Barkov
7a8402c98d Replacing iconv converstion routines in libdoc.cc to MariaDB routines.
TODO: Access to Field->charset() instead of hardcoded latin1 conversion.

modified:
  storage/connect/libdoc.cpp
  storage/connect/libdoc.h
2013-02-01 17:28:13 +04:00
Alexander Barkov
d7143a4160 Adding the CONNECT storage engine sources. 2013-01-18 19:21:44 +04:00