Commit graph

37 commits

Author SHA1 Message Date
Mikhail Chalov
19af1890b5 Use memory safe snprintf() in Connect Engine
This commit replaces sprintf(buf, ...) with
snprintf(buf, sizeof(buf), ...),
specifically in the "easy" cases where buf is allocated with a size
known at compile time.

The changes make sure we are not write outside array/string bounds which
will lead to undefined behaviour. In case the code is trying to write
outside bounds - safe version of functions simply cut the string
messages so we process this gracefully.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license.  I am contributing on behalf of my employer Amazon Web Services,
Inc.

bsonudf.cpp warnings cleanup by Daniel Black

Reviewer: Daniel Black
2022-07-26 16:28:59 +10:00
Dmitry Shulga
bc7ba7afee MDEV-27758: Errors when building Connect engine on os x 11.6.2
Added checking for support of vfork by a platform where
building being done. Set HAVE_VFORK macros in case vfork()
system call is supported. Use vfork() system call if the
macros HAVE_VFORK is set, else use fork().
2022-04-22 18:47:19 +07:00
Sergei Krivonos
2df99f2193 Revert "MDEV-19129: Xcode compatibility update: deprecated vfork -> fork"
This reverts commit 5d6f3cebca.
2021-11-03 18:14:02 +02:00
Sergei Krivonos
5d6f3cebca MDEV-19129: Xcode compatibility update: deprecated vfork -> fork 2021-10-31 00:23:53 +03:00
Olivier Bertrand
ed70f76cf7 - Make function strz return null when LEX_STRING is null
modified:   storage/connect/ha_connect.cc

- Use NOTE instead of WARNING in connect_assisted_discovery
  This because MariaDB raise an error when doing so
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/tabrest.cpp

- Make MONGO tables recognize STRINGIFY and JsonAllPath
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/mongo.h
  modified:   storage/connect/tabcmg.h
  modified:   storage/connect/tabjmg.cpp
  modified:   storage/connect/tabcmg.cpp
  modified:   storage/connect/tabjmg.h

- Fix OBJECT option for Pretty != 2 JSN and BSON tables
  Accept all syntaxes  for the OBJECT option
  modified:   storage/connect/tabbson.cpp
  modified:   storage/connect/tabjson.cpp

- Use my_snprintf in function xcurl (by vuvova)
  modified:   storage/connect/tabrest.cpp

- Format dates entered as integer when formatted
  modified:   storage/connect/value.cpp
  modified:   storage/connect/value.h
2021-06-24 00:46:12 +02:00
Olivier Bertrand
5f64276fb2 - Fix MDEV-25863 : Replace __WIN__ by _WIN32
modified:   storage/connect/array.cpp
  modified:   storage/connect/blkfil.cpp
  modified:   storage/connect/block.h
  modified:   storage/connect/bson.cpp
  modified:   storage/connect/cmgoconn.cpp
  modified:   storage/connect/colblk.cpp
  modified:   storage/connect/domdoc.cpp
  modified:   storage/connect/filamap.cpp
  modified:   storage/connect/filamdbf.cpp
  modified:   storage/connect/filamfix.cpp
  modified:   storage/connect/filamgz.cpp
  modified:   storage/connect/filamtxt.cpp
  modified:   storage/connect/filamvct.cpp
  modified:   storage/connect/filamzip.cpp
  modified:   storage/connect/filter.cpp
  modified:   storage/connect/filter.h
  modified:   storage/connect/fmdlex.c
  modified:   storage/connect/global.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/javaconn.cpp
  modified:   storage/connect/javaconn.h
  modified:   storage/connect/jdbconn.cpp
  modified:   storage/connect/jmgfam.cpp
  modified:   storage/connect/json.cpp
  modified:   storage/connect/macutil.cpp
  modified:   storage/connect/macutil.h
  modified:   storage/connect/maputil.cpp
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/myconn.cpp
  modified:   storage/connect/myconn.h
  modified:   storage/connect/myutil.cpp
  modified:   storage/connect/odbconn.cpp
  modified:   storage/connect/odbconn.h
  modified:   storage/connect/os.h
  modified:   storage/connect/osutil.c
  modified:   storage/connect/plgdbsem.h
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/plugutil.cpp
  modified:   storage/connect/rcmsg.c
  modified:   storage/connect/reldef.cpp
  modified:   storage/connect/reldef.h
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabext.cpp
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfmt.cpp
  modified:   storage/connect/tabjdbc.cpp
  modified:   storage/connect/tabmac.cpp
  modified:   storage/connect/tabmac.h
  modified:   storage/connect/tabmul.cpp
  modified:   storage/connect/tabmul.h
  modified:   storage/connect/tabmysql.cpp
  modified:   storage/connect/taboccur.cpp
  modified:   storage/connect/tabodbc.cpp
  modified:   storage/connect/tabpivot.cpp
  modified:   storage/connect/tabrest.cpp
  modified:   storage/connect/tabrest.h
  modified:   storage/connect/tabsys.cpp
  modified:   storage/connect/tabtbl.cpp
  modified:   storage/connect/tabutil.cpp
  modified:   storage/connect/tabvct.cpp
  modified:   storage/connect/tabwmi.cpp
  modified:   storage/connect/tabxcl.cpp
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/valblk.cpp
  modified:   storage/connect/value.cpp
  modified:   storage/connect/xindex.cpp
  modified:   storage/connect/xindex.h

- Fix Date errors and SSL warnings
  modified:   storage/connect/mysql-test/connect/r/jdbc.result
  modified:   storage/connect/mysql-test/connect/r/jdbc_new.result
  modified:   storage/connect/mysql-test/connect/t/jdbc.test
  modified:   storage/connect/mysql-test/connect/t/jdbc_new.test

- Update java source files
  modified:   storage/connect/Mongo2Interface.java
  modified:   storage/connect/Mongo3Interface.java
  added:      storage/connect/Client2.java
  added:      storage/connect/Client3.java
  added:      storage/connect/TestInsert2.java
  added:      storage/connect/TestInsert3.java
2021-06-08 17:44:43 +02:00
Olivier Bertrand
bbec4aafda typo tabrest.cpp 2021-04-08 19:05:11 +02:00
Olivier Bertrand
710e1bac5a tabrest.cpp 2021-04-08 19:05:10 +02:00
Olivier Bertrand
757aaa3bcc tabrest.cpp 2021-04-08 19:05:09 +02:00
Olivier Bertrand
e4c23fe2d0 tabrest.cpp 2021-04-08 17:35:21 +02:00
Olivier Bertrand
953c84657b try to fix tabrest.cpp compile error 2021-04-08 17:35:19 +02:00
Olivier Bertrand
26e5ba4b51 - Fix Linux compile errors
modified:   storage/connect/tabrest.cpp
  modified:   storage/connect/CMakeLists.txt

- Fix cmake error
 modified:   libmariadb/cmake/ConnectorName.cmake
2021-04-08 17:35:17 +02:00
Olivier Bertrand
d1b6cad028 Test tabrest.cpp fix 2021-04-08 17:35:16 +02:00
Olivier Bertrand
28b76afcea - Fix(?) Linux compile errors
modified:   storage/connect/tabrest.cpp
  modified:   storage/connect/CMakeLists.txt

-fix MDEV-24794
 modified:   storage/connect/valblk.h
2021-04-06 00:10:07 +02:00
Olivier Bertrand
2aefe0bee1 - Fix crash when not specifying the collection for MongoDB
modified:   storage/connect/cmgoconn.cpp

- Fix(?) Linux compile errors
  modified:   storage/connect/tabrest.cpp
2021-04-05 18:33:37 +02:00
Olivier Bertrand
caff19ada5 - Copy Mongo2.jar and Mongo3.jar in plugin directory
modified:   storage/connect/CMakeLists.txt
  modified:   storage/connect/javaconn.cpp

- Check privileges while creating tables with Discovery
  modified:   storage/connect/ha_connect.cc

- Calculate LRECL for JSON tables created with Discovery
  modified:   storage/connect/tabjson.cpp

- Use CreateProcess (Windows) or fork/exec (linux)
  to retrieve the result from REST queries
  modified:   storage/connect/tabrest.cpp

- Typo
  modified:   storage/connect/jmgoconn.cpp
2021-04-05 17:01:43 +02:00
Olivier Bertrand
e85006e670 - Fix bug making REST table fail using CURL
This when the HTTP contains & characters
  modified:   storage/connect/tabbson.cpp
  modified:   storage/connect/tabjson.cpp

- Make stringfy option work on only one Json item
  modified:   storage/connect/tabbson.cpp
  modified:   storage/connect/tabbson.h
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h

- Make Json/Bson DATE columns accept JSON date syntax
  modified:   storage/connect/tabbson.cpp
  modified:   storage/connect/tabjson.cpp

- Fix bug making REST table default file not being
  erased when dropping the table
  modified:   storage/connect/tabbson.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabrest.cpp
  modified:   storage/connect/tabxml.cpp

- Suppress CHAR(36) --> VARCHAR(36) when DEVELOPMENT
  This was fixed in MyClient
  modified:   storage/connect/ha_connect.cc
2021-03-16 23:32:58 +01:00
Olivier Bertrand
2113cab7ec Make REST tables default file name. Commit before continuing BSON development 2020-12-22 22:50:12 +01:00
Olivier Bertrand
9055db73d5 Commit new source and all recent changes. 2020-11-18 14:37:44 +01:00
Olivier Bertrand
8771390dfd Change cURL option from > to -o. modified storage/connect/tabrest.cpp 2020-11-14 18:28:16 +01:00
Olivier Bertrand
17ea1efe0a Fix using a null pointer. modified storage/connect/tabrest.cpp 2020-11-11 17:41:11 +01:00
Olivier Bertrand
8c617e9901 Add getting REST query answer via curl. modified storage/connect/tabrest.cpp 2020-11-11 12:55:07 +01:00
Olivier Bertrand
99ab562a92 - Make possible to allocate work space larger than 4GB
All variables handling sizes that were uint are now size_t.
  The variable connect_work_size is now ulong (was uint);
  Also make Json functiosn to allocate a larger memory (M=9 was 7)
  modified:   storage/connect/global.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/json.cpp
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/plgdbutl.cpp
  modified:   storage/connect/plugutil.cpp
  modified:   storage/connect/user_connect.cc

- Fix uninitialised variable (pretty) in Json_File.
  Make Jbin_file accept the same arguments as Json_File ones.
  modified:   storage/connect/jsonudf.cpp

- Change the Level option to Depth (the word currently used)
  (Level being still accepted)
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabxml.cpp

- Suppress 2nd argument default value for MYSQLtoPLG function
  modified:   storage/connect/myutil.h

- Allow REST tables to be create not specifying a file_name
  modified:   storage/connect/tabrest.cpp
2020-10-01 19:18:26 +02:00
Olivier Bertrand
8ff3eb417c - Fix MDEV-21450
Failed compile when XML table type is not supported.
  Was because XMLDEF was unconditionally called from REST table.
  modified:   storage/connect/tabrest.cpp

- Make cmake less verbose
  modified:   storage/connect/CMakeLists.txt

- Hide Switch_to_definer_security_ctx not defined for 10.1 and 10.0
  modified:   storage/connect/ha_connect.cc
2020-01-12 19:59:07 +01:00
Olivier Bertrand
817675459a Fix tabrest.cpp when used for OEM 2019-12-03 19:20:26 +01:00
Olivier Bertrand
579d572d25 Make restGetFile extern C 2019-12-03 18:06:05 +01:00
Olivier Bertrand
64b1b959f1 comment out <dlfnc.h> in tabrest.cppc 2019-12-01 01:16:23 +01:00
Olivier Bertrand
65310f5855 <dlfnc.h> in tabrest.cpp and redef.cpp 2019-12-01 00:14:24 +01:00
Olivier Bertrand
1f7f533144 Add include <dlfnc.h> in tabrest.cpp 2019-11-30 19:03:29 +01:00
Olivier Bertrand
f0da39be7f - Fix MDEV-13782
Problem with NOT LIKE queries.
  modified:   storage/connect/ha_connect.cc
  modified:   sql/item_cmpfunc.h

- Fix MDEV-21084
  Misusage of strncat could cause buffer overflow.
  modified:   storage/connect/reldef.cpp
  modified:   storage/connect/tabcmg.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabrest.cpp
  modified:   storage/connect/tabxml.cpp
2019-11-26 19:22:46 +01:00
Olivier Bertrand
b4bfa12b00 Fix compile error (missing declaration) in tabrest.cpp 2019-11-21 20:12:06 +01:00
Olivier Bertrand
420789512f Fix compile error (missing declaration) in tabrest.cpp 2019-11-21 19:38:17 +01:00
Olivier Bertrand
9c0e462ff2 Fix missing declaration in tabrest.cpp causing compiling failure on Linux 2019-11-16 15:59:16 +01:00
Olivier Bertrand
2cb4b152c8 =====================================================================
This new CONNECT version 1.07 fully implements NOSQL support.
It allows working on JSON or XML data retrieved as REST query results
from all binary distributions of MariaDB when cpprestsdk is installed
and the GetRest library is available.
=====================================================================
- Make Rest available for MariaDB binary distributed versions.
  Change RestGet function so it can be called from a library.
  modified:   storage/connect/CMakeLists.txt
  modified:   storage/connect/restget.cpp
  modified:   storage/connect/tabrest.cpp

- Make column FLAG option available to discovery functions.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/plgdbsem.h

- Update CONNECT version number and date.
  modified:   storage/connect/ha_connect.cc

- Move OEMColumns function from mycat.cc to reldef.cpp.
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/reldef.cpp

- Allocate tables as TABREF (was RELDEF)
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/mycat.h

- Fix MDEV-20845 by commenting out TIMEOUT setting.
  modified:   storage/connect/myconn.cpp

- Call DefineAM before calling GetColCatInfo. Column offset
  is now based on record format instead of table type.
  The RECFM_VCT format was added.
  This enables tables to specify the record format and is
  useful in particular for OEM tables.
  modified:   storage/connect/plgdbsem.h
  modified:   storage/connect/reldef.cpp
  modified:   storage/connect/reldef.h
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabfix.cpp
  modified:   storage/connect/tabfmt.cpp
  modified:   storage/connect/tabmysql.cpp
  modified:   storage/connect/tabutil.cpp
  modified:   storage/connect/tabutil.h
  modified:   storage/connect/tabvct.cpp
  modified:   storage/connect/xindex.cpp
2019-11-16 14:59:54 +01:00
Olivier Bertrand
c0f9042500 Some small changes.
===================
- Modify tracing to use htrc to be compatible with old versions
  when this code is used to make an EOM module.
  modified:   storage/connect/restget.cpp
  modified:   storage/connect/tabrest.cpp

- Path apparently not needed for the cpprest lib on Linux
  modified:   storage/connect/CMakeLists.txt
2019-08-19 18:06:34 +02:00
Olivier Bertrand
4d93c7f3b0 In CONNECT version 1.6.10 NOSQL facility is enhanced by a new way to retrieve NOSQL data.
In addition to files and Mongo collections, JSON as well as XML and CSV data can be retrieved
from the net as answers from REST queries. Because it uses and external package (cpprestsdk)
this is currently available only to MariaDB servers compiled from source.

-- Add compile flags needed on Windows /MD or /MDd (debug)
-- Also include some changes needed on Linux
  modified:   storage/connect/CMakeLists.txt

- Add the xtrc tracing function
  modified:   storage/connect/global.h
  modified:   storage/connect/plugutil.cpp

- Modify tracing to use xtrc and some typo
  modified:   storage/connect/array.cpp
  modified:   storage/connect/block.h
  modified:   storage/connect/restget.cpp

- Fix compilation error when ZIP is not supported
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/tabfmt.cpp

- Add some tracing + typo
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/tabjson.cpp

- Add conditional code based on MARIADB
  This to be able to use the same code in CONNECT and EOM modules
  modified:   storage/connect/osutil.h
  modified:   storage/connect/tabrest.cpp

- Replace PlugSetPath by some concat (crashed on Fedora) + typo
  modified:   storage/connect/reldef.cpp

- Try to fix test failures
  modified:   zlib/CMakeLists.txt
2019-08-17 16:58:58 +02:00
Olivier Bertrand
e4797a991f In CONNECT version 1.6.10 NOSQL facility is enhanced by a new way to retrieve NOSQL data.
In addition to files and Mongo collections, JSON as well as XML and CSV data can be retrieved
from the net as answers from REST queries. Because it uses and external package (cpprestsdk)
this is currently available only to MariaDB servers compiled from source.

-- Add the REST support when Microsoft Casablanca package (cpprestsdk) is installed.
-- Also include some changes specific to MariaDB 10.3.
  modified:   storage/connect/CMakeLists.txt

-- Add conditional REST support
-- Added string options HTTP and URI.
-- Added added internal table type TAB_REST.
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/mycat.cc
  modified:   storage/connect/mycat.h
  modified:   storage/connect/plgdbsem.h

-- Fix MDEV-19648 Variable connect_conv_size doesn't change
-- Change Variable wrong block parameter from 8169 to 1.
-- Also change connect_conv_size default value to 1024.
  modified:   storage/connect/ha_connect.cc

-- Avoid possible buffer overflow
-- In particular by the function ShowValue.
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabfmt.cpp
  modified:   storage/connect/value.cpp
  modified:   storage/connect/value.h

-- Add some cast to avoid some compiler warnings
  modified:   storage/connect/filamdbf.cpp

-- Fix some C++ error
  modified:   storage/connect/javaconn.cpp
  modified:   storage/connect/jmgoconn.cpp
  modified:   storage/connect/plugutil.cpp

-- Miscellaneous Typo and warning suppressing changes
  modified:   storage/connect/connect.cpp
  modified:   storage/connect/connect.h
  modified:   storage/connect/filamvct.cpp
  modified:   storage/connect/inihandl.cpp
  modified:   storage/connect/jsonudf.cpp
  modified:   storage/connect/libdoc.cpp
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabtbl.cpp
  modified:   storage/connect/tabxml.cpp
  modified:   storage/connect/user_connect.cc
  modified:   storage/connect/user_connect.h

-- Update failing test results and disbling
  modified:   storage/connect/mysql-test/connect/disabled.def
  modified:   storage/connect/mysql-test/connect/r/dir.result
  modified:   storage/connect/mysql-test/connect/r/grant.result
  modified:   storage/connect/mysql-test/connect/r/jdbc.result
  modified:   storage/connect/mysql-test/connect/r/jdbc_postgresql.result
  modified:   storage/connect/mysql-test/connect/r/xml.result
  modified:   storage/connect/mysql-test/connect/r/xml2.result
  modified:   storage/connect/mysql-test/connect/r/xml2_mult.result
  modified:   storage/connect/mysql-test/connect/r/xml_mult.result

-- Add an option
  modified:   storage/connect/mysql-test/connect/t/grant.test
2019-07-30 22:45:04 +02:00