Add the end of file marker x1A to the DBF file and handle it
correctly to preserve interoperability with Libreoffice, and others
that have followed the DBF spec.
The file open mode of "a+" was problematic because Linux and the OSX,
the previous main development mode are inconsistent (see man fopen).
The main problem per the bug report was the inability to fseek back to the
beginning to update the records in the header.
As such the "a+" mode is remove and "w+b" is used inserting to a new file
and "r+b" is used for appending to the file.
In DBFFAM::CloseTableFile move PlugCloseFile down to close the file in
all modes.
The year unlike the comments is always since 1900. Use the
YYYY-MM-DD as an unabigious form during tracing.
Thanks for Mr. Zoltan Duna for the descriptive bug report.
The MariaDB code base uses strcat() and strcpy() in several
places. These are known to have memory safety issues and their usage is
discouraged. Common security scanners like Flawfinder flags them. In MariaDB we
should start using modern and safer variants on these functions.
This is similar to memory issues fixes in 19af1890b5
and 9de9f105b5 but now replace use of strcat()
and strcpy() with safer options strncat() and strncpy().
However, add '\0' forcefully to make sure the result string is correct since
for these two functions it is not guaranteed what new string will be null-terminated.
Example:
size_t dest_len = sizeof(g->Message);
strncpy(g->Message, "Null json tree", dest_len); strncat(g->Message, ":",
sizeof(g->Message) - strlen(g->Message)); size_t wrote_sz = strlen(g->Message);
size_t cur_len = wrote_sz >= dest_len ? dest_len - 1 : wrote_sz;
g->Message[cur_len] = '\0';
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services
-- Reviewer and co-author Vicențiu Ciorbaru <vicentiu@mariadb.org>
-- Reviewer additions:
* The initial function implementation was flawed. Replaced with a simpler
and also correct version.
* Simplified code by making use of snprintf instead of chaining strcat.
* Simplified code by removing dynamic string construction in the first
place and using static strings if possible. See connect storage engine
changes.
This commit replaces sprintf(buf, ...) with
snprintf(buf, sizeof(buf), ...),
specifically in the "easy" cases where buf is allocated with a size
known at compile time.
The changes make sure we are not write outside array/string bounds which
will lead to undefined behaviour. In case the code is trying to write
outside bounds - safe version of functions simply cut the string
messages so we process this gracefully.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services,
Inc.
bsonudf.cpp warnings cleanup by Daniel Black
Reviewer: Daniel Black
- clang complains about register in C++
- Removed not used variables
- Fixed bug when printing date in filamdbf.cpp
- Added {} to fix warning about dangling else
- Changed connect_done_func() to be global to remove conflict with header
files
- Added extra () around assignment in if
extra/mariabackup/fil_cur.cc:361:42: warning: format specifies type 'unsigned long' but the argument has type 'ib_int64_t' (aka 'long long') [-Wformat]
extra/mariabackup/fil_cur.cc:376:9: warning: format specifies type 'unsigned long' but the argument has type 'ib_int64_t' (aka 'long long') [-Wformat]
sql/handler.cc:6196:45: warning: format specifies type 'unsigned long' but the argument has type 'wsrep_trx_id_t' (aka 'unsigned long long') [-Wformat]
sql/log.cc:1681:16: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
sql/log.cc:1687:16: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
sql/wsrep_sst.cc:1388:86: warning: format specifies type 'long' but the argument has type 'wsrep_seqno_t' (aka 'long long') [-Wformat]
sql/wsrep_sst.cc:232:86: warning: format specifies type 'long' but the argument has type 'wsrep_seqno_t' (aka 'long long') [-Wformat]
storage/connect/filamdbf.cpp:450:47: warning: format specifies type 'short' but the argument has type 'int' [-Wformat]
storage/connect/filamdbf.cpp:970:47: warning: format specifies type 'short' but the argument has type 'int' [-Wformat]
storage/connect/inihandl.cpp:197:16: warning: address of array 'key->name' will always evaluate to 'true' [-Wpointer-bool-conversion]
storage/innobase/btr/btr0scrub.cc:151:17: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/innobase/buf/buf0buf.cc:5085:8: warning: nonnull parameter 'bpage' will evaluate to 'true' on first encounter [-Wpointer-bool-conversion]
storage/innobase/fil/fil0crypt.cc:2454:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/innobase/handler/ha_innodb.cc:18685:7: warning: format specifies type 'unsigned long' but the argument has type 'wsrep_trx_id_t' (aka 'unsigned long long') [-Wformat]
storage/innobase/row/row0mysql.cc:3319:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/innobase/row/row0mysql.cc:3327:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/maria/ma_norec.c:35:10: warning: implicit conversion from 'int' to 'my_bool' (aka 'char') changes value from 131 to -125 [-Wconstant-conversion]
storage/maria/ma_norec.c:42:10: warning: implicit conversion from 'int' to 'my_bool' (aka 'char') changes value from 131 to -125 [-Wconstant-conversion]
storage/maria/ma_test2.c:1009:12: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
storage/maria/ma_test2.c:1010:12: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
storage/mroonga/ha_mroonga.cpp:9189:44: warning: use of logical '&&' with constant operand [-Wconstant-logical-operand]
storage/mroonga/vendor/groonga/lib/expr.c:4987:22: warning: comparison of constant -1 with expression of type 'grn_operator' is always false [-Wtautological-constant-out-of-range-compare]
storage/xtradb/btr/btr0scrub.cc:151:17: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/xtradb/buf/buf0buf.cc:5047:8: warning: nonnull parameter 'bpage' will evaluate to 'true' on first encounter [-Wpointer-bool-conversion]
storage/xtradb/fil/fil0crypt.cc:2454:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/xtradb/row/row0mysql.cc:3324:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/xtradb/row/row0mysql.cc:3332:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
unittest/sql/mf_iocache-t.cc:120:35: warning: format specifies type 'unsigned long' but the argument has type 'int' [-Wformat]
unittest/sql/mf_iocache-t.cc:96:35: note: expanded from macro 'INFO_TAIL'
Do not silence uncertain cases, or fix any bugs.
The only functional change should be that ha_federated::extra()
is not calling DBUG_PRINT to report an unhandled case for
HA_EXTRA_PREPARE_FOR_DROP.
In particular enable to set length and endian setting.
This should solve all problems on IBM390s machines.
modified: storage/connect/ha_connect.cc
modified: storage/connect/tabfix.cpp
modified: storage/connect/tabfix.h
Make DBF tables to be usable in big-endian machines (test version)
modified: storage/connect/filamdbf.cpp
Ignore git files on storage/connect
modified: .gitignore
They are still experimental and should not be used in production.
added:
storage/connect/encas.h
storage/connect/english.msg
storage/connect/enids.h
storage/connect/frcas.h
storage/connect/french.msg
storage/connect/frids.h
storage/connect/frmsg.h
modified:
storage/connect/connect.cc
storage/connect/engmsg.h
storage/connect/filamdbf.cpp
storage/connect/global.h
storage/connect/ha_connect.cc
storage/connect/msgid.h
storage/connect/plgdbutl.cpp
storage/connect/plugutil.c
storage/connect/rcmsg.c
storage/connect/resource.h
storage/connect/tabfmt.h
- Fix global variable not being properly initialized (MDEV-6690, MDEV-7094)
modified:
storage/connect/ha_connect.cc
storage/connect/plugutil.c
storage/connect/user_connect.cc
storage/connect/xindex.cpp
- Implement Rewind for ODBC tables (MDEV-7097)
modified:
storage/connect/odbconn.cpp
storage/connect/odbconn.h
storage/connect/tabmysql.cpp
storage/connect/tabodbc.cpp
- Reset N when reopening MYSQL tables
modified:
storage/connect/tabmysql.cpp
the beginning. Defining the STRING class and begining to use it (MYSQL)
2) Change the xtrace, use_tempfile and exact_info connect variables from
GLOBAL to SESSION. Remaining GLOBAL variables have been made readonly.
3) Take care of LEX_STRING variables. The .str should not be regarded as
allways being 0 terminated. This is handled by the Strz functions that
make sure to return 0 terminated strings.
Bug fix:
- When inserting in MYSQL table with special column(s) a query such as:
insert into t2 values(0,4,'new04'),(0,5,'new05');
failed saying: column id (the special column) not found in t2.
It is now accepted but must be counted in values (these 0 are ignored)
- ROWID was returning row numbers based 0. Now it is from base 1.
modified:
storage/connect/array.cpp
storage/connect/blkfil.cpp
storage/connect/colblk.cpp
storage/connect/connect.cc
storage/connect/filamap.cpp
storage/connect/filamdbf.cpp
storage/connect/filamfix.cpp
storage/connect/filamtxt.cpp
storage/connect/filamvct.cpp
storage/connect/filamzip.cpp
storage/connect/filamzip.h
storage/connect/filter.cpp
storage/connect/global.h
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/libdoc.cpp
storage/connect/mycat.cc
storage/connect/myconn.cpp
storage/connect/odbconn.cpp
storage/connect/plgdbutl.cpp
storage/connect/plugutil.c
storage/connect/reldef.cpp
storage/connect/tabcol.cpp
storage/connect/tabdos.cpp
storage/connect/tabfix.cpp
storage/connect/tabfmt.cpp
storage/connect/table.cpp
storage/connect/tabmul.cpp
storage/connect/tabmysql.cpp
storage/connect/tabmysql.h
storage/connect/taboccur.cpp
storage/connect/tabodbc.cpp
storage/connect/tabpivot.cpp
storage/connect/tabsys.cpp
storage/connect/tabtbl.cpp
storage/connect/tabutil.cpp
storage/connect/tabvct.cpp
storage/connect/tabwmi.cpp
storage/connect/tabwmi.h
storage/connect/tabxcl.cpp
storage/connect/tabxml.cpp
storage/connect/user_connect.cc
storage/connect/valblk.cpp
storage/connect/value.cpp
storage/connect/value.h
storage/connect/xindex.cpp
storage/connect/xobject.cpp
storage/connect/xobject.h
storage/connect/xtable.h
that uninitialised characters be written in op file.
modified:
storage/connect/tabdos.cpp
- Typo
modified:
storage/connect/filamap.cpp
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/valblk.cpp
storage/connect/xindex.cpp
it belongs to each tables and the catalog being share between several
instances of CONNECT, when a query implied several tables belonging to
different databases, some where pointing on the wrong database. This
fix bugs occuring in queries such as:
INSERT into db1.t1 select * from db2.t2;
Where the t1 data file was made in db2.
modified:
storage/connect/catalog.h
storage/connect/connect.cc
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/mycat.cc
storage/connect/mycat.h
storage/connect/plgdbsem.h
storage/connect/plgdbutl.cpp
storage/connect/reldef.cpp
storage/connect/reldef.h
storage/connect/tabfix.h
storage/connect/tabfmt.cpp
storage/connect/tabfmt.h
storage/connect/tabmul.cpp
a temporary file should be used for UPDATE/DELETE of file tables.
Also use the "sorted" argument of index_init to help decide if
sorting of positions must be done.
modified:
storage/connect/checklvl.h
storage/connect/connect.cc
storage/connect/connect.h
storage/connect/filamdbf.cpp
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/ha_connect.cc
storage/connect/mysql-test/connect/r/part_table.result
storage/connect/plgdbsem.h
storage/connect/plgdbutl.cpp
storage/connect/reldef.cpp
storage/connect/tabdos.cpp
storage/connect/tabdos.h
storage/connect/tabfix.cpp
storage/connect/tabfmt.cpp
storage/connect/tabvct.cpp
storage/connect/tabvct.h
storage/connect/xindex.cpp
- Fix a bug in TDBASE::ColDB that caused some special columns not to
be found in the column list and reallocated without their Value
causing a crash of some queries.
modified:
storage/connect/table.cpp
- Fix a bug causing RestoreNrec to be called before closing a table
causing a wrong value given to Spos
modified:
storage/connect/tabdos.cpp
storage/connect/xindex.cpp
- Add a new CONNECT global variable connect_exact_info. Set to ON, it
tells CONNECT to return exact record numbers on info queries. If OFF
it just gives an estimate. In version 10.0.13 this was unconditionally
ON and caused info queries on remote tables to be extremely long and
was the subject of MDEV-6612.
modified:
storage/connect/ha_connect.cc
storage/connect/tabdos.cpp
storage/connect/tabmysql.cpp
storage/connect/tabodbc.cpp
them sorted by file position. Firstly a new value is stored in indexes
to know if they are sorted, preventing to do the sorting when it is not
needed. Secondly, almost all in now done in connect instead of being
done by the different file access method classes. This pepares the future
use of temporary files for all table types and also fix the bug that was
occuring when partially using a multi-column index because of false MRR
like call of position followed by unsorted rnd_pos no more using indexing.
modified:
storage/connect/connect.cc
storage/connect/filamap.cpp
storage/connect/filamap.h
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/filamtxt.h
storage/connect/filamvct.cpp
storage/connect/filamvct.h
storage/connect/tabdos.cpp
storage/connect/tabdos.h
storage/connect/tabfix.h
storage/connect/tabfmt.cpp
storage/connect/tabfmt.h
storage/connect/xindex.cpp
storage/connect/xindex.h
storage/connect/xtable.h
have been created by the last addition of new CONNECT features.
The version previous to this one is a preliminary test version and
should not be distributed.
- Handle indexed UPDATE/DELETE. Previously this was just tested and
an error message send when it could not be done. Now CONNECT can
do it in all the cases. It is done by a MRR like tchnique by making
a list of all update or delete to do, sort them, then execute them.
modified:
storage/connect/array.cpp
storage/connect/array.h
storage/connect/filamap.cpp
storage/connect/filamap.h
storage/connect/filamdbf.cpp
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/filamtxt.h
storage/connect/filamvct.cpp
storage/connect/filamvct.h
storage/connect/filamzip.cpp
storage/connect/filamzip.h
storage/connect/global.h
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
- Differenciate Cardinality that returns a true or estimated table size
and GetMaxSize that return a value equal or greater than the table
row number. This fixes the errors of non matching opt files.
modified:
storage/connect/connect.cc
storage/connect/tabdos.cpp
storage/connect/tabdos.h
storage/connect/tabfix.cpp
storage/connect/table.cpp
storage/connect/tabmac.h
storage/connect/tabmysql.cpp
storage/connect/tabmysql.h
storage/connect/tabodbc.cpp
storage/connect/tabodbc.h
storage/connect/tabpivot.h
storage/connect/tabtbl.cpp
storage/connect/tabtbl.h
storage/connect/tabutil.cpp
storage/connect/tabutil.h
storage/connect/tabwmi.h
storage/connect/xtable.h
- Fix some errors and issues when making index and opt files.
Erase opt and index files for void tables.
Fix wrong calculation of Block and Last in MakeBlockValues.
Invalidate indexes before making opt file.
Fully handle blocked variable tables. Make opt file for blocked
variable tables even when they have no optimised colums.
modified:
storage/connect/tabdos.cpp
storage/connect/xindex.h
- Fix some errors making index
Return an error when the allocation is too small (should not
really occur now that GetMaxSize is sure)
Don't use XXROW index for DBF tables because of soft deleted lines.
modified:
storage/connect/xindex.cpp
- Typo
modified:
storage/connect/macutil.cpp
storage/connect/tabdos.h
storage/connect/tabsys.cpp
storage/connect/tabsys.h
1) Support of partitioning by connect. A table can be partitioned
by files, this is an enhanced MULTIPLE table. It can be also
partitioned by sub-tables like TBL and this enables table sharding.
2) Handling a CONNECT bug that causes in some cases extraneous rows
to remain in the table after an UPDATE or DELETE when the command
uses indexing (for not fixed file tables). Until a real fix is
done, CONNECT tries to ignore indexing and if it cannot do it
abort the command with an error message.
- Add tests on partitioning
added:
storage/connect/mysql-test/connect/r/part_file.result
storage/connect/mysql-test/connect/r/part_table.result
storage/connect/mysql-test/connect/t/part_file.test
storage/connect/mysql-test/connect/t/part_table.test
- Temporary fix
modified:
sql/sql_partition.cc
- Add partition support
modified:
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/reldef.cpp
storage/connect/reldef.h
storage/connect/tabdos.cpp
- Add functions ha_connect::IsUnique and ha_connect::CheckColumnList
modified:
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
- Prevent updating a partition table column that is part of
the partition function (outward tables only)
modified:
storage/connect/ha_connect.cc
- Support INSERT/UPDATE/DELETE for PROXY tables
modified:
storage/connect/tabutil.cpp
- Handle the bug on updating rows via indexing. Waiting for a real fix,
Don't use indexing when possible else raise an error and abort.
modified:
storage/connect/ha_connect.cc
- dbuserp->UseTemp set to TMP_AUTO
modified:
storage/connect/connect.cc
- Add members nox, abort and only
modified:
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
- Add arguments nox and abort to CntCloseTable
modified:
storage/connect/connect.cc
storage/connect/connect.h
storage/connect/filamap.cpp
storage/connect/filamap.h
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/filamtxt.h
storage/connect/filamvct.cpp
storage/connect/filamvct.h
storage/connect/filamzip.cpp
storage/connect/filamzip.h
storage/connect/ha_connect.cc
- Add arguments abort to CloseTableFile and RenameTempFile
modified:
storage/connect/filamap.cpp
storage/connect/filamap.h
storage/connect/filamdbf.cpp
storage/connect/filamdbf.h
storage/connect/filamfix.cpp
storage/connect/filamfix.h
storage/connect/filamtxt.cpp
storage/connect/filamtxt.h
storage/connect/filamvct.cpp
storage/connect/filamvct.h
storage/connect/filamzip.cpp
storage/connect/filamzip.h
storage/connect/tabdos.cpp
storage/connect/tabdos.h
storage/connect/tabvct.cpp
storage/connect/xtable.h
- Fix info->records when file does not exists
modified:
storage/connect/connect.cc
- Close XML table when opened for info
modified:
storage/connect/connect.cc
- Add function VCTFAM::GetFileLength
modified:
storage/connect/filamvct.cpp
storage/connect/filamvct.h
- Column option DISTRIB -> ENUM
modified:
storage/connect/ha_connect.cc
- Options connect, query_string and partname allways available
modified:
storage/connect/ha_connect.cc
- Add function MYSQLC::GetTableSize
modified:
storage/connect/myconn.cpp
storage/connect/myconn.h
- Add new special columns (PARTNAME, FNAME, FPATH, FTYPE and FDISK)
modified:
storage/connect/colblk.cpp
storage/connect/colblk.h
storage/connect/plgdbsem.h
storage/connect/table.cpp
- Add function ExtractFromPath
modified:
storage/connect/colblk.cpp
storage/connect/plgdbsem.h
storage/connect/plgdbutl.cpp
- Enhance Cardinality for some table types
modified:
storage/connect/tabdos.cpp
storage/connect/tabmysql.cpp
storage/connect/tabmysql.h
storage/connect/tabodbc.cpp
storage/connect/tabodbc.h
storage/connect/tabsys.cpp
storage/connect/tabsys.h
storage/connect/xindex.cpp
storage/connect/xindex.h
storage/connect/xtable.h
- Add test on special column
modified:
storage/connect/tabfmt.cpp
- Add new files (added for block indexing)
modified:
storage/connect/CMakeLists.txt
modified:
storage/connect/ha_connect.cc
storage/connect/ha_connect.h
storage/connect/reldef.cpp
- Add INSERT/UPDATE support to PROXY tables
modified:
storage/connect/tabutil.cpp
storage/connect/tabutil.h
- Take care of SPECIAL columns
modified:
storage/connect/filamdbf.cpp
storage/connect/reldef.h
storage/connect/tabfmt.cpp
-Typo and misc
modified:
storage/connect/odbconn.cpp
storage/connect/tabfix.cpp
storage/connect/xindex.cpp
modified:
storage/connect/ha_connect.cc
- Suppress the use of connect.in by adding the connect_xtrace system variable.
modified:
storage/connect/ha_connect.cc
- Make column length, varchar, and temporal column types recognized in discovery
when using SRCDEF,or PIVOT.
modified:
storage/connect/ha_connect.cc
storage/connect/myconn.cpp
storage/connect/myutil.cpp
storage/connect/plgdbsem.h
- Avoid (rare) crash when using DECIMAL type. (buf was too small)
modified:
storage/connect/tabfmt.cpp
storage/connect/tabmysql.cpp
storage/connect/tabodbc.cpp
storage/connect/tabpivot.cpp
storage/connect/valblk.cpp
storage/connect/value.cpp
- General cleaning of unused code, standardize tracing, and update version number
modified:
storage/connect/block.h
storage/connect/colblk.cpp
storage/connect/connect.cc
storage/connect/csort.h
storage/connect/filamap.cpp
storage/connect/filamdbf.cpp
storage/connect/filamfix.cpp
storage/connect/filamzip.cpp
storage/connect/ha_connect.cc
storage/connect/mycat.cc
storage/connect/myconn.cpp
storage/connect/mysql-test/connect/r/alter.result
storage/connect/mysql-test/connect/r/xml.result
storage/connect/myutil.cpp
storage/connect/osutil.c
storage/connect/plgdbsem.h
storage/connect/plgdbutl.cpp
storage/connect/plugutil.c
storage/connect/reldef.cpp
storage/connect/tabcol.cpp
storage/connect/tabfmt.cpp
storage/connect/tabmysql.cpp
storage/connect/tabodbc.cpp
storage/connect/tabpivot.cpp
storage/connect/tabvct.cpp
storage/connect/user_connect.cc
storage/connect/valblk.cpp
storage/connect/value.cpp
storage/connect/xindex.cpp
