Commit graph

198057 commits

Author SHA1 Message Date
Diogo Teles Sant'Anna
e37a7197c7 Create SECURITY.md 2023-06-05 08:28:06 +10:00
Rucha Deodhar
4e5b771e98 MDEV-30677: Incorrect result for "SELECT JSON_SCHEMA_VALID('{}', NULL)"
Analysis: null_value is not set if any one of the arguments is NULL. So it
returns 1.
Fix: when either argument is NULL, set null_value to true, so that null can
be returned
2023-05-03 12:33:11 +05:30
Rucha Deodhar
97675570ca MDEV-30689: JSON_SCHEMA_VALID for type=array return 1 for any string that
starts with '['

Analysis:
When type is non-scalar and the json document has syntax error
then it is not detected during validating type. And Since other validate
functions take const argument, the error state is not stored eventually.
Fix:
After we run out of all schemas (in case of no error during validation) from
the schema list, go over the json document until there is error in parsing
or json doc has ended.
2023-05-03 12:31:45 +05:30
Daniel Lenski
3ef111610b [MDEV-29827] Clear error when --event-scheduler=ON is combined with --skip-grant-tables
When the server is started with `--event-scheduler=ON` and with
`--skip-grant-tables` (or built as embedded server which has no grant
tables at all), the event scheduler *appears* to be enabled (`SELECT
@@global.event_scheduler` returns `'ON'`), but attempting to
manipulate it in any way returns a misleading error message:

  "Cannot proceed, because event scheduler is disabled"

Possible solutions:

1. Fast-fail: fail immediately on startup if `EVENT_SCHEDULER` is set to
   any value other than `DISABLED` when starting up without grant
   tables, then prevent `SET GLOBAL event_scheduler` while running.

   Problem: there are existing setup scripts and code which start with
   this combination and assume it will not fail.

2. Warn and change value: if `EVENT_SCHEDULER` is set to any value
   other than `DISABLED` when starting, print a warning and change it
   immediately to `DISABLED`.

   Advantage: The value of the `EVENT_SCHEDULER` system variable after
   startup will be consistent with its functional unavailability.

3. Display a clear error: if `EVENT_SCHEDULER` is enabled, but grant
   tables are not enabled, then ensure error messages clearly explain
   the fact that the combination is not supported.

   Advantage: The error message encountered by the end user when
   attempting to manipulate the event scheduler (such as `CREATE
   EVENT`) is clear and explicit.

This commit implements the combination of solutions (2) and (3): it
will set `EVENT_SCHEDULER=DISABLED` on startup (reflecting the
functional reality) and it will print a startup warning, *and* it will
print a *distinct* error message each time that an end user attempts to
manipulate the event scheduler, so that the end user will clearly understand
the problem even if the startup messages are not visible at that point.

It also adds an MTR test `main.events_skip_grant_tables` to verify the
expected behavior, and the unmodified `main.events_restart` test
continues to demonstrate no change in the error message when the event
scheduler is non-functional for *different* reasons.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services
2023-04-27 15:28:24 +10:00
Rucha Deodhar
7321c71aa1 MDEV-31032: UBSAN|downcast of address X which does not point to an
object of type 'Item_string' in sql/json_schema.cc

Analysis: make_string_literal() returns pointer of type
Item_basic_constant which is converted to pointer of type Item_string. Now,
Item_string is base class of Item_basic_constant, so the error about
downcasting.
Fix: using constructor of Item_string type directly instead of
downcasting would be more appropriate.
2023-04-26 15:15:47 +05:30
Rucha Deodhar
4b67ff3b25 MDEV-30705: JSON_SCHEMA_VALID: schema with multipleOf for big value
always return 1

Analysis: Implementation used double to store value. longlong is better
choice
Fix: Use longlong for multiple_of and the value to store it and num_flag to
check for decimals.
2023-04-26 11:00:09 +05:30
Rucha Deodhar
2c4c7c8b02 MDEV-30704: JSON_SCHEMA_VALID: multipleOf must be greater than zero
Analysis: multipleOf must be strictly greater then 0. However the
implementation only disallowed values strcitly less than 0
Fix: check if value is less than or equal to 0 instead of less than 0 and
return true.
Also fixed the incorrect return value for some other keywords.
2023-04-26 11:00:09 +05:30
Rucha Deodhar
dffd1679ba MDEV-30703: JSON_SCHEMA_VALID : Enum array must have at least one value
Analysis: Current implementation does not check the number of elements in
the enum array and whether they are unique or not.
Fix: Add a counter that counts number of elements and before inserting the
element in the enum hash check whether it exists.
2023-04-26 11:00:09 +05:30
Rucha Deodhar
d555f38af8 MDEV-30690: Server crashed on function JSON_SCHEMA_VALID with incorrect
input json schema

Analysis: In case of syntax error while scanning json schema, true is
returned inspite of it being wanring and not error.
Fix: return true instead of false.
2023-04-26 11:00:09 +05:30
Rucha Deodhar
1c25b5c026 MDEV-30977: Additional key values are not validating properly when using
unevaluatedProperties with properties declared in subschemas

Analysis:
When a key fails to validate for "properties" when "properties" is being
treated as alternate schema, it needs to fall back on alternate schema
for "properites" itself ("unevaluatedProperties" in context of the bug).
But that doesn't happen and we end up returning false (=validated)
Fix:
When "properties" fails to validate as an alternate schema, fall back on
alternate schema for "properties" itself.
2023-04-26 11:00:08 +05:30
Rucha Deodhar
ee41fa38fc MDEV-30995: JSON_SCHEMA_VALID is not validating case sensitive when using
regex

Analysis:
When initializing Regexp_processor_pcre object, we set the PCRE2_CASELESS
flag which is responsible for case insensitive comparison.
Fix:
Unset the flag after initializing.
2023-04-26 11:00:08 +05:30
Rucha Deodhar
8939e21dc5 MDEV-30795: JSON_SCHEMA_VALID bugs mentioned in comment
comment 2)
Analysis:
flag to check unique gets reset every time. Hence unique values cannot be
correctly checked
Fix:
do not reset the flag that checks unique for null, true and false
values.

comment 3)
Analysis:
current implementation checks for appropriate value but does not
return true.
Fix:
return true on error

comment 4)
Analysis:
Current implementation did not check for value type for values
inside required array.
Fix:
Check values inside required array
2023-04-26 11:00:08 +05:30
Rucha Deodhar
358b8495f5 MDEV-27128: Implement JSON Schema Validation FUNCTION
Implementation:
Implementation is made according to json schema validation draft 2020

JSON schema basically has same structure as that of json object, consisting
of key-value pairs. So it can be parsed in the same manner as
any json object.

However, none of the keywords are mandatory, so making guess about the
json value type based only on the keywords would be incorrect.
Hence we need separate objects denoting each keyword.

So during create_object_and_handle_keyword() we create appropriate objects
based on the keywords and validate each of them individually on the json
document by calling respective validate() function if the type matches.
If any of them fails, return false, else return true.
2023-04-26 11:00:08 +05:30
Oleg Smirnov
af0e0ad18d MDEV-30946 Index usage for DATE(datetime_column) = const does not work for DELETE and UPDATE
Add date conditions transformation to the execution paths
of DELETE and UPDATE commands.

Strip excessive prepared statements from the test file
2023-04-25 20:21:36 +07:00
Oleg Smirnov
9f9a53be40 MDEV-30901 Index usage for DATE(datetime_column) = const does not work for engine Memory
Fix incorrect approach to determine whether a field is part of an index.
Remove "force index" hints from tests.
Add tests with composite indexes
2023-04-25 20:21:35 +07:00
Oleg Smirnov
f0b665f880 MDEV-8320 Allow index usage for DATE(col) <=> const and YEAR <=> const
Rewrite datetime comparison conditions into sargeable. For example,
    YEAR(col) <= val  ->  col <= YEAR_END(val)
    YEAR(col) <  val  ->  col <  YEAR_START(val)
    YEAR(col) >= val  ->  col >= YEAR_START(val)
    YEAR(col) >  val  ->  col >  YEAR_END(val)
    YEAR(col) =  val  ->  col BETWEEN YEAR_START(val) AND YEAR_END(val)
Do the same with DATE(col), for example:
    DATE(col) <= val  ->  col <= DAY_END(val)

After such a rewrite index lookup on column "col" can be employed
2023-04-25 20:21:35 +07:00
Yuchen Pei
54c11273e3
MDEV-28363 remove #ifdef SPIDER_use_LEX_CSTRING_for_Field_blob_constructor 2023-04-14 10:44:10 +10:00
Junqi Xie
d20a96f9c1 MDEV-21921 Make transaction_isolation and transaction_read_only into system variables
In MariaDB, we have a confusing problem where:
* The transaction_isolation option can be set in a configuration file, but it cannot be set dynamically.
* The tx_isolation system variable can be set dynamically, but it cannot be set in a configuration file.

Therefore, we have two different names for the same thing in different contexts. This is needlessly confusing, and it complicates the documentation. The same thing applys for transaction_read_only.

MySQL 5.7 solved this problem by making them into system variables. https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-20.html

This commit takes a similar approach by adding new system variables and marking the original ones as deprecated. This commit also resolves some legacy problems related to SET STATEMENT and transaction_isolation.
2023-04-12 11:04:29 +10:00
Daniel Black
4472a7b4ff MDEV-30205: /usr/share/mysql-test -> mariadb-test (fix)
A suppression was needed for encryption.innodb-bad-key-change
due to the path change.
2023-04-11 17:47:34 +10:00
Daniel Black
320a4b52c9 MDEV-30205 Move /usr/share/mysql to /usr/share/mariadb
These are mainly internal files so is a low impact change.

The few scripts/mysql*sql where renames to mariadb_* prefix
on the name.

mysql-test renamed to mariadb-test in the final packages
2023-04-11 07:59:02 +10:00
Marko Mäkelä
2b61ff8f22 Merge 11.0 into 11.1 2023-03-29 17:23:21 +03:00
Marko Mäkelä
5e01255732 Merge 10.11 into 11.0 2023-03-29 17:20:42 +03:00
Marko Mäkelä
d84a282629 Merge 10.10 into 10.11 2023-03-29 16:53:37 +03:00
Marko Mäkelä
191821f7df Merge 10.9 into 10.10 2023-03-29 15:29:02 +03:00
Marko Mäkelä
55e78ebf41 Merge 10.8 into 10.9 2023-03-29 15:28:13 +03:00
Marko Mäkelä
dd2fe81122 Merge 10.6 into 10.8 2023-03-29 15:16:42 +03:00
Marko Mäkelä
0760ad3336 Merge 10.5 into 10.6 2023-03-28 15:25:52 +03:00
Marko Mäkelä
402f36dd65 MDEV-30936 fixup
fil_space_t::~fil_space_t(): Invoke ut_free(name) because
doing so in the callers would trip MSAN_OPTIONS=poison_in_dtor=1
2023-03-28 15:10:32 +03:00
Marko Mäkelä
dfa90257f6 MDEV-30936 clang 15.0.7 -fsanitize=memory fails massively
handle_slave_io(), handle_slave_sql(), os_thread_exit():
Remove a redundant pthread_exit(nullptr) call, because it
would cause SIGSEGV.

mysql_print_status(): Add MEM_MAKE_DEFINED() to work around
some missing instrumentation around mallinfo2().

que_graph_free_stat_list(): Invoke que_node_get_next(node) before
que_graph_free_recursive(node). That is the logical and
MSAN_OPTIONS=poison_in_dtor=1 compatible way of freeing memory.

ins_node_t::~ins_node_t(): Invoke mem_heap_free(entry_sys_heap).

que_graph_free_recursive(): Rely on ins_node_t::~ins_node_t().

fts_t::~fts_t(): Invoke mem_heap_free(fts_heap).

fts_free(): Replace with direct calls to fts_t::~fts_t().

The failures in free_root() due to MSAN_OPTIONS=poison_in_dtor=1
will be covered in MDEV-30942.
2023-03-28 11:44:24 +03:00
Shivam Choudhary
6c3b1dced4 Fixed some typos in optimizer_costs.txt 2023-03-28 18:41:09 +11:00
Yuchen Pei
7bd225e129
MDEV-30920 Remove need_lock and table from spider_close_sys_table()
They became obsolete after commit cfd145faed:

commit cfd145faed
Author: Nayuta Yanagisawa <nayuta.yanagisawa@hey.com>
Date:   Fri Jan 28 01:03:06 2022 +0900

    MDEV-27641 Spider: remove #if MYSQL_VERSION_ID < ${VERSION}
2023-03-28 16:31:53 +11:00
Sergei Golubchik
c2b6916393 MDEV-19629 post-merge fixes
* it isn't "pfs" function, don't call it Item_func_pfs,
  don't use item_pfsfunc.*
* tests don't depend on performance schema, put in the main suite
* inherit from Item_str_ascii_func
* use connection collation, not utf8mb3_general_ci
* set result length in fix_length_and_dec
* do not set maybe_null
* use my_snprintf() where possible
* don't set m_value.ptr on every invocation
* update sys schema to use the format_pico_time()
* len must be size_t (compilation error on Windows)
* the correct function name for double->double is fabs()
* drop volatile hack
2023-03-27 21:27:27 +02:00
Ahmed Ibrahim
d9808f79de MDEV-19629: format_pico_time implementation 2023-03-27 16:34:29 +02:00
Marko Mäkelä
5a1f7522a5 Fix ColumnStore again
This fixes up commit e371b1e264
that accidentally reverted d77aaa6994.
2023-03-27 13:43:18 +03:00
Marko Mäkelä
f6c5e917e0 Merge 11.0 into 11.1 2023-03-27 12:53:57 +03:00
Tuukka Pasanen
31487f4b2b MDEV-30837: Remove usage of AWK in autobake-debs.sh
AWK is used in autobake-debs.sh for printing information
about created DEB packages.

This can be rewrite with bash inner commands read and echo.
2023-03-27 16:28:48 +11:00
Tuukka Pasanen
fe32a4a5bb MDEV-30837: Remove usage of AWK from Debian init and postinst scripts
AWK in used in Debian SysV-init and postinst scripts to determine
is there enough space starting MariaDB database or create new
database to target destination.

These AWK scripts can be rewrited to use pure SH or help
using Coreutils which is mandatory for usage of MariaDB currently.

Reasoning behind this is to get rid of one very less used dependency
2023-03-27 16:28:48 +11:00
Andrei
216d99bb39 MDEV-26071: rpl.rpl_perfschema_applier_status_by_worker failed in bb …
…with: Test assertion failed

Problem:
=======
Assertion text: 'Value returned by SSS and PS table for Last_Error_Number
should be same.'
Assertion condition: '"1146" = "0"'
Assertion condition, interpolated: '"1146" = "0"'
Assertion result: '0'

Analysis:
========
In parallel replication when slave is started the worker pool gets
activated and it gets cleared when slave stops. Each time the worker pool
gets activated a backup worker pool also gets created to store worker
specific perforance schema information in case of errors. On error, all
relevant information is copied from rpl_parallel_thread to rli and it gets
cleared from thread.  Then server waits for all workers to complete their
work, during this stage performance schema table specific worker info is
stored into the backup pool and finally the actual pool gets cleared. If
users query the performance schema table to know the status of workers the
information from backup pool will be used. The test simulates
ER_NO_SUCH_TABLE error and verifies the worker information in pfs table.
Test works fine if execution occurs in following order.

Step 1. Error occurred 'worker information is copied to backup pool'.
Step 2. handle_slave_sql invokes 'rpl_parallel_resize_pool_if_no_slaves' to
deactivate worker pool, it marks the pool->count=0
Step 3. PFS table is queried, since actual pool is deactivated backup pool
information is read.

If the Step 3 happens prior to Step2 the pool is yet to be deactivated and
the actual pool is read, which doesn't have any error details as they were
cleared. Hence test ocasionally fails.

Fix:
===
Upon error mark the back pool as being active so that if PFS table is
quried since the backup pool is flagged as valid its information will be
read, in case it is not flagged regular pool will be read.

This work is one of the last pieces created by the late Sujatha Sivakumar.
2023-03-24 15:56:24 +02:00
Marko Mäkelä
e371b1e264 MDEV-28883 fixup: clang -Winconsistent-missing-override 2023-03-24 15:12:08 +02:00
Thirunarayanan Balathandayuthapani
e06c6046d2 MDEV-29545 InnoDB: Can't find record during replace stmt
Problem:
========
- InnoDB replace statement returns can't find record as result during
bulk insert operation. InnoDB returns DB_END_OF_INDEX blindly when
bulk transaction is visible to current transaction even though
the search tuple is inserted as a part of current replace statement.

Solution:
=========
row_search_mvcc(): InnoDB should allow the transaction to read
all the rows when innodb intends to do any locking on the
record even though bulk insert transaction changes are
visible to the current transaction
2023-03-24 15:20:21 +05:30
Marko Mäkelä
07460c31e3 MDEV-30900 Crash on macOS due to zero-initialized buf_dblwr.write_cond
buf_dblwr_t::init(), buf_dblwr_t::close(): Cover also write_cond,
which was added in commit a55b951e60
without explicit initialization. On GNU/Linux, PTHREAD_COND_INITIALIZER
is a zero-initializer. That is why the default zero initialization
happened to work on that platform.
2023-03-24 07:59:27 +02:00
Tuukka Pasanen
a79abb6517 MDEV-30778: Remove Awk from mysql_install_db
Commit reduces need of AWK-command at least
for Debian mariadb-server-compat package.

Commit removes need of AWK-command from
scripts/mysql_install_db.sh script.

AWK command is replace by purely Posix sh compiliant version.
2023-03-24 12:54:21 +11:00
Otto Kekalainen
50c8ef01fc Fix trivial spelling errors
- agressively -> aggressively
- exising -> existing
- occured -> occurred
- releated -> related
- seperated -> separated
- sucess -> success
- use use -> use

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
2023-03-24 12:54:05 +11:00
Daniel Black
15ca6c5a2f rpl_reporting: sprintf -> snprintf
This was failing to compile with AppleClang 14.0.0.14000029.

Thanks to Arunesh Choudhary for noticing.
2023-03-24 12:04:16 +11:00
Aleksey Midenkov
a8b616d1e9 MDEV-30421 rpl_parallel_*.test cleanup
Moved rpl_parallel_*.inc to rpl_parallel_*.test
2023-03-23 22:31:55 +03:00
Marko Mäkelä
9aa098c46b Merge 11.0 into 11.1 2023-03-23 12:38:46 +02:00
Marko Mäkelä
92772485b7 MDEV-30911 Multi-batch recovery of ROW_FORMAT=COMPRESSED table hangs
In commit d6aed21621 a condition at
the start of buf_read_ahead_random() was refactored. Only the caller
buf_read_recv_pages() was adjusted for this. We must in fact adjust
every caller and make sure that spare blocks will be allocated
while crash recovery is in progress. This is the simplest fix;
ideally recovery would operate on the compressed page frame.

The observed recovery hang occurred because pages 0 and 3 of a
tablespace were being read due to buf_page_get_gen() calls by
trx_resurrect_table_locks() before the log records for these pages
had been applied. In buf_page_t::read_complete() we would skip
the call to recv_recover_page() because no uncompressed page frame
had been allocated for the block.
2023-03-23 12:21:48 +02:00
Nayuta Yanagisawa
22392b36ee
MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HS
The HandlerSocket support of Spider has been deleted by MDEV-26858.
Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.
2023-03-23 10:07:52 +11:00
Marko Mäkelä
1efdf67e60 Merge 10.5 into 10.6 2023-03-22 15:54:45 +02:00
Marko Mäkelä
701399ad37 MDEV-30882 Crash on ROLLBACK in a ROW_FORMAT=COMPRESSED table
btr_cur_upd_rec_in_place(): Avoid calling page_zip_write_rec() if we
are not modifying any fields that are stored in compressed format.

btr_cur_update_in_place_zip_check(): New function to check if a
ROW_FORMAT=COMPRESSED record can actually be updated in place.

btr_cur_pessimistic_update(): If the BTR_KEEP_POS_FLAG is not set
(we are in a ROLLBACK and cannot write any BLOBs), ignore the potential
overflow and let page_zip_reorganize() or page_zip_compress() handle it.
This avoids a failure when an attempted UPDATE of an NULL column to 0 is
rolled back. During the ROLLBACK, we would try to move a non-updated
long column to off-page storage in order to avoid a compression failure
of the ROW_FORMAT=COMPRESSED page.

page_zip_write_trx_id_and_roll_ptr(): Remove an assertion that would fail
in row_upd_rec_in_place() because the uncompressed page would already
have been modified there.

This is a 10.5 version of commit ff3d4395d8
(different because of commit 08ba388713).
2023-03-22 15:19:52 +02:00