Commit graph

420 commits

Author SHA1 Message Date
Sergei Golubchik
2010cfab2a remove GET_ADJUST_VALUE
avoid contaminating my_getopt with sysvar implementation details.
adjust variable values after my_getopt, like it's done for others.
this fixes --help to show correct values.
2023-02-10 12:59:36 +02:00
Monty
b66cdbd1ea Changing all cost calculation to be given in milliseconds
This makes it easier to compare different costs and also allows
the optimizer to optimizer different storage engines more reliably.

- Added tests/check_costs.pl, a tool to verify optimizer cost calculations.
  - Most engine costs has been found with this program. All steps to
    calculate the new costs are documented in Docs/optimizer_costs.txt

- User optimizer_cost variables are given in microseconds (as individual
  costs can be very small). Internally they are stored in ms.
- Changed DISK_READ_COST (was DISK_SEEK_BASE_COST) from a hard disk cost
  (9 ms) to common SSD cost (400MB/sec).
- Removed cost calculations for hard disks (rotation etc).
- Changed the following handler functions to return IO_AND_CPU_COST.
  This makes it easy to apply different cost modifiers in ha_..time()
  functions for io and cpu costs.
  - scan_time()
  - rnd_pos_time() & rnd_pos_call_time()
  - keyread_time()
- Enhanched keyread_time() to calculate the full cost of reading of a set
  of keys with a given number of ranges and optional number of blocks that
  need to be accessed.
- Removed read_time() as keyread_time() + rnd_pos_time() can do the same
  thing and more.
- Tuned cost for: heap, myisam, Aria, InnoDB, archive and MyRocks.
  Used heap table costs for json_table. The rest are using default engine
  costs.
- Added the following new optimizer variables:
  - optimizer_disk_read_ratio
  - optimizer_disk_read_cost
  - optimizer_key_lookup_cost
  - optimizer_row_lookup_cost
  - optimizer_row_next_find_cost
  - optimizer_scan_cost
- Moved all engine specific cost to OPTIMIZER_COSTS structure.
- Changed costs to use 'records_out' instead of 'records_read' when
  recalculating costs.
- Split optimizer_costs.h to optimizer_costs.h and optimizer_defaults.h.
  This allows one to change costs without having to compile a lot of
  files.
- Updated costs for filter lookup.
- Use a better cost estimate in best_extension_by_limited_search()
  for the sorting cost.
- Fixed previous issues with 'filtered' explain column as we are now
  using 'records_out' (min rows seen for table) to calculate filtering.
  This greatly simplifies the filtering code in
  JOIN_TAB::save_explain_data().

This change caused a lot of queries to be optimized differently than
before, which exposed different issues in the optimizer that needs to
be fixed.  These fixes are in the following commits.  To not have to
change the same test case over and over again, the changes in the test
cases are done in a single commit after all the critical change sets
are done.

InnoDB changes:
- Updated InnoDB to not divide big range cost with 2.
- Added cost for InnoDB (innobase_update_optimizer_costs()).
- Don't mark clustered primary key with HA_KEYREAD_ONLY. This will
  prevent that the optimizer is trying to use index-only scans on
  the clustered key.
- Disabled ha_innobase::scan_time() and ha_innobase::read_time() and
  ha_innobase::rnd_pos_time() as the default engine cost functions now
  works good for InnoDB.

Other things:
- Added  --show-query-costs (\Q) option to mysql.cc to show the query
  cost after each query (good when working with query costs).
- Extended my_getopt with GET_ADJUSTED_VALUE which allows one to adjust
  the value that user is given. This is used to change cost from
  microseconds (user input) to milliseconds (what the server is
  internally using).
- Added include/my_tracker.h  ; Useful include file to quickly test
  costs of a function.
- Use handler::set_table() in all places instead of 'table= arg'.
- Added SHOW_OPTIMIZER_COSTS to sys variables. These are input and
  shown in microseconds for the user but stored as milliseconds.
  This is to make the numbers easier to read for the user (less
  pre-zeros).  Implemented in 'Sys_var_optimizer_cost' class.
- In test_quick_select() do not use index scans if 'no_keyread' is set
  for the table. This is what we do in other places of the server.
- Added THD parameter to Unique::get_use_cost() and
  check_index_intersect_extension() and similar functions to be able
  to provide costs to called functions.
- Changed 'records' to 'rows' in optimizer_trace.
- Write more information to optimizer_trace.
- Added INDEX_BLOCK_FILL_FACTOR_MUL (4) and INDEX_BLOCK_FILL_FACTOR_DIV (3)
  to calculate usage space of keys in b-trees. (Before we used numeric
  constants).
- Removed code that assumed that b-trees has similar costs as binary
  trees. Replaced with engine calls that returns the cost.
- Added Bitmap::find_first_bit()
- Added timings to join_cache for ANALYZE table (patch by Sergei Petrunia).
- Added records_init and records_after_filter to POSITION to remember
  more of what best_access_patch() calculates.
- table_after_join_selectivity() changed to recalculate 'records_out'
  based on the new fields from best_access_patch()

Bug fixes:
- Some queries did not update last_query_cost (was 0). Fixed by moving
  setting thd->...last_query_cost in JOIN::optimize().
- Write '0' as number of rows for const tables with a matching row.

Some internals:
- Engine cost are stored in OPTIMIZER_COSTS structure.  When a
  handlerton is created, we also created a new cost variable for the
  handlerton. We also create a new variable if the user changes a
  optimizer cost for a not yet loaded handlerton either with command
  line arguments or with SET
  @@global.engine.optimizer_cost_variable=xx.
- There are 3 global OPTIMIZER_COSTS variables:
  default_optimizer_costs   The default costs + changes from the
                            command line without an engine specifier.
  heap_optimizer_costs      Heap table costs, used for temporary tables
  tmp_table_optimizer_costs The cost for the default on disk internal
                            temporary table (MyISAM or Aria)
- The engine cost for a table is stored in table_share. To speed up
  accesses the handler has a pointer to this. The cost is copied
  to the table on first access. If one wants to change the cost one
  must first update the global engine cost and then do a FLUSH TABLES.
  This was done to be able to access the costs for an open table
  without any locks.
- When a handlerton is created, the cost are updated the following way:
  See sql/keycaches.cc for details:
  - Use 'default_optimizer_costs' as a base
  - Call hton->update_optimizer_costs() to override with the engines
    default costs.
  - Override the costs that the user has specified for the engine.
  - One handler open, copy the engine cost from handlerton to TABLE_SHARE.
  - Call handler::update_optimizer_costs() to allow the engine to update
    cost for this particular table.
  - There are two costs stored in THD. These are copied to the handler
    when the table is used in a query:
    - optimizer_where_cost
    - optimizer_scan_setup_cost
- Simply code in best_access_path() by storing all cost result in a
  structure. (Idea/Suggestion by Igor)
2023-02-02 23:54:45 +03:00
Oleksandr Byelkin
50b270525a Merge branch '10.7' into 10.8 2022-08-08 17:15:13 +02:00
Oleksandr Byelkin
d2f1c3ed6c Merge branch '10.5' into bb-10.6-release 2022-08-03 12:19:59 +02:00
Sergei Golubchik
086eb8e23c my_getopt --help for "bit" options
print "Defaults to on" not only for GET_BOOL, but also for GET_BIT
2022-08-02 17:49:21 +02:00
Vladislav Vaintroub
2e48fbe3f5 MDEV-27525 Invalid (non-UTF8) characters found for option 'plugin_dir'
Two Problems
1. Upgrade wizard failed to retrieve path to service executable,
if it contained non-ASCII.
Fixed by setlocale(LC_ALL, "en_US.UTF8"), which was missing in upgrade wizard

2.mysql_upgrade_service only updated (converted to UTF8) the server's sections
leaving client's as-is

Corrected typo.

3. Fixed assertion in my_getopt, turns out to be too strict.
2022-01-18 17:32:53 +01:00
Vladislav Vaintroub
ba9d231b5a MDEV-26713 Set activeCodePage=UTF8 for windows programs
- Use corresponding entry in the manifest, as described in
https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page

- If if ANSI codepage is UTF8 (i.e for Windows 1903 and later)
  Use UTF8 as default client charset
  Set console codepage(s) to UTF8, in case process is using console

- Allow some previously disabled MTR tests, that used Unicode for in "exec",
  for the recent Windows versions
2021-12-15 19:13:57 +01:00
Vladislav Vaintroub
0102732686 Revert "MDEV-26713 Windows - improve utf8 support for command line tools"
This reverts commit several commits pushed by mistake.
2021-11-19 09:46:57 +01:00
Vladislav Vaintroub
220dc1fd59 xxx 2021-11-18 17:25:41 +01:00
Daniel Black
46852b3bbb MDEV-22219: error on parsing negative unsigned options 2021-04-07 14:04:55 +10:00
Marko Mäkelä
c515b1d092 Merge 10.4 into 10.5 2020-06-18 13:58:54 +03:00
Vlad Lesin
9bdf35e90f MDEV-18215: mariabackup does not report unknown command line options
MDEV-21298: mariabackup doesn't read from the [mariadbd] and [mariadbd-X.Y]
server option groups from configuration files
MDEV-21301: mariabackup doesn't read [mariadb-backup] option group in
configuration file

All three issues require to change the same code, that is why their
fixes are joined in one commit.

The fix is in invoking load_defaults_or_exit() and handle_options() for
backup-specific groups separately from client-server groups to let the last
handle_options() call fail on unknown backup-specific options.

The order of options procesing is the following:
1) Load server groups and process server options, ignore unknown
options
2) Load client groups and process client options, ignore unknown
options
3) Load backup groups and process client-server options, exit on
unknown option
4) Process --mysqld-args command line options, ignore unknown options

New global flag my_handle_options_init_variables was added to have
ability to invoke handle_options() for the same allowed options set
several times without re-initialising previously set option values.

--password value destroying is moved from option processing callback to
mariabackup's handle_options() function to have ability to invoke server's
handle_options() several times for the same possible allowed options
set.

Galera invokes wsrep_sst_mariabackup.sh with mysqld command line
options to configure mariabackup as close to the server as possible.
It is not known what server options are supported by mariabackup when the
script is invoked. That is why new mariabackup option "--mysqld-args" is added,
all unknown options that follow this option will be silently ignored.

wsrep_sst_mariabackup.sh was also changed to:
- use "--mysqld-args" mariabackup option to pass mysqld options,
- remove deprecated innobackupex mode,
- remove unsupported mariabackup options:
    --encrypt
    --encrypt-key
    --rebuild-indexes
    --rebuild-threads
2020-06-14 13:23:07 +03:00
Sergei Golubchik
7c58e97bf6 perfschema memory related instrumentation changes 2020-03-10 19:24:22 +01:00
Marko Mäkelä
adae286e35 Amend 3ea51b518b
Remove redundant NULL checks for __attribute__((nonnull)) parameters,
because clang warns about them.
2019-10-28 16:21:42 +02:00
Sergei Golubchik
f217612fad MDEV-12684 Show what config file a sysvar got a value from
change get_one_option() prototype to pass the filename and
not to pass the redundant optid.
2019-10-14 10:29:30 +02:00
Sergei Golubchik
927521a2c1 MDEV-12684 Show what config file a sysvar got a value from
make load_defaults() store the file name in the generated option list
using a special marker ---file-marker--- option.

Pick up this filename in handle_options().

Remove ---args-separator---, use ---file-marker--- with an empty file
name instead - this simplifies checks on the caller, only one special
option to recognize.
2019-10-14 10:29:30 +02:00
Sergei Golubchik
f7b8d14490 cleanup: don't use my_getopt_is_args_separator()
only my_getopt should use it, because it changes my_getopt's behavior.
If one simply wants to skip the separator - don't ask it to be added
in the first place
2019-10-14 10:29:30 +02:00
Sergei Golubchik
3ea51b518b cleanup: my_getopt, get_one_option isn't optional
it turns out that practically every single user of handle_options()
used the get_one_option callback. Simplify the code,
make it mandatory, adjust unit tests.
2019-10-14 10:29:30 +02:00
Sergei Golubchik
eb3431d529 cleanup: my_getopt, consistency
almost all my_getopt settings and callbacks are global variables,
directly assignable to configure my_getopt. Only getopt_get_addr
was using a setter function. Get rid of it, make it a global
directly assignable variable like all other settings.

Also make getopt_compare_strings() static.
2019-10-14 10:29:30 +02:00
Marko Mäkelä
be85d3e61b Merge 10.2 into 10.3 2019-05-14 17:18:46 +03:00
Marko Mäkelä
26a14ee130 Merge 10.1 into 10.2 2019-05-13 17:54:04 +03:00
Vicențiu Ciorbaru
cb248f8806 Merge branch '5.5' into 10.1 2019-05-11 22:19:05 +03:00
Vicențiu Ciorbaru
5543b75550 Update FSF Address
* Update wrong zip-code
2019-05-11 21:29:06 +03:00
Vladislav Vaintroub
6c279ad6a7 MDEV-15091 : Windows, 64bit: reenable and fix warning C4267 (conversion from 'size_t' to 'type', possible loss of data)
Handle string length as size_t, consistently (almost always:))
Change function prototypes to accept size_t, where in the past
ulong or uint were used. change local/member variables to size_t
when appropriate.

This fix excludes rocksdb, spider,spider, sphinx and connect for now.
2018-02-06 12:55:58 +00:00
Otto Kekäläinen
c9c28bef3c Minor spelling fixes in code comments, docs and output
This commit does not touch any variable names or any other actual code,
and thus should not in any way affect how the code works.
2018-01-12 16:49:02 +02:00
Daniel Black
3464b67025 mysys: handle T, P, E suffixs
Signed-off-by: Daniel Black <daniel@linux.vnet.ibm.com>
2017-12-19 13:40:05 +04:00
Marko Mäkelä
8409f721ff setval(): Do not fall through from GET_FLAGSET to GET_BIT 2017-11-10 16:11:15 +02:00
Sergey Vojtovich
3722372ae5 MDEV-11549 - Twin include
Removed duplicate include.
2017-10-20 13:56:49 +04:00
Marko Mäkelä
2c1067166d Merge bb-10.2-ext into 10.3 2017-10-04 08:24:06 +03:00
Vladislav Vaintroub
7354dc6773 MDEV-13384 - misc Windows warnings fixed 2017-09-28 17:20:46 +00:00
Monty
21518ab2e4 New option for slow logging (log_slow_disable_statements)
This fixes MDEV-7742 and MDEV-8305 (Allow user to specify if stored
procedures should be logged in the slow and general log)

New functionality:
- Added new variables log_slow_disable_statements and log_disable_statements
  that can be used to disable logging of certain queries to slow and
  general log. Currently supported options are 'admin', 'call', 'slave'
  and 'sp'.
  Defaults are as before. Only 'sp' (stored procedure statements) is
  disabled for  slow and general_log.
- Slow log to files now includes the following new information:
  - When logging stored procedure statements the name of stored
    procedure is logged.
  - Number of created tmp_tables, tmp_disk_tables and the space used
    by temporary tables.
- When logging 'call', the logged status now contains the sum of all
  included statements.  Before only 'time' was correct.
- Added filsort_priority_queue as an option for log_slow_filter (this
  variable existed before, but was not exposed)
- Added support for BIT types in my_getopt()

Mapped some old variables to bitmaps (old variables can still be used)
- Variable 'log_queries_not_using_indexes' is mapped to
  log_slow_filter='not_using_index'
- Variable 'log_slow_slave_statements' is mapped to
  log_slow_disabled_statements='slave'
- Variable 'log_slow_admin_statements' is mapped to
  log_slow_disabled_statements='admin'
- All the above variables are changed to session variables from global
  variables

Other things:
- Simplified LOGGER::log_command. We don't need to check for super if
  OPTION_LOG_OFF is set as this flag can only be set if one is a super
  user.
- Removed some setting of enable_slow_log as it's guaranteed to be set by
  mysql_parse()
- mysql_admin_table() now sets thd->enable_slow_log
- Added prepare_logs_for_admin_command() to reset thd->enable_slow_log if
  needed.
- Added new functions to store, restore and add slow query status
- Added new functions to store and restore query start time
- Reorganized Sub_statement_state according to types
- Added code in dispatch_command() to ensure that
  thd->reset_for_next_command() is always called for a query.
- Added thd->last_sql_command to simplify checking of what was the type
  of the last command. Needed when logging to slow log as lex->sql_command
  may have changed before slow logging is called.
- Moved QPLAN_TMP_... to where status for tmp tables are updated
- Added new THD variable, affected_rows, to be able to correctly log
  number of affected rows to slow log.
2017-08-24 01:05:51 +02:00
Marko Mäkelä
f1f8ebc325 Merge 10.0 into 10.1 2017-01-26 23:40:11 +02:00
Sachin Setiya
18ef02b04d MDEV-4774 Strangeness with max_binlog_stmt_cache_size Settings
Problem:- When setting max_binlog_stmt_cache_size=18446744073709547520
from either command line or .cnf file, server fails to start.

Solution:- Added one more function eval_num_suffix_ull , which uses
strtoull to get unsigned ulonglong from string. And getopt_ull calls this
function instead of eval_num_suffix. Also changed previous eval_num_suffix to
eval_num_suffix_ll to remain consistent.
2017-01-20 21:33:36 +05:30
Sergei Golubchik
a2bcee626d Merge branch '10.0' into 10.1 2015-12-21 21:24:22 +01:00
Sergei Golubchik
1623995158 Merge branch '5.5' into 10.0 2015-12-13 00:10:40 +01:00
Sergei Golubchik
abf9d35213 Merge branch 'mysql/5.5' into 5.5 2015-12-09 10:00:49 +01:00
Sergei Golubchik
daf3551cce add missing DBUG_RETURN-s 2015-11-16 07:55:55 +01:00
Sergei Golubchik
bc12d5fd43 MDEV-6066: Merge new defaults from 5.6 and 5.7
cosmetic fixes. test fixes.
2015-09-04 10:33:56 +02:00
Oleksandr Byelkin
21daa7b929 MDEV-6066: Merge new defaults from 5.6 and 5.7 (autoset)
--autoset- command line prefix added
2015-09-04 10:33:55 +02:00
Arun Kuruvila
ddcad361d6 Bug #21503595 : --QUERY-ALLOC-BLOCK-SIZE=-1125899906842624 +
PID_FILE CHECK LEADS TO OOM SIG 11

Description:- A server started with 'query_alloc_block_size'
option set to a certain range of negative values on a
machine without enough memory may lead to OOM.

Analysis:- Server uses 'strtoull()' to convert server
variable values of type 'GET_UINT', 'GET_ULONG' or 'GET_ULL'
from string to unsigned long long. According to the man
page, 'strtoull()' function returns either the result of the
conversion or, if there was a leading minus sign, the
negation of the result of the conversion represented as an
unsigned value, unless the original(nonnegated) value would
overflow; in the latter case, strtoull() returns ULLONG_MAX
and sets errno to ERANGE. So 'strtoull()' converts a small
negative value to a larger postive value. For example string
'-1125899906842624' will be converted to an unsigned value,
'18445618173802708992' (ulonglong typecast of
'-1125899906842624'). So a
server started with 'query_alloc_block_size' set to
"-1125899906842624" on a machine without enough memory will
lead to OOM since server allocates '18445618173802708992'
bytes(17178820608 GB) for query allocation block.

Fix:- When server is started with any server variable, of
type "GET_UINT", "GET_ULONG" or "GET_ULL", set to a negative
value, a warning, "option xxx: value -yyy adjusted to zzz"
is thrown and the value is adjusted to the lowest possible
value for that variable. The dynamic server variable which
is configured through the client exhibit the same behavior
as fix made for variables configured during the server
start up.
2015-09-04 11:10:57 +05:30
Monty
7332af49e4 - Renaming variables so that they don't shadow others (After this patch one can compile with -Wshadow and get much fewer warnings)
- Changed ER(ER_...) to ER_THD(thd, ER_...) when thd was known or if there was many calls to current_thd in the same function.
- Changed ER(ER_..) to ER_THD_OR_DEFAULT(current_thd, ER...) in some places where current_thd is not necessary defined.
- Removing calls to current_thd when we have access to thd

Part of this is optimization (not calling current_thd when not needed),
but part is bug fixing for error condition when current_thd is not defined
(For example on startup and end of mysqld)

Notable renames done as otherwise a lot of functions would have to be changed:
- In JOIN structure renamed:
   examined_rows -> join_examined_rows
   record_count -> join_record_count
- In Field, renamed new_field() to make_new_field()

Other things:
- Added DBUG_ASSERT(thd == tmp_thd) in Item_singlerow_subselect() just to be safe.
- Removed old 'tab' prefix in JOIN_TAB::save_explain_data() and use members directly
- Added 'thd' as argument to a few functions to avoid calling current_thd.
2015-07-06 20:24:14 +03:00
Sergei Golubchik
4280b25ed8 --getopt-prefix-matching command-line option 2015-02-10 10:21:15 +01:00
Sergei Golubchik
b050354ffb compiler warnings 2015-01-31 21:51:45 +01:00
Sergei Golubchik
dc64ba2187 MDEV-6137 better help for SET/ENUM sysvars
Auto-generate the allowed list of values for enum/set/flagset options
in --help output. But don't do that when the help text already has them.

Also, remove lists of values from help strings of various options, where
they were simply listed without any additional information.
2014-06-19 12:02:23 +02:00
Sergey Vojtovich
e5a80a307c MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the
MariaDB 10.0 server
2013-11-25 18:49:40 +04:00
Nirbhay Choubey
1297a9ef19 Fix for a compiler warning. 2013-11-25 13:01:57 -05:00
unknown
57a267a8c0 Merge from 10.0-base to 10.0 the feature MDEV-4506: Parallel replication.
The merge is still missing a few hunks related to temporary tables and
InnoDB log file size. The associated code did not seem to exist in
10.0, so the merge of that needs more work. Until this is fixed, there
are a number of test failures as a result.
2013-11-01 12:00:11 +01:00
unknown
cb86ce60b9 Merge MDEV-4506: Parallel replication into 10.0-base. 2013-11-01 09:17:06 +01:00
Michael Widenius
b722aae621 Allow unique prefix for command line options, like any GNU program. 2013-09-23 20:17:46 +03:00
unknown
e5746665c9 merge 10.0-base -> 10.0 2013-09-26 21:20:15 +03:00