create_process() temporarily changes STDOUT/STDERR output to error log file
This might redirect mtr output on Windows, so avoid it by holding
flush_lock.
--skip-not-found switch tells mtr to skip not found tests instead of aborting.
But it failed to skip the test if the suite name was not found.
This problem also made the *last-N-failed builbot builders fail
to run `mtr --skip-not-found` if the last commit removed a file in
the mysql-test/include/ directory.
This commit fixes it, now the not found test is properly skipped,
no matter what component of the test name was not found:
$ ./mtr main.foo --skip-not-found foo.main
...
==============================================================================
TEST WORKER RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------
foo.main [ skipped ] not found
main.foo [ skipped ] not found
--------------------------------------------------------------------------
With the result like
encryption.innochecksum 'debug' [ skipped ] combination not found
instead of
*** ERROR: Could not run encryption.innochecksum with 'debug' combination(s)
New mtr option --skip-not-found makes it to show not found tests
as skipped
main.a [ skipped ] not found
(but only if the test was specified with the suite name)
and not error out early with
mysql-test-run: *** ERROR: Could not find 'a' in 'main' suite
This is useful in buildbot, on builders that generate the list
of tests dynamically.
Windows C runtime does not implement line buffering mode for stdio.
This sometimes makes output from different tests interleaved in MTR
MTR relies on this buffering (lines won't output until "\n") to correctly
work in parallel scenarios.
Implement do-it-yourself line buffering on Windows, to workaround.
otherwise, e.g.
./mtr main.mysql_install_db_win_admin,innodb
results in sporadic
mysql-test-run: *** ERROR: Could not run main.mysql_install_db_win_admin with 'innodb' combination(s)
depending on whether it'll process the skip (not windows admin)
or the innodb.combinations first (if skip is processed first,
innodb combination wasn't, making the further code think that the test
doesn't have innodb combination)
MSYS2 is basically Cygwin, except it has more easy installation (but
with tools which are not used) and it has some more control of path
conversion via MSYS2_ARG_CONV_EXCL and MSYS2_ENV_CONV_EXCL. So it
should be more Windows-friendly than Cygwin.
Installation
Similar to Cygwin, except installing patch requires additional command
run from shell:
pacman -S patch
MSYS2 still doesn't work as it returns wierd "Bad address" when
exec-ing forked process from create_process(). Same exec from
standalone perl -e runs just fine... :(
Cygwin is more Unix-oriented. It does not treat \n as \r\n in regexps
(fixed by \R), it supplies Unix-style paths (fixed by
mixed_path()). It does some cleanup on paths when running exe, so it
will be different in exe output (like with $exe_mysqld, comparing
basename() is enough).
Cygwin installation
1. Just install latest perl version (only base package) and
patchutils from cygwin-setup;
2. Don't forget to add c:\cygwin64\bin into system path
before any other perl flavors;
3. There is path-style conflict (see below), you must replace
c:\cygwin64\bin\sh.exe with the wrapper. Run MTR with
--cygwin-subshell-fix=do for that. Make sure you are running Cygwin
perl for the option to work.
4. Restart buildbot via net stop buildbot; net start buildbot
Path-style conflict of Cygwin-ish Perl
Some exe paths are passed to mysqltest which are executed by a native
call. This requires native-style paths (\-style). These exe paths also
executed by Perl itself. Either by MTR itself which is not so
critical, but also by tests' --perl blocks which is impossible to
change. And if Perl detects shell-expansion or uses pipe command it
passess this exe path to /bin/sh which is Cygwin-compiled bash that
cannot work with \-style (or at least in -c processing). Thus we require
\-style on some parts of MTR execution and /-style on another parts.
The examples of tests which cover these different parts are:
main.mysqlbinlog_row_compressed \
main.sp_trans_log
That could be great to force Perl to use something different from
/bin/sh, but unfortunately /bin/sh is compiled-in into binary. So the
only solution left is to overwrite /bin/sh with some wrapper script
which passes the command to cmd.exe instead of bash.
See "Path-style conflict" in "MDEV-30836 MTR Cygwin fix" for explanation.
To install subshell fix use --cygwin-subshell-fix=do
To uninstall use --cygwin-subshell-fix=remove
This works only from Cygwin environment. As long as perl on PATH is
from Cygwin you are on Cygwin environment. Check it with
perl --version
This is perl 5, version 36, subversion 1 (v5.36.1) built for
x86_64-cygwin-threads-multi
This ensures that no mtr test can change install.db after it's initial
creation as changing it while as another thread is coping it will lead to
failures in at least InnoDB and Aria recovery.
Fixed spider/bugfix.mdev_30370 that was wrongly used install.db
An unfortunate change to the default behavior of the handling of
core dumps was implemented in
commit e9be5428a2
by making MTR_PRINT_CORE=small the default value, that is, to
only display the stack trace of one thread in crash reports.
Many if not most failures that occur in regression tests are
sporadic and involve race conditions or deadlocks. To be able to
analyze such failures, having the stack traces of all active threads
is a must, because CI environments typically do not save any core dumps.
While the environment variable MTR_PRINT_CORE could be set in CI
environments to compensate for the unfortunate change, it is better
to revert to the old default (dumping all threads) so that
no explicit action will be required from maintainers of independent
CI systems. In that case, if something fails once in a blue moon,
we can have some hope of diagnosing it based on the output.
We fix this regression by defaulting the unset environment variable
MTR_PRINT_CORE to "medium".
first SIGTERM and if the process didn't die in 10 seconds, SIGKILL it.
This allows various tools like `rr`, `gcov`, `gprof`, etc to flush
their data to disk properly
66832e3a introduced change that prints core dumps in very detailed
format. That's completely out of user-friendliness but serves as a
measure for debugging hard-reproducible bugs.
The proper way to implement this:
1. it must be controlled by command-line and environment variable;
2. detailed traces must be default for buildbots only, for user
invocations normal stack traces should be printed.
Options for control are: MTR_PRINT_CORE and --print-core that accept
the following values:
no Don't print core
short Print stack trace of failed thread
medium Print stack traces of all threads
detailed Print all stack traces with debug context
custom:<code> Use debugger commands <code> to print stack trace
Default setting is: short (see env_or_default() call in pre_setup())
For environment variable wrong values are silently ignored (falls back
to default setting, see env_or_default()).
Command-line option --print-core (or -C) overrides environment
variable. Its default value is 'short' if not specified explicitly
(same env_or_default() call in pre_setup()). Explicit values are
checked for validity.
--print-method option can specify by which debugger we print
cores. For Windows there is only one choice: cdb. For Unix the values
are: gdb, dbx, lldb, auto. Default value is: auto
In 'auto' we try to use all possible debuggers until success.
setup_boot_args(), setup_client_args(), setup_args() traversing
datastructures on each invocation. Even if performance is not
important to perl script (though it definitely saves some CO2), this
nonetheless provokes some code-reading questions. Reading and
debugging such code is not convenient.
The better way is to prepare all the data in advance in an easily
readable form as well as do the validation step before any further
processing.
Use mtr_report() instead of die() like the other code does.
TODO: do_args() does even more data processing magic. Prepare that
data according the above strategy in advance in pre_setup() if possible.
Summary of changes
- MD_CTX_SIZE is increased
- EVP_CIPHER_CTX_buf_noconst(ctx) does not work anymore, points
to nobody knows where. The assumption made previously was that
(since the function does not seem to be documented)
was that it points to the last partial source block.
Add own partial block buffer for NOPAD encryption instead
- SECLEVEL in CipherString in openssl.cnf
had been downgraded to 0, from 1, to make TLSv1.0 and TLSv1.1 possible
(according to https://github.com/openssl/openssl/blob/openssl-3.0.0/NEWS.md
even though the manual for SSL_CTX_get_security_level claims that it
should not be necessary)
- Workaround Ssl_cipher_list issue, it now returns TLSv1.3 ciphers,
in addition to what was set in --ssl-cipher
- ctx_buf buffer now must be aligned to 16 bytes with openssl(
previously with WolfSSL only), ot crashes will happen
- updated aes-t , to be better debuggable
using function, rather than a huge multiline macro
added test that does "nopad" encryption piece-wise, to test
replacement of EVP_CIPHER_CTX_buf_noconst
part of MDEV-29000
bt full - to include args and locals.
set print sevenbit on
- it is more useful to be able to see the exact bytes
(in case something is dumped as a string and not hexadecimal digits)
set print static-members off
- there are many interesting (non-const) static members
set frame-arguments all
- even non-printables are useful to see.
Let's make our bb logs give a little bit more detail on those
hard to reproduce bugs.
Tests on rhel7's gdb-7.6.1-120.el7
