> After May 31, 2024, CentOS Stream 8 will be archived and no further
updates will be provided. [1]
CentOS Stream 8 is now EOL and should be updated to using CentOS Stream
9 for compatibility testing in GitLab CI.
[1] https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/https://www.centos.org/centos-linux-eol/
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services.
Line numbers had to be removed from the ignorelists in order to be
diffed against since locations of the same findings can differ
across runs. Therefore preprocessing has to be done on the CI findings
so that it can be compared to the ignorelist and new findings can be
outputted. However, since line numbers have to be removed, a situation
occurs where it is difficult to reference the location of findings
in code given the output of the CI job.
To lessen this pain, change the cppcheck template to include
code snippets which make it easier to reference where in the code
the finding is referring to, even in the absence of line numbers.
Ignorelisting works as before since locations of the finding may
change but not the code it is referring to.
Furthermore, due to the innate difficulty in maintaining ignorelists
across branches and triaging new findings, allow failure as to not
have constantly failing pipelines as a result of a new findings that
have not been addressed yet.
Lastly, update SAST ignorelists to match the newly refactored cppcheck
job and the current state of the codebase.
All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.
A new GitLab MTR run is added which uses libfaketime to simulate
starting the server and running the test suite in the future.
The job simulates running in two times in the year 2038,
once before the signed 32 bit UNIX limit and once after. This change
helps to ensure that all future changes preserve the ability for the
engine start and function past the year 2038.
All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.
The mini-benchmark.sh script failed to run in the latest Fedora
distributions in GitLab CI. Executing the benchmark inside a Docker
container had failed because the check for `perf` was done in a way that
caused the benchmark to exit because of the `set -e` option. Test and
skip `perf` to allowing the remaining benchmark activities to proceed.
This check was added in acb6684 but inadvertantly reverted in 42a1f94.
Logic was corrected to only run perf when the flag is enabled, and to
prevent perf stat and perf record from being simultaneously enabled.
Set -ex is also added to enable easier identification of mini-benchmark
issues in the future.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Performs an upgrade of mariadb from an earlier version to the rpms built
in CI. Then checks whether log contains evidence of upgrade in the form
of "Needs upgrade" or "Table rebuild required". Designed to check minor
version upgrades which should not trigger rebuilds.
The test is written in bash script so it can be executed from other CI
systems.
$ test_upgrade.sh source_version target_version
$ test_upgrade.sh source_version # defaults to rpm/
$ test_upgrade.sh source_version --rpm-dir <directory>
Binaries must be created with performance schema enabled, or
mysql_upgrade complains about missing tables.
Upgrade testing is parallelized with a "matrix" of source versions.
Others can be introduced later.
This was partially designed to catch issues like that seen in
https://jira.mariadb.org/browse/MDEV-28727 where a minor version upgrade
(e.g. 10.4.8 -> 10.4.26) triggered a system table rebuild.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
Add .gitlab-ci.yml file to earliest supported branch to enable
automated building and testing for all MariaDB major branches.
Note to mergers:
GitLab CI is available for branches >= 10.6. This commit includes a
GitLab CI file identical to that in branches >= 10.6, except for the
MARIADB_MAJOR_VERSION variable which should reflect the branch version.
A modified CI will be included in branches 10.4 with PR !2418.
Also changed is the `allow_failure: true` for the MSAN build,
which should be merged up to later branches.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
Note to mergers: Do not merge this commit to 10.5+. An additional PR
will be created for the 10.5 branch which is compatible with later
branches.
Include cppcheck and FlawFinder for SAST scanning.
From 10.6, cherry-picked 12bf5c46 (Remove unused French translations in
Connect engine) and c6072ed9 (Ensure that source files contain only
valid UTF8 encodings). Necessary for FlawFinder to execute and useful
anyway.
Removing MSAN build and test as it was not introduced until 10.5 and
does not successfully build.
Remove failing upgrade test since Fedora installs MariaDB 10.5 and the
10.5->10.4 upgrade rightfully complains
Add to skiplist failing test: main.func_math (MDEV-20966)
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
These are mainly internal files so is a low impact change.
The few scripts/mysql*sql where renames to mariadb_* prefix
on the name.
mysql-test renamed to mariadb-test in the final packages
Include cppcheck and FlawFinder for SAST scanning.
Ignorelists are present for both, so only new problems will trigger a CI
failure.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
The mini-benchmark.sh script failed to run in the latest Fedora
distributions in GitLab CI. It requires `lscpu` resolved by installing
util-linux.
Additionally, executing the benchmark inside a Docker container had
failed because of increased Docker security in recent updates. In
particular the `renice` and `taskset` operations are not permitted.
Neither are the required `perf` operations.
https://docs.docker.com/engine/security/seccomp/
Allow these operations to fail gracefully, and test then skip `perf`,
allowing the remaining benchmark activities to proceed.
Other minor changes to the CI are included such as allowing sanitizer
jobs to fail and using "needs" in the mini-benchmark pipeline.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
Add .gitlab-ci.yml file to earliest supported branch to enable
automated building and testing for all MariaDB major branches.
For 10.4 we use the bundled SSL to build MariaDB when the platform
does not have OpenSSL 1.1 available. This requires the installation of
gnutls-devel as a dependency of MariaDB Connector/C.
OpenSSL 3.0 support was backported to MariaDB 10.5
(see https://github.com/MariaDB/server/pull/2036, f0fa40ef, 8a9c1e9c)
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
The MDEV-25004 test innodb_fts.versioning is omitted because ever since
commit 685d958e38 InnoDB would not allow
writes to a database where the redo log file ib_logfile0 is missing.
Sanitizer tests were introduced in 617f45b for GitLab CI, but started
failing on latest Fedora version with error:
$ yum install -y /usr/lib64/libasan.so.6.0.0 /usr/lib64/libtsan.so.0.0.0 /usr/lib64/libubsan.so.1.0.0
Last metadata expiration check: 0:00:51 ago on Fri Dec 9 20:05:01 2022.
No match for argument: /usr/lib64/libasan.so.6.0.0
No match for argument: /usr/lib64/libtsan.so.0.0.0
Error: Unable to find a match: /usr/lib64/libasan.so.6.0.0 /usr/lib64/libtsan.so.0.0.0
The reason for using specific library versions is unknown. Switch to
simply using latest package versions, as is works and is likely to work
best in the long run.
Also, enclose "../rpmlist-$CI_JOB_NAME-$CI_COMMIT_REF_SLUG.log" in
quotes to avoid `ambiguous redirect` error when $CI_JOB_NAME has spaces.
Additionally use "needs" statements to allow tests to run immediately
after dependent jobs passed instead of waiting for the full stage to
complete.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services
The version was fixed to be Fedora 36 due to previous issues on
Gitlab-CI, but those seem to be solved now.
Use 'mariadb' name in scripts and server binary as Fedora switched name in
df76620f9e
Switch to using the `default:` section supported by newer Gitlab-CI,
see https://docs.gitlab.com/ee/ci/yaml/#default.
Also define an explicit timeout of 3 hours to ensure builds don't time
out if the default timeout is too short.
NOTE TO MERGERS: These changes are version independent and should be
merged up on all MariaDB branches 10.6 -> 10.11.
Add a build and test job for each of ASAN, MSAN, TSAN, and UBSAN to the
GitLab pipeline such that current vulnerabilities will be more easily
visible and on each new commit, we can ensure that there are no
additional errors introduced. Furthermore, sanitizer test runs are run
separate from the existing mysql-test-run to isolate sanitizer error
from functional errors.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
In previous versions it was stated that MDEV-25968 was causing other
jobs in the pipeline to fail if not run with "-j 2" but this bug was not
affecting fedora-ninja. This is still true for the public gitlab runners.
However, running the fedora-ninja job on custom runners with more processors
without the "-j 2" flag will cause the compiler to crash.
When running the build with 2,4,8,16,32 threads, build times were
consistent indicating that the typical bottleneck is I/O and not CPU
cores. Therefore, "-j 2" is not a big drawback and it was chosen in
order to remain consistent with the other builds affected by MDEV-25968.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
The commits a73acf6c06 and
4d74bac8bc updated the PCRE library to a new
version, which in turn requires CMake 3.0. That does not exist in CentOS 7
nor 8, so builds started failing.
Actually the build should not be downloading anything at all. The root
cause was that pcre2-devel was missing from the dependencies. This was
originally not detected, as the download fallback had masked the issue.
Implement new mini-benchmark script for simple CPU bound benchmark for the
duration of 5 minutes. The script can be run stand-alone or as part of a
CI pipeline.
Extend Gitlab-CI to run mini-benchmark on every commit to catch if there
are severe performance regressions.
Also bump MARIADB_MAJOR_VERSION to 10.8 which is needed on the 10.8 branch.
- Add new Ninja and Clang build jobs. This helps to ensure those
toolchains also work in addition to default CMake/gcc.
- Generate dependencies.dot/png to illustrate the CMake/Make/Ninja
build dependencies. Viewing this image and identifying bottle necks
in parallelism can help make the build run faster.
- Enable CUnit tests now as they are fixed on 10.6 (MDEV-25820).
- Limit parallel builds to 2 CPUs (full parallelism needs MDEV-25968) on
CMake/Make. Now only the Ninja builds run full parallel builds as only
Ninja is smart enough to prevent builds failing on resource
over-consumption.
- Enable Gitlab-CI cache for job 'centos8' for ccache so that it builds
faster. Don't use Gitlab-CI cache for other jobs, as it would too easily
use up all free tier storage on Gitlab.com and force users to get a paid
account just for MariaDB builds.
- On other jobs clean away ccache, as it only had a 5% hit rate on single
builds with no downloaded cache.
- Dump full database contents during the test install so that one can
use diff to compare the database contents at different stages and thus
track/debug potential bugs in mariadb-install-db and mariadb-upgrade
code.
Bugfixes:
- Zero out ccache stats before each run so that 'ccache -s' would actually
show the stats for the latest run.
As Travis-CI has stopped offering free testing for open source projects,
and they don't seem to have any plans to revert their new restrictions,
MariaDB no longer has a good CI system outside contributors could run
independently for basic validation before submitting Pull Requests.
Implement a simple Gitlab-CI pipeline that runs basic RPM builds on
one old, one less old and one very new distro release and then do some
basic tests on the RPM packages to validate they installed and the
server actually runs.
