The mini-benchmark.sh script failed to run in the latest Fedora
distributions in GitLab CI. It requires `lscpu` resolved by installing
util-linux.
Additionally, executing the benchmark inside a Docker container had
failed because of increased Docker security in recent updates. In
particular the `renice` and `taskset` operations are not permitted.
Neither are the required `perf` operations.
https://docs.docker.com/engine/security/seccomp/
Allow these operations to fail gracefully, and test then skip `perf`,
allowing the remaining benchmark activities to proceed.
Other minor changes to the CI are included such as allowing sanitizer
jobs to fail and using "needs" in the mini-benchmark pipeline.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
The MDEV-25004 test innodb_fts.versioning is omitted because ever since
commit 685d958e38 InnoDB would not allow
writes to a database where the redo log file ib_logfile0 is missing.
Sanitizer tests were introduced in 617f45b for GitLab CI, but started
failing on latest Fedora version with error:
$ yum install -y /usr/lib64/libasan.so.6.0.0 /usr/lib64/libtsan.so.0.0.0 /usr/lib64/libubsan.so.1.0.0
Last metadata expiration check: 0:00:51 ago on Fri Dec 9 20:05:01 2022.
No match for argument: /usr/lib64/libasan.so.6.0.0
No match for argument: /usr/lib64/libtsan.so.0.0.0
Error: Unable to find a match: /usr/lib64/libasan.so.6.0.0 /usr/lib64/libtsan.so.0.0.0
The reason for using specific library versions is unknown. Switch to
simply using latest package versions, as is works and is likely to work
best in the long run.
Also, enclose "../rpmlist-$CI_JOB_NAME-$CI_COMMIT_REF_SLUG.log" in
quotes to avoid `ambiguous redirect` error when $CI_JOB_NAME has spaces.
Additionally use "needs" statements to allow tests to run immediately
after dependent jobs passed instead of waiting for the full stage to
complete.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the BSD-new
license. I am contributing on behalf of my employer Amazon Web Services
The version was fixed to be Fedora 36 due to previous issues on
Gitlab-CI, but those seem to be solved now.
Use 'mariadb' name in scripts and server binary as Fedora switched name in
df76620f9e
Switch to using the `default:` section supported by newer Gitlab-CI,
see https://docs.gitlab.com/ee/ci/yaml/#default.
Also define an explicit timeout of 3 hours to ensure builds don't time
out if the default timeout is too short.
NOTE TO MERGERS: These changes are version independent and should be
merged up on all MariaDB branches 10.6 -> 10.11.
Add a build and test job for each of ASAN, MSAN, TSAN, and UBSAN to the
GitLab pipeline such that current vulnerabilities will be more easily
visible and on each new commit, we can ensure that there are no
additional errors introduced. Furthermore, sanitizer test runs are run
separate from the existing mysql-test-run to isolate sanitizer error
from functional errors.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
In previous versions it was stated that MDEV-25968 was causing other
jobs in the pipeline to fail if not run with "-j 2" but this bug was not
affecting fedora-ninja. This is still true for the public gitlab runners.
However, running the fedora-ninja job on custom runners with more processors
without the "-j 2" flag will cause the compiler to crash.
When running the build with 2,4,8,16,32 threads, build times were
consistent indicating that the typical bottleneck is I/O and not CPU
cores. Therefore, "-j 2" is not a big drawback and it was chosen in
order to remain consistent with the other builds affected by MDEV-25968.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
The commits a73acf6c06 and
4d74bac8bc updated the PCRE library to a new
version, which in turn requires CMake 3.0. That does not exist in CentOS 7
nor 8, so builds started failing.
Actually the build should not be downloading anything at all. The root
cause was that pcre2-devel was missing from the dependencies. This was
originally not detected, as the download fallback had masked the issue.
Implement new mini-benchmark script for simple CPU bound benchmark for the
duration of 5 minutes. The script can be run stand-alone or as part of a
CI pipeline.
Extend Gitlab-CI to run mini-benchmark on every commit to catch if there
are severe performance regressions.
Also bump MARIADB_MAJOR_VERSION to 10.8 which is needed on the 10.8 branch.
- Add new Ninja and Clang build jobs. This helps to ensure those
toolchains also work in addition to default CMake/gcc.
- Generate dependencies.dot/png to illustrate the CMake/Make/Ninja
build dependencies. Viewing this image and identifying bottle necks
in parallelism can help make the build run faster.
- Enable CUnit tests now as they are fixed on 10.6 (MDEV-25820).
- Limit parallel builds to 2 CPUs (full parallelism needs MDEV-25968) on
CMake/Make. Now only the Ninja builds run full parallel builds as only
Ninja is smart enough to prevent builds failing on resource
over-consumption.
- Enable Gitlab-CI cache for job 'centos8' for ccache so that it builds
faster. Don't use Gitlab-CI cache for other jobs, as it would too easily
use up all free tier storage on Gitlab.com and force users to get a paid
account just for MariaDB builds.
- On other jobs clean away ccache, as it only had a 5% hit rate on single
builds with no downloaded cache.
- Dump full database contents during the test install so that one can
use diff to compare the database contents at different stages and thus
track/debug potential bugs in mariadb-install-db and mariadb-upgrade
code.
Bugfixes:
- Zero out ccache stats before each run so that 'ccache -s' would actually
show the stats for the latest run.
As Travis-CI has stopped offering free testing for open source projects,
and they don't seem to have any plans to revert their new restrictions,
MariaDB no longer has a good CI system outside contributors could run
independently for basic validation before submitting Pull Requests.
Implement a simple Gitlab-CI pipeline that runs basic RPM builds on
one old, one less old and one very new distro release and then do some
basic tests on the RPM packages to validate they installed and the
server actually runs.
