Comparison Operators
buildtest supports several comparison operators as part of status check such as >, >=, <=, <, ==, !=. Each metric is compared with a reference value that can be useful when running performance checks. In this section we will cover the following comparison:
assert_ge: Greater Equal
buildtest can determine status check based on performance check. In this next example, we will run the
STREAM memory benchmark and capture metrics named copy, scale
add and triad from the output and perform an Assertion Greater Equal (assert_ge) with a reference value.
The assert_ge contains a list of assertions in the comparisons property where each metric name is
referenced via name that is compared with the reference value defined by ref property. The comparison
is metric_value >= ref, where metric_value is the value assigned to the metric name captured by the regular
expression. The type field in the metric section is used for the type conversion which can be float, int, or string.
The item is a numeric field used in match.group to retrieve the output
from the regular expression search. The item must be non-negative number.
buildspecs:
stream_test:
type: script
executor: generic.local.bash
description: Run stream test with metrics example using assert greater equal
env:
OMP_NUM_THREADS: 4
run: |
wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
gcc -openmp -o stream stream.c
./stream
metrics:
copy:
type: float
regex:
exp: 'Copy:\s+(\S+)\s+.*'
stream: stdout
item: 1
scale:
type: float
regex:
exp: 'Scale:\s+(\S+)\s+.*'
stream: stdout
item: 1
add:
type: float
regex:
exp: 'Add:\s+(\S+)\s+.*'
stream: stdout
item: 1
triad:
type: float
regex:
exp: 'Triad:\s+(\S+)\s+.*'
stream: stdout
item: 1
status:
assert_ge:
mode: and
comparisons:
- name: copy
ref: 5000
- name: scale
ref: 5500
- name: add
ref: 6000
- name: triad
ref: 6500
buildtest will evaluate each assertion in the list and use a logical AND to determine the final
status of assert_ge. The keyword mode is used to determine whether to perform a logical
OR / AND operation when evaluating the final expression. The mode can be any of the
values: [AND, OR, and, or]. If mode is ommitted the default is logical AND.
Let’s build this test, take a close look at the output of buildtest build and take note of the assertion
statement.
buildtest build -b tutorials/perf_checks/assert_ge.yml
$ buildtest build -b tutorials/perf_checks/assert_ge.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24413852-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/05/17 18:00:10 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.18 │
│ Configuration File: /tmp/tmpqzxycqdh/config.yml │
│ Test Directory: /tmp/tmpqzxycqdh/var/tests │
│ Report File: /tmp/tmpqzxycqdh/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_ge.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_ge.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ stream_t │ script │ generic. │ None │ None │ None │ Run │ /home/d │
│ est/2e66 │ │ local.ba │ │ │ │ stream │ ocs/che │
│ 15d4 │ │ sh │ │ │ │ test │ ckouts/ │
│ │ │ │ │ │ │ with │ readthe │
│ │ │ │ │ │ │ metrics │ docs.or │
│ │ │ │ │ │ │ example │ g/user_ │
│ │ │ │ │ │ │ using │ builds/ │
│ │ │ │ │ │ │ assert │ buildte │
│ │ │ │ │ │ │ greater │ st/chec │
│ │ │ │ │ │ │ equal │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/a │
│ │ │ │ │ │ │ │ ssert_g │
│ │ │ │ │ │ │ │ e.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
stream_test/2e6615d4: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
stream_test/2e6615d4 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━┩
│ stream_test/2e6615d4 │
└──────────────────────┘
stream_test/2e6615d4: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stage
stream_test/2e6615d4: Running Test via command: bash stream_test_build.sh
─────────────────── Output Message for stream_test/2e6615d4 ────────────────────
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 6099.5 0.026338 0.026232 0.026442
Scale: 6917.4 0.023245 0.023130 0.023740
Add: 9201.4 0.026464 0.026083 0.026781
Triad: 8690.6 0.027863 0.027616 0.028184
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
stream_test/2e6615d4: Test completed in 1.637986 seconds with returncode: 0
stream_test/2e6615d4: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stream_test.out
stream_test/2e6615d4: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stream_test.err
stream_test/2e6615d4: testing metric: copy if 6099.5 >= 5000.0 - Check: True
stream_test/2e6615d4: testing metric: scale if 6917.4 >= 5500.0 - Check: True
stream_test/2e6615d4: testing metric: add if 9201.4 >= 6000.0 - Check: True
stream_test/2e6615d4: testing metric: triad if 8690.6 >= 6500.0 - Check: True
stream_test/2e6615d4: ge check: True
stream_test/2e6615d4: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stage/stream_test_postrun.sh
stream_test/2e6615d4: Post run script exit code: 1
───────────────── stream_test/2e6615d4: Post Run Script Output ─────────────────
───────────────── stream_test/2e6615d4: Post Run Script Error ──────────────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stage/stream_test_postrun.sh not found.
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ stream_test/2e6615d4 │ generic.local.bash │ PASS │ 0 │ 1.638 │
└──────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpqzxycqdh/var/report.json
Writing Logfile to /tmp/tmpqzxycqdh/var/logs/buildtest_vtbj1k3b.log
Let’s run buildtest inspect query -o stream_test to retrieve the test details and output of STREAM test.
buildtest inspect query -o stream_test
$ buildtest inspect query -o stream_test
─────────────── stream_test/2e6615d4-8340-43f6-aec9-e487a5087aed ───────────────
Executor: generic.local.bash
Description: Run stream test with metrics example using assert greater equal
State: PASS
Returncode: 0
Runtime: 1.637986 sec
Starttime: 2024/05/17 18:00:10
Endtime: 2024/05/17 18:00:12
Command: bash stream_test_build.sh
Test Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stream_test.sh
Build Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stream_test_build.sh
Output File: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stream_test.out
Error File: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/stream_test/2e6615d4/stream_test.err
Log File: /tmp/tmpqzxycqdh/var/logs/buildtest_vtbj1k3b.log
Metrics
┏━━━━━━━┳━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━╇━━━━━━━━┩
│ copy │ 6099.5 │
│ scale │ 6917.4 │
│ add │ 9201.4 │
│ triad │ 8690.6 │
└───────┴────────┘
─ Output File: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ge/strea… ─
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 10000000 (elements), Offset = 0 (elements)
Memory per array = 76.3 MiB (= 0.1 GiB).
Total memory required = 228.9 MiB (= 0.2 GiB).
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 23999 microseconds.
(= 23999 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 6099.5 0.026338 0.026232 0.026442
Scale: 6917.4 0.023245 0.023130 0.023740
Add: 9201.4 0.026464 0.026083 0.026781
Triad: 8690.6 0.027863 0.027616 0.028184
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
assert_gt: Greater Than
In this example, we perform a > operation, this can be done via assert_gt property
buildspecs:
assert_gt_example:
type: script
executor: generic.local.bash
description: Run stream test with metrics example using assert greater than.
env:
OMP_NUM_THREADS: 4
run: |
wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
gcc -openmp -o stream stream.c
./stream
metrics:
copy:
type: float
regex:
exp: 'Copy:\s+(\S+)\s+.*'
stream: stdout
item: 1
scale:
type: float
regex:
exp: 'Scale:\s+(\S+)\s+.*'
stream: stdout
item: 1
add:
type: float
regex:
exp: 'Add:\s+(\S+)\s+.*'
stream: stdout
item: 1
triad:
type: float
regex:
exp: 'Triad:\s+(\S+)\s+.*'
stream: stdout
item: 1
status:
assert_gt:
comparisons:
- name: copy
ref: 5000
- name: scale
ref: 5500
- name: add
ref: 6000
- name: triad
ref: 6500
assert_le: Less Than Equal
In this example, we perform a <= operation, this can be done via assert_le property
buildspecs:
assert_le_example:
type: script
executor: generic.local.bash
description: Run stream test with metrics example using assert less than equal
env:
OMP_NUM_THREADS: 4
run: |
wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
gcc -openmp -o stream stream.c
./stream
metrics:
copy:
type: float
regex:
exp: 'Copy:\s+(\S+)\s+.*'
stream: stdout
item: 1
scale:
type: float
regex:
exp: 'Scale:\s+(\S+)\s+.*'
stream: stdout
item: 1
add:
type: float
regex:
exp: 'Add:\s+(\S+)\s+.*'
stream: stdout
item: 1
triad:
type: float
regex:
exp: 'Triad:\s+(\S+)\s+.*'
stream: stdout
item: 1
status:
assert_le:
comparisons:
- name: copy
ref: 5000
- name: scale
ref: 5500
- name: add
ref: 6000
- name: triad
ref: 6500
assert_lt: Less Than
In this example, we perform a < operation, this can be done via assert_lt property
buildspecs:
assert_lt_example:
type: script
executor: generic.local.bash
description: Run stream test with metrics example using assert less than
env:
OMP_NUM_THREADS: 4
run: |
wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
gcc -openmp -o stream stream.c
./stream
metrics:
copy:
type: float
regex:
exp: 'Copy:\s+(\S+)\s+.*'
stream: stdout
item: 1
scale:
type: float
regex:
exp: 'Scale:\s+(\S+)\s+.*'
stream: stdout
item: 1
add:
type: float
regex:
exp: 'Add:\s+(\S+)\s+.*'
stream: stdout
item: 1
triad:
type: float
regex:
exp: 'Triad:\s+(\S+)\s+.*'
stream: stdout
item: 1
status:
assert_lt:
comparisons:
- name: copy
ref: 5000
- name: scale
ref: 5500
- name: add
ref: 6000
- name: triad
ref: 6500
assert_eq: Equal
buildtest can perform assert equality check with metrics to determine status of test. In this next example, we define
four metrics x, y, first, and last which will be compared with its reference value. We introduce a new
property assert_eq which is composed of list of assertions. Each reference is converted to its appropriate
type (int, float, str).
buildspecs:
assert_eq_example:
type: script
executor: generic.local.bash
description: Test for assert equality
vars:
X: 1
Y: 1.5
first: John
last: Smith
run: |
echo "X: $X"
echo "Y: $Y"
echo "Name: $first $last"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
y:
type: float
regex:
stream: stdout
exp: 'Y:\s+(\S+)\s+.*'
item: 1
first:
type: str
regex:
stream: stdout
exp: 'Name:\s+(\S+)\s+.*'
item: 1
last:
type: str
regex:
stream: stdout
exp: '(Smith)$'
item: 1
status:
assert_eq:
comparisons:
- name: x
ref: 1
- name: y
ref: 1.5
- name: first
ref: John
- name: last
ref: Smith
This test is expected to pass where all assertions are True. Let’s build the test and see the output
buildtest build -b tutorials/perf_checks/assert_eq.yml
$ buildtest build -b tutorials/perf_checks/assert_eq.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24413852-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/05/17 18:00:13 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.18 │
│ Configuration File: /tmp/tmpqzxycqdh/config.yml │
│ Test Directory: /tmp/tmpqzxycqdh/var/tests │
│ Report File: /tmp/tmpqzxycqdh/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_eq.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_eq.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_e │ script │ generic. │ None │ None │ None │ Test for │ /home/d │
│ q_exampl │ │ local.ba │ │ │ │ assert │ ocs/che │
│ e/055041 │ │ sh │ │ │ │ equality │ ckouts/ │
│ fd │ │ │ │ │ │ │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/a │
│ │ │ │ │ │ │ │ ssert_e │
│ │ │ │ │ │ │ │ q.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_eq_example/055041fd: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_eq_example/055041fd does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_eq_example/055041fd │
└────────────────────────────┘
assert_eq_example/055041fd: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/stage
assert_eq_example/055041fd: Running Test via command: bash assert_eq_example_build.sh
──────────────── Output Message for assert_eq_example/055041fd ─────────────────
X: 1
Y: 1.5
Name: John Smith
assert_eq_example/055041fd: Test completed in 0.007249 seconds with returncode: 0
assert_eq_example/055041fd: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/assert_eq_example.out
assert_eq_example/055041fd: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/assert_eq_example.err
assert_eq_example/055041fd: testing metric: x if 1 == 1 - Check: True
assert_eq_example/055041fd: testing metric: y if 1.5 == 1.5 - Check: True
assert_eq_example/055041fd: testing metric: first if John == John - Check: True
assert_eq_example/055041fd: testing metric: last if Smith == Smith - Check: True
assert_eq_example/055041fd: eq check: True
assert_eq_example/055041fd: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/stage/assert_eq_example_postrun.sh
assert_eq_example/055041fd: Post run script exit code: 1
────────────── assert_eq_example/055041fd: Post Run Script Output ──────────────
────────────── assert_eq_example/055041fd: Post Run Script Error ───────────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/stage/assert_eq_example_postrun.sh not found.
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_eq_example/05504 │ generic.local.bash │ PASS │ 0 │ 0.007 │
│ 1fd │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpqzxycqdh/var/report.json
Writing Logfile to /tmp/tmpqzxycqdh/var/logs/buildtest_lbpx6s3u.log
buildtest inspect query -o assert_eq_example
$ buildtest inspect query -o assert_eq_example
──────────── assert_eq_example/055041fd-91f4-41a1-b7d3-3278e103ad75 ────────────
Executor: generic.local.bash
Description: Test for assert equality
State: PASS
Returncode: 0
Runtime: 0.007249 sec
Starttime: 2024/05/17 18:00:13
Endtime: 2024/05/17 18:00:13
Command: bash assert_eq_example_build.sh
Test Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/assert_eq_example.sh
Build Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/assert_eq_example_build.sh
Output File: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/assert_eq_example.out
Error File: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/assert_eq_example/055041fd/assert_eq_example.err
Log File: /tmp/tmpqzxycqdh/var/logs/buildtest_lbpx6s3u.log
Metrics
┏━━━━━━━┳━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━╇━━━━━━━┩
│ x │ 1 │
│ y │ 1.5 │
│ first │ John │
│ last │ Smith │
└───────┴───────┘
─ Output File: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq/asser… ─
X: 1
Y: 1.5
Name: John Smith
In the next example, we have two tests to highlight some exceptions. In the first test, we define an invalid metric name invalid_metric
in assert_eq since this metric was not defined in metrics field, therefore this test will fail. The second test will fail because we have
a mismatch in value captured by metric x which is 1 however the reference value is 2.
buildspecs:
assert_eq_invalid_metric:
type: script
executor: generic.local.bash
description: An invalid metric name will cause failure
vars:
X: 1
run: |
echo "X: $X"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
status:
assert_eq:
comparisons:
- name: x
ref: 1
- name: invalid_metric
ref: 'hello'
assert_eq_mismatch:
type: script
executor: generic.local.bash
description: This test will fail because there is a mismatch in metric x assert equality
vars:
X: 1
run: |
echo "X: $X"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
status:
assert_eq:
comparisons:
- name: x
ref: 2
Let’s build this test and see the output.
buildtest build -b tutorials/perf_checks/assert_eq_exceptions.yml
$ buildtest build -b tutorials/perf_checks/assert_eq_exceptions.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24413852-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/05/17 18:00:14 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.18 │
│ Configuration File: /tmp/tmpqzxycqdh/config.yml │
│ Test Directory: /tmp/tmpqzxycqdh/var/tests │
│ Report File: /tmp/tmpqzxycqdh/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_eq_exceptions.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_eq_exceptions.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_e │ script │ generic. │ None │ None │ None │ An │ /home/d │
│ q_invali │ │ local.ba │ │ │ │ invalid │ ocs/che │
│ d_metric │ │ sh │ │ │ │ metric │ ckouts/ │
│ /43a2f60 │ │ │ │ │ │ name │ readthe │
│ 6 │ │ │ │ │ │ will │ docs.or │
│ │ │ │ │ │ │ cause │ g/user_ │
│ │ │ │ │ │ │ failure │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/a │
│ │ │ │ │ │ │ │ ssert_e │
│ │ │ │ │ │ │ │ q_excep │
│ │ │ │ │ │ │ │ tions.y │
│ │ │ │ │ │ │ │ ml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ assert_e │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ q_mismat │ │ local.ba │ │ │ │ test │ ocs/che │
│ ch/52afa │ │ sh │ │ │ │ will │ ckouts/ │
│ 266 │ │ │ │ │ │ fail │ readthe │
│ │ │ │ │ │ │ because │ docs.or │
│ │ │ │ │ │ │ there is │ g/user_ │
│ │ │ │ │ │ │ a │ builds/ │
│ │ │ │ │ │ │ mismatch │ buildte │
│ │ │ │ │ │ │ in │ st/chec │
│ │ │ │ │ │ │ metric x │ kouts/d │
│ │ │ │ │ │ │ assert │ evel/tu │
│ │ │ │ │ │ │ equality │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/a │
│ │ │ │ │ │ │ │ ssert_e │
│ │ │ │ │ │ │ │ q_excep │
│ │ │ │ │ │ │ │ tions.y │
│ │ │ │ │ │ │ │ ml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_eq_invalid_metric/43a2f606: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/43a2f606
assert_eq_mismatch/52afa266: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/52afa266
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_eq_invalid_metric/43a2f606 does not have any dependencies adding test to queue
assert_eq_mismatch/52afa266 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_eq_invalid_metric/43a2f606 │
│ assert_eq_mismatch/52afa266 │
└───────────────────────────────────┘
assert_eq_invalid_metric/43a2f606: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/43a2f606/stage
assert_eq_invalid_metric/43a2f606: Running Test via command: bash assert_eq_invalid_metric_build.sh
───────────── Output Message for assert_eq_invalid_metric/43a2f606 ─────────────
X: 1
assert_eq_invalid_metric/43a2f606: Test completed in 0.007185 seconds with returncode: 0
assert_eq_invalid_metric/43a2f606: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/43a2f606/assert_eq_invalid_metric.out
assert_eq_invalid_metric/43a2f606: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/43a2f606/assert_eq_invalid_metric.err
assert_eq_invalid_metric/43a2f606: testing metric: x if 1 == 1 - Check: True
assert_eq_invalid_metric/43a2f606: Unable to find metric: invalid_metric. List of valid metrics are the following: ['x']
assert_eq_invalid_metric/43a2f606: eq check: False
assert_eq_invalid_metric/43a2f606: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/43a2f606/stage/assert_eq_invalid_metric_postrun.sh
assert_eq_invalid_metric/43a2f606: Post run script exit code: 1
────────── assert_eq_invalid_metric/43a2f606: Post Run Script Output ───────────
─────────── assert_eq_invalid_metric/43a2f606: Post Run Script Error ───────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/43a2f606/stage/assert_eq_invalid_metric_postrun.sh not found.
assert_eq_mismatch/52afa266: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/52afa266/stage
assert_eq_mismatch/52afa266: Running Test via command: bash assert_eq_mismatch_build.sh
──────────────── Output Message for assert_eq_mismatch/52afa266 ────────────────
X: 1
assert_eq_mismatch/52afa266: Test completed in 0.006682 seconds with returncode: 0
assert_eq_mismatch/52afa266: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/52afa266/assert_eq_mismatch.out
assert_eq_mismatch/52afa266: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/52afa266/assert_eq_mismatch.err
assert_eq_mismatch/52afa266: testing metric: x if 1 == 2 - Check: False
assert_eq_mismatch/52afa266: eq check: False
assert_eq_mismatch/52afa266: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/52afa266/stage/assert_eq_mismatch_postrun.sh
assert_eq_mismatch/52afa266: Post run script exit code: 1
───────────── assert_eq_mismatch/52afa266: Post Run Script Output ──────────────
────────────── assert_eq_mismatch/52afa266: Post Run Script Error ──────────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/52afa266/stage/assert_eq_mismatch_postrun.sh not found.
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_eq_mismatch/52af │ generic.local.bash │ FAIL │ 0 │ 0.007 │
│ a266 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ assert_eq_invalid_metri │ generic.local.bash │ FAIL │ 0 │ 0.007 │
│ c/43a2f606 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 0/2 Percentage: 0.000%
Failed Tests: 2/2 Percentage: 100.000%
Adding 2 test results to report file: /tmp/tmpqzxycqdh/var/report.json
Writing Logfile to /tmp/tmpqzxycqdh/var/logs/buildtest_2f0di03_.log
assert_ne: Not Equal
In this section, we will discuss the inverse equality operation Not Equal check (!=) with reference value.
We can use assert_ne property to perform != check, it works similar to assert_eq with data types int,
float and str. In this example, we check the metrics x, y, first and last and each metric
should pass. The reference value is converted to the data-type (type field) for each metrics
buildspecs:
assert_ne_example:
type: script
executor: generic.local.bash
description: Test for assert not equal
vars:
X: 1
Y: 1.5
first: John
last: Smith
run: |
echo "X: $X"
echo "Y: $Y"
echo "Name: $first $last"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
y:
type: float
regex:
stream: stdout
exp: 'Y:\s+(\S+)\s+.*'
item: 1
first:
type: str
regex:
stream: stdout
exp: 'Name:\s+(\S+)\s+.*'
item: 1
last:
type: str
regex:
stream: stdout
exp: '(Smith)$'
item: 1
status:
assert_ne:
comparisons:
- name: x
ref: 2
- name: y
ref: 2.5
- name: first
ref: Robert
- name: last
ref: Brown
We expect this test to pass. In order to run this test, you can do the following
buildtest build -b tutorials/perf_checks/assert_ne.yml
$ buildtest build -b tutorials/perf_checks/assert_ne.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24413852-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/05/17 18:00:15 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.18 │
│ Configuration File: /tmp/tmpqzxycqdh/config.yml │
│ Test Directory: /tmp/tmpqzxycqdh/var/tests │
│ Report File: /tmp/tmpqzxycqdh/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_ne.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_ne.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_n │ script │ generic. │ None │ None │ None │ Test for │ /home/d │
│ e_exampl │ │ local.ba │ │ │ │ assert │ ocs/che │
│ e/a9457b │ │ sh │ │ │ │ not │ ckouts/ │
│ d0 │ │ │ │ │ │ equal │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/a │
│ │ │ │ │ │ │ │ ssert_n │
│ │ │ │ │ │ │ │ e.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_ne_example/a9457bd0: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ne/assert_ne_example/a9457bd0
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_ne_example/a9457bd0 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_ne_example/a9457bd0 │
└────────────────────────────┘
assert_ne_example/a9457bd0: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ne/assert_ne_example/a9457bd0/stage
assert_ne_example/a9457bd0: Running Test via command: bash assert_ne_example_build.sh
──────────────── Output Message for assert_ne_example/a9457bd0 ─────────────────
X: 1
Y: 1.5
Name: John Smith
assert_ne_example/a9457bd0: Test completed in 0.007149 seconds with returncode: 0
assert_ne_example/a9457bd0: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ne/assert_ne_example/a9457bd0/assert_ne_example.out
assert_ne_example/a9457bd0: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ne/assert_ne_example/a9457bd0/assert_ne_example.err
assert_ne_example/a9457bd0: testing metric: x if 1 != 2 - Check: True
assert_ne_example/a9457bd0: testing metric: y if 1.5 != 2.5 - Check: True
assert_ne_example/a9457bd0: testing metric: first if John != Robert - Check: True
assert_ne_example/a9457bd0: testing metric: last if Smith != Brown - Check: True
assert_ne_example/a9457bd0: ne check: True
assert_ne_example/a9457bd0: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ne/assert_ne_example/a9457bd0/stage/assert_ne_example_postrun.sh
assert_ne_example/a9457bd0: Post run script exit code: 1
────────────── assert_ne_example/a9457bd0: Post Run Script Output ──────────────
────────────── assert_ne_example/a9457bd0: Post Run Script Error ───────────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_ne/assert_ne_example/a9457bd0/stage/assert_ne_example_postrun.sh not found.
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_ne_example/a9457 │ generic.local.bash │ PASS │ 0 │ 0.007 │
│ bd0 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpqzxycqdh/var/report.json
Writing Logfile to /tmp/tmpqzxycqdh/var/logs/buildtest_4gs877zj.log
assert_range: Upper and Lower Bound
The assert_range property can be used to test performance for a metric given a lower and upper bound. This property expects
one to specify lower and upper field which must be an integer or floating point number to perform comparison. buildtest will
perform an assertion, if metric value is in the range specified by lower and upper, then test will pass. Shown below
is an example using the assert_range property with stream benchmark.
buildspecs:
assert_range_ex:
type: script
executor: generic.local.bash
description: Example on assert_range
env:
OMP_NUM_THREADS: 4
run: |
wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
gcc -openmp -o stream stream.c
./stream
metrics:
copy:
type: float
regex:
exp: 'Copy:\s+(\S+)\s+.*'
stream: stdout
item: 1
scale:
type: float
regex:
exp: 'Scale:\s+(\S+)\s+.*'
stream: stdout
item: 1
add:
type: float
regex:
exp: 'Add:\s+(\S+)\s+.*'
stream: stdout
item: 1
triad:
type: float
regex:
exp: 'Triad:\s+(\S+)\s+.*'
stream: stdout
item: 1
status:
assert_range:
comparisons:
- name: copy
lower: 5000
upper: 20000
- name: scale
lower: 4500
upper: 20000
- name: add
lower: 4300
upper: 20000
- name: triad
lower: 5600
upper: 20000
Let’s build this test and see the output
buildtest build -b tutorials/perf_checks/assert_range.yml
$ buildtest build -b tutorials/perf_checks/assert_range.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24413852-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/05/17 18:00:16 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.18 │
│ Configuration File: /tmp/tmpqzxycqdh/config.yml │
│ Test Directory: /tmp/tmpqzxycqdh/var/tests │
│ Report File: /tmp/tmpqzxycqdh/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_range.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_range.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_r │ script │ generic. │ None │ None │ None │ Example │ /home/d │
│ ange_ex/ │ │ local.ba │ │ │ │ on │ ocs/che │
│ 4846b93a │ │ sh │ │ │ │ assert_r │ ckouts/ │
│ │ │ │ │ │ │ ange │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/a │
│ │ │ │ │ │ │ │ ssert_r │
│ │ │ │ │ │ │ │ ange.ym │
│ │ │ │ │ │ │ │ l │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_range_ex/4846b93a: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_range/assert_range_ex/4846b93a
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_range_ex/4846b93a does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_range_ex/4846b93a │
└──────────────────────────┘
assert_range_ex/4846b93a: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_range/assert_range_ex/4846b93a/stage
assert_range_ex/4846b93a: Running Test via command: bash assert_range_ex_build.sh
───────────────── Output Message for assert_range_ex/4846b93a ──────────────────
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 6120.6 0.026217 0.026141 0.026354
Scale: 6938.1 0.023133 0.023061 0.023297
Add: 9197.9 0.026293 0.026093 0.026599
Triad: 8729.8 0.027629 0.027492 0.028010
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
assert_range_ex/4846b93a: Test completed in 1.343466 seconds with returncode: 0
assert_range_ex/4846b93a: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_range/assert_range_ex/4846b93a/assert_range_ex.out
assert_range_ex/4846b93a: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_range/assert_range_ex/4846b93a/assert_range_ex.err
assert_range_ex/4846b93a: testing metric: copy if 5000.0 <= 6120.6 <= 20000.0 - Check: True
assert_range_ex/4846b93a: testing metric: scale if 4500.0 <= 6938.1 <= 20000.0 - Check: True
assert_range_ex/4846b93a: testing metric: add if 4300.0 <= 9197.9 <= 20000.0 - Check: True
assert_range_ex/4846b93a: testing metric: triad if 5600.0 <= 8729.8 <= 20000.0 - Check: True
assert_range_ex/4846b93a: Range Check: True
assert_range_ex/4846b93a: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_range/assert_range_ex/4846b93a/stage/assert_range_ex_postrun.sh
assert_range_ex/4846b93a: Post run script exit code: 1
─────────────── assert_range_ex/4846b93a: Post Run Script Output ───────────────
─────────────── assert_range_ex/4846b93a: Post Run Script Error ────────────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/assert_range/assert_range_ex/4846b93a/stage/assert_range_ex_postrun.sh not found.
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_range_ex/4846b93 │ generic.local.bash │ PASS │ 0 │ 1.343 │
│ a │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpqzxycqdh/var/report.json
Writing Logfile to /tmp/tmpqzxycqdh/var/logs/buildtest_2b_ewa5n.log
Note that performance results may vary on your system and depending on the metric value you may want to adjust the lower and upper bound to match your requirement.
contains, not_contains: Contains and Not Contains
Buildtest can perform status check with a list of reference values and check if metrics value is in the list. The
property contains and not_contains can be used to perform this type of check. The ref property is a list of
reference values that a metric must have to pass metrics check.
In example below we have two tests, the first test perform contains and not_contains on metrics x. We expect both
status check will pass. The second test is expected to fail because metric x will store integer value 1 but the list has
string equivalent ‘1’.
buildspecs:
contains_and_not_contains:
type: script
executor: generic.local.bash
description: Status check based on contains and not contains where test pass
vars:
X: 1
run: |
echo "X: $X"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
status:
contains:
comparisons:
- name: x
ref: [1, 2, 4, 8]
not_contains:
comparisons:
- name: x
ref: [2, 4]
assert_contains_fail:
type: script
executor: generic.local.bash
description: Status check based on contains where test fails
vars:
X: 1
run: |
echo "X: $X"
metrics:
x:
type: int
regex:
stream: stdout
exp: 'X:\s+(\S+)\s+.*'
item: 1
status:
contains:
comparisons:
- name: x
ref: ['1', 2, 4, 8]
You can run this test, by running the following command
buildtest build -b tutorials/perf_checks/contains.yml
$ buildtest build -b tutorials/perf_checks/contains.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24413852-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/05/17 18:00:18 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.18 │
│ Configuration File: /tmp/tmpqzxycqdh/config.yml │
│ Test Directory: /tmp/tmpqzxycqdh/var/tests │
│ Report File: /tmp/tmpqzxycqdh/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/contains.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/contains.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ contains │ script │ generic. │ None │ None │ None │ Status │ /home/d │
│ _and_not │ │ local.ba │ │ │ │ check │ ocs/che │
│ _contain │ │ sh │ │ │ │ based on │ ckouts/ │
│ s/decbdd │ │ │ │ │ │ contains │ readthe │
│ d1 │ │ │ │ │ │ and not │ docs.or │
│ │ │ │ │ │ │ contains │ g/user_ │
│ │ │ │ │ │ │ where │ builds/ │
│ │ │ │ │ │ │ test │ buildte │
│ │ │ │ │ │ │ pass │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/c │
│ │ │ │ │ │ │ │ ontains │
│ │ │ │ │ │ │ │ .yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ assert_c │ script │ generic. │ None │ None │ None │ Status │ /home/d │
│ ontains_ │ │ local.ba │ │ │ │ check │ ocs/che │
│ fail/ff7 │ │ sh │ │ │ │ based on │ ckouts/ │
│ 307ae │ │ │ │ │ │ contains │ readthe │
│ │ │ │ │ │ │ where │ docs.or │
│ │ │ │ │ │ │ test │ g/user_ │
│ │ │ │ │ │ │ fails │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /perf_c │
│ │ │ │ │ │ │ │ hecks/c │
│ │ │ │ │ │ │ │ ontains │
│ │ │ │ │ │ │ │ .yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
contains_and_not_contains/decbddd1: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/contains_and_not_contains/decbddd1
assert_contains_fail/ff7307ae: Creating Test Directory: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/assert_contains_fail/ff7307ae
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
contains_and_not_contains/decbddd1 does not have any dependencies adding test to queue
assert_contains_fail/ff7307ae does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ contains_and_not_contains/decbddd1 │
│ assert_contains_fail/ff7307ae │
└────────────────────────────────────┘
contains_and_not_contains/decbddd1: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/contains_and_not_contains/decbddd1/stage
contains_and_not_contains/decbddd1: Running Test via command: bash contains_and_not_contains_build.sh
──────────── Output Message for contains_and_not_contains/decbddd1 ─────────────
X: 1
contains_and_not_contains/decbddd1: Test completed in 0.007138 seconds with returncode: 0
contains_and_not_contains/decbddd1: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/contains_and_not_contains/decbddd1/contains_and_not_contains.out
contains_and_not_contains/decbddd1: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/contains_and_not_contains/decbddd1/contains_and_not_contains.err
contains_and_not_contains/decbddd1: testing metric: x if 1 in [1, 2, 4, 8] - Check: True
contains_and_not_contains/decbddd1: Contains Check: True
contains_and_not_contains/decbddd1: testing metric: x if 1 not in [2, 4] - Check: True
contains_and_not_contains/decbddd1: Not Contains Check: True
contains_and_not_contains/decbddd1: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/contains_and_not_contains/decbddd1/stage/contains_and_not_contains_postrun.sh
contains_and_not_contains/decbddd1: Post run script exit code: 1
────────── contains_and_not_contains/decbddd1: Post Run Script Output ──────────
────────── contains_and_not_contains/decbddd1: Post Run Script Error ───────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/contains_and_not_contains/decbddd1/stage/contains_and_not_contains_postrun.sh not found.
assert_contains_fail/ff7307ae: Current Working Directory : /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/assert_contains_fail/ff7307ae/stage
assert_contains_fail/ff7307ae: Running Test via command: bash assert_contains_fail_build.sh
─────────────── Output Message for assert_contains_fail/ff7307ae ───────────────
X: 1
assert_contains_fail/ff7307ae: Test completed in 0.006861 seconds with returncode: 0
assert_contains_fail/ff7307ae: Writing output file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/assert_contains_fail/ff7307ae/assert_contains_fail.out
assert_contains_fail/ff7307ae: Writing error file - /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/assert_contains_fail/ff7307ae/assert_contains_fail.err
assert_contains_fail/ff7307ae: testing metric: x if 1 in ['1', 2, 4, 8] - Check: False
assert_contains_fail/ff7307ae: Contains Check: False
assert_contains_fail/ff7307ae: Running Post Run Script: /tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/assert_contains_fail/ff7307ae/stage/assert_contains_fail_postrun.sh
assert_contains_fail/ff7307ae: Post run script exit code: 1
──────────── assert_contains_fail/ff7307ae: Post Run Script Output ─────────────
───────────── assert_contains_fail/ff7307ae: Post Run Script Error ─────────────
/tmp/tmpqzxycqdh/var/tests/generic.local.bash/contains/assert_contains_fail/ff7307ae/stage/assert_contains_fail_postrun.sh not found.
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_contains_fail/ff │ generic.local.bash │ FAIL │ 0 │ 0.007 │
│ 7307ae │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ contains_and_not_contai │ generic.local.bash │ PASS │ 0 │ 0.007 │
│ ns/decbddd1 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to report file: /tmp/tmpqzxycqdh/var/report.json
Writing Logfile to /tmp/tmpqzxycqdh/var/logs/buildtest__nyzwbq_.log