Compiler Schema¶
The compiler schema is used for compilation of programs, currently we support single source file compilation.
Schema Files¶
Compilation Examples¶
In order to use the compiler schema you must set type: compiler
in your
sub-schema. We assume the reader has basic understanding of Global Schema
validation.
The type, compiler, and executor are required keys for the schema.
Shown below are 6 test examples performing Hello World compilation with C, C++, and Fortran using GNU compiler
version: "1.0"
buildspecs:
hello_f:
type: compiler
description: "Hello World Fortran Compilation"
executor: local.bash
tags: [tutorials]
build:
source: "src/hello.f90"
name: gnu
fflags: -Wall
hello_c:
type: compiler
description: "Hello World C Compilation"
executor: local.bash
tags: [tutorials]
build:
source: "src/hello.c"
name: gnu
cflags: -Wall
hello_cplusplus:
type: compiler
description: "Hello World C++ Compilation"
executor: local.bash
tags: [tutorials]
build:
source: "src/hello.cpp"
name: gnu
cxxflags: -Wall
cc_example:
type: compiler
description: Example by using cc to set C compiler
executor: local.bash
tags: [tutorials]
build:
source: "src/hello.c"
name: gnu
cc: gcc
fc_example:
type: compiler
description: Example by using fc to set Fortran compiler
executor: local.bash
tags: [tutorials]
build:
source: "src/hello.f90"
name: gnu
fc: gfortran
cxx_example:
type: compiler
description: Example by using cxx to set C++ compiler
executor: local.bash
tags: [tutorials]
build:
source: "src/hello.cpp"
name: gnu
cxx: g++
The tests hello_f
, hello_c
and hello_cplusplus
rely on buildtest to
detect compiler wrappers while tests cc_example
, fc_example
, cxx_example
rely on user to specify compiler wrappers manually.
The compiler
object is start of compilation section, the required
keys are source
and name
. The source key requires an input program for
compilation, this can be a file relative to buildspec file or an absolute path.
In this example our source examples are in src
directory. The name
field
informs buildtest to auto-detect compiler wrappers (cc
, fc
, cxx
).
The compilation pattern buildtest utilizes is the following:
# C example
$cc $cppflags $cflags -o <executable> $SOURCE $ldflags
# Fortran example
$cxx $cppflags $cxxflags -o <executable> $SOURCE $ldflags
# Fortran example
$fc $cppflags $fflags -o <executable> $SOURCE $ldflags
If you specify cc
, fc
and cxx
field attributes you are responsible for
selecting the correct compiler wrapper. You can use cflags
, cxxflags
and
fflags
field to pass compiler options to C, C++ and Fortran compilers.
Shown below is an example build for the buildspec example
$ buildtest build -b tutorials/compilers/gnu_hello.yml
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+-----------------------------------------------------------------------
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
-----------------+---------------------------+------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------
hello_f | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
hello_c | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
hello_cplusplus | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
cc_example | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
fc_example | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
cxx_example | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
-----------------+------------+----------+--------------+------------------------------------------------------------------------------------------------
hello_f | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/generate.sh
hello_c | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/generate.sh
hello_cplusplus | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/generate.sh
cc_example | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/generate.sh
fc_example | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/generate.sh
cxx_example | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 6 tests
Passed Tests: 6/6 Percentage: 100.000%
Failed Tests: 0/6 Percentage: 0.000%
The generated test for test name hello_f is the following:
#!/bin/bash
gfortran -Wall -o hello.f90.exe /global/u1/s/siddiq90/tutorials/examples/serial/src/hello.f90
./hello.f90.exe
buildtest will fill in the compilation line based on compilation pattern. buildtest, will detect the file extensions and perform a lookup to find the programming language, and finally generate the appropriate C, C++, or Fortran compilation based on language detected.
buildtest detects the programming language and it finds .f90 file extension
and infers it must be Fortran program, hence gfortran
was selected. The
executable name is generated by adding .exe
to end of source file name
so we get hello.f90.exe
. Finally, we run the executable.
File Extension Language Table¶
Shown below is the file extension table for your reference
Language |
File Extension |
---|---|
C |
.c |
C++ |
.cc .cxx .cpp .c++ |
Fortran |
.f90 .F90 .f95 .f .F .FOR .for .FTN .ftn |
Passing Arguments¶
If you want to pass options to executable command use the args
key. Shown
below is an example test
version: "1.0"
buildspecs:
executable_arguments:
type: compiler
description: Passing arguments example
executor: local.bash
tags: [tutorials]
build:
source: "src/argc.c"
name: gnu
cflags: -Wall
run:
args: "1 2 3"
The exec_args will pass options to the executable, use this if your binary requires input arguments. Shown below is a generated test:
#!/bin/bash
gcc -Wall -o argc.c.exe /global/u1/s/siddiq90/tutorials/examples/serial/src/argc.c
./argc.c.exe 1 2 3
OpenMP Example¶
Here is an example OpenMP reduction test that runs on 1 node using 32 tasks on a haswell node:
version: "1.0"
buildspecs:
reduction:
type: compiler
executor: slurm.debug
sbatch: ["-N 1", "--ntasks-per-node 32", "-C haswell", "-t 1"]
module:
- "module load PrgEnv-gnu"
env:
OMP_NUM_THREADS: 32
OMP_PROC_BIND: spread
OMP_PLACES: cores
build:
source: src/reduction.c
name: gnu
cflags: -fopenmp
tags: [openmp]
In this example, we use the SlurmExecutor slurm.debug
, the source file is
src/reduction.c
that is relative to buildspec file. The environment variables
are defined using env
section. To enable openmp flag, for GNU compilers we
pass -fopenmp
to C compiler. Finally we classify this test using tags
key which is set to openmp.
The generated test looks as follows:
#!/bin/bash
#SBATCH -N 1
#SBATCH --ntasks-per-node 32
#SBATCH -C haswell
#SBATCH -t 1
export OMP_NUM_THREADS=32
export OMP_PROC_BIND=spread
export OMP_PLACES=cores
module load PrgEnv-gnu
gcc -fopenmp -o reduction.c.exe /global/u1/s/siddiq90/buildtest-cori/apps/openmp/src/reduction.c
./reduction.c.exe
MPI Example¶
In this example we run a MPI Laplace code using 4 process on a KNL node using
the module PrgEnv-intel
. The executable is launched using srun
, that
is set via launcher
field. The source code src/laplace_mpi.c
must be run
with 4 process, for this test we allocate 1 node with 4 tasks.
The name
field is a required field, buildtest uses this field to select the
appropriate subclass, when you set name: intel
buildtest will select the IntelCompiler
subclass which sets the cc
, fc
and cxx
variables automatically. If you
want to specify your compiler variables you can use cc
, fc
and cxx
fields
and buildtest will honor your options.
version: "1.0"
buildspecs:
laplace_mpi:
type: compiler
description: Laplace MPI code in C
sbatch: ["-C knl", "-N 1", "-n 4"]
executor: slurm.debug
tags: ["mpi"]
module:
- "module load PrgEnv-intel"
build:
name: intel
source: src/laplace_mpi.c
cflags: -O3
run:
launcher: srun -n 4
The generated test is as follows:
#!/bin/bash
#SBATCH -C knl
#SBATCH -N 1
#SBATCH -n 4
module load PrgEnv-intel
icc -O3 -o laplace_mpi.c.exe /global/u1/s/siddiq90/buildtest-cori/apps/mpi/src/laplace_mpi.c
srun -n 4 ./laplace_mpi.c.exe
Shown below is a sample build for this buildspec:
$ buildtest build -b mpi/laplace_mpi.yml
Paths:
__________
Prefix: /global/u1/s/siddiq90/cache
Buildspec Search Path: ['/global/u1/s/siddiq90/buildtest/tutorials']
Test Directory: /global/u1/s/siddiq90/cache/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/global/u1/s/siddiq90/buildtest-cori/apps/mpi/laplace_mpi.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
-------------+---------------------------+--------------------------------------------------------------+---------------------------------------------------------------
laplace_mpi | compiler-v1.0.schema.json | /global/u1/s/siddiq90/cache/tests/laplace_mpi/laplace_mpi.sh | /global/u1/s/siddiq90/buildtest-cori/apps/mpi/laplace_mpi.yml
+----------------------+
| Stage: Running Test |
+----------------------+
[laplace_mpi] job dispatched to scheduler
[laplace_mpi] acquiring job id in 2 seconds
name | executor | status | returncode | testpath
-------------+-------------+----------+--------------+--------------------------------------------------------------
laplace_mpi | slurm.debug | N/A | 0 | /global/u1/s/siddiq90/cache/tests/laplace_mpi/laplace_mpi.sh
Polling Jobs in 10 seconds
________________________________________
[laplace_mpi]: JobID 33306420 in COMPLETED state
Polling Jobs in 10 seconds
________________________________________
+---------------------------------------------+
| Stage: Final Results after Polling all Jobs |
+---------------------------------------------+
name | executor | status | returncode | testpath
-------------+-------------+----------+--------------+--------------------------------------------------------------
laplace_mpi | slurm.debug | PASS | 0 | /global/u1/s/siddiq90/cache/tests/laplace_mpi/laplace_mpi.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 1 tests
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
OpenACC Examples¶
Next, we will make use of an OpenACC vector addition example shown below is an example test
version: "1.0"
buildspecs:
vecadd_gnu:
type: compiler
description: Vector Addition example with GNU compiler
tags: [tutorials]
executor: local.bash
build:
name: gnu
source: src/vecAdd.c
cflags: -fopenacc
ldflags: -lm
status:
regex:
stream: stdout
exp: "^final result: 1.000000$"
To compile OpenACC program with gnu compiler we must use -fopenacc
flag, this
program requires linking with math library so we can specify linker flags (ldflags)
using ldflags: -lm
.
The output of this test will generate a single line output as follows:
final result: 1.000000
The status
field with regex
is used for checking output stream using stream: stdout
and exp
key to specify regular expression to use. If we are to build this test,
you will notice the run section will have a Status of PASS
$ buildtest build -b tutorials/compilers/vecadd.yml
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
------------+---------------------------+----------------------------------------------------------------------------------------+--------------------------------------------------------------------
vecadd_gnu | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
------------+------------+----------+--------------+----------------------------------------------------------------------------------------
vecadd_gnu | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 1 tests
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
The regular expression is performed using re.search, for example if we can change
the exp
field as follows:
exp: "^final result: 0.99$"
Next if we re-run test we will notice the Status is FAIL
even though we
have a Return Code of 0:
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
------------+------------+----------+--------------+------------------------------------------------------------------------------------------
vecadd_gnu | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/run_script.sh
In the next example, we extend the previous buildspec test to run at Cori GPU
machine using Slurm scheduler. We use the executor slurm.gpu
where our executor
is defined as follows:
gpu:
description: submit jobs to GPU partition
options: ["-C gpu"]
cluster: escori
In order to submit job to the Cori GPU cluster we must use sbatch -C gpu -M escori
which
is what slurm.gpu
executor is doing.
In this example we make use of module
field to load modules into the test, for
this test we load the modules cuda
and gcc/8.1.1-openacc-gcc-8-branch-20190215
.
This test will launch job via srun
and check job state code is COMPLETED
.
version: "1.0"
buildspecs:
vecadd_openacc_gnu:
type: compiler
description: Vector Addition example with GNU compiler
executor: slurm.gpu
sbatch: ["-G 1", "-t 5", "-N 1"]
module:
- "module load cuda"
- "module load gcc/8.1.1-openacc-gcc-8-branch-20190215"
build:
name: gnu
source: src/vecAdd.c
cflags: -fopenacc
ldflags: -lm
run:
launcher: srun
status:
slurm_job_state_codes: COMPLETED
buildtest will generate the following test, buildtest will add the #SBATCH directives
followed by module commands. The executable is run via srun
because we specify the launcher
field.
#!/bin/bash
#SBATCH -G 1
#SBATCH -t 5
#SBATCH -N 1
module load cuda
module load gcc/8.1.1-openacc-gcc-8-branch-20190215
gcc -fopenacc -o vecAdd.c.exe /global/u1/s/siddiq90/buildtest-cori/apps/openacc/src/vecAdd.c -lm
srun ./vecAdd.c.exe
In this next example, we build same test using hpcsdk
compiler by NVIDIA that acquired PGI compiler. At cori, we must load hpcsdk
and cuda
module in order to use the hpcsdk compiler. The name
is a
required field however buildtest will ignore since we specify
cc
field. NVIDIA changed their compiler names instead of pgcc
we must use
nvc
with flag -acc
to offload to GPU. For CoriGPU we must use
srun
to acquire GPU access hence launcher
field is set to srun.
version: "1.0"
buildspecs:
vecadd_hpcsdk_gnu:
type: compiler
description: Vector Addition example with hpcsdk (pgi) compiler
executor: slurm.gpu
sbatch: ["-G 1", "-t 5", "-N 1"]
module:
- "module load hpcsdk"
- "module load cuda"
build:
name: pgi
cc: nvc
source: src/vecAdd.c
cflags: -acc
ldflags: -lm
run:
launcher: srun
Pre/Post sections for build and run section¶
The compiler schema comes with pre_build
, post_build
, pre_run
and
post_run
fields where you can insert commands before and after build
or
run
section. The build section is where we compile code, and run
section is where compiled binary is executed.
Shown below is an example to illustrate this behavior:
version: "1.0"
buildspecs:
executable_arguments:
type: compiler
description: example using pre_build, post_build, pre_run, post_run example
executor: local.bash
tags: [tutorials]
pre_build: |
echo "This is a pre-build section"
gcc --version
build:
source: "src/hello.c"
name: gnu
cflags: -Wall
post_build: |
echo "This is post-build section"
pre_run: |
echo "This is pre-run section"
export FOO=BAR
post_run: |
echo "This is post-run section"
The format of the test structure is the following:
#!{shebang path} -- defaults to #!/bin/bash depends on executor name (local.bash, local.sh)
{job directives} -- sbatch or bsub field
{environment variables} -- env field
{variable declaration} -- vars field
{module commands} -- modules field
{pre build commands} -- pre_build field
{compile program} -- build field
{post build commands} -- post_build field
{pre run commands} -- pre_run field
{run executable} -- run field
{post run commands} -- post_run field
The generated test for this buildspec is the following:
#!/bin/bash
echo "This is a pre-build section"
gcc --version
gcc -Wall -o hello.c.exe /Users/siddiq90/Documents/buildtest/tutorials/compilers/src/hello.c
echo "This is post-build section"
echo "This is pre-run section"
export FOO=BAR
./hello.c.exe
echo "This is post-run section"
Compiler Schema Examples¶
The compiler schema examples can be retrieved via buildtest schema -n compiler-v1.0.schema.json -e
which shows a list of valid/invalid buildspec examples using type: compiler
.
Each example is validated with schema compiler-v1.0.schema.json
and error
message from invalid examples are also shown in example output.
$ buildtest schema -n compiler-v1.0.schema.json -e
File: /Users/siddiq90/Documents/buildtest/buildtest/schemas/examples/compiler-v1.0.schema.json/valid/examples.yml
Valid State: True
________________________________________________________________________________
version: "1.0"
buildspecs:
gnu_example:
executor: local.bash
type: compiler
description: "gnu example with modules, and cflags example"
module:
- "module purge && module load gcc/4.0"
- "module purge && module load gcc/6.0"
build:
name: gnu
source: src/hello.c
cflags: "-O1"
intel_example:
executor: local.bash
type: compiler
description: "intel example using cflags"
module:
- "module purge && module load intel/17"
- "module purge && module load intel/18"
build:
name: intel
source: src/hello.c
cflags: "-O1"
pgi_example:
executor: local.bash
type: compiler
description: "pgi example using cxxflags, ldflags key"
module:
- "module purge && module load pgi"
build:
source: src/hello.cpp
name: pgi
cxxflags: "-O1"
ldflags: "-lm"
cray_example:
executor: local.bash
type: compiler
description: "cray example using fflags and cppflags"
sbatch: ["-C knl", "-q normal", "-t 01:00"]
build:
name: cray
source: src/hello.f90
fflags: "-O1"
cppflags: "-DFOO"
cc_example:
type: compiler
description: Example by using cc to set C compiler
executor: local.bash
build:
source: "src/hello.c"
name: gnu
cc: gcc
fc_example:
type: compiler
description: Example by using fc to set Fortran compiler
executor: local.bash
build:
source: "src/hello.f90"
name: gnu
fc: gfortran
cxx_example:
type: compiler
description: Example by using cxx to set C++ compiler
executor: local.bash
build:
source: "src/hello.cpp"
name: gnu
cxx: g++
args_example:
type: compiler
description: Launcher example
executor: local.bash
build:
source: "src/hello.cpp"
name: gnu
run:
args: "1 2 4"
mpi_launcher_example:
type: compiler
description: Launcher example
executor: local.bash
build:
source: "src/hello.cpp"
name: gnu
cxx: mpicxx
cxxflags: "-O3"
run:
launcher: mpirun -np 2
File: /Users/siddiq90/Documents/buildtest/buildtest/schemas/examples/compiler-v1.0.schema.json/invalid/examples.yml
Valid State: FAIL
________________________________________________________________________________
version: "1.0"
buildspecs:
missing_type:
executor: local.bash
description: "type key is missing, this is a required field"
module:
- "module purge && module load intel/17"
- "module purge && module load intel/18"
build:
source: src/hello.c
name: intel
cflags: "-O1"
missing_build:
executor: local.bash
type: compiler
description: "build key is missing, this is a required field"
module:
- "module purge && module load intel/17"
- "module purge && module load intel/18"
invalid_type_value:
executor: local.bash
type: script
description: "invalid value for type field must be 'compiler' "
module:
- "module purge && module load gcc/4.0"
- "module purge && module load gcc/6.0"
build:
source: src/hello.c
name: gnu
cflags: "-O1"
invalid_description_value:
executor: local.bash
type: compiler
description: 1
module:
- "module purge && module load gcc/4.0"
- "module purge && module load gcc/6.0"
build:
source: src/hello.c
name: gnu
cflags: "-O1"
invalid_type_module:
executor: local.bash
type: compiler
description: "type for 'module' key, expecting type 'array' but received 'string' "
module: "module purge && module load gcc/4.0"
build:
source: src/hello.c
name: gnu
cflags: "-O1"
module_mismatch_array_items:
executor: local.bash
type: compiler
description: "The module is an array of string items, this test as a mix of numbers and string"
module:
- 1
- "module purge && module load intel"
build:
source: src/hello.c
name: intel
cflags: "-O1"
missing_source_in_compiler:
executor: local.bash
type: compiler
description: "missing source key in compiler object"
module:
- "module purge && module load gcc/4.0"
build:
name: gnu
cflags: "-O1"
missing_name_in_build:
executor: local.bash
type: compiler
description: "missing name key in build object"
module:
- "module purge && module load gcc/4.0"
build:
source: src/hello.c
name_type_mismatch:
executor: local.bash
type: compiler
description: "compiler 'name' expects a string but received a list"
module:
- "module purge && module load gcc/4.0"
build:
source: src/hello.c
name: ["gnu", "intel"]
cflags: "-O1"
ldflags: "-lm"
test_additionalProperties_compiler:
executor: local.bash
type: compiler
description: "test additionalProperties in compiler object. Schema does not allow for additional keys"
module:
- "module purge && module load gcc/4.0"
build:
source: src/hello.c
foo: bar
name: gnu
cflags: "-O1"
ldflags: "-lm"
test_additionalProperties_main_schema:
executor: local.bash
type: compiler
description: "test additionalProperties in main schema"
foo: bar
module:
- "module purge && module load gcc/4.0"
build:
source: src/hello.c
name: gnu
cflags: "-O1"
ldflags: "-lm"
type_mismatch_args:
executor: local.bash
type: compiler
description: "type mismatch on args key"
module:
- "module purge && module load gcc/4.0"
build:
source: src/hello.c
name: gnu
cflags: "-O1"
ldflags: "-lm"
run:
args: 1
________________________________________ Validation Error ________________________________________
Did not find 'type' key in test section: missing_type