5.2.2. Regression test¶
The regression test executes a series of test cases which intend to fully describe OpenFAST and its module’s capabilities. Each locally computed result is compared to a static set of baseline results. To account for system, hardware, and compiler differences, the regression test attempts to match the current machine and compiler type to the appropriate solution set from these combinations
- macOS with GNU compiler (default)
- Red Hat Enterprise Linux with Intel compiler
- Windows with Intel compiler
The compiler versions, specific math libraries, and hardware used to generate these baseline solutions are documented in the r-test repository documentation. Currently, the regression test supports only double precision solutions, so it is required to build OpenFAST in double precision for testing. All baseline solutions are generated with a double precision build.
The regression test system can be executed with CMake and CTest or manually with an included Python driver. Both systems provide similar functionality with respect to testing, but CTest integration provides access to multithreading, automation, and test reporting via CDash. Both modes of execution require some configuration as outlined below.
In both modes of execution a subdirectory is created in the build directory
called reg_tests
where all of the input files for the test cases are copied
and all of the locally generated outputs are stored.
Ultimately, both CTest and the manual execution program call a series of Python
scripts and libraries in reg_tests
and reg_tests/lib
. One such script is
lib/pass_fail.py
which reads the output files and computes a norm on each
channel reported. If the maximum norm is greater than a preset tolerance, that particular
test is reported as failed. The failure criteria is outlined in pseudocode below.
difference = abs(testData-baselineData)
for i in nChannels
if channelRange < 1 {
norm[i] = MaxNorm( difference[:,i] )
} else {
norm[i] = MaxNorm( difference[:,i] ) / channelRange
}
if max(norm) < tolerance:
success
5.2.2.1. Dependencies¶
- Python 3+
- Numpy
- CMake and CTest (Optional)
- matplotlib (Optional)
5.2.2.2. Manual driver configuration¶
The regression test can be executed manually with the included driver
openfast/reg_tests/manualRegressionTest.py
. This program reads a case list file at
openfast/reg_tests/r-test/glue-codes/fast/CaseList.md
. Cases can be removed
or ignored with a #
. This driver program includes multiple optional flags
which can be obtained by executing with the help option:
openfast/reg_tests/manualRegressionTest.py -h
For the NREL 5MW turbine test cases, an external ServoDyn controller must be compiled and included in the appropriate directory or all NREL 5MW cases will fail without starting. More information is available in the documentation for the r-test repository.
5.2.2.3. CTest configuration¶
CTest is included with CMake and is mostly a set of preconfigured targets and
commands. To use the CTest driver for the regression test, CMake must be run with
one of two CMakeLists.txt
‘s:
- openfast/CMakeList.txt
- openfast/reg_tests/CMakeLists.txt
CMake variables can be configured in the CMake
GUI or through the command line interface with
ccmake
.
The regression test specific CMake variables are
- BUILD_TESTING
- CTEST_OPENFAST_EXECUTABLE
- CTEST_[MODULE]_EXECUTABLE
- CTEST_PLOT_ERRORS
- CTEST_REGRESSION_TOL
IT IS IMPORTANT to verify that NREL 5MW turbine external controllers are compiled and placed in the correct location. More information is available in the documentation for the r-test repository, but be aware that these three DISCON controllers must exist
openfast/build/reg_tests/glue-codes/fast/5MW_Baseline/ServoDyn/DISCON.dll
openfast/build/reg_tests/glue-codes/fast/5MW_Baseline/ServoDyn/DISCON_ITIBarge.dll
openfast/build/reg_tests/glue-codes/fast/5MW_Baseline/ServoDyn/DISCON_OC3Hywind.dll
This can be accomplished manually with the CMake projects included with the DISCON source codes
at openfast/reg_tests/r-test/glue-codes/fast/5MW_Baseline/ServoDyn/
or during CMake configuration by setting the CMAKE_INSTALL_PREFIX
CMake variable.
If using this method, the install prefix variable should point to an existing and appropriate
location for CMake to place the compiled binaries. This is important because the NREL 5MW turbine external
controller CMake projects are preconfigured to install themselves in the appropriate
location in the build directory. Then, it is important to execute make install
rather than simply make
. If CMAKE_INSTALL_PREFIX
is not appropriately configured,
the install step may fail or openfast binaries may be placed in some inappropriate default location.
After CMake configuration, the automated regression test can be executed
by running either of the commands make test
or ctest
from the build
directory. If the entire OpenFAST package is to be built, CMake will configure
CTest to find the new binary at openfast/build/glue-codes/fast/openfast
.
However, if the intention is to build only the test suite, the OpenFAST binary
should be specified in the CMake configuration under the CTEST_OPENFAST_EXECUTABLE
flag. There is also a corresponding CTEST_[MODULE]_NAME
flag for each module
included in the regression test.
5.2.2.4. Running the regression test with CTest¶
When driven by CTest, the regression test can be executed by running various
forms of the command ctest
from the build directory. The basic commands are
ctest
- Run the entire regression testctest -V
- Run the entire regression test with verbose outputctest -R [TestName]
- Run a test by namectest -j [N]
- Run all tests with N tests executing in parallelctest -N
- List all of the tests available
Each regression test case contains a series of labels associating all of the
modules used. The labeling can be seen in the test instantiation in
reg_tests/CTestList.cmake
and called directly with
ctest -L [Label]
These flags can be compounded making useful variations of ctest
such as
ctest -V -L aerodyn14
- Runs all cases that use AeroDyn14 with verbose outputctest -j 16 -L aerodyn14
- Runs all cases that use AeroDyn14 in 16 concurrent processesctest -V -R 5MW_DLL_Potential_WTurb
- Runs the case with name “5MW_DLL_Potential_WTurb”ctest -N -L beamdyn
- Lists all tests with the “beamdyn” label
The automated regression test writes new files only into the build directory. Specifically,
all locally generated solutions are located in the corresponding glue-code or module within
openfast/build/reg_tests
. The baseline solutions contained in openfast/reg_tests/r-test
are strictly read not modified by the automated process.
5.2.2.5. Regression test from scratch¶
- Build OpenFAST and the test suite
git clone --recursive https://github.com/openfast/openfast.git
# The default git branch is 'master'. If necessary, switch to your target branch:
# git checkout dev
mkdir build && cd build
# Configure CMake with openfast/CMakeLists.txt
# - BUILD_TESTING
# - CTEST_OPENFAST_EXECUTABLE
# - CTEST_[MODULE]_EXECUTABLE
cmake ..
make install
ctest
- Build only the test suite
git clone --recursive https://github.com/openfast/openfast.git
# The default git branch is 'master'. If necessary, switch to your target branch:
# git checkout dev
mkdir build && cd build
# Configure CMake with openfast/reg_tests/CMakeLists.txt
# - BUILD_TESTING
# - CTEST_OPENFAST_EXECUTABLE
# - CTEST_[MODULE]_EXECUTABLE
cmake ../reg_tests
make install
ctest
Follow the link above for a detailed procedure. It is summarized below though excluding the procedure to build OpenFAST itself.
git clone --recursive https://github.com/openfast/openfast.git
cd openfast
## Build the ServoDyn external controller libraries
# Open the Visual Studio Solution (DISCON.sln) located in 'openfast\vs-build\DISCON'
# Choose Release and x64 for the Solutions Configuration and Solutions Platform
# Build Solution
## Execute the OpenFAST regression Tests
# Open a command prompt which is configured for Python (like Anaconda)
cd openfast\reg_tests
python manualRegressionTest.py ..\build\bin\openfast_x64.exe Windows Intel 1e-5