automatic-test-scripts.md

+# Automatic evaluation of students submission.
+
+## Create entry in dbabgabe.
+
+1.  Go to [dbabgabe](https://dbabgabe.cosy.sbg.ac.at/admin/).
+2.  Add a new Duty (mostly self-explained):
+    - add Relpath - subdirectory on the server where evaluation files will be stored
+    - do not set Isevaluate
+
+## Write necessary scripts.
+
+3.  Implement the correct solution. It must confirm to all requirements given to students.
+4.  Implement a script to generate test cases and produce correct solutions.
+5.  Implement a verification script that takes student's solution as the only argument (e.g. tester.py).
+    - This script should print to stdout a status code ('1': success, '2': fail, '3': points). Code '3' must be followed by the number of obtained points (in a new line). Each code may be followed by a message to the student (starting in a new line after codes '1' and '2'; starting in a new line after the points in case of code '3').
+    - This script must execute student's solution externally using the following script (e.g. execute_student.sh):
+
+~~~ { .bash }
+#!/bin/bash
+
+# Exit immediately if a command exits with a non-zero status.
+set -e
+
+# Set paths to test areas. 
+# Example local config file:
+#   testarea_local=/tmp
+#   testarea=/tmp
+#   savetest=
+source config
+
+# First parameter is the stunedt's script that is to be executed.
+scripttoexecute="$1"
+# Second parameter is the input to the student's script.
+testinput="$2"
+
+# Copy student's script and input to the safe test area.
+cp "$scripttoexecute" "$testinput" $testarea
+chmod +rx "$testarea/$(basename $scripttoexecute)"
+
+# Execute the student's script in the safe tet area.
+$savetest python -B $testarea_local/"$(basename $scripttoexecute)" $testarea_local/"$(basename $testinput)"
+# '$?' returns the last executed command's return value. Store it ant return at the end.
+ret=$?
+
+# Clean up/
+rm -f $testarea/"$(basename $scripttoexecute)" $testarea/"$(basename $testinput)"
+
+exit $ret
+~~~
+
+## Enable automatic evaluation.
+
+6. Copy to `inhander@dodo ~/testscripts/<COURSE>/<RELPATH>` (RELPATH given in point 2) all files from point 5 and test cases with correct outputs (generated with a script from point 4).
+7. Make necessary file executable.
+8. Create a symbolic link from tester.sh to the verification script (if it has a different name), for example, `ln -s tester.py tester.sh`.
+9. Set Isevaluate in dbabgabe.
+
+## Examples.
+
+- [Validation protocol for ADB.](https://frosch.cosy.sbg.ac.at/mpawlik/adb-2016ws/tree/master/labs/src/validation-protocol)
\ No newline at end of file