Quick Start Guides

How to set up a file-based exercise


In How to set up your first exercise we saw how to set up a simple exercise using the CoudeJudge web interface (it's recommended to read that guide before this one). In the following we will show how to do this using files instead. The motivation for using files to manage your exercises is that you can easily use your existing workflows for sharing exercises with your colleagues, versioning, etc.

Please note that this is a quick start guide, and CodeJudge supports many more features than described here. This guide should however get you started on using files for managing your exercises.

The problem

In this guide we will use the following problem again:

Create a python function sum(a, b) that returns the sum of the integers a and b, and a function factorial(a) that returns 1 * 2 * ... * a = a! (should work for a >= 0).

Files structure

To get an overview, let us start from the end, and see what files we end up having:

collection.yml describes the exercise collection the exercise belongs to (for instance an assignment or a weekly exercise collection). exercise.yml describes fundamental things about the exercise. contains the modal solution for the problem. contains the tests which the users' programs will be tested on.

The exercise collection: collection.yml

Let us create an exercise group for the first week of a course. The file should contain: (see documentation for details)

name: Week 1
type: exerciseGroup
visibleFrom: 2000-01-01 12:34:56

The exercise: exercise.yml

Similarly we create a file describing the exercise itself: (see documentation for details)

name: Calculator
suggestedSolutionVisibility: afterSolved
description: >
    Create a python function **sum(a, b)** that returns the sum
    of the integers **a** and **b**, and a function **factorial(a)** that
    returns **1 \* 2 \* ... \* a = a!** (should work for **a >= 0**).
  • The problem description can be formatted using Markdown. The problem description can also be put in a file called instead.

The solution:

Next, we'll create a solution to the problem.

def sum(a, b):
    return a + b

def factorial(a):
    if a <= 1:
        return 1

    return factorial(a - 1) * a


Finally, we have to create the tests we want to use:

#/@ Test
print(sum(3, 6))
#/@ Test
print(sum(-5, -9))
#/@ Test
print(sum(33, 64))
#/@ Test
#/@ Test
#/@ Test
#/@ Test
#/@ Test
#/@ Hint: Remember 0! = 1

Hopefully, it is relatively clear what the above test cases are, so now we will look at the structure of the file. Note the above test code is a fully correct and runnable python program when combined with the solution, so we can easily develop both the solution and the test file directly in our favorite IDE.

The beginning of a test should be marked with "#/@ Test" ("//@ Test" in most other languages). A test can consists of multiple code lines. All lines not inside a test are considered common to all tests. Finally a hint can be added with "#/@ Hint: The hint text here" - this hint will only be shown, when a program fails the test. Hints are especially useful for special/tricky test cases.

When the file is uploaded to CodeJudge it is split into a runnable program for each test case. For instance the first test case will be (in a file called

print(sum(3, 6))

You might now be thinking that we also need to specify what the expected output of our test cases are. However, in this example we will let CodeJudge produce the expected output by uploading just the "input" of the test cases and the solution. (It is possible to manually specify the expected output instead - see the documentation for this)

Test cases can be much more complex than this example; they may involve input from the console, command line arguments, files, multiple correct outputs, etc. For more about this, see the documentation.

Upload to CodeJudge

It is now time to setup the exercise on CodeJudge, so we will navigate to our course on CodeJudge.

  1. Select "Exercises" in the menu.
  2. Select "Sync" in the page tabs.
  3. Drag and drop the 4 files (including directory structure) to upload it. If your browser does not support this, you can zip the files before uploading.
  4. A list of changes will be shown if everything looks fine. Otherwise an error will be shown.
  5. If you are happy with the changes, accept them and you are done.

After a short while of processing, the exercise is ready!

Updating the exercise

If you found a mistake in your exercise, do not worry (too much). Simply change the files as needed and upload to CodeJudge again. CodeJudge will recognize this exercise already exists (based on the name), and show what changes you are about to upload. You can even do this on an active exercise. CodeJudge will automatically detect if any of the changes require submissions to be regraded.

Test it!

As a final step, it is a good idea to see if the exercise actually works as expected (especially when it is the first time you create one). Navigate to the "View"-tab of the exercise and try to upload your solution.