Quick Start Guides

File-based exercise management

There are two ways to manage exercises in CodeJudge. Either one can directly use the web interface, or one can use the file-based way. The first way is simpler to get started with, but the latter is often more useful for larger and more complex courses.

To get started with file-based exercise mangement we suggest you look at How to set up a file-based exercise.


When you have created new exercises or edited existing ones, you will need the changes to be reflected on CodeJudge. We call this step synchronization. In this step, you upload your changed files (either on the Sync-tab in the web interface or by using the CLI), and these are compared to version of the exercises we currently have. After this, you will be shown a list of the changes there have been detected. You should check these changes corresponds to what you expect, and then apply them.

CodeJudge CLI (preview)

NOTE: This is a preview and is likely to contain some bugs. Please contact us if you encounter any problems, or have suggestions for improvements. The preview version currently does not check for updates, so make sure to check for updates if you encounter any problems.

We have created a python program that can synchronize files on your local file system with CodeJudge without using the web interface. The latest version of the program can be found on PyPi, and can easily be installed using pip:

pip install codejudge

You must authenticate with CodeJudge to use the CLI. Log in on CodeJudge, click on your profile icon in the upper right corner, and select API Tokens. Click generate to make a new token, and copy paste it into the following command:

codejudge auth [API token]

You can now initialize a directory to be synced with CodeJudge by navigating to the directory and running (this generates a .codejudge file that stores the identifier of the course, this file does not contain secrets):

codejudge init "[name of course]"

Now everytime you want to sync changes from this directory, simply run the command:

codejudge sync

This will give you a list of detected changes that you can accept to apply or abort (verify the changes seem reasonable before accepting).

The CLI tries to determine which files are related to CodeJudge and only upload these (it looks for collection.yml/json, exercise.yml/json, and testgroup.yml/json files to determine directories of interest). This means you can have other files in these directories as well.

We strongly suggest using the CodeJudge CLI in combination with git or similar in order to easily collaborate on and manage your exercises.


It is possible to export the exercises of a course to the file-based format from the Sync-tab in the web interface. However, the exported files will most likely not look similar to the files you originally synchronized. This is due to the fact, when you synchronize we transform your files in to an internal format, in order for us to allow online editing of exercises and similar. Thus you should in general keep your original files, and not rely on the export function.

File hierarchy overview

Below are listed all files that are currently supported by the system.

Language overrides

It is in general possible to override configuration settings and test files for specific languages. To do this, one simply prefix the path of the language specific file with "[lang]/" (for instance "py/"). For regular files, an eventual existing file will simply be overwritten. For config files, the files will be merged.

Single file tests

The single file tests format allows you to collect all tests into a single file while still working with valid python/java/... files. In most cases, this greatly simplify creating tests. See How to create test data for my language? for more examples.


Let's see a simple example

//@ TestGroup: Samples
//@ OnTestGroupFailue: Break

//@ Test: Sample
1 3
//@ Ans: 4
//@ TestGroup: Tests
//@ Test
100 200
//@ Ans: 300

In this example we create two test groups: Samples and Tests. For Samples the property OnTestGroupFailue is set to Break and a test named "Sample" with input "1 3" and ans "4" is created. For the test group Tests a test named Test01 with input "100 200" and ans "300" is created. Tests are named Test01..N when a name is not specified.

Special files can now also be created:

//@ Test: FileTest
//@ wkdir/text-file.txt
Hello world

will create the file FileTest/wkdir/text-file.txt with "Hello world" as content.

For test scripts all lines before the first //@ Test(Group) and after a //@ End command will be written to all test files:

class Exercise {
  public static void main(String[] args) {
    public calculator = new Calculator();
    //@ TestGroup: Samples
    //@ Test
    System.out.println(Calculator.Sum(1, 3));
    //@ TestGroup: Tests
    //@ Test
    System.out.println(Calculator.Sum(100, 200));
    //@ End

will output

class Exercise {
  public static void main(String[] args) {
    public calculator = new Calculator();
    System.out.println(Calculator.Sum(1, 3));
class Exercise {
  public static void main(String[] args) {
    public calculator = new Calculator();
    System.out.println(Calculator.Sum(100, 200));

(Here the multiline prefix //$ could be useful if one would create a wkdir file for instance, as the class could then still be compiled.)


Files are parsed on a line by line basis. Lines starting with one of the following two prefixes have a special meaning:

(For languages where "//" is not a comment, we also support that the first character can be "#" or "%" i.e. "#/@", "%/$" etc.)

There are two types of commands:

//@ [State]: [Name]
//@ [State]
//@ [Property name]: [Value]
//@ [Property name]
//$ with optional multiline prefix

File name

The single file must be named Tests.[ext] where ext can be any extension of a test file. The Tests file can either be placed at exercise level (in the exercise folder) if it is specifying test groups, or in a test group folder if it is only specifying tests. Tests.[in|ans|lang|args|score|group|hint]

The value specified after //@ Test will get the extension of the single test file, e.g. if the single file is named, the values after //@ Test will be written to [test name].in.


For more complex exercises the test data is often generated by code e.g. a python script. Instead of having to run the script locally and upload/sync all the generated files, CodeJudge now supports uploading the generator code (e.g. the python script) instead when syncing. See the guide How to use generators to get started.

The program/script should be in a subfolder of the exercise called generator or if it is only a single file it can be called generator.[lang] e.g.

The generator is the executed as a pre-step to the normal synchronization, and all the generated files will be passed on to sync as if uploaded. This for instance means that a generator can output a single test file, which will then be split up into the individual test files.

Configuration files

Configuration files can either be json- or YAML-files. Properties can be either PascalCase or camelCase.

Collections: collection.yml

All exercise collection properties can be set in this file.

Exercises: exercise.yml

All exercise properties and exercise-level test configuration can be set in this file.

TestGroups: testgroup.yml

All test group properties and test group-level test configuration can be set in this file.

Test Configuration: config.yml

All test configuration can be set in this file. Should generally not be used when having test configuration in either exercise.yml or testgroup.yml.