# The TIRA platform

TIRA will be used to evaluate your submitted systems. You will install your systems in a dedicated virtual machine provided by TIRA. During evaluation phases, the systems will get access to the test data and process the test data inside the VM. The evaluation script will run there as well.

There is some flexibility to the operating system and hardware resources available in the VM. Once registered, you will be contacted by TIRA administrators and asked about your preferences. You will also be offered a private Github repository where you can keep and version-control the source code of your parser (and increase future reproducibility of your results).

Typically, you will train your models on your own hardware. Once ready, you will upload both your parsing system and the models to the VM. It is not forbidden to train the models directly in the VM but note that the resources there are limited.

When your system and models have been deployed in your VM, proceed to the TIRA web interface (same login as to your VM), register there the shell command to run your system, and run it. Note that your VM will not be accessible while your system is running – it will be “sandboxed”, detached from the internet, and after the run the state of the VM before the run will be restored. Your run can then be reviewed and evaluated by the organizers.

Note that your system is expected to read two paths to the input and output folders from the command line. When you register the command to run your system, put variables in positions where you expect to see these paths. Thus if your system expects to get the options -i and -o, followed by input and output path respectively, the command you register may look like this:

/home/my-user-name/my-software/run.sh -i $inputDataset -o$outputDir

The actually executed command will then look something like this:

/home/my-user-name/my-software/run.sh -i /media/training-datasets/chat-analytics-for-twitch/chat20-training-dataset-2020-03-24/ -o /tmp/chat-analytics-for-twitch/2020-04-22-09-35-53/output

The folder provided via the \$inputDataset argument contains files called train.json or train_small.json (for the train runs) or test.json (for the test runs). All files have the same structure as the train.json from the training dataset, i.e. they have fields such as the channel, user, and messages. For each of the channel-user combinations in this file, you need to predict the subscription status. You then output a file called predictions.csv to the provided output directory. This file should contain the fields user, channel, subscribed, as in the ground truth file of the training dataset.

See the links below for more details.

# Processing the data on TIRA

Within your VM, you can see the training data and another file consisting of the first few thousand training data lines mounted read-only at /media/training-datasets/chat-analytics-for-twitch/. We recommend to first try running your system on the subset from within your VM (no sandboxing), then try the same through the web interface. When invoked from the web interface, your system will be given path to the input file and path to the output folder, where it is supposed to generate all output files. When you run the system on training data, the input path will lead to the location mentioned above. But during the test phase, it will be a different path to which you normally don’t have access.

# Test phase

In this challenge, there will be an early bird as well as the final evaluation phase. After the early bird evaluation phase, you will get notified of your performance privately, while after the final evaluation phase, a leaderboard with all team’s performances will be provided. In general, we will not tell you the score of your system before the test phase closes. The exception is when your score on one or more files is anomalously low, which probably means a bug or another technical problem.

You can register more than one system (“software”) per virtual machine. You can have all of them officially scored and use those numbers in your system description paper. However, you have to decide what is the primary system that will represent your team in the challenge (you have to decide that based on your experiments with the development data, without knowing the actual performance on the test data). TIRA gives systems automatic names “Software 1”, “Software 2” etc. If possible please make sure that “Software 1” is your primary system. If this is not possible because you deleted “Software 1” or because you changed your decision after completing runs that you do not want to delete, send us a message with the name of your primary system.

You can run one system multiple times (especially if we tell you that there is a problem with your previous run). If there are multiple successful runs of your primary system, the last one will be considered your submission to the challenge. You can delete a run if you want a previous one to be submitted.

If you modify your software between runs, please make sure you can reinstate the version that was used in the previous run (you can use the private Github repository offered by TIRA for this purpose). This is important for reproducibility of the final results of the challenge. If your last run is not successful and the previous run becomes your official submission, you should be able to return to the version of your system that generated the submission.

If your system requires more resources than available in the default VM (memory, disk space, CPUs), please estimate what you need and discuss it with Martin Potthast, the administrator of TIRA. You can get a VM with more resources. Note however that accommodating such requests takes time, so act early. The sooner you complete at least one successful run, the safer you are.