The TIRA platform

TIRA will be used to evaluate your submitted systems. You will install your systems in a dedicated virtual machine provided by TIRA. During evaluation phases, the systems will get access to the test data and process the test data inside the VM. The evaluation script will run there as well.

There is some flexibility to the operating system and hardware resources available in the VM. Once registered, you will be contacted by TIRA administrators and asked about your preferences. You will also be offered a private Github repository where you can keep and version-control the source code of your parser (and increase future reproducibility of your results).

Typically, you will train your models on your own hardware. Once ready, you will upload both your parsing system and the models to the VM. It is not forbidden to train the models directly in the VM but note that the resources there are limited.

When your system and models have been deployed in your VM, proceed to the TIRA web interface (same login as to your VM), register there the shell command to run your system, and run it. Note that your VM will not be accessible while your system is running – it will be “sandboxed”, detached from the internet, and after the run the state of the VM before the run will be restored. Your run can then be reviewed and evaluated by the organizers.

Note that your system is expected to read two paths to the input and output folders from the command line. When you register the command to run your system, put variables in positions where you expect to see these paths. Thus if your system expects to get the options -i and -o, followed by input and output path respectively, the command you register may look like this:

/home/my-user-name/my-software/run.sh -i $inputDataset -o $outputDir

The actually executed command will then look something like this:

/home/my-user-name/my-software/run.sh -i /media/training-datasets/chat-analytics-for-twitch/chat20-training-dataset-2020-03-24/ -o /tmp/chat-analytics-for-twitch/2020-04-22-09-35-53/output

The folder provided via the $inputDataset argument contains files called train.json or train_small.json (for the train runs) or test.json (for the test runs). All files have the same structure as the train.json from the training dataset, i.e. they have fields such as the channel, user, and messages. For each of the channel-user combinations in this file, you need to predict the subscription status. You then output a file called predictions.csv to the provided output directory. This file should contain the fields user, channel, subscribed, as in the ground truth file of the training dataset.

See the links below for more details.

• TIRA user guide
• TIRA paper (Section 2)

Processing the data on TIRA

Within your VM, you can see the training data and another file consisting of the first few thousand training data lines mounted read-only at /media/training-datasets/chat-analytics-for-twitch/. We recommend to first try running your system on the subset from within your VM (no sandboxing), then try the same through the web interface. When invoked from the web interface, your system will be given path to the input file and path to the output folder, where it is supposed to generate all output files. When you run the system on training data, the input path will lead to the location mentioned above. But during the test phase, it will be a different path to which you normally don’t have access.

Test phase

In this challenge, there will be an early bird as well as the final evaluation phase. After the early bird evaluation phase, you will get notified of your performance privately, while after the final evaluation phase, a leaderboard with all team’s performances will be provided. In general, we will not tell you the score of your system before the test phase closes. The exception is when your score on one or more files is anomalously low, which probably means a bug or another technical problem.

You can register more than one system (“software”) per virtual machine. You can have all of them officially scored and use those numbers in your system description paper. However, you have to decide what is the primary system that will represent your team in the challenge (you have to decide that based on your experiments with the development data, without knowing the actual performance on the test data). TIRA gives systems automatic names “Software 1”, “Software 2” etc. If possible please make sure that “Software 1” is your primary system. If this is not possible because you deleted “Software 1” or because you changed your decision after completing runs that you do not want to delete, send us a message with the name of your primary system.

You can run one system multiple times (especially if we tell you that there is a problem with your previous run). If there are multiple successful runs of your primary system, the last one will be considered your submission to the challenge. You can delete a run if you want a previous one to be submitted.

If you modify your software between runs, please make sure you can reinstate the version that was used in the previous run (you can use the private Github repository offered by TIRA for this purpose). This is important for reproducibility of the final results of the challenge. If your last run is not successful and the previous run becomes your official submission, you should be able to return to the version of your system that generated the submission.

If your system requires more resources than available in the default VM (memory, disk space, CPUs), please estimate what you need and discuss it with Martin Potthast, the administrator of TIRA. You can get a VM with more resources. Note however that accommodating such requests takes time, so act early. The sooner you complete at least one successful run, the safer you are.

Access to the Virtual Machines and Intellectual Property Rights

The VMs are distributed across different hosts. The only people who have access to the participant VMs are TIRA admins (a very small group of people operating the service) and the organizers of the challenge.

We can guarantee that we will never deliberately share your VM or its contents, nor use it for anything else but for the purpose of evaluating your software as part of the challenge, unless you give us written permission. We ask that you give the challenge organizers and the TIRA operators usage rights for your software for this purpose only.

However, we cannot guarantee that no content of the VM will leak accidentally and we shall not be held liable for damages caused by such leaks. In particular, we cannot vouch that the software packages and operating systems TIRA depends on are free of zero day exploits.

The performance results and output of your software will become part of public record, for which we ask for indefinite, irrevocable, and transferable rights to publish them within any scientific publication as well as on the TIRA web service.

By deploying your system in your VM and running it through the TIRA interface you express your consent to these conditions and give us the rights as described above.

We understand that for industry-based participants, protecting their software is an important matter. If you want to learn more about the TIRA procedures and about your options, please get in touch with the TIRA administrators (tira at webis dot de). TIRA has been used by a number of companies so far, some small ones but also some big ones. The involvement of industry in scientific events should not be foreclosed. If we are to improve reproducibility at large, however, there is no way around venturing more openness on either side.