Directory structure

Django apps

EvalAI is a Django-based application, hence it leverages the concept of Django apps to properly namespace the functionalities. All the apps can be found in the apps directory situated in the root folder.

Some important apps along with their main uses are:

  • Challenges

This app handles all the workflow related to creating, modifying, and deleting challenges.

  • Hosts

This app is responsible for providing functionalities to the challenge hosts/organizers.

  • Participants

This app serves users who want to take part in any challenge. It contains code for creating a Participant Team, through which they can participate in any challenge.

  • Jobs

One of the most important apps, responsible for processing and evaluating submissions made by participants. It contains code for creating a submission, changing the visibility of the submission and populating the leaderboard for any challenge.

  • Web

This app serves some basic functionalities like providing support for contact us or adding a new contributor to the team, etc.

  • Accounts

As the name indicates, this app deals with storing and managing data related to user accounts.

  • Base

A placeholder app which contains the code that is used across various other apps.

Settings

Settings are used across the backend codebase by Django to provide config values on a per-environment basis. Currently, the following settings are available:

  • dev

Used in development environment

  • testing

Used whenever test cases are run

  • staging

Used on staging server

  • production

Used on production server

URLs

The base URLs for the project are present in evalai/urls.py. This file includes URLs of various applications, which are also namespaced by the app name. So URLs for the challenges app will have its app namespace in the URL as challenges. This actually helps us separate our API based on the app.

Frontend

The whole codebase for the frontend resides in a folder named frontend in the root directory

Scripts

Scripts contain various helper scripts, utilities, python workers. It contains the following folders:

  • migration

Contains some of the scripts which are used for one-time migration or formatting of data.

  • tools

A folder for storing helper scripts, e.g. a script to fetch pull request

  • workers

One of the main directories, which contains the code for submission worker. Submission worker is a normal python worker which is responsible for processing and evaluating submission of a user. The command to start a worker is:

python scripts/workers/submission_worker.py

Test Suite

All of the codebase related to testing resides in tests folder in the root directory. In this directory, tests are namespaced according to the app, e.g. tests for challenges app lives in a folder named challenges.

Management Commands

To perform certain actions like seeding the database, we use Django management commands. Since the management commands are common throughout the project, they are present in base application directory. At the moment, the only management command is seed, which is used to populate the database with some random values. The command can be invoked by calling

python manage.py seed