A Survey on Online Judge Systems and Their Applications

Szymon Wąsik; Maciej Antczak; Jan Badura; Artur Laskowski; Tomasz Sternal

doi:10.1145/3143560

Scientific Information System of the Poznań University of Technology

PL EN

Main page / Publications / A Survey on Online Judge Systems and Their Applications

Submit a comment

Article

Download BibTeX

Title

A Survey on Online Judge Systems and Their Applications

Authors

Szymon Wąsik (WI) ^{[ 1 ][ 2.3 ][ P ]}
Maciej Antczak (WI) ^{[ 1 ][ 2.3 ][ P ]}
Jan Badura (WI) ^{[ 1 ][ 2.3 ][ P ]}
Artur Laskowski (WI) ^{[ 1 ][ 2.3 ][ P ]}
Tomasz Sternal (WI) ^{[ 1 ][ P ]}

^{[ 1 ]} Instytut Informatyki, Wydział Informatyki, Politechnika Poznańska | ^{[ P ]} employee

Scientific discipline (Law 2.0)

[2.3] Information and communication technology

Year of publication

2018

Published in

ACM Computing Surveys

Journal year: 2018 | Journal volume: vol. 51 | Journal number: no. 1

Article type

scientific article

Publication language

english

Keywords

EN

online judge
crowdsourcing
evaluation as a service
challenge
contest

Abstract

EN Online judges are systems designed for the reliable evaluation of algorithm source code submitted by users, which is next compiled and tested in a homogeneous environment. Online judges are becoming popular in various applications. Thus, we would like to review the state of the art for these systems. We classify them according to their principal objectives into systems supporting organization of competitive programming contests, enhancing education and recruitment processes, facilitating the solving of data mining challenges, online compilers and development platforms integrated as components of other custom systems. Moreover, we introduce a formal definition of an online judge system and summarize the common evaluation methodology supported by such systems. Finally, we briefly discuss an Optil.io platform as an example of an online judge system, which has been proposed for the solving of complex optimization problems. We also analyze the competition results conducted using this platform. The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.

Pages (from - to)

3-1 - 3-34

DOI

10.1145/3143560

URL

https://dl.acm.org/doi/10.1145/3143560

Comments

Article number: 3

Ministry points / journal

50

Ministry points / journal in years 2017-2021

50