Hybrid Human-Machine Classification System for Cultural Heritage Data

Shabani, Shaban and Sokhn, Maria and Schuldt, Heiko. (2020) Hybrid Human-Machine Classification System for Cultural Heritage Data. In: Proceedings of the 2nd workshop on Structuring and Understanding of Multimedia heritAge Contents (SUMAC 20). New York, pp. 49-56.

Full text not available from this repository.

Official URL: https://edoc.unibas.ch/81637/

Downloads: Statistics Overview


The advancement of digital technologies has helped cultural heritage organizations to digitize their data collections and improve the accessibility via online platforms. These platforms have enabled citizens to contribute to the process of digital preservation of cultural heritage by sharing documents and their knowledge. However, many historical datasets have problems due to incomplete metadata. To solve this issue, cultural heritage organizations heavily depend on domain experts. In this paper, we address the issue of completing the metadata of historical digital collections. For this, we introduce a new hybrid human-machine model. This model jointly integrates predictions of a deep multi-input model and inferred labels from multiple crowd judgements. The multi-input model uses visual features extracted from the images and textual features from the metadata, complemented with Wikipedia classes of concepts extracted in the text. On the crowd answer aggregation, our method considers the workers' reliability scores. This score is based on the performance of workers' task history and their performance in our task. We have applied our hybrid approach to a culture heritage platform and the evaluations show that it outperforms both deep learning and crowdsourcing when applied individually.
Faculties and Departments:05 Faculty of Science > Departement Mathematik und Informatik > Informatik > Databases and Information Systems (Schuldt)
UniBasel Contributors:Schuldt, Heiko and Shabani, Shaban
Item Type:Conference or Workshop Item, refereed
Conference or workshop item Subtype:Conference Paper
Publisher:Association for Computing Machinery
Note:Publication type according to Uni Basel Research Database: Conference paper
Identification Number:
Last Modified:02 Feb 2021 12:46
Deposited On:02 Feb 2021 12:46

Repository Staff Only: item control page