City-Stories: A Multimedia Hybrid Content and Entity Retrieval System for Historical Data

Information systems used in tourism rely mostly on up-to-date content on attractive places. In addition, these systems increasingly make use of archived photographs, documents, films, or even ancient paintings and other artwork by integrating such curated content from museums and memory institutions, possibly enriched with user-provided content. Hence the distinction between cultural heritage applications and tourism more and more blurs. Users are not only interested in the current appearance of landscapes, monuments, or buildings, but also in the evolution of these places over time. This requires large multimedia collections which integrate content from several cultural heritage institutions. As a consequence, interactive retrieval systems for historical multimedia are needed that support homogeneous content-based and semantic querying despite the heterogeneity of these collections. In this paper we present City-Stories, a multimedia hybrid content and entity retrieval system. City-Stories is based on a state-of-the-art open source multimedia retrieval system. Multimedia features in City-Stories represent multiple semantic levels: low-level (e.g., color, edge, motion), mid-level (e.g., date, location, objects), and high-level features (e.g., semantic entities, scene category). For the latter, City-Stories applies entity recognition and entity linking for identifying semantic concepts and linking objects across media types. Consequently, City-Stories supports various types of cross-modal queries. Moreover, City-Stories uses a map-based visualization layer that facilitates spatial queries and browsing. Finally, City-Stories follows a crowdsourcing approach for content annotation and for enriching curated content with multimedia objects and documents provided by users. The paper shows how the City-Stories system seamlessly combines content-based search with entity-based navigation and leverages the wisdom of the crowd for manual annotations.