(Mis)Matching Metadata: Improving Accessibility in Digital Visual Archives through the EyCon Project

1.1 Connecting the Data

The problem with missing or thin record data is not just affecting historical photographs. The visual records of our current history are at risk of being poorly recorded too, if recorded at all. Most of our photographs are now born-digital, as many of our mobile phones double as digital cameras and the quality of film-based photography has been far surpassed by digital capabilities. According to Anderson Almeida Firmino et al., hundreds of millions of photographs are added to Facebook daily, as well as 80 million photos and videos on Instagram, but few of these visual records are annotated by their creators [24]. Most smartphones will now record basic metadata when their users capture audio or visual images, such as the time and location; some will recognize faces in photographs. But how do these technologies translate within a complex archival context? What can the advances in AI do to help improve the accessibility and the data records of historical photo archives?

Efforts to digitize archival collections are often done in a silo. The disconnect between visual archives, where collections or similar images may be scattered between numerous institutions or even countries, means their usability as historical objects becomes confounded. While this problem is being recognized and addressed by several current projects, the broader issue remains that many historical records, in all formats, have been affected by contemporary prejudices, nationalistic interpretations, and/or racist discrimination, and specific contextual details may have been lost as a result. Without researchers being able to access and critically approach these images, the relevance of the metadata will continue to remain a sticking point.

For example, when searching for First World War photographs on the Imperial War Museum's UK (IWM) website, users are given a search bar or the initial options of filtering results by type, “photograph,” and period, “First World War” [25]. These two selections give over 135,000 results. Subsequent filters, including “Format,” “Creator,” and “Keywords,” provide users with predetermined search terms, or the search bar can be used for more specific keywords. By searching for “colonial” within these filtered records, the results are reduced to 507. Each of these images are labeled with the category “photographs,” and yet the record “Journée de l'Armée d'Afrique et des Troupes Coloniales [African Army and Colonial Troops Day]” is an illustrated poster, not a photograph [26]. The mis-categorized record is therefore not included in the search for “Posters” and “First World War,” potentially skewing the search results. Such issues can be corrected by direct communication with the host organization, but they also remind us that the creation of record metadata will never be a flawless process, even with the use of AI.

The driving force behind the EyCon project is that such disparity in visual records of early conflict, particularly concerning colonial or imperial warfare, has led to the entrenchment of national exceptionalism and an inability for researchers and public users to cross-examine potentially connected photographic collections. To offer an example of national exceptionalism and how a photographic record can become distorted when processed in a silo, we can examine an image currently held by two different French archives.

The black-and-white photograph in Figure 1, dated October 23, 1917, depicts three male soldiers in the First World War, wearing uniform and helmets, standing in a muddy field. One man appears to be injured. The photograph is held by La Contemporaine and the Etablissement de Communication et de Production Audiovisuelle de la Défense (ECPAD). The ECPAD description reads “Un tirailleur sénégalais est blessé au ‘Balcon’, position allemande conquise par les alliés près de Soupir” [A Senegalese rifleman was wounded at the “Balcon,” a German position conquered by the Allies near Soupir]. However, La Contemporaine includes the description handwritten under their image, which is part of an album: “Blessé français évacué sur l'arrière” [French wounded evacuated to the rear] [27]. These different descriptions have automatically become the title of the same image and demonstrate the necessity for a collaborative approach to the recording of image data and historical context. If users were able to compare these images visually and analyze their different descriptions together, such potentially problematic metadata and semantic issues would be more easily identified. For EyCon, especially in the inclusion of photograph albums in the database, where different metadata is held for a photograph or multiple photographs of the same event, all the variant records are being maintained to preserve the contextual labeling of these historical documents for research and discoverability.

2.1 What Is “Problematic” Metadata?

The EyCon team has had to identify what metadata is important to collect for the purposes of the project and its afterlife. However, assessing the relevancy of a digital image's record is not a straightforward process. But as current approaches are being built around existing records and their metadata, there must be a serious consideration for how this data is approached and applied. Where problematic metadata can be identified in existing records, such as inconsistencies with the object description or the use of outdated or insensitive language, as researchers and users of archival records, we must first ask whether any existing data should be corrected. And if so, by whom?

Many archives are issuing policies and statements regarding offensive language or terminology used in their records. Taking an example from The Keep, an archive center in East Sussex, UK, partnered with the East Sussex Record Office, Brighton, and the Hove Record Office and the University of Sussex, their “Inclusive Cataloguing” decolonization statement outlines that they hold material with direct links to enslavement and British imperialism, and that the “catalogue does not always accurately reflect the reality of these relationships.” The statement also acknowledges that such terminology can be a “barrier to access” [41]. The University of Sussex's policy states that “Locally, we will identify the use of problematic or incorrect language and take steps to change it, updating terminology and subject headings” [42]. Many decolonization statements will include similar approaches in an attempt to improve awareness for users and to break down accessibility issues. Indeed, the Archives & Records Association (ARA, UK and Ireland) has a recent series of blog posts addressing inclusive cataloguing [43].

However, if archives are tackling these issues in isolated and targeted ways within their own institutions, what decisions are being made regarding problematic or incorrect language in catalog data? And who is making these updates? Without national or even international guidance and collaborative approaches, are archives not still at risk of furthering cultural and national biases by “correcting” metadata? An object's metadata is itself a historical resource. While more is being done to address sensitive or offensive record data through trigger warnings and archival policy updates regarding the decolonizing of collections, we cannot simply erase or correct the anachronistic language or potentially misleading descriptions, as they are part of that photograph's history. These details can tell us about how the image was recorded and about the views of those who wrote its original record and enable analyses of how that data has impacted historical context. Although there will be controversial decisions made, especially in the choice to remove outdated and racist language, there is also a strong argument to preserve the semantics of the original record to keep history accountable.

The 2019 Le Modèle Noir exhibition at the Musée d'Orsay, Paris, stands as an example. Orsay's president at the time, Laurence des Cars, made the decision to change some of the titles of the exhibited works to include the names of black subjects, demonstrating a shift in power. The title Portrait d'une négresse was changed to Portrait femme noire [44]. While the accusation of rewriting history will ring in many ears, the solution to problematic contextual data is to find a balance between retaining original descriptions for research purposes and enriching the metadata to allow modern users to discover images in new lights. The recorded history of an object is what Hannah Turner refers to as “legacy data,” a term that encourages us to think critically about how an object's documentation can embed some narratives while excluding others. Indeed, Turner argues that the most important metadata for an image is tied to its historical or original situation [45]. The EyCon project is therefore working on the principle that outdated or mismatching metadata should be preserved, but it should also be enriched by more recent information, and even additional sources of knowledge where contexts may have been lost [46].

3.1 Object Detection and Identifying Sensitive Content

Image recognition and object detection tools can be used to create textual data from the automated visual analysis of an image. Such tools are widely used in the conservation of cultural heritage records and aim to define a picture by identifying and locating a predefined element during training, to which a semantic label can be attached [53]. In this sense, computer vision performs the descriptive tagging work normally done by archivists and record managers, which in turn makes it possible to reference and connect the content of a corpus of images and to carry out searches based on these tags.

One of the aims of the EyCon project is to use descriptive tagging to link, analyze, and comment on their curated collections to facilitate discovery and connect these neglected and scattered visual materials. As the corpus includes hugely varying content, the necessary diversity of the created metadata requires the use of several complementary approaches and newly developed tools [54]. The methodology used by the EyCon project for developing a pre-trained model for object detection includes two main steps. The first prerequisite is the choice of specific classes that correspond to the objects to be detected. The second is to annotate a part of the corpus with all the elements we need and their location to allow automatic recognition. The training is then semi-supervised and the annotations are made for the algorithm to learn from.

However, the largest issue in automated detection is the relationship between the visual content and the vocabulary used to describe it [55]. For the conflict pictures analyzed by EyCon, we have envisaged using several main classes for recognition, which will be integrated into the metadata: weapons, ships, landscapes, and the gender of the individuals depicted. The alleged objectivity of AI tools is therefore continually juxtaposed with the manual choice of vocabulary, which will ultimately constitute the ontology of the image database created through the project. The subjectivity of this choice can be reduced by using existing ontologies that are considered to be as neutral as possible, although any bias in the results will be amplified by the scale of the corpus and will depend on the selection of the descriptors and their interpretation.

Most libraries, for example, will follow one of several indexing systems based on chosen vocabularies, which force the interpretation of certain events simply by selecting one name at the expense of others. One example of the Rameau indexation used by the Bibliothèque Nationale de France is the title “Boxeurs, Guerre des (1899–1901),” which is used to describe a large variety of realities (such as the dates) that could differ from one ontology to another [56]. Likewise, the term “war” could be replaced by “conflict,” “rebellion,” or other descriptors, depending on the approach, the point of view, and the language used. The ontology used by the EyCon team is based on Iconclass [57], even though its vision is predominantly Eurocentric. For example, the annotations made during the digitization of the Victor Forbin collection at the  Service Historique de la Défense (SHD) using the Iconclass categories still must be implemented at scale. We will be testing the possibilities offered by the Iconclass test set and apply ImageNet on the classes that make the most sense, keeping only the highest recall rates.

The fact that the images chosen to select the suited vocabulary are only from Europe also poses a problem for a project such as EyCon, which aims to de-Westernize the view of visually archived events [58]. Where the detection of some features, such as landscapes or objects, might seem unproblematic, ethical issues soon arise when we want to identify and name colonial troops through semantic choices. It is difficult to select vocabularies that will allow us to keep the cultural and geographical context of objects while avoiding the occidental or Westernized perspective. This is more significant when it comes to outdated descriptions: for example, the term “colonial army” does not cover all historical realities. The questions concerning controlled vocabularies are particularly important to consider upstream, as they determine the level of interoperability of the delivered database. A tension therefore surfaces in the decision-making process for the choice of ontology. The universalization of the vocabulary used by Iconclass leads to a harmful over-classification, but an interoperable ontology is necessary for the dissemination and usability of the EyCon database. This is further problematized when one is dealing with populations or events that are unrecorded or even erased from the archive, stunting the emergence of new thinking on controversial issues [59].

The work of Mrinalini Luthra, Konstantin Todorov, Charles Jeurgens, and Giovanni Colavizza on their recent Unsilencing Colonial Archives project has approached the issues of problematic archival classification and metadata through automated entity recognition for content-based indexing. An annotation typology was applied to a colonial archive of the Dutch East India Company, and annotations were completed as a shared task, based on state-of-the-art neural network models [60]. While this process is beyond the scope of the EyCon project, we share in the wider aims of the dataset creation to train entity recognition models to be more inclusive. The training of new models on more accurate corpora will allow machine learning to target visual content more accurately and reduce digital biases, such as image qualities that are too different or object labeling that is too anachronistic [61]. Pictures depicting colonial conflicts are particularly affected by this lack of semantic precision. As potential bias lies in the choice of vocabulary used by the annotation, we focus on two prerequisites: prior knowledge of the corpus and its content (to define the terms) as well as the connection to a controlled vocabulary linked to a pre-defined and pre-conceptualized ontology [62]. The constitution of a corpus is based on thematic, sometimes geographical and temporal criteria on which the visual content depends. The more training a model has on a homogeneous (and therefore characteristically restricted) dataset, the more accurate it will be when detecting the same type of documents [63]. However, while such categorization is necessary, it must be as flexible as possible to avoid locking the image into its network of visual descriptors. For this reason, the wider EyCon team is formed of experts and consultants within archival science, humanities, and computer science, who have been consulted regarding the choices of vocabularies used in the project's training models.

We are also using object detection to help identify the sensitive content of photographs in the database. As the collection is intended to be published online, we will include trigger warnings to identify sensitive content within the collection. However, with a corpus of thousands of images, it is impossible to annotate each sensitive image individually, and this process is also subjective, considering the potentially traumatic nature of the images and their use as research materials. The advantage of automatic detection tools is that they can allow for many images to be recognized as sensitive by an algorithm. For example, Google Vision's “Detect Explicit content - Safe search” [64] is being used to help us classify certain photographs in our corpus, most notably those featuring corpses, which have been detected correctly as “violent” but, on occasion, wrongly as “racy.” For example, out of a selected corpus of 200 images containing human corpses, only 5 were recognized as “likely” to be violent by Google Vision, and only 22 as “possibly” violent. We are now experimenting with the common errors made by the algorithm and how to keep a constant dialogue with the shortcomings of existing sensitive content detection tools applied to heritage images. We envision a potential pipeline to improve automated sensitivity classification by using the project's dataset. This pipeline would identify sensitive or contested contents across visual archives using pre-trained models that have been re-trained on high-quality annotations completed by experts from a wide consortium of GLAM sector institutions who manage potentially sensitive photographic materials. These experts could identify textual captions and visual contents that can be tested and improved by users through a crowdsourcing platform, such as Zooniverse. The Living with Machines project is using this platform to check their keyword search results, offering insight into the reliability of automated results [65].

3.2 Layout Analysis to Extract Metadata from Captions

Another part of the EyCon project involves studying and extracting images from photo albums and the layouts of newspapers to enrich our photo database. We have gathered numerous titles of French and English periodicals and magazines from the late 19th and early 20th centuries. This period is host to an outburst of photographic production and mass-produced visual news and constituted the first generation of photojournalists. Comparing this corpus with photographs of the period's conflicts makes it possible to identify the trajectories of several of the images that fall within the project's perimeters.

Walter Benjamin has claimed that the caption to a photograph is equally as important as the photograph itself. Benjamin questions, must we not “count as illiterate the photographer who cannot read his own pictures? Will not the caption become the most important component of the shot?” [66] For images in newspapers, a picture makes no real sense without a caption, whether it is used to illustrate a news item or as the basis of a media discourse. Images need context to provide a certain point of view on the events to be disseminated. The captions that accompany such images are therefore intrinsically valuable for describing and identifying them. In fact, these captions are a necessary basis for generating metadata for the image. To process images included in text-based records, we need to extract the images and their related captions, and these two different tasks can be processed by layout analysis technologies.

For example, some relevant datasets for the extraction of periodical layouts are directly implemented in Layout Parser [67], the tool that seems best suited to EyCon's challenges [68]. The datasets are composed of the image files of the periodicals’ pages, as well as files in XML-ALTO [69] format, indicating each layout element [70], including titles, images, text blocks, maps, graphics, and advertisements. The latter hold a problematic place in our workflow as they are intended to be neither directly extracted and retained as part of our final database nor the subject of semantic enrichment. When extracting the images from the newspapers without distinction, the associated captions will allow us to sort out those that are thematically relevant and that will be included in our final database.

Several projects concerning layout analysis have already been carried out [71], and there are numerous pipelines for document layout analysis [72]. The extraction of the photographs from the albums was done with Layout Parser without much of a challenge: the images are well recognized, extracted, and quickly integrated into the database. Some of the images are more difficult to crop, and these represent almost 20% of the images, but they are still usable for object detection and image similarity, although cropping sometimes results in a loss of information and the image is not as well detected, such as in Figure 2. On the other hand, more than 75% of the photographs in the albums are perfectly detected, even when the albums have complex layouts.

Fig. 1.

View Figure

Fig. 2.

View Figure

However, the layouts of periodicals and magazines, especially in the late 19th and early 20th centuries, can be very different from one another. One example is the Excelsior, whose images overlap in a hierarchical jumble (see Figure 3) [73]. Despite the various tools created for the analysis of historical document layouts, few have demonstrated accurate and rigorous predictions when it comes to extracting images and their captions from periodicals [74]. It should be noted that, even within a single issue, each page can follow very different editorial rules. This in turn contributes to the inappropriate development of heterogeneous training datasets that are not suitable for applying automated algorithms. We find here the same strategic problem as object detection [75]. To remedy this, we will continue to improve existing newspaper datasets with other periodicals and magazine titles from different regions and time periods to add variety to the types of document layouts [76].

Fig. 3.

View Figure

That said, providing a layout analysis with a dataset is still uncommon [77], i.e., XML-ALTO files attached to the images described. Thus, the illustrated periodicals available with XML-ALTO files produced by the Bibliothèque Nationale de France are particularly valuable to the project, as most of the captions are defined and associated with the image they describe [78]. A dataset of 30,000 images has been created thanks to these annotations, which are converted into the COCO format (JSON file). This format is adapted to the training of visual recognition models, as it contains the dimensions of an object as well as its placement, allowing for the automated creation of metadata. In linking captions directly to an image's metadata, we can avoid severing the image from its physical and editorial context [79]. Moreover, Layout Parser enables OCR for detected text blocks, including image captions. While OCR tools are now widely developed and can provide predictions with a high accuracy rate, it is particularly useful to associate the extracted text directly with its location, as defined by an identifier. Layout Parser detects the region of a text block (corresponding to a paragraph) and extracts the text at the same time. Each recognized element is distinguished by an identifier, and by automating the addition of the same identifier to an image and a caption, they can be associated as soon as they are segmented.

To generate the layout analysis for the dataset, considering the diversity of the newspaper and magazine layouts from the beginning of the 20th century, it was necessary to first make classifications for the pages. The EyCon team has used Pixplot [80] to help visualize the different layouts, including cover pages, blank pages or pages with one or several images, and advertisements. Pixplot represents the similarities between images in a whole corpus and vectors them by assigning a numerical value. It then shows the images according to their similarity and can be used to get a first look at an unknown corpus. The aim for EyCon was to provide a pre-classifier of layout pages and isolate particular ones. As the layouts are very different from one page to another, pre-classifying the pages by their layout and developing a model for each type of layout can be a way of improving accuracy. Even if the regions of the layout page are overlapping or messy, the accuracy rate of the images extracted is better thanks to a model trained on more precise features. Such results will enable the study of page layouts and their chronological evolution.

Having visual descriptions of images and their positions in relation to text is invaluable when considering the creation of new metadata from scratch. The evolution of new tools makes it possible to link, in the most precise and objective way possible, an image and its metadata with multimodal (text and vision) analysis tools [81]. Models such as CLIP [82] or Flamingo [83] are potential solutions to automatically create very detailed descriptions of images. CLIP is trained on pairs of images and text to associate the linguistic concepts and the visual semantics of a picture. It can be used to recreate missing captions for pictures published in newspapers, for example. This bypasses the pre-normalization of the vocabulary used and avoids the modern and potentially anachronical judgment of the researcher or curator. However, by preserving the original captions, CLIP simply repeats the contemporary description. When it comes to navigating potential biases, the same difficulties affect all approaches, from the most manual to the most automated.

3.3 Representing Similarity

The third computational entry point proposed by the EyCon project is to employ visual image searches that do not rely on existing metadata. This appears to somewhat mitigate the dangers of pre-determined vocabularies and semantic biases in the previous approaches. To study the circulation of conflict images in the contemporary press, we are using computer vision to identify the visual similarity between all image types. The pre-trained models do not use the associated metadata but work solely by digital visual matching. The features of each image in the corpora are extracted as vectors, which are used to measure the distance between them, thus obtaining a similarity score. The higher the ratio, the closer the two images are visually. It is thus possible to identify reuses with identical similarity, but also photographs that are similar and may have been taken at the same event or at the same time. It is important, however, to account for the quality of the images in the collected corpora when considering the similarity scores between two photographs [84]. Indeed, our photographs extracted from the press are of a far more degraded quality compared to the original prints preserved by institutions and digitized with high-quality cameras.

Different methods have been tested to match images by visual similarity. The poor quality of halftone reproductions of photographs is a hindrance for the AI, so we have enhanced the halftone images with the Jarvis algorithm to preserve the details of the original image [85]. The imitation of the photographic grain of an analog camera was simulated using a homogeneous Boolean model filter. To increase the robustness of the model, the data can be augmented: this was done in the first similarity tests on a single corpus of thousands of photographs from the First World War [86]. The zoom technique, for example, can be used to simulate variations in the distance between the object and the camera, or the noise technique to simulate defective pixels. A CNN architecture is used (the same as ImageNet) with an unsupervised K-means clustering algorithm that matches the nearest neighbors. It is possible to reach an optimal number of clusters after several training sessions. Thus, thanks to a visualization (using the t-SNE algorithm), it is possible to identify interesting clusters: a sensitive image cluster is the “soldiers with facial disfigurement” cluster, for example, which can help to identify scattered photographs that were taken at the same time or aggregate images of the same location, taken at different times, from slightly different angles and mounted in different albums.

The mixture of photographs and prints within the EyCon project is imperative. A photograph of the Battle of the Somme during World War I kept at the IWM [87], which was re-used by La Guerre illustrée [88] (Figure 4), offers a further example of the disparity between visual records held in different institutions. The French magazine captioned the photograph “The British advance on the Somme: the ‘London Scottish’ going to the trenches” [89], while it is described in the IWM as “A Company, 1/14th Battalion, London Regiment (London Scottish) marching to the trenches on Doullens-Amiens road at Pas-en-Artois, 26th June 1916.” To incorporate both examples, the primary metadata needs to be completed. The image metadata from La Guerre illustrée, for example, can be enriched with an exact date and place, as well as the name of the photographer, as provided in the IWM record. Then, a layer of secondary metadata can be added to both records, providing information regarding the duplication of the image.

Fig. 4.

View Figure

As both descriptions need to be retained in the EyCon project's metadata, whether they were created for publication or for conservation purposes, it is necessary to find a conservation format that can differentiate between the layers of information and their origins without removing the additional contexts. Like other types of semi-automated metadata enrichment, the results must be saved in a structured format that the XML requires for their durability and exploitation. The extracted information can then be cross-referenced [90], allowing places, events, or characters to be identified. The extraction of named entities—ideally with a model trained on its own data—allows this information to be aligned with external databases through controlled vocabulary. This process ensures the veracity of the information cited and its cross-referencing under a single identifier. Continuing with the previous example, the IWM has listed the photographer as “Brooks, Ernest (Lieutenant).” By aligning this data, all photos attributed to Brooks can be linked together. This linked construction, through text and image, will allow for the curation of relevant information within the interface, to automate historical photograph analysis and enable images to be collated within subject headings.

This brings us to the heart of the issues concerning the reliability of metadata for image records. Most important is the form of the information: first, the format in which the metadata is stored, and second, the ways in which these new computationally developed results are displayed. For similar images discovered between different archives, it is essential to include this information on each image's record so that users can, regardless of potentially different descriptions, consider these two images together. In this approach, EyCon is building on existing visualization projects that emphasize visual similarity, such as Snoop [91]. Snoop allows searching by visual similarity only, without any need for semantic description, and puts the image at the center. Additionally, the computer vision tools developed within the ModOAP  [92] project allow the calculation of a similarity score of over 80%. For projects such as Impresso [93], NewsEye [94], Newspaper Navigator [95], or Chronicling America [96], the information extracted from an image makes quantitative analyses possible. Being able to approach a photographic database via its spatial characteristics, for example, is relevant to users studying potentially contested pasts in their direct geographical context [97].

The improvement of automated metadata creation for image archives will in turn improve the accessibility of visual materials, particularly for humanities researchers. As Dahlgren suggests, humanities scholars comprise a core user group for image collections in cultural heritage organizations, and yet often they feel their needs are not being addressed by existent description practices within image collecting institutions [98]. Indeed, the demand for historical approaches to be applied to the processing of visual materials–that is, including the necessary information to make photos “readable” as historical artefacts–has been ongoing, but often without addressing the practicalities of such labor-intensive endeavors [99]. For EyCon, we will continue to consider what details are needed to ensure a photograph is found and to avoid it being misinterpreted, or taken out of context, as the way an image is labeled remains an essential part of its discoverability.