Incidental vocabulary learning with subtitles in a new language: Orthographic markedness and number of exposures

Mercedes Pérez-Serrano; Marta Nogueroles-López; Jon Andoni Duñabeitia

doi:10.1371/journal.pone.0246933

Abstract

The present study is set to explore the way the orthographic distributional properties of novel written words and the number of exposures to these words affect their incidental learning in terms of recall and recognition. To that end, two experiments were conducted using videos with captions. These videos included written nonwords (orthographically marked language-specific items) and pseudowords (orthographically unmarked items) as captions paired to the spoken targets, presented either in isolation (Experiment 1) or within sentences (Experiment 2). Our results consistently show that items containing legal letter combinations (i.e., pseudowords) are better recalled and recognized than those with illegal combinations (i.e., nonwords). Further analysis in the recall task indicate that frequency modulates the learning of pseudowords and nonwords in a different way. The learning of pseudowords increases linearly with repetitions, while nonwords are equally learned across frequencies. These differential effects found in the recall task do not show up in the recognition task. Although participants took more time to recognize nonwords in the recognition task, increased exposure to the items similarly modulated reading times and accuracy for nonwords and pseudowords. Additionally, higher accuracy rates were found in Experiment 2, which underscores the beneficial effect of supportive visual information.

Citation: Pérez-Serrano M, Nogueroles-López M, Duñabeitia JA (2021) Incidental vocabulary learning with subtitles in a new language: Orthographic markedness and number of exposures. PLoS ONE 16(2): e0246933. https://doi.org/10.1371/journal.pone.0246933

Editor: José A. Hinojosa, Universidad Complutense Madrid, SPAIN

Received: May 20, 2020; Accepted: January 28, 2021; Published: February 16, 2021

Copyright: © 2021 Pérez-Serrano et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript.

Funding: This study was partially supported by grants RED2018-102615-T and PGC2018-097145-B-I00 from the Spanish Government, and H2019/HUM-5705 from the Comunidad de Madrid.

Competing interests: The authors have declared that no competing interests exist.

Introduction

The acquisition of new vocabulary is a foundational part of language learning, and the only requirement to learn vocabulary is input [1]. There is general agreement that a significant percentage of vocabulary is acquired incidentally, without the objective of learning, that is through mere exposure to the language. In fact, in one’s native language, incidental vocabulary acquisition is regarded as the primary mode of vocabulary acquisition [see 2, 3].

Some factors have been found to have an impact on the incidental acquisition of vocabulary from a foreign language, such as the type of word [4, 5], the morphological predictability of the words [5], the association of the words with pictorial material depicting the concepts [6], and the number of exposures to those new words [7]. In the current study, we explored how some properties of the nonnative words that have been found to change the manner in which they are processed and learned modulate foreign word incidental vocabulary learning. To do so, we focused on orthographic markedness, which refers to effects found in the context of the presence of particular letters or letter combinations in a particular language that are not plausible in another language known to the learner (e.g., the combination of the graphemes C and K does not exist in Spanish, while it does in English). By comparing the incidental learning of marked and unmarked words and how repetition or frequency effects modulate it (i.e., the number of times a given item has been presented during an exposure phase), we aimed at gaining knowledge on how these two factors interact during incidental foreign word learning and their impact in future recall and recognition of the words.

In general, vocabulary learning requires input to happen, be it written, oral or bimodal [1]. Hence, incidental vocabulary also requires a form of exposure to unimodal or multimodal input that will ultimately allow learning a consequence of a gradual process that involves the accumulation of knowledge through the repeated encounters with the words. After the first few encounters, speakers typically learn the word form, but it is not until a higher number of encounters have taken place that the connection between form and meaning stabilizes, since that requires a greater number of repetitions [8, 9].

A great deal of previous studies on incidental vocabulary learning has focused on frequency of appearance in written texts. Preceding research has shown that incidental vocabulary learning through reading occurs for both native language learners [e.g. 2, 10] and second language learners [e.g. 11, 12]. It is now well established that readers need many encounters with novel single words that are set in meaningful contexts in order to observe vocabulary knowledge growth as a function of reading [9, 13–15]. However, it should be considered that word knowledge is a multidimensional construct [16], and that the amount of exposure to or encounters with a new item needed to develop a certain type of word knowledge will not be uniform. For example, according to Webb [9], more than ten repetitions may be needed to develop full knowledge of a word, whereas as little as one encounter might suffice to observe gains in receptive lexical knowledge [9, 15].

Recently, Hulme, Barsky and Rodd [17] investigated whether adult readers could learn new meanings incidentally attributed to known words while reading stories and explored the role of the frequency of exposure (namely, the number of encounters; two, four, six, or eight times) to the novel word meaning. Results showed that incidental learning while reading developed linearly with an increase in the number of exposures, following a cumulative incremental trajectory. These findings demonstrate the remarkable success with which adults learn new meanings for known words incidentally while completing a common everyday action like reading.

Likewise, auditory processing has also proved to be a good source of incidental vocabulary growth [5, 18–20]. As in the case of the reading studies, research on auditory processing suggests that frequency is a strong predictor of vocabulary learning while listening. Learners can gain knowledge of novel words related to their meaning, grammatical functions and collocations while listening to sentences or speech [21]. Nonetheless, some differences have been also shown between auditory processing and reading in incidental learning. Overall, when compared to learning while reading, learning while listening is less effective and far more encounters are needed. Brown, Waring and Donkaewbua [18] concluded that it is very unlikely for a word to be learnt just by listening to it less than 20 times [see also 19]. Feng and Webb [22] pointed out that this disadvantage of aural input against written input in incidental vocabulary learning could respond to the faster processing demands required by listening, making it more challenging for learners to notice unknown words in this modality [see also 23].

With this in mind, audiovisual integration using bimodal input during incidental word learning could serve to boost the benefits of the two modalities. Studies focusing on incidental learning while participants follow the written words as they listen to the oral text suggest that this is indeed the case [see 7, 13, 18, 24–26]. Bisson, Van Heuven, Conklin, and Tunney [7] investigated the effect of repeated exposures to multimodal stimuli on the incidental acquisition of new vocabulary from the foreign language by manipulating the number of appearances of the new words (2, 4, 6, and 8 exposures). Following an incidental learning phase, participants completed an explicit learning task in which they learned the mappings between translation equivalents in their native and foreign language. Critically, some of the items had been previously presented in the incidental learning phase, while others were totally new. The results showed that participants performed better on the words they had previously been exposed to, and that this incidental learning effect took place even for items that were only presented twice.

The importance of exposure to bimodal input for foreign language incidental learning should be further considered, given that it opens doors to ecologically valid learning settings that could boost vocabulary acquisition. Watching television or videos is beneficial for foreign language learners’ vocabulary growth [e.g. 27, 28], and research has shown that vocabulary learning in these contexts is also governed by frequency effects [29]. On-screen text aids such as native or foreign language subtitles or captions could be effective ways to turn video materials into effective didactic tools for foreign language incidental word learning, given the evidence showing their effectiveness in vocabulary learning [27, 29, 30]. Hence, it is worth exploring the mediating role of the amount of exposures (i.e., the frequency effect) to a new item in audiovisual settings with bimodal presentation to understand how incidental vocabulary learning occurs in naturalistic learning contexts. The current study was set to provide evidence in this regard.

Over and above external manipulations involving the amount of exposure to a series of items, additional factors that are intrinsic to these items are also expected to modulate incidental learning effects. One of these factors that has been extensively studied in recent years in the domain of multilingual language processing is the orthotactic pattern of a written item. The orthotactics of a word corresponds to the distributional properties of the orthographic representations of that word (e.g., letters or bigrams) within the language it belongs to, and also across the languages known to a bilingual person. The orthotactic properties of a word may hint at the language to which it belongs. For instance, in the case of an English-Spanish bilingual, words with a WH-onset would point to English lexemes.

Preceding studies have demonstrated that bilinguals benefit from the presence of orthographic cues (namely, orthographic markedness) to detect the language of a given written word [31–35]. When bilingual readers are required to identify as quickly as possible the language of a series of stimuli they are presented with, response latencies are faster and accuracy rates are higher for orthographically marked than unmarked words [31, 34]. These findings, together with recent evidence from neuroimaging data [see 35] demonstrates that language-specific orthography guides single word identification, and thus suggests that access to the lexicon might be guided by the extraction of language-specific orthotactic combinatorial rules [32].

Focusing specifically on the distinction between incidental and intentional vocabulary learning, Bordag, Kirschenbaum, Rogahn and Tschirner [36] examined the role of orthotactics (understood as the relative frequencies of the letters and their sequences within words) in the early stages of vocabulary acquisition by adult native German speakers and advanced learners of German. They explored in which way low versus high orthotactic probabilities affected the acquisition of the meaning of new words in a native and foreign language and whether the effect differed between intentional and incidental acquisition modes. On the one hand, the results showed that low orthotactic probabilities (namely, high orthographic markedness levels) contributed positively to incidental acquisition of novel word meanings in the first language, but no such effect was found in the foreign language. That is, the meaning of words from the native language was incidentally acquired better when these words were orthographically marked. On the other hand, high orthographic probabilities (namely, low orthographic markedness levels) positively affected intentional learning in a foreign language, but not in the native language.

The current study was set in order to shed further light on the way the orthographic distributional properties of novel written words impact their incidental learning in a setting that could resemble to those used in foreign language learning contexts. Besides, the manner in which orthographic markedness and number of encounters or frequency interact during incidental word learning and how they modulate the ability to recognize and recall these words was also explored. To this end, and following the line of research using subtitles and captions to study incidental language learning, two experiments were conducted using videos with captions. These videos included written nonwords (orthographically marked language-specific items) and pseudowords (orthographically unmarked items) as captions paired to the spoken targets, presented either in isolation (Experiment 1) or within sentences (Experiment 2). Using nonwords and pseudowords enabled us to control for the prior lexical knowledge of the participants. We were especially interested in exploring to what extent novel words that are orthotactically illegal in the native language would benefit from a higher number of encounters during incidental learning settings.

Experiment 1: Captions of isolated novel words

In Experiment 1, participants were visually presented with two videos containing images that were unrelated to the synchronously presented audio tracks containing information about some animals and some fruits. Each of the target animal and fruit name presented auditorily was paired with a written element that was displayed on the screen, consisting either of a pseudoword or a nonword. Critically, some of the elements were presented more often than others (namely, a frequency manipulation). After the passive perception of the videos and soundtracks, participants completed a recall and a recognition test.

Materials and methods

Participants

Forty Spanish students (33 women; mean age = 19.6 years) majoring in Modern Languages and Psychology at Nebrija University took part in the experiment. All the students had Spanish as their native language and participated voluntarily in exchange for monetary compensation. Prior to the experimental session, all participants gave their informed consent in accordance with guidelines approved by the Ethics and Research Committees of the Nebrija University.

Design and stimuli

We used a repeated‐measures design for this experiment with Type of Item (nonword | pseudoword) and Number of Exposures (1 | 4 | 8) as within‐subject factors. The stimuli used were two videos created by the researchers with successive images of landscapes that did not include any piece of information that could be related in any way to the content of the soundtrack. The soundtracks of each of the two videos consisted of 28 sentences in Spanish containing a total of 12 targets. One of the soundtracks presented information about 6 animals (cat, mouse, rabbit, bird, fish and turtle), and the other soundtrack referred to 6 fruits (mango, pineapple, melon, pear, grape, strawberry). By creating the material ex profeso, we ensured having full control over the factors that could influence the results, such as the precise number of exposures to the targets and their characteristics.

Each target word was paired with a corresponding nonword (orthotactically marked) or pseudoword (orthotactically unmarked) as shown in Table 1.

Download:

Table 1. Target words paired with a nonword (NW) or pseudoword (PW) item.

https://doi.org/10.1371/journal.pone.0246933.t001

Each soundtrack consisted of 28 sentences. 26 sentences contained a single mention to one of the Spanish targets, and one opening and one closing sentences were added too. The critical 26 sentences comprised objective information on the fruits and animals. The two texts that were used to create the soundtracks were highly similar in length (fruits: 488 words; animals: 479 words) and had a very similar structure. The texts were transformed into speech using an online software and the resulting audio clips were included in the videos using video editing software. Three different versions of each final video clip were created in order to ensure that the place of appearance of the targets did not affect the results. To do so, the order of appearance of the sentences in the audio files was randomized. The resulting 6 video clips ranged from 3:33 to 3:41 minutes in length. Each participant was randomly assigned one version of each video. The pseudowords and nonwords appeared on the lower central part of the screen during the videos when the corresponding target was produced in the audios. They were written in uppercase and displayed for 2 seconds. As shown in Table 1, the items were split into three subsets such that words were presented either once, four times, or eight times in each video clip. For every one of the levels of the Number of Exposure factor, each video clip included one specific pairing between a Spanish word and a nonword (orthotactically marked), and another pairing of a different Spanish word and a pseudoword (orthotactically unmarked). The pseudowords were created in accordance with the orthotactic rules of Spanish, and contained legal bigrams in this language (e.g., mastu). The nonwords did not follow the Spanish orthotactic rules and each contained an illegal bigram in Spanish (e.g., the bigram ZF in rezfa). All the nonwords and pseudowords were pronounceable disyllabic 5-letter strings with the same orthographic structure (namely, CVCCV).

Procedure

The experiment was created using Gorilla [37]. All participants were tested in individual PCs with headsets. After signing the consent forms, the participants completed a short sociodemographic questionnaire, and then they watched the two corresponding video clips consecutively. The participants were naive to the phases of the experiment and the experimenters did not mention the subtitles in the videos or the tasks that would come after. The only instruction that the experimenters gave to the participants was to pay attention to what was played on the screen. This provided us with an incidental learning situation in which participants could associate the aural input (namely, the Spanish words for the fruits and animals) with the on-screen text (namely, the pseudowords and the nonwords). This allowed us to operationalize the construct of awareness non-concurrently at the reconstruction stage [38].

Once the two video clips had ended, participants completed a simple working memory test (a timed N-back task in which they had to indicate via button presses whether or not a given letter had already been presented two trials before). This test was aimed at diverting the attention from the content of the video clips and was used as a filler task. At the end of the N-back task, participants were asked to complete the two critical tests that were devised to measure recall and recognition of the correspondences between the displayed new items (pseudowords and nonwords) and the played Spanish words (animal and fruit names). First, the recall test was completed. In this task, participants were presented with the picture and the Spanish name of each of the fruits and animals used in the videos, and they were required to write the name they think they saw during the video accompanying each of them. Each trial started with the centered presentation of a fixation cross for 1000ms, immediately followed by the concurrent presentation of a picture and the lowercase Spanish name of one of the items that were mentioned in the video clips. Participants had to type the string they remembered, and they were explicitly told that they could not advance without typing anything. Once they had pressed the enter button, they had to complete a trial from a recognition task, in which another fixation cross was displayed for 500ms, followed by the same picture and Spanish word in the same location, this time accompanied by two alternatives to select from. The two displayed alternatives were lowercase strings corresponding to the correct string (namely, the one matching the target) and a competitor taken from the same video that had been presented in the exposure phase the same number of times. For instance, when the picture of a mouse was displayed together with the Spanish word ratón, the correct target rezfa was displayed with the mismatching alternative mastu, since the latter was presented the same number of times in the video (8 times, in this example). Participants completed the 2-alternative forced choice (2AFC) task without time pressure, and accuracy was prioritized and no time limit was set for the responses. The whole experiment lasted for about 20 minutes.

Results and discussion

An analysis of the data from the recall and the recognition tests was carried out for the group of 40 participants with Type of Item (nonword | pseudoword) and Number of Exposures (1 | 4 | 8) as within‐subject factors in a series of ANOVAs. Descriptive statistics split per condition are reported in Table 2.

Download:

Table 2. Descriptive statistics for each of the dependent variables in each of the conditions tested in the recall and recognition tests.

https://doi.org/10.1371/journal.pone.0246933.t002

Recall test

In a first analysis, we explored the success or failure in the recall task by considering exclusively the coincidence between the typed strings and the expected ones. As shown in Table 2, the exact recall was relatively low in general, with a mean error rate of 87.9% (SD = 16.1, range: 41.7–100). The ANOVA showed a significant main effect of Type of Item (F(1,39) = 8.86, p = .005), demonstrating that pseudowords were recalled with higher accuracy levels than nonwords. The main effect of the Number of Exposures was also significant (F(1,39) = 13.28, p < .001), showing that the recall increased as a function of repetitions. More importantly, the interaction between the two factors was significant (F(1,78) = 9.04, p < .001). Follow-up post hoc tests (Holm-corrected pairwise comparisons) showed that the recall for the pseudowords increased with repetitions (1 vs. 4: t(156) = 2.86, p = .048; 4 vs. 8: t(156) = 3.72, p = .003). However, this was not the case for nonword items, since no differences were found as a function of increased exposure (all ts<1.2 and ps>.99; see Fig 1).

Download:

Fig 1. Results in the recall tasks of Experiment 1 (upper panels) and Experiment 2 (lower panels): Percentage of errors (left panels) and number of edits (right panels).

Error bars correspond to the 95% confidence intervals.

https://doi.org/10.1371/journal.pone.0246933.g001

Admittedly, precise recall in an incidental vocabulary learning task such as the present one may not be expected. With this in mind, we carried out an additional analysis on the recall data based on the Levenshtein distance (LD) between the participants’ responses and the actual targets. LD is an index of the number of edits (replacements, insertions or additions) needed to get from one text string to another, and thus allows for an estimation of the degree of similarity between the typed response and the expected one, with larger values representing greater deviance. As shown in Table 2, the mean deviance was of 3.75 edits (SD = 1.03). An ANOVA with the same factors and levels was carried out and results revealed a significant Number of Exposures effect (F(2,78) = 29.59, p < .001) and no effect of Type of Item (F<1 and p>.35). Critically, the interaction between the two factors was significant (F(2,78) = 4.88, p = .010). Holm-corrected pairwise comparisons showed that the number of edits needed to convert the typed responses into the expected ones decreased as a function of exposure, but that this decrease was significantly more marked for pseudowords than for nonwords (pseudowords: 1vs. 4: t(151) = 3.83, p = .002; 4 vs. 8: t(151) = 3.98, p = .001; nonwords: 1 vs. 4: t(151) = 2.92, p = .028; 4 vs. 8: t(151) = 1.06, p>.99; see Table 2 and Fig 1).

Recognition test

The results of the accuracy rates in the recognition test were analyzed following the same design. Results showed that participants recognized the correct responses with a good level of accuracy, since the mean percentage of errors was fairly low (Mean = 30.8%, SD = 19.7; see Table 2). The ANOVA showed a significant main effect of Number of Exposures, so that recognition accuracy increased as a function of repetitions (F(2,78) = 8.27, p < .001). The main effect of Type of Item and the interaction between the two factors were not significant (Fs<2 and ps>.16).

A parallel analysis was carried out on the latency data to shed further light on the effects, even if participants were not told to provide speeded responses. Reaction times associated with erroneous responses were not included in the analysis, and latencies above 5000ms or beyond 300ms were also discarded (representing only 4.82% of the data). The ANOVA showed a main effect of Type of Item (F(1,20) = 8.17, p = .010), demonstrating that pseudoword items were recognized faster than nonword items (see Table 2 for descriptives). The effect of Number of Exposures was also significant (F(2,40) = 4.13, p = .023), showing that reaction times varied as an inverse function of exposure. As in the case of the accuracy data, the interaction between the two factors was not significant (F<1 and p>.40).

In terms of recall, our results confirm that orthotactically unmarked novel words (namely, the pseudowords) are better retrieved and benefit more from an increased exposure than orthotactically marked words (namely, the nonwords). Across measures, our results demonstrate a main frequency effect showing an increase in recall performance as a function of repetitions (i.e., strings encountered more times showed better recall patterns). More importantly, this frequency effect was significantly more marked for pseudowords than for nonwords (with the latter type of strings showing negligible frequency effects in direct recall). The analysis of the recognition task showed a different pattern, suggesting that accuracy improved with repetitions, and that reaction times also decreased as a function of repetitions. A general recognition latency benefit was found for legal novel words over illegal strings, but no clear signs of an interaction was found either in the accuracy or in the response times.

One issue that should be kept in mind is that the videos that participants viewed did not contain any visual information directly related with the auditorily presented information. This was done in order to artificially create an incidental learning scenario, but this admittedly created an unnatural set-up that notably deviates from real life settings (e.g., watching a movie with subtitles). Besides, the presentation of the nonwords and pseudowords in isolation could have made their processing more salient, given that these were the only written elements displayed. And if this were the case, the fully incidental nature of the learning process could be questioned. For these reasons, and in an attempt to replicate the findings and extend them to more realistic scenarios, a new experiment was conducted.

Experiment 2: Captions with novel words embedded

In Experiment 2, participants were presented with two videos including the same verbal content as in Experiment 1, but this time accompanied by supporting related visual information depicting the animals and fruits mentioned. Besides, the captions of individual novel words used in Experiment 1 were replaced by captions of the whole sentences in which the critical words had been changed for the novel words (either nonwords or pseudowords). By making these changes, we intended to create a more realistic set-up and an incidental learning scenario.