betaFeedback

WorksComputer Science

Flash Word : ESL Listening Support with Subtitle Skimming

Abstract

Listening to English audio with English subtitles is thought to be effective for ESL students to improve listening comprehension skills. However, for those who learn English as a second language it can be hard to finish English subtitles. Also they are unlikely to focus on listening while being busy reading subtitles. We propose Flash Word, a subtitling system which generates and presents shortened intralingual subtitles for English audio by detecting keywords from the script. Flash Word extracts keywords from scripts using an existing algorithm, and shows them one at a time like a flashcard. The system calculates when the keywords are being pronounced and aligns them correctly. Flash Word aims to enable ESL students to focus better on listening by minimising the cost of reading subtitles.

Introduction

The English language is spoken globally today and there is a growing need for effective ways to study English as a second language. In learning English as a second language (ESL), listening comprehension skill is crucial among others. In fact, out of the time spent in English communication, it is said that more than half of the total time is spent on listening [1]. One of the methods that are proved to improve listening comprehension is to listen to English audio with English subtitles [2, 6].

However, the speed of reading subtitles differs from individual to individual, and ESL students are assumed to be slower in reading English than native speakers. Though professional subtitle writers often omit unnecessary words from original scripts to shorten subtitles [4], ESL students are still likely not to be able to finish the English subtitle of the audio they listen to. Thus, there is a possibility where a student pay too much attentions to subtitles, neglecting listening concentration.

Based on this observation, we would like to propose a hypothesis that presenting only important words should be more helpful for ESL learners to better understand English audio than showing the whole script. We also consider that this approach is valuable in terms of human augmentation. Instead of spoiling the listening skills by full subtitles, this approach rather supplements or augments the user’s listening ability with carefully selected keywords. In order to test and support this hypothesis, we have designed a system called Flash Word, which generates a shortened subtitle by detecting keywords in the English audio.

In this paper, we describe the idea of subtitle skimming as listening support, proposing two possibilities of the system design, and explain the system architecture of the prototype of Flash Word. Then we describe the pilot study using the present prototype, analyze its results and finally we discuss its future work.

Listening Support with Skimmed Subtitling

Figure 1 shows the comparison of typical English listening training titles with the full subtitle (above) and the skimmed one (below). As shown in this figure, the full subtitle version might pose too much attention to the listeners for reading the whole sentence, while the skimmed version the listener can concentrate on audio and glancing the keywords. These skimmed words synchronously appear on the screen with the audio, so that the listener can promptly know the important word even when the listener missed it.

A comparison of a subtitle with and without skimming

Figure 1. A comparison of a subtitle with and without skimming (video source[3]). (above) Normal subtitling. (below) Our proposed method that only displays selected words from the script, synchronized with the audio, letting ESL students concentrate on listening, rather than reading subtitles.

Flash Word can be designed either for pre-subtitling with scripts or for real-time subtitling without scripts.

Pre-subtitling with Scripts

When an English audio has its script, Flash Word prepares a shortened subtitle beforehand. In order to shorten the subtitle, the present system plans to utilize three methods of detecting important words in the context; it quantifies each word’s importance (1) by tf-idf (term frequency - inverse document frequency) formula [8], (2) by Rapid Automatic Keyword Extraction (RAKE) algorithm [12], which detects key phrases based on registered stop-words, and (3) by using TextRank[10], an algorithm which quantifies each word’s significance based on the context.

To correctly synchronize speech with the subtitle, the system also provides an alignment mechanism for detecting word pronunciation timing based on the given audio and its corresponding script. To do this, we use a technique developed for our previous system [13]. The system first applies a speech recognition over the audio file using Julius [9]. As the Julius system provides a time point of each recognized word, this information can be used for the alignment. Then, the compares the script with recognized text and correct miss-recognitions. Finally, we get the correct time point of each word in the script.

System Architecture

We have developed a pre-subtitling prototype of Flash Word. In this section, we explain how the prototype is implemented. Figure 3 shows the whole system architecture. The pre-subtitling prototype of Flash Word takes a movie and its transcript as inputs and returns a Flash Word subtitle file for the movie. The prototype is designed for use in experiments, so that it can work best for relatively short movies such as news stories, not for long ones such as films.

Flash Word system architecture showing movie and transcript input, speech recognition, word alignment, keyword detection, FlashWord SRT file generation, and movie output with FlashWord subtitles

Figure 3. Flash Word takes a movie which includes speech in English and its transcript. It aligns the words in the script to the movie by speech recognition. Then the system combines the alignment and the detected keywords and generates a subtitle file for the movie.

Speech recognition and Word alignment

As described in the previous section, in order to adjust the timing of flashing subtitles to audio, the present system utilizes Julius[9], which is an open-source large vocabulary continuous speech recognition engine. Julius needs an acoustic model which links acoustic features and linguistic features, and a language model which decides the probability that a certain word string appears in the language in order to recognize speech correctly[13]. In our prototype, we employ a free English acoustic model provided online for an acoustic model. For a language model, the prototype system generates an English language model based on the transcript of the audio. That is, the system takes the transcript sentences as a grammar pattern which can be uttered. Flash Word needs to detect when the keywords are pronounced in the given audio. So as to realize this, the prototype system operates Forced Alignment in Julius, which calculates when in the audio each word from the transcript is being spoken. By Forced Alignment, the system generates a list of all the words from the transcript and their timings of pronunciation.

Keyword detection

As mentioned above, there are several possibilities of how to detect keywords in the English speech. We tried extracting keywords using Tf-idf, RAKE and TextRank. Out of those three methods, RAKE algorithm turned out the best for short scripts. Both Tf-idf and TextRank quantify word importance based heavily on its occurrence frequency. Since not many words appear multiple times in short scripts, neither of them extracted important keywords. Tf-idf with the short scripts resulted in most of the words in the scripts having the same score, and TextRank was unable to generate a sufficient graph. For example, the sentence shown in Figure 2, which is "The royal couple feed the animals, some of which are orphaned by poachers" results in no keywords by Tf-idf calculation, and in only "royal couple" by TextRank.

An example of subtitle skimming with RAKE algorithm

Figure 2. An example of subtitle skimming with RAKE algorithm

Thus, the prototype utilizes only RAKE algorithm to detect keywords from a script. RAKE algorithm recognizes phrases by separating a script with pre-registered stop-words, such as prepositions, articles and basic adjectives[12]. And it scores phrases based on the frequency of words in a phrase and length of a phrase[12].

Subtitle generation

After aligning the script with the audio and detecting the keywords, by combining the results the system generates a subtitle file in an SRT format, which can be integrated into videos as subtitles easily on different platforms. The system runs a ruby script which iterates through the list of words and pronunciation timings and generates a new list that only contains keywords and their pronunciation timings. In this process the timings of the keywords pronunciation are adjusted so that a keyword is being shown as a subtitle until the next keyword appears. The system converts the generated list into an SRT format, and integrates it into the input video.

Pilot Study

We conducted a pilot study in order to test if Flash Word actually enables ESL students to focus on audio better, and also to collect feedback from participants for the purpose of future implementation.

Experimental design

We recruited 12 participants, aged 22-25 years old (average: 23.6 years, SD: 0.96 years), nine male and four female, from the local university. The participants had no prior association with our laboratory or with our project. Students’ majors included physics, architecture and art. No participants were from English speaking countries and all the participants identified themselves as ESL students.

The pilot study consisted of four parts: (1) pre-test questionnaire, (2) introduction of the system, (3) three listening comprehension tests, and (4) post-test questionnaire and interview.

Before conducting listening comprehension tests, we asked the participants to answer a demographic questionnaire. The questions also included TOEIC, TOEFL scores so that we could know the English skills.

For listening comprehension tests, we prepared three different listening audio from CNN Student News[5]. Each audio is cut down to 19 seconds and contains (A)57, (B)54 and (C)52 words. Readability of each transcript measured on online readability calculator[11] is (A)6.8, (B)7.0 and (C)7.0. None of the audio contents were related to each other. From audio (A), (B) and (C), we prepared videos which have white blank background with each audio. For each video, we prepared three subtitle option: no subtitles, normal subtitles and Flash Word subtitles.

We divided 12 participants into three groups: Group 1 watched (A) with no subtitles, (B) with normal subtitles, and (C) with Flash Word subtitles. Group 2 watched (A) with normal subtitles, (B) with Flash Word subtitles, and (C) with no subtitles. Group 3 watched (A) with Flash Word subtitles, (B) with no subtitles, and (C) with normal subtitles. We changed the order in which each group sees three types of subtitles so that the order or the difference in difficulties of questions would not matter.

We asked the participants to watch videos three times each, and then answer three questions we prepared beforehand for each video. So, each participant watched video(A), (B) and (C) three times, and answered three questions each for (A), (B) and (C) after watching each of them.

After the tests, we asked the participants to answer post-test questionnaire. We also interviewed them and recorded their feedback.

Subtitles for test videos

We prepared three subtitle options for each (A), (B) and (C) video. Normal subtitles show the exact transcript, around 15 to 20 words at a time with the timings of pronunciation. Flash Word subtitles show only the keywords extracted with RAKE algorithm. The numbers of the keywords shown for each video are (A)18, (B)23, and (C) 23.

Listening comprehension tests

One of the purposes of this pilot study was to test if Flash Word enables participants to focus on audio than normal subtitles. In order to examine it, we prepared three questions for each video that asks about audio information. Table 1 shows an example of the questions we actually asked. The questions were design to focus on how words are pronounced in the audio, so that a participant would not be able to answer them correctly if he or she reads subtitles and remember them without listening to the audio. All the questions are multiple choice problems.

Normal Subtitle0600 hours, NASA prepares to fly.
PronunciationO[ou] six hundred hours, NASA prepares to fly.
QuestionHow was the word "0600" pronounced in the video? (1) O[ou] six double O[ou] (2) Zero six hundred (3) O[ou] six hundred (4) Zero six double zero

Table 1. Example of listening comprehension questions

Results of tests

The result of the listening comprehension tests is shown in Table 2. The result shows that in this pilot study, the participants performed best with no subtitles, followed by normal subtitles, and worst with Flash Word subtitles.

No subtitlesNormal subtitlesFlash Word Subtitles
1.421.251.17

Table 2. Result of listening comprehension test: numbers in the table show mean scores out of three points

Feedback from questionnaire and interview

We asked questions about Flash Word via questionnaire after the listening tests. The results are shown in Table 3, which shows the mean, SD, and percentage of positive responses (>4 on a 7-point Likert scale) to each question.

QuestionMeanSD%
Was it easy to listen to the audio with Flash Word?4.460.969/13
Do you want to recommend Flash Word to other ESL students?5.001.227/13

Table 3. Result of Questionnaire

Some of the comments from the post-test interview are show in Table 4. The table lists the participants’ TOEIC English score next to their feedback.

Participant’s TOEIC scoreFeedback
400"To be honest I’m not used to English listening so I just read subtitles when they were there.
550"I understood the audio with Flash Word subtitles, but couldn’t really focus on listening."
760"It was easy to guess what was being said in the audio with Flash Word."
900"I would want this Flash Word for studying English listening. I think ESL students who are already good at English would benefit the most from this."

Table 4. Feedback from Interview

Analysis

As a result of the pilot study, we were not able to show that the participants focus better on audio with Flash Word than with normal subtitles. However, it is assumed from the interview that ESL students with relatively higher scores appreciate Flash Word better. In this pilot study we do not have all the participants’ TOEIC scores so it is hard to classify the test performance based on the English ability, but in future experiments we plan to expand the experiments to more participants so that we could have more data to show whether Flash Word system works good for those who are at specific English levels.

Discussion and future work

We have developed a pre-subtitling prototype of Flash Word. One possible way of enhancing the proposed system is to determine keywords to be displayed in accordance with the student’s vocabulary level. By measuring it, the system would be able to show difficult or unknown words tailored for the particular student.

Other than ESL students support, we also consider that this system would be beneficial to senior people who tend to miss the spoken words even in their own mother tongue. Instead of showing full subtitles, our skimmed version can be used as an alternative text captioning support for TV broadcasting.

Real-time Subtitling without scripts

We also consider a real-time version, where no script is available as the source of subtitle. In this case, the system solely relies on the speech recognition and keyword skimming is also incrementally applied. We are planning to use the Google Cloud Speech API [7] that provides a function where the recognized word sequence can be available before the pronounced sentence ends. Such a real-time system should be useful as a wearable device that continuously supports the wearer’s listening effort in daily life.

With such a real-time version, there might be a slight delay between text captioning and the word utterance, even with the real-time speech recognition. In this case, a listener might miss an important word and the subtitle of the corresponding word appears after some delay. Thus the listener might have a cognitive load for integrating these two events. We are planning to investigate this issue more closely to reveal the effect caused by the delay.

Conclusion

We introduced Flash Word, which is aim to improve ESL students’ listening skills. With the prototype system we implemented we were unable to show that the system helped the participants to listen to English audio better than normal subtitles. However, it is suggested from the interviews that advanced ESL students appreciate Flash Word better. In future we plan to show if there is any English level which benefits from the system

References

References

  1. [1]Mohammad Reza Ahmadi Abbas Pourhossein Gilakjani. 2011. A Study of Factors Affecting EFL Learners’ English Listening Comprehension and the Strategies for Improvement. Journal of Language Teaching and Research 2 (2011), 977–988.
  2. [2]Antar Sohly Abdellah. 2008. Can Intra-lingual subtitling enhance English majors’ listening comprehension of literary texts?. In Journal of Qena Faculty of Education. Vol 11, No 1. 2008 [Egypt]. 34 pages.
  3. [3]BBC News. 2016. BBC News Website 'Royals feed baby elephants and discuss poaching in India'. http://www.bbc.com/news/uk-36033063.
  4. [4]Annamaria Caimi. 2006. Audiovisual Translation and Language Learning: The Promotion of Intralingual Subtitles. In The Journal of Specialised Translation Issue 6 - July 2006. 85–98.DOI
  5. [5]CNN Student News. 2016. CNN Student News webpage. http://edition.cnn.com/studentnews.
  6. [6]John Field. 2009. Listening in the Language Classroom. Cambridge University Press.
  7. [7]Google, Inc. 2016. Google Cloud Speech API. https://cloud.google.com/speech/.
  8. [8]Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28 (1972), 11–21.
  9. [9]Akinobu Lee, Tatsuya Kawahara, and Kiyohiro Shikano. 2001. Julius - an open source realtime large vocabulary recognition engine. In EUROSPEECH. 1691–1694.DOI
  10. [10]Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing Order into Texts. In EMNLP. 8 pages.
  11. [11]Readability Formulas. 2016. Free Text Readability Consensus Calculator. http://www.readabilityformulas.com/free-readability-formula-tests.php.
  12. [12]S. Rose, D. Engel, N. Cramer, and W.Cowley. 2010. Automatic Keyword Extraction from Individual Documents. Text Mining: Theory and Applications: John Wiley & Sons (2010).DOI
  13. [13]Xinlei Zhang, Takashi Miyaki, and Jun Rekimoto. 2016. WithYou: An Interactive Shadowing Coach with Speech Recognition. In UIST 2016 Demonstration.

Paper details

vlnc:W_ctvcr3jt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents