The Panlingual Camera Phone: Internet Prototype

Martin Hecko, Kinsley Ogunmola, Jonathan Pool, Tim Wong, Peter Woodman

Project sites:
http://www.cs.washington.edu/education/courses/cse490f/07wi/project_files/camera/
http://panlingual.org/ (hosted by Utilika Foundation)

Report URLs:
http://www.cs.washington.edu/education/courses/cse490f/07wi/project_files/camera/rereproto/
http://panlingual.org/rereproto/

Introduction

We report here an increment in the development and testing of a user interface for a panlingual camera phone. In this increment, we made our interactive prototype testable anywhere by means of a Web browser and integrated the prototype into a questionnaire that gave instructions to a user, tested the user's success in performing tasks, elicited information about the user, and elicited opinions and comments from the user. Thus, this increment, while adding no features to the prototype, added to our repertoire of methods for testing its features.

Tasks

The tasks that test users were asked to perform were as follows:

"Taking a Bus" (easy): While sightseeing, you are at a bus stop. A bus approaches, with a foreign-language destination sign. You want an English translation, to decide whether to take the bus.

"Ordering a Meal" (moderate): You are traveling in France and are at a restaurant with a menu in Chinese and waiters who are fluent only in Chinese. You have a note in English asking for the vegetarian dishes to be pointed out. You want your note translated into Chinese, to show it to the waiter.

"Exploring Your Ancestry" (difficult): You are visiting a cemetery where an ancestor is believed buried and have found the likely tombstone, with a foreign-language inscription. You want an English translation and, after seeing it, want to know which word in the original was translated into a particular English word.

Study Construction and Deployment

We configured the prototype on a server as a set of HTML documents, in such a way as to capture records of the HTTP queries from respondent clients. These records would tell us which pages of the prototype each user visited, in which order, and at which times. For this purpose, we made the Web server issue zero-duration expiration headers with all pages of the prototype, in order to induce browsers to fetch all page instances from the server and none from browser caches.

After constructing a draft questionnaire, we pre-tested it on ourselves and on one acquaintance. We amended the questionnaire on the basis of problems discovered during these pretests. However, the pace of construction was such as to preclude some improvements that we would have undertaken if more time had been available.

We recruited uncompensated volunteers to undergo the amended questionnaire. We informed individuals with whom we were acquainted, posted messages on newsgroups to which we subscribed, and invited readers of these messages to inform others.

The questionnaire was available for public access for about one day. Response activity took place from 21:09 on 24 January until 18:41 on 25 January 2007.

Questionnaire Contents

Introduction

We designed the questionnaire in conformity with the requirements of the University of Washington Catalyst WebQ system. Respondents entered our test by beginning our questionnaire. At three points, the questionnaire instructed the respondents to follow links, which opened new browser windows or tabs containing states of the prototype, and to use the prototype to perform tasks.

We summarize the questionnaire items here. A copy of the questionnaire is attached as Addendum 1.

Orientation

The questionnaire began with an orientation text describing the purpose of the study as evaluating a prototype of a mobile product that would translate texts for users, and in particular finding and correcting problems with the prototype's interface. Respondents were informed that they would be anonymous and that data from their participation would be used academically and possibly in publications.

Respondent Characteristics

The first four items in the questionnaire were questions eliciting information about the respondents: sex, age, portable electronic devices they use, and comfort with particular features of mobile phones.

Tasks

The rest of the questionnaire was partitioned into three sections. Each section began with a task description. Next, an instruction gave a link to the respondent and told the respondent to open the linked page and use the interface to perform the task. Next, a factual question tested the user's successful task performance. Next, three questions asked for the respondent's opinion on how easy the task was to complete, how easy the interface was to use, and whether the interface helped to complete the task. Finally, a question invited the respondent to provide any comments on, or suggestions for, the interface with respect to the current scenario.

The question on the "Taking a Bus" task asked the respondent to determine the English translation of the bus's destination sign. This required operating the "Capture and Translate" control and correctly reading the result.

The question on the "Ordering a Meal" task asked the respondent to determine how many characters appear on the first line of the Chinese translation of the English note. This required operating the "Capture and Translate" control, then activating the target-language dropdown list, then selecting "Chinese" as the target language, and then correctly reading the result.

The question on the "Exploring Your Ancestry" task asked the user to determine the Hattanese word that means "school" in English. This required operating the "Capture and Translate" control, then noticing a context-sensitive hint inviting the respondent to touch a word, then clicking on the word "school" in the English translation, then noticing that the source-language text now contained annotations of one word, and then correctly inferring that these annotations identified that as the Hattanese word whose translation into English is "school".

Results

Introduction

Our data consist of records from the questionnaires and from the prototype's server logs. The questionnaire system did not disclose to us the IP addresses of respondents. IP addresses were the identifiers that permitted us to collect server-log entries into respondent sessions. To match questionnaire respondents with server-log sessions, we relied on the requirement that each respondent's questionnaire session was required to begin before, and end after, that same respondent's interactions with the prototype. However, this method of record matching was fallible, because of some nearly simultaneous sessions and because a respondent could continue to manipulate the prototype after completing the questionnaire.

Univariate Results

After discarding pretest respondents and respondents for whom we could not match questionnaire and log data, we had data for 24 respondents, of whom 75% were male and who ranged in age from 18 to 61 years. The mean age was 27.54, the median age was 22, and the standard deviation of the ages was 12.29.

The fractions of the respondents reporting they used various mobile devices were as follows:

The mean respondent reported using 2.13 of these devices, and the median respondent reported using 2 of them.

The respondents' average reported comfort levels with various uses of mobile phones, with 1 as the minimum and 5 as the maximum comfort, were as follows:

The respondents' performances on the three tasks were as follows:

Their judgments about the usability of the interface in each task, with responses to three questions aggregated, could range from 3, for the least, to 15, for the greatest, usability. The average judgments were as follows:

When we aggregate the three tasks, we find that the mean respondent succeeded in 2.46 of the 3 tasks. The all-task aggregate judgments of usability could range from 9 to 45. They had a mean value of 38.38 and a median value of 41.5.

The performance and judgment results suggest that objective performance may tend to differ substantially more from task to task than subjective usability judgments do. If so, respondents may be substantially unreliable judges of their own proficiency, of user interfaces' usability levels, and of the relative difficulties of tasks.

The time spent by respondents undergoing the test ranged from 2 minutes to 75 minutes. The mean duration was 9.63 minutes, and the median duration was 6 minutes. The questionnaire platform did not disclose to us the times at which particular questionnaire pages were submitted. A respondent could wait, or be distracted, for a long time after completing the final page before submitting it. Our inability to observe respondents gives us reason for hesitation in drawing conclusions from the duration data.

The tasks, in the aggregate, if performed with maximal efficiency, required respondents to request a total of 9 prototype pages. The server logs show that respondents in fact requested from 7 to 39 pages. The mean number of page requests was 14.88, and the median was 12.5. Without being able to observe the respondents, we cannot know why most of them requested more or fewer pages than the maximally efficient number. They could have aborted the questionnaire after viewing several pages and then restarted the questionnaire with an advantage. Or they could have viewed additional pages out of curiosity or to exercise the prototype. Thus, it would be unreasonable to believe confidently that a larger number of page views is an indicator of a lower level of skill in manipulating this interface, or any other single attribute.

In Addendum 2, we reproduce the comments entered by respondents in the questionnaire, after excluding comments apparently not relevant to the user interface or system.

Bivariate Results

In light of our finding that objective performance differences were much greater than differences among subjective usability judgments, we were curious as to whether objective and subjective differences were associated. Our data show a (slight) positive association, with the most successful respondents having predicted (mean) aggregate subjective usability judgments of 40, versus 34 for the least successful respondents. The correlation (R²) between these variables was 0.07. The joint distribution is shown in this figure:

Respondents who made more prototype page requests tended to be more successful in task performance (R² = 0.15). Those who made few page requests varied in success, but those who made the most page requests were invariably successful, as shown in this figure:

Respondents with experience in the use of more types of mobile information devices tended to be more successful in using our interface, although experience had little predictive power (R² = 0.05), as shown in this figure:

By contrast, respondents who reported more aggregate comfort with various tasks performed on mobile phones exhibited a mild tendency (R² = 0.06) to be less successful in the tasks, as shown in this figure:

Inspection of the respondents' task successes revealed imperfect scalability. If the tasks differed in difficulty on a single dimension, we would expect respondents who failed on any task to fail on all tasks of greater difficulty. In fact, 22 out of the 24 respondents fit this pattern, but the other 2 respondents deviated from it. In both those cases, the respondents failed at the "easiest" task (Taking a Bus), while succeeding at both of the more "difficult" tasks. However, deeper inspection revealed that, in both these cases, the respondents' "failures" were not true failures. Our task definition was "You want an English translation .... Open this page, and use the interface to figure out what the sign says. What does the sign say?" Our expected response was the English translation ("Elementary School 17"). But a literal interpretation of our instruction could make a respondent give us the sign's original text, and that is what both these respondents did (we can surmise that in one case the respondent entered that text from memory and misremembered one of the words in the foreign language.) This result suggests that it may have been more appropriate to count these responses as correct in the above analyses. It also indicates a need for more care in (and thus more time for) the formulation of instructions to respondents.

Recommendations for Changes

The results suggest that further refinement and testing should emphasize these aspects of the interface:

• The help subsystem. Adding content to the currently empty help pages will give users deeper context-sensitive advice.
• Capture-translation integration. The current integration of these functionalities may disorient users and may add complexity when the translation of saved images is added to the feature set and when realistic delays in translation are experienced by users.
• Intra-text navigation. The model, vocabulary (e.g., "zoom"), gestures, and icons that provide for user redefinition of the focal region of the image, and of the texts, should be tested and tuned for intelligibility and efficiency.
• Annotation. The colors and other conventions adopted by the interface for marking correspondences between original and translated text segments should be tested and tuned for intelligibility.
• Document management. The interface should be further developed to include the management of captured images and translations, and the translation of previously captured images. Further postponement of this functionality could make the interface too dependent on the false assumption that only just-taken photographs will be translated.

Conclusions

The deployment of our panlingual camera phone user-interface prototype as a Web-based questionnaire and user test was effective in permitting the inexpensive and rapid acquisition of participants. The number of respondents (24) was greater than the numbers of respondents that it had been practical for us to recruit and test in laboratory environments. Conversely, the unobservability of the respondents made some of their behaviors more difficult to interpret than would have been the case in a laboratory or in contextual testing.

The Web-based test approach seems likely to be valuable in the making of decisions about alternative solutions to interface problems. Issues like how to zoom in and out, and how to annotate text selections, are likely to motivate us to test several alternatives. Web-based randomization of treatments could be a (or perhaps the only) practical means for accomplishing such experimentation. If we forego such testing, we may arbitrarily and prematurely select among alternatives, or reject features that haven't yet worked well but could be refined to be effective with adequate testing.

Addendum 1: Questionnaire

Panlingual Camera Usability Survey

The purpose of this survey is to evaluate and gather information on proposed product that would help people by translating text for which they are not a native speaker on the fly. This survey will ask you interact with the prototype of the system. Since this is a prototype it will have problems and we are interested in finding and correcting them.

The data that is collected will be anonymous, and any personally identifying information will be removed. However, the results and your feedback will be used in preparing a report that will be then presented to other participants on the CSE 490f Winter course at University of Washington. This data might also be used in a further publication.

You are under no obligation to complete the survey and you decide withdraw from the survey at any time.

What are you?

Required.

◯Female

◯Male

How old are you?

Required. Enter an integer (without commas).

Which of the following do you use? (check all that apply)

☐ Cell Phone

☐ PDA (Palm)

☐ Portable music player (Zune, iPod, etc)

☐ Smart Phone (Blackberry, Palm Treo, Windows Mobile Phone)

On a scale of 1 to 5, 1 being the least comfortable and 5 being very comfortable, rate how comfortable you are using the following features of your phone-

Required.

While sightseeing, you are at a bus stop. A bus approaches, with a foreign-language destination sign. You want an English translation, to decide whether to take the bus.

Open this page, and use the interface to figure out what the sign says.

What does the sign say?

Required.

Rank your agreement to the following questions 1-5, 1 being disagreement and 5 being agreement.

Required.

Any comments or suggestions for the interface in this situation?

You are traveling in a foreign country and are at a restaurant with a menu in, and waiters who know, a different foreign language. You have a note in English asking for the vegetarian dishes to be pointed out. You want your note translated into the restaurant's language (Chinese), to show it to the waiter.

Open this page and use the interface to give a translation.

How many characters are in the first line of the Chinese translation?

Required. Enter an integer (without commas).

Rank your agreement to the following questions 1-5, 1 being disagreement and 5 being agreement.

Required.

Any comments or suggestions for the interface in this situation?

You are visiting a cemetery where an ancestor is believed buried and have found the likely tombstone, with a foreign-language inscription. You want an English translation and want to explore the inscription and its translation, to see parts of the inscription more clearly, estimate the translation's reliability, and learn some words in the language.

Open this page and use the interface to accomplish this task.

What is the Hattanese word for 'school'?

Rank your agreement to the following questions 1-5, 1 being disagreement and 5 being agreement.

Required.

Any comments or suggestions for the interface in this situation?

Addendum 2: Questionnaire Comments

All respondent comments relevant to the user interface, the system, or both, prefixed by the respondents' identifiers, are shown below.

The comments on task 1 (Taking a Bus) were:

• 1391973: The scenario you present is kind of an edge case because there is only one instance of text in the picture. This seems like a highly unlikely scenario. Your interface ought provide some means of selecting what text it is you need translated. I'm also not sure i support the idea of combining the actions of capturing and translating into a single control. I suppose it depends on how quickly the translation is processed, but an incorrect capture (ie: the users aim was slightly off and they cut off the text) followed by a wait for translation would be VERY frustrating. Thus i propose a capture, review, translate experience flow.
• 1391982: Red is ugly
• 1392095: The graphic is absolute crap. Seriously. But it was easy to use.
• 1392147: It's effective. It is ugly, though.
• 1392304: Looks pretty easy to use, although I would get tired of the "capture and translate" button when I'm just around Seattle.
• 1392304: Note that my cell phone doesn't have all the features (ie camera, email) you presumed it did on the last page.
• 1392419: A contextual sort of link around the area to 'capture and translate' would have been better than a link in the corner.
• 1392900: Different font... but I expect that's just for the prototype.
• 1392999: I couldn't tell what I was supposed to do. It wasn't really clear that I was looking at a UI?

The comments on task 2 (Ordering a Meal) were:

• 1391973: As users are apt to do i started clicking without really paying attention, and clicked on the "from english" menu. This put me into a never-never land with no back button. I used the Browser controls to get back to where i knew i needed to go. It was not clear to me that by clicking the menu header again I could get back (is that the intended behavior?) On small displays like this you often have to use lots of iconography to get your point across. I understood the downward arrow icon, but that icon simply dissappears ... maybe it should change state in order to represent the users options.
• 1391982: I'm not sure, how does it know what to translate it FROM? is this auto?
• 1392147: Make it a bit more obvious, the arrow is kind of ambiguous. Good in general, though.
• 1392173: I like the flow from one language (top) to another (bottom).
• 1392304: It took a second to figure out the triangles were drop down menus.
• 1392999: It was easier this time because I had seen that it was an interface. The interface is very similar to babelfish.

The comments on task 3 (Exploring Your Ancestry) were:

• 1391973: I like the way you used colors to indicate a relationship between the highlighted word in the translated text and the highlighted text in the image. I might pick a color that's not used elsewhere in the interface.
• 1391982: why hattanese?
• 1392095: The translation was a little hard to read and the "zoom" feature made the words blurry.
• 1392147: Good.
• 1392157: It would have been much harder to pick out the word "school" if it was't in blue.
• 1392173: I was confused by the use of the Zoom options in this case. Clicking on the words seemed intuitive, and zooming back out after clicking a work made sense. I didn't feel like I knew what clicking to zoom IN would do though, since I had not clicked a word yet. Well, in hindsight it seems like the blue word would be the one that is active to be zoomed in upon. So maybe it does make sense, and my misunderstanding was an artifact of me using the mouse instead of cursor-driving buttons.
• 1392397: The zoom feature didnt work very well, I could not get it to zoom. The word zoom was surrournded by a green box but I could not figure out how to zoom in to see the highlighted word.
• 1392419: Highlighting the keyword is nice, but many languages do not have a one-to-one word equivalency with English.
• 1392999: Why was it highlighted?

Addendum 3: Other Questionnaire Data

The questionnaire data are shown in the table below. The comments, shown in Addendum 2, are omitted from this table.

Addendum 4: Server Log Data

The server log data are shown in the table below. We have removed the IP addresses from the table in order to comply with our promise of confidentiality to the respondents.