Keywords

1 Introduction

Eating habits play an important role in health and wellness. Research shows that not only the types of nutrients that we eat but also their quantity (amount consumed) has an impact on our health. Therefore, it is important for people to know type and quantity of nutrients in their food so that they can make healthy food choices. This is particularly important for people with dietary restrictions and/or those with chronic illnesses [10]. When it comes to packaged food items, making healthy decisions would be extremely difficult if not impossible without nutrition fact labels.

Nutrition facts labels typically list all the nutrients and their respective amounts in a tabular form. These labels, which convey nutrition information in form of numbers, are only effective if consumers know how to use them [21]. These labels provide information; they do not provide guidance or advice [11]. The numeric format of nutrition labels requires simple calculations to interpret the provided information. While such calculations are not inherently difficult for interpreting a single nutrient, they become relatively more complex when multiple nutrients are track at once (e.g., sugar, fat, protein, sodium, etc.) [16]. In United States, nutrition facts label typically include ‘percent daily value’ to convey information regarding the amount of nutrients in one serving. Unfortunately, consumers often do not know how to interpret percent daily values, i.e., how to use them to make healthy decisions [12].

In this research, we created a decision support system to make it easier for people to make healthy decisions. To achieve this goal, we developed a mobile application for smartphone that can scan nutrition information from labels and convert it into an easy to understand personalized feedback. We believe that such an easy to understand personalized feedback will help consumers to make healthier decisions.

We developed this decision support system in two steps. In the first step we used the optical character recognition (OCR) technology to scan the nutrition information. We then converted this information into an easy to process feedback. In the second step we worked on visualizing the feedback using two different prototypes: one porotype showed the advice in augmented reality (AR) and the other showed it in a static pop-up format. We used the 5-20 rule recommended by the FDA to create feedback signifying whether the value of a particular nutrient was in a “good” or “bad” range [29].

2 Theoretical Background

Different countries have different government agencies to regulate food packaging and provide guidelines for listing ingredients and nutrition information on packaged food products. The objective of these guidelines is to make the general public aware of nutrients in the food and recommend the healthy quantity for each nutrient. In the United States, ingredients and nutrition information on packaged food products are regulated by the Food and Drug Administration (FDA). In this research we focused on nutrition facts labels, more specifically the macronutrient information on packaged food items available in the United States. Therefore, we used FDA’s regulations, US Department of Health and Human Services and U.S. Department of Agriculture guidelines, e.g., Dietary Guidelines for Americans [28] throughout this research.

2.1 Nutrition Facts Label

Multiple studies on nutrition facts labels found a positive relation between the use of nutrition facts labels and diet and health. For example, increased use of nutrition facts labels improved healthier dietary pattern [9, 13, 15, 20]. Studies show a strong relationship between health beliefs and nutrition label usage or intention to use labels [9, 21]. They also show that consumers with health conditions and special dietary requirements are more likely to use nutrition facts labels [18,19,20]. Additionally, research indicates that nutrition label knowledge has a positive impact on consumer’s ability to distinguish between nutritional value of different food products [17].

While providing useful information, generally nutrition facts labels tend to be hard to read and understand; they can be confusing [11, 12, 14, 16]. Factors such as age, level of education and income can further affect understanding of nutrition labels and hence negatively affect their usefulness in guiding consumers’ decisions [16]. Additionally, eye tracking studies reveal that provided information on nutrition labels is not processed equally or fully. Components near the top of a nutrition label grab considerably more attention than components at the bottom [14, 23], in other words, consumers tend to read only the top lines on the label. The placement of the nutrition facts labels on a package can also contribute to their usefulness as it can affect consumers’ ability to locate the provided information [14].

One way to make nutrition facts labels more useful is by providing percent daily values for nutrients. Percent daily values are calculated based on the type of nutrients and their recommended amount. While percent daily values can provide useful information, they are not always present for every nutrient on nutrition facts labels. Moreover, percent daily values are typically calculated based on a 2000 calorie intake threshold. If consumers are not familiar with the 2000 calorie intake guideline or follow a different one, the percent values on the label can be confusing. For example, research shows that consumers with dietary restrictions, such as those with medical conditions, tend to find percent daily values hard to understand [10].

Another important factor about nutrition facts labels is that updating them on food packages is not a quick process. For example, while in May 2016 the FDA announced new guidelines for providing nutrition facts, consumers may have to wait for a few years before they see updates on every food product because the compliance dates for manufacturers was set to a much later date, e.g., January 2020 and January 2021. The updated guidelines in May 2016 seem to have made little difference in engaging consumers to pay attention to provided information. For example, an eye tracking study comparing the old and new labels, found no difference in terms of attention given to different components of the new nutrition labels nor did it reveal any difference in consumers’ decisions for making healthier food purchases [22].

2.2 Augmented Reality (AR) Feedback in Food and Nutrition

Various attempts were made in the past to improve the delivery of nutrition or ingredient information on the label with the help of augmented reality for making them easier to understand and more helpful in making decisions. Augmented reality tends to elicit more engagement from user and tend to enhance their perception of reality thereby create positive experiences for them [24, 25]. The review of prior AR research in food and nutrition shows that AR technology has been successfully used to teach individuals with intellectual disabilities to identify food allergens [1]. Using an AR app participants were able to identify potential food allergens successfully and many participants learned to use the app right away. In another study, researchers proposed the development of an application that can scan unpackaged food items and display their nutrition information using color coded gauges [2]. They argued that users can better manage their diet with such an easy to understand feedback. Another group of researchers proposed an augmented reality application that could scan unpackaged grocery items and display feedback in the form of ranking bars based on a user’s dietary profile [3]. Ahn et al. developed an AR grocery shopping application that recommends healthy products and highlights products to avoid on a grocery store aisle based on user preferences [4]. Their results showed that their app significantly reduced the amount of time to select healthy products and helped users to avoid unhealthy products.

Prior research in augmented reality in other contexts shows that not all AR applications provide significant positive experiences [26, 27]. For example, in a recent study the AR app providing social media reviews (textual information) for a product, received relatively low ease of use ratings [27].

2.3 Optical Character Recognition (OCR)

To get the nutrition data from packaged food products, there are two methods available. The first method is using or maintaining a product database, which can be accessed by scanning barcode of the product. The second method utilizes OCR to capture information directly from nutrition facts labels (e.g., using the phone camera). Various algorithms have been developed that make OCR useful in extracting information from nutrition facts labels. For example, Kulyukin et al. developed an algorithm that can correct the OCR output on mobile phones in real time [5]. They also developed an algorithm for extracting text chunks from nutrition facts labels [6]. Additionally, other algorithms have been developed to detect text skew angles [7] and localization of skewed nutrition facts labels [8].

3 Improve User Experience of Nutrition Facts

Given the difficulty of users to understand nutrition labels, in this study we propose and develop an app to help users process nutrition facts based on their dietary restrictions and/or preferences. Our proposed app scans nutrition facts labels using OCR to extract nutrition information. Our app applies FDA’s 5-20 rule on the percent daily value of nutrients and displays the results as thumbs up or thumbs down if the amount is healthy or unhealthy, respectively. Prior research suggests the use of traffic light style color coding on labels to make them more understandable [11]. Hence, we used green to color code the thumbs up feedback and red to color code the thumps down feedback in our app.

We provided this feedback using two different formats. Research shows that AR can enhance the way we engage with the world around us [24]. However, previous research also suggests that AR interfaces in some situations may be harder to use than static interfaces such as pop ups [27]. Therefore, we created two different interfaces for the feedback generated by the app to see which one is more effective in communicating information to users. In augmented reality interface, feedback was augmented next to the nutrition name on the label while it was being scanned by the smart phone. In static popup interface, a static screen popped up on smartphone screen with nutrient names and feedback after the label was scanned by the smart phone.

We argue that regardless of the feedback interface, the app will make it easier for the consumers to make healthy decisions as compared to using just the label. This in turn is likely to improve overall experience of using nutrition labels. We also examine which of the two feedback format (AR vs. pop up) can provide a better user experience.

4 Methodology

In this experiment we focused on three nutrients: saturated fat, sugar and protein. Foods containing fats and sugars are highly preferred [30] and consuming them in high amounts have adverse effects on health [31, 32]. High protein is generally desirable particularly for weight management [33, 34]. We created two anonymous food products (Product A and B). For each of the two food products we created a set of four nutrition fact labels. The nutrition labels in this set were designed to represent products with different nutrition values. For example, in one of the nutrition labels all three nutrients were in healthy amount, in another one of the three nutrients was in an unhealthy amount, in another two of the nutrients were in an unhealthy amount, and in another all three nutrients were in an unhealthy amount (Fig. 1).

Fig. 1.
figure 1

Types of products and labels

Full size image

We recruited twenty-two participants for the study. All participants were students from a university in Northeast US. All participants completed the task of choosing a product (product I or II) among a set of four alternatives twice, hence, they looked at 8 labels in total (4 for each product). Participants completed these two tasks one time by looking at the labels without using the app that was developed in our study (Task A) and one time by using the app (Task B). We assigned the order of the two tasks in a Latin Square fashion. For the portion of the experiment where participants used the app (Task B), we used Latin Square to randomly assign the participants into two groups. One group used the app with the AR interface (Task B1), while the other group used the app with the static popup interface (Task B2). Therefore, we used a with-in subject design for comparing user reaction to traditional way of looking at nutrition labels and using the apps, and a between subject study design to compare reactions to AR and static pop-up interfaces (Fig. 2).

Fig. 2.
figure 2

Study design

Full size image

Before the task, participants answered a few pre-task questions regarding frequency of using nutrition facts labels, frequency of finding nutritional facts labels helpful, difficulty in understanding nutrition facts labels, and frequency of trouble in understanding labels. Participants then were given a scenario that asked them to imagine that they recently decided to start eating healthier by reducing saturated fat, sugar, and increasing protein intake while staying within a two-thousand calories daily limit. In each task, they saw a set of four generic boxes in front of them with nutrition facts labels displaying their ingredients. They were told that these products contained the same type of food from four different undisclosed brands. Participants were required to look at the labels one by one and decide which food product was healthy for them to consume. They were again reminded that they were on a low saturated fat, low sugar, high protein diet, and wish to limit their total calorie intake to two-thousand per day. Once they made a selection, they received another set of four products. As mentioned earlier, one of the two sets of products was viewed via an app that was designed in our study. In that portion of the experiment we provided an android smartphone with the app for participant to view the labels. Participants were instructed to think out loud while performing the task.

Once participants completed a task, they were asked to assess their decision-making process in terms of difficulty in making decisions, confidence in the decisions, ease of use, helpfulness of label and app, and overall experience. We also asked participants whether they would be willing to use the app in future. Participants rated overall experience by giving stars (out of 5), all other factors were assessed on 7-point scale. Demographic information (e.g., age and gender) was also collected at this part of the experiment. Finally, we conducted an interview asking open ended questions regarding factors considered while making decisions, the overall experience of using label and app, knowledge and use of the percent daily value, use of other nutrition related apps, and suggestions for our app.

5 Results

Ten males and 12 females participated in our study. Age range of participants varied from 22 years to 47 years, with average age of 27.1 years. During pre-task survey participants reported that while buying a food product they typically look at the nutrition facts ‘half of the time’, with mean rating of 3.2 out of 5. They found nutrition labels helpful ‘most of the time’, with mean rating of 3.7 out of 5. They indicated that it was ‘slightly easy’ for them to understand nutrition facts labels for any food product, with mean rating of 4.6 out of 7 and they indicated that ‘sometimes’ they had trouble understanding the label, with mean rating of 2.2 out of 5. Ratings for pre-task questions are summarized in the Figs. 3 and 4. Percentages may not total 100 due to rounding.

Fig. 3.
figure 3

Result of frequency related pre-task questions

Full size image
Fig. 4.
figure 4

Result of difficulty related pre-task question

Full size image

The results showed that 91% of participants made healthiest selection using the app compared to only 45% participants without the app. Participants found that decision making was significantly easier with the app (p = 0.000) and were significantly more confident in decisions that were made with the app (p = 0.001). In both treatments participants found that the provided information, via labels or through app feedback, was helpful in making decisions. However, participants found the information provided though the app was significantly easier to use (p = 0.004). Similarly, participants reported that the overall experience of using labels was significantly better when they used the app (0.002) (Table 1).

Table 1. Results of t-test comparing user reactions with and without the app
Full size table

Next, we compared user reactions to AR and pop up interfaces. The results did not show significant differences between the two interfaces. Out of 5, mean overall rating for using the app with AR interface was 4.3 compared to 4.2 for using the app with static popup app. Participants reported that both interfaces were easy to use, with mean ratings of 5.9 out of 7 for both. The results showed favorable ratings for adopting the app. Participants’ rating for willingness to use the app with AR interface was 6 out of 7, a slightly lower (5.4) for the app with pop up. These results are shown in Table 2.

Table 2. Results of t-tests comparing AR and static interfaces
Full size table

The interviews at the end of the experimental session revealed that only 41% of participants looked at percent daily values on nutrition facts labels when they purchase products. Out of those only 22% knew what percent daily values signified; 22% used it incorrectly and 56% used percent daily values as a way to compare food products without knowing what these values signified (Fig. 5).

Fig. 5.
figure 5

Knowledge and use of percent daily value (of participants who look at percent daily value)

Full size image

The interviews also revealed that the color coding and thumbs up and thumbs down images were the deciding cues for participants to select healthy products. Participants reported that their experience with nutrition facts label without using the app was confusing, time consuming, and required a great deal of cognitive effort (calculations and comparisons). While most participants (91%) found the app easy to use and helpful a few (9%) preferred using the labels without the app. Some participants (9%) indicated that more detailed feedback (other than color-coded thumps) would be helpful in conveying information.

6 Discussion

Our results show evidence that the app had significant advantage over using just the label. The results showed that 91% of participants made healthiest selection using the app while only 45% of participants were able to do the same without the app. Participants reported confusion and complexity in using the labels during the task. These findings are consistent with the previous research that nutrition facts labels are hard to understand and confusing in general [11, 12, 14, 16].

With significantly higher ratings for the app in regard to ease of decision making, confidence in decision, and ease of use along with 95% of participants being able to make healthiest choice, it is reasonable to argue that the app had a significant positive impact on decision making. Our app also received significantly higher overall experience rating compared to just using the label.

The results did not show significant differences in ease of use and overall experience between the two feedback interfaces. The average scores for these measures were relatively high and very close for both interfaces. While participants exhibited willingness to use both interfaces in future, they rated their willingness to use the AR interface slightly higher than their willingness to use the popup interface. Although this difference was not significant, they indicate the possibility that augmented reality may be a more enticing adoption factor. Future research is needed to explore these possibilities.

Our results regarding the perception and use of percent daily value are also consistent with previous research showing low understanding of these important values [12]. Only 22% of participants in our study had a correct understanding of percent daily values. This highlights the importance of our app, which provides feedback using FDA’s 5-20 rule that is based on percent daily values.

7 Limitations and Future Research

As in any experimental study, our research is limited to its setting and tasks. Our results show that our app has a potential to change the behavior of consumers in the long term towards making healthy decisions. However, more research is needed to extend the generalizability of our results and to verify the behavioral impact that our app can have to help consumers make healthy decisions.

Some participants pointed out that the app feedback was not detailed enough. Adding more information to the feedback and testing its effectiveness provides a direction for future research. While we found significant differences in decision making and overall experience in decision making with and without the app, we did not find significant differences between the two feedback interfaces. Repeating the experiment with a larger sample size is likely to provide more insights about the impact of these two feedback interfaces on consumer behavior regarding nutrition information and decision making.

In this study, our app provided feedback based on a two-thousand calorie diet. Future studies using personalized dietary plans are needed to increase confidence in generalizability of our results. Moreover, the app was developed only for Android smartphones. More platform compatibility is required in the future for maximum consumer reach.

8 Conclusion

As one revision of official nutrition label design could take a considerable amount of time to be completely implemented, our approach of transforming the nutrition facts label in real-time with the help of augmented reality provides an effective and efficient way to keep consumers informed. No matter what the physical form of the label is, we can modify and personalize that information using the most updated government health guidelines. The use of OCR technology in our app creates a seamless experience and eliminates the dependency on a food product database. Consumers can use the app on any nutritional facts label of any product.

Our app can help consumers to make healthy decisions. Consumers seem to have little knowledge about various components of the nutrition facts labels, especially percent daily value. Research indicates that for regular consumers nutrition facts labels are confusing, complex and difficult to use. Our app scans information from nutrition labels, applies calculations based on FDA’s recommendations, and visualizes the results in an intuitive way to help consumers make healthier decisions.