Keywords

1 Introduction

The hazards of operating on the smartphone while walking has become common knowledge today, and the behavior has become an object of public concern, frequently causing collisions between individuals and bicycles or cars. This is because smartphone users tend to concentrate on their phones and not notice the approaching cars.

Today, meeting others without a smartphone is unthinkable. When meeting outside of their homes, people try to meet by communicating with each other. There is no guarantee they will be able to meet without operating on their smartphones while walking. For this reason, there is no end to collisions caused by users looking at their phones.

Moreover, on rainy days, one hand would be taken with the umbrella. Holding the smartphone in the other hand for contacting others will fill up both hands, and the hands will not be available for other purposes. To perform other tasks, individuals would have to hold both the umbrella and the smartphone in the same hand, which makes it difficult to operate on the phone.

People operating on the smartphones while walking will inevitably gaze downward. Not being able to see what is in the front, they become involved in collisions. In addition if they are wearing headphones they will not be able to hear, and the risk of causing an accident becomes even more elevated. However, although smartphone operation while walking is a known hazard, the numbers of these accidents do not decrease. On the other hand, it is impossible to prevent people from using their devices while walking. Taking these points into account, the solution would be to make it possible for smartphone users to operate on their phone whilst directing their attention to their surroundings. The risk would decrease drastically if individuals could operate their phone facing forwards, for example, by displaying the smartphone screen on the umbrella. Additionally, even in the rain, users would be able to hold the umbrella without using up both hands. If smartphones could be operated using the umbrella, users will not need to look down at the smartphone screen in their hands.

2 Related Work

Wang et al. noticed that smartphone users cross the road while operating on their phones [1]. Those who cross the road and operate on their phones simultaneously are in more danger than those who don’t. Thus, Wang et al. developed “WalkSafe” to support those who operate on their phones while walking. WalkSafe photographs the surrounding scenery on the smartphone camera. The system uses machine learning on the photographed images and detects approaching cars. When cars are approaching, the system will warn the user through sounds and vibrations.

Kodama et al. have pointed out that although people are aware that smartphone operation while walking is dangerous, they are unable to stop [2]. In order to make this operation safer, they suggested a support system that uses an image sensor (Fig. 1). Their support system obtains the frontal image via a range image sensor, and shows the surrounding elements of danger in the overlay display on the upper part of the smartphone screen. When the distance between the user and the object is close, the color will be red, and when there are no objects approaching the user, the color will be blue. For example, if the user’s left side is closer to the object, then the color on the smartphone’s upper left will be red, and if there are no objects in front of the user, the upper center of the smartphone will be blue. The system warns the user using color and location on the screen.

Fig. 1.
figure 1

Safe-walking support system [2]. (Color figure online)

Full size image

Ito et al. assumed smartphone use by motorbike riders [3]. They attached an accelerometer on riding gloves, and suggested controlling the smartphone by detecting the hand gestures. The results of the operation would be outputted by sound or voice, and users would not need to stare at the smartphone during input or output. Their experiment demonstrated that acceleration is not influenced by vibration during the motorbike ride. The system detected accelerations without problem, and the user was able to operate the smartphone.

National Geographic conducted a study where a part of the sidewalk was experimentally converted into a “smartphone operators only lane.” [4] The left side of the sidewalk was designated to smartphone users, and the right lane to other pedestrians. (Fig. 2) Results showed that very few people changed lanes while walking. Although many people operated on their phones while walking, they were so absorbed in the smartphone screen that they did not notice the smartphone users’ lane.

Fig. 2.
figure 2

Cellphones lane [4]

Full size image

In this study, we will make it possible to operate smartphones using umbrellas. For this purpose, we will develop a system and interaction that will turn the umbrella into smartphones (allow the umbrella to perform the input and output operations of the smartphones). Our aim is to suggest and validate the interaction between the user and his/her umbrella. By using the umbrella as a display, users will be able to face forward while walking. Since users will be operating their phones by tilting or moving the umbrella, they will not hold anything besides the umbrella. We will validate through several apps whether it is possible to operate the smartphone using an umbrella.

3 Examples of Application

In this study, we will not implement the OS functions for activating the app. (rancher functions), and assume that each application has been activated, and is ready for use. This time, we will make validations regarding the below apps.

3.1 Communication Apps Such as SNS

We validated using communication apps such as LINE and iMessage. Each communication consists of one screen per dialogue partner, with screens connected horizontally. Additionally, in each communication, older conversations exist in the upper section, and the newest conversation in the lowest section.

Switching between conversation partners is done by scrolling the screen horizontally with the left and right movements of the umbrella. Scrolling within each conversation will be done by the front and backward movement of the umbrella. The functions will mainly be for browsing, and not for input.

3.2 Map Apps

The map app will display the map surrounding the present location, and users will be able to scroll by moving the umbrella forwards, backwards, left, and right. Moving the umbrella upwards will reduce, and drawing it downwards will enlarge the size of the map (Fig. 3)

Fig. 3.
figure 3

Map application on the canopy of umbrella

Full size image

3.3 Movie and Music Player Apps

The display will consist of 3 panes. On the right 1/4 of the display, thumbnails of videos and music will be displayed vertically. User will be able to switch between videos by swaying the umbrella front and backwards. The left side of the display will be divided into the upper and lower halves, with SeekBar on the lower half. By swaying the umbrella left and right, user will be able to designate the section of the video they wish to watch. The upper left section will be for displaying the videos. User can control the volume (sound) by swaying the umbrella up and down.

4 Interaction Models

In utilizing the umbrella as a mobile device, the canopy would be the part suitable for the display. On the other hand, it is not realistic to connect a keyboard or a mouse to the umbrella for input purposes. Furthermore, having to touch the canopy as one would the screen of the smartphone will not be a fundamental improvement. Therefore, we will display the screen on the canopy and use the movement of the umbrella for input. With an umbrella used for input, it would be difficult to perform diverse operations like on the phones, since we would not be able to recreate the multi touch operations people perform with their fingers.

In order to come up with an interface that remains practical even with simplified operations, we will consider an interaction designed especially for umbrellas, which would be different from that for smartphones. Thus, we designated the selection (direction), decision, and cancel operations to different movements of the umbrella. Rocking of the umbrella to the front, back, left, and right will correspond to actions for pointing to directions such as the flick and slide on the smartphone. Input action for decision will be lifting the umbrella or hitting the shaft. Users can go back to the previous action by pulling the umbrella downwards.

Apps will be developed according to this interaction model. However, some of the apps may have their own individual interactions not necessarily be based on this model.

5 Implementation

The suggested interactions will be used in specific apps. The reactions to each action will depend on the app, but the basic actions will depend on the interaction model.

We used the following devices in creating the system.

5.1 Umbrella

We used a white umbrella with 92 cm length, 116.5 cm diameter, and 324 g weight. To create the system, we used an umbrella larger than normal. When the length of the umbrella is shorter, the focal length of the projector will be shorter, and thus the displayed image would be smaller. By using a short focus projector, we will be able to display large enough images.

5.2 PC

We used pc where in real-life situation smartphones would be used. With a smartphone, we would be able to use the built-in accelerometer. Since PCs do not have a built-in accelerometer, we connected the PC to an accelerometer using Arduino. The PC converts the values from the accelerometer into inputs, and processes them as operation on the app being displayed on the screen. The system should display the smartphone screen, but for the prototype, the PC screen will be displayed.

5.3 Acceleration Sensor

We used a 3-axis accelerometer. Although accelerometer would be enough in itself, adding a 3-axis gyro sensor may improve the accuracy of acceleration measurement. However because we were aiming for a simple system, we used the accelerometer only.

5.4 Projector

We selected a 168 g light-weight projector. Because the projector is light, it will not add too much weight to the umbrella once attached (Fig. 4). The projector has a built-in battery, and does not need an electrical power source. The compact projector has short focus distance, and can display a 37 inch image from 1.0 m distance.

Fig. 4.
figure 4

Projector attached to umbrella

Full size image

5.5 Arduino

The accelerometer will be attached to the tip of the umbrella, and will be connected to Arduino. Arduino outputs the acceleration values to the PC. Although we used the affordable Arduino Uno, a more compact Arduino or WiFi could be used.

6 Experiment of Movement Identification

6.1 Experiment Method

This system will be used while walking. When the user walks, naturally the umbrella will rock. The system needs to distinguish between the natural and the unnatural (deliberate operation by the user) rock. In order to distinguish between the two, we conducted an experiment. We attached the accelerometer to the tip of the umbrella, and asked 13 participants to move the umbrella. From the observed acceleration values, we aimed to calculate the acceleration values that would differentiate walking and deliberate operation.

The coordinate system for the accelerometer in this study is as follows: the back and forth movements will be shown on the x axis (the backward movements represented as positive values), the left and right movements will be shown on the y axis (left movements will be represented as positive values), and the up and down movements, on the z axis (downward movements will be represented as positive values). The reason the axes are inverted is because the accelerometer was attached to the umbrella tip in a vertically inverted position.

6.2 Result

The Fig. 5 shows the acceleration for the 5 times when participants rocked the umbrella forwards. We measured the acceleration at 30 Hz sampling rate. Dotted line stands for the acceleration on the x axis, the broken line, on the y axis, and the solid line, on the z axis.

Fig. 5.
figure 5

The acceleration of umbrella in a forward direction.

Full size image

For the forward and backward directions, the threshold for differentiating natural movements (unintended movements) and deliberate movements was 0.33G. If the accelerometer senses acceleration surpassing 0.33G on the x axis and immediately afterwards detect acceleration surpassing −0.33G, the system will assess that the umbrella was rocked forwards by the participant. Backward rocking can be detected in the same manner.

The Fig. 6 shows the acceleration for the 5 times when participants rocked the umbrella towards the right. We measured the acceleration at 30 Hz sampling rate. Dotted line stands for the acceleration on the x axis, the broken line, on the y axis, and the solid line, on the z axis.

Fig. 6.
figure 6

The acceleration of umbrella in a rightward direction.

Full size image

For the left and right directions, the threshold for differentiating natural movements (unintended movements) and deliberate movements was 0.33G. Similar to the forward and backward movement, we were able to distinguish the deliberate movements.

The Fig. 7 shows the acceleration for the 5 times when participants lifted the umbrella. We measured the acceleration at 30 Hz sampling rate. Dotted line stands for the acceleration on the x axis, the broken line, on the y axis, and the solid line, on the z axis.

Fig. 7.
figure 7

The acceleration of umbrella in an upward direction.

Full size image

For the upwards and downwards directions, the threshold for differentiating natural movements (unintended movements) and deliberate movements was 0.16G. Similar to the forward and backward movement, we were able to detect the deliberate movements. The system can assess that the participant has lifted the umbrella when it detects an acceleration surpassing −0.16G acceleration on the z axis and immediately afterwards, acceleration that surpasses 0.16G in the opposite direction on the same axis. Acceleration surpassing 0.16G is detected when the umbrella is rocked in the forward direction, but because in this case the acceleration in the opposite direction will not be detected immediately afterwards, distinction between forward movement and upward movement is possible.

7 Evaluation Experiment

In the future, we will conduct an evaluation experiment following the below procedures. Ideally we should conduct the experiment outdoors in rainy weather. However, because our prototype is not waterproof, we are planning to conduct the experiment indoors.

We will explain the operation procedures to the participants of the experiment. We will explain them how to perform operations for each app. Participants will operate the umbrella and attempt to complete the goals we have set, such as turning the volume down 3 notches on music player, or displaying destination on the map app, etc. Participants will complete the goals through trial and error.

We will be recording the acceleration of the umbrella during the experiment. The participants will be asked to voice the intent for moving the umbrella. We will record the intent of the participant, the operation the participant made in order to realize the intent, and the results of the movement. Lastly, we will evaluate the results, including whether the participant was able to achieve the goal. We will conduct a questionnaire survey after the experiment and perform subjective evaluation.

Furthermore, we will calculate the movement of the umbrella based on the acceleration data attained, and analyze the participants’ intent and the movement of the umbrella.

8 Conclusion

We implemented a system that utilizes umbrellas for input and output on the smartphone. Using the system, we suggested an interaction for operating on the smartphones through umbrellas. In the future, we will perform a usability evaluation based on the data obtained in the evaluation experiment. We will then analyze whether the participants of the experiment accepted the interaction we implemented, and whether were able to recall appropriate interactions on the umbrella.

9 Future Work

We are hoping to find out whether the users were able operate this umbrella-based system with one hand. We also hope to evaluate whether they had more ability to grasp their environment using our umbrella-based operating system, compared to operating on their phones while walking.