Introduction

In the current situation, everyone is petrified of the word COVID-19. This pandemic is directly or indirectly affecting each and every person across the globe. Overcoming this misfortune is strenuous until there is an inoculation. Inevitably, pandemic has become a part of life. Nevertheless, time and technology are directly proportional as technology grows in accordance with time. New methods like online classes are being conducted by schools and colleges so that students are not deprived of education. Similarly, the standard of living is changing day by day. Envisaging the world as the dream is fun to think of. What if this can be true! Fortunately, there is a technology called Mixed Reality, which helps a person to envision the virtual world created by them over the real world. By adapting this technology many applications can be built in different domains, for instance, healthcare, education, entertainment, gaming, etc.

Utilizing this futuristic approach in the education system can benefit in numerous ways. The most important upshot of education is to gain knowledge, and there are many ways to implement interactive pedagogy. Virtual reality as well as augmented reality is already being used for educational purposes. In virtual reality, the projection will be simulated in a virtual world. In augmented reality, the projection will be limited to the screen and is not that interactive. To overcome this, a technology called Mixed Reality [1] is deployed, which can be considered as a union of virtual reality and augmented reality as shown in Fig. 1. This technology provides an interactive and immersive user interface (UI) and user experience, respectively. Henceforth, there can be a transfiguration in the education system.

Fig. 1
figure 1

Mixed Reality

Full size image

Literature Survey

V-Touch Immersive Experience

Normally, the virtual objects are less interactive which in turn affects over a good immersive experience. To subdue this situation, vision tangible objects can be employed which will provide an excellent experience with its tactile feedback. This kind of interactivity with objects can be achieved using a touch interface which works without physical touch called V-Touch [2].

Telepresence with a Fine UI

Number simulations of 3D generated avatars are used in telepresence [3] which basically contains 2 main approaches: Video-based reconstruction and 3D model rendering. In video-based reconstruction, first, the whole avatar is captured via camera and analysed. Next, the static and moving objects in the background are eliminated, offering a complete 3D avatar as a result. Then, the generated 3D avatar is subjected to 3D model rendering to overcome the visible lines in the polygon edges, hence providing a clear avatar. In order to provide a sublime user interface, it would be ideal to merge the factors of the virtual and physical world. This can be employed in Mixed Reality successively providing a better immersive experience [4].

Light-weight Image Rendering

Image rendering can also be interpreted as conversion of two-dimensional objects into three- dimensional objects. Two types of image rendering can be observed offline rendering and real-time rendering. In offline rendering, each and every frame of a video captured is converted from 2 to 3D, providing a high-quality photo-realistic image which on combining gives a high-resolution video that is expensive. Light-weight online rendering [5] can be used, where the weight of the polygons is reduced in order to provide a cost-effective photo-realistic image.

Protection of Sensitive Information

While providing an input, sensitive information can be unknowingly indulged to the device. In the same way, while using an application, the sensitive information can be extracted. To circumvent this, the method of input sanitization can be employed which essentially eliminates all the sensitive information from the input stream of data. While displaying, the device might want to check the surface information to decide the displayability of the surface. In that instant, sensitive information breach can emerge. To elude that, a technique called privacy-preserving rendering can be implemented which prevents the device from checking the information on the surface. Rather it just displays the output if in case the surface is found displayable [6].

Walk-Through MR World Experience

Mixed Reality can be considered as an eminent technology for the world today. Using this technology in education, tremendous changes can be observed like providing an interactive classroom experience to the students. To design a mixed reality project, tools such as Unity Editor and Mixed Reality Toolkit (MRTK) [1] can be used. Due to this COVID-19, there has been a shift in the education environment. Abode has become the new school; online classes have become a new trend. Applications like Zoom, Google Meet, Microsoft Teams, Discord, etc., are being used. These applications rank down when it comes to interactivity, in turn students end up losing focus toward education.

As this pandemic may take a while to fade away, a much more interactive application is required to impart fairness to students in educational aspects. Even though the world recovers from the pandemic, students can make use of this technology to level up the education system and learning means. When it comes to education, it includes primary school students till the postgraduation students. For the school children providing education properly that is in the way they understand is important but when considering the graduation or postgraduation students, along with pursuing the education they need to do many projects in order to enhance their skill-sets. For instance, in technical education like electrical engineering, robotics engineering, mechanical engineering and aeronautical engineering, the circuits and robots are simulated in the virtually [7], using MATLAB, 3Ds MAX, GNU Octave, Virtual Robotic Toolkit, CAD [8], V-REP, Microsoft Robotic Developer Studio, etc. Again, using Mixed Reality, a more interactive application can be developed so that the user has an effective immersive experience and gains comprehensive knowledge, both theoretically and practically. First let us consider vanquishing the obstacles faced by the school children. Here, mixed reality can be used as a boon. It is illustrated in Fig. 2. When a student opens the book of a subject and when he/she opens chapter 1, the concepts will be displayed via telepresence where the students can feel their presence just like they are inside the classroom.

Fig. 2
figure 2

Flow of proposed system for education

Full size image

For instance, let us consider the student is opening science textbook, once they open the page where they want to start the learning, the figure/diagrams in that page will be visible with 3D projections when the student wears the head-mounted device and also the teacher will be teaching the concepts accordingly hence the student will be able to feel the presence of the teacher in their environment and gain the knowledge effectively. This can be achieved using the following steps. First, the video must be captured in all the angles using a camera. Unity Editor provides an excellent user interface, and hence, MRTK as well as the captured video is imported into it. MRTK provides many APIs, using which the background of the video containing both static and moving objects can be segmented from the foreground. Next, the 3D avatar is generated which can be used in telepresence so that the students experience immersiveness and engage themselves in the class. 3D model rendering and light-weight rendering are deployed to generate a photo-realistic image or frame removing the visible lines in the polygon. In order to provide input protection, the technique of input sanitization can be employed. Therefore, the students have privileges to learn and gain knowledge sitting at home without any difficulties.

In a similar manner, this technology can be implemented for technical pedagogy to make the learning more interactive as shown in Fig. 3. Suppose the student wants to build a layout for the civil building and check the architectural view of the building in that case first they can build the layout on Unity using MRTK and then project it on to the land where they are planning to construct, so that the user will have an idea of how the interior dimensions/layout of the house may look. They can also project the architectural design on to the construction area and check the exterior looks of the house.

Fig. 3
figure 3

Flow of proposed system for simulation

Full size image

To implement this, first, MRTK is imported to the Unity Editor to generate a UI. The UI is constructed with two parts, i.e. components and tools area and construction area, using some MRTK APIs and Unity Editor in order to provide a good immersive experience for the users. To construct circuits, buildings or robots, few components and tools are required which can be located in the user interface. Once the user starts constructing, the user can easily select the tools and components and drag and drop it down to the construction area. In the construction area, virtual objects can be simulated that have to be projected into the real world and accuracy of the system can be measured. With a view to provide an excellent quality in project of the virtual circuits, buildings or robots, light-weight rendering can be implemented which scrapes out visible lines from the polygons which in turn provides a photo-realistic projection. Next, V touch is employed so that there is improvisation in the haptic sense, providing an excellent immersive experience. To dispense output protection, a technique called privacy-preserving rendering can be employed. Hence, relinquishing a perfect environment for learning both theoretically and practically.

Results

In Fig. 4, the sample prototype of the output can be seen. It comprises students who are the end-users and they use the application that is suggested in the proposed system. The user must use the HMD (Head-Mounted Device) in order to observe the virtual projection onto the physical world. If the student wants to attend regular classes, they can choose the classroom visualization option which will lead the student to projection of the classroom where they can interact with the teacher and learn. If the student wants to conduct any simulation, then the simulation environment option can be chosen.

Fig. 4
figure 4

Prototype of sample output of the proposed system

Full size image

Case-Study Analysis

When the above-mentioned idea was applied to create an mobile application which provided information about various plants so that it would help the agriculturist/farmer to know the ideal time for growing the crop so maximum gain can be expected. It was chosen four basic conditions which are required for the proper crop growth like suitable/ideal season for its growth, soil fertility of the land where the crop is grown, ideal weather conditions required to grow that crop and water requirement/plant. The outcomes of this application can be seen in Figs. 5 and 6.

Fig. 5
figure 5

Image scanned using regular mobile camera

Full size image
Fig. 6
figure 6

Information about Tomato crop projected on to the real world when image was scanned through mobile camera using the application

Full size image

Here, pictures of various crops are using and once the mobile application scans the image of the crop it automatically projects the output. In Fig. 5, it can observe that initially, there is no projection on to the real world, when image was scanned with regular camera, but when the camera scanned the same image by opening the application, a virtual projection of an image on to the real world can be seen which is shown in Fig. 6. Here, it can be observed that are projecting the images as output but in the proposed system, which are going to project the images, 3D objects as well as videos which helps the user to feel the immersive experience.

Hence, using the above application, it can easily build an MR application which will help the user to see the world in a completely different way where they are able to recreate a virtual world which can be completely different from real world, but without affecting the real world.

Conclusions

Learning can be impressive when there is interactivity and immersivity is indulged in the system. The above-proposed methods provide a futuristic approach for education. Therefore, the students are benefited with a new viewpoint, making the learning much easier and more enthusiastic. In the future, these methods can be implemented for the whole education system amidst COVID-19 or similar pandemic and in the time when the world is cured from the pandemic. There are few limitations in this system, for example, when the network slowdowns it directly affects the immersive experience causing whole process to slow down or on continuous usage of head-mounted device may lead to irritation in the eye which may also lead to headache nausea, etc. In near future, it is sure to build a system which has good visual experience and cause less harm to our eye. Despite these limitations, it can be said that it is efficient teaching–learning method with respect to the recent situation.