Benefits of 3D live streaming The next area of interest is what benefits can 3D live streaming provide that will push consumers away from the current low level 2D streaming technology or physically being at an event or environment. For consumers, there are numerous benefits that can be achieved through a 3D live stream event. One can think of it as the closest thing to actually being present at an event or environment. Instead of viewing a static 2D screen, or being on location with no real interaction or things to explore, one could feel fully immersed in a 3D environment with the added perk of being able to interact with other consumers attending the event. In terms of gaming, it will be possible to actually “be inside” a game and to witness the game in its true 3D beauty at any desired location. A similar thing can be said for fans of virtual reality – to have a 3D live stream of a virtual environment or event is essentially the holy grail.
What is 3D live streaming? First thing is first – what is 3D live streaming? As the technology is relatively new, it might be surprising to find that there are many people who do not quite grasp the concept of 3D live streaming. In simplest terms, to 3D live stream an event or environment means to capture all aspects of said event or environment in 3D and live stream the data to an end user. When the end user receives the data, they will be able to view the event or environment in 3D from their computer or device, and in some recent cases with use of a virtual reality HMD (head mounted display). The end goal of the technology is to have the end user feel fully immersed in a distant event or environment in true 3D. With the recent demand for virtual experiences and the push for virtual reality, many believe 3D live streaming will be an integral part of our shift from actual to virtual experiences in the modern era.
What is 3D Live Streaming?
But what place does live streaming have in the 3D world? There are virtual worlds entirely dedicated to live events, and some are so elaborate that they involve the participation of thousands of residents. The 3D Web, built on technology such as X3D, is the standard for web-based virtual worlds and web content. The content of the 3D Web often mirrors the nature of reality, with an abundance of cities, meeting places, educational facilities, corporate environments, and entertainment venues. Many individuals within these communities are interested in viewing and creating live 3D content, and it’s said that the demand for live streaming has always been present. That’s where 3D live streaming comes into play.
Live streaming is growing to become a valuable niche in the virtual worlds. Live streaming is a way of broadcasting events, sharing views, indulging in chats on all sorts of topics, and reaching the masses, which have taken a sharp turn with the arrival of social media. For the everyday consumer, live streaming is a means to view content that is not broadcast in their area or that requires an event ticket, such as music concerts. And for organizations or individuals, live streaming serves as a tool to make events, lectures, or workshops accessible to those who are unable to attend in person.
Benefits of 3D Live Streaming
One of the main advantages of 3D live streaming is giving the viewers an option to take part in events that they cannot physically attend. The most obvious example of this would be sporting events. It’s fairly unrealistic to think that you could get a ticket to the World Cup final or the Super Bowl. Though most people would like to attend such events, most can’t, either because the events are held in a foreign country or because the cost of tickets is too high. All the same, the majority of fans are very eager to take in these events, and 3D live streaming allows them to do so in a way that is more involved than traditional TV broadcast. The broadcasts could be controlled in the same way as a game that is being played, with the camera angles being changed as the action takes place. This would give virtual spectators a very involved experience. For games like the World Cup, special virtual reality game clients could be developed allowing a viewer to take part in a 3D live streamed game while the game being played decides the actual outcome of the event. This could attract many fans of the sport who are also gamers and would give fans of competing nations an opportunity to participate in some friendly virtual warfare.
The Rise of 3D Virtual Live Streaming Services
The VR market, at its current state, is limited by its lack of content and its relatively small user base compared to traditional media platforms. Despite this, it is a market that is growing at a rapid pace, and many believe it to be the future of media consumption. With the increased functionality of 3D live streaming, it has the potential to be the enabling factor for VR to become a mainstream media platform.
Static 2D or 2.5D media cannot achieve the full sense of realism and immersion offered by high presence and varied dimension content. With recent advances in VR technology, there is a strong push for the development of 3D virtual live streaming to suit the needs of VR users. High presence and varied dimension 3D content can be more closely matched to its real-world equivalent. This content can then be viewed within VR with a much more seamless transition between real-world and virtual experiences.
By varying the dimensions of 3D virtual live streaming, it can add a greater sense of depth to the user’s experience. Depth can be varied by the range of viewing angles that users have and how far they can move within a CG environment. With 3D live streaming, the range of viewing angles can more closely match that of a real-world event with a pan-able and zoom-able camera rig. This can bring the viewer closer to the action compared to a traditional fixed camera setup. Increasing the viewer’s capacity to move around within an environment can be used for applications such as streaming a music festival or sports event in which a user may have a particular area they want to view or move around to capture the complete experience.
The main benefits of 3D virtual live streaming service is that it can provide an immersive experience and allow for a greater sense of presence. In the VR context, where a user is experiencing a computer-generated environment, an event, or a live performance, the higher the sense of presence, the greater the perceived realism of the content. High presence can lead to a psychological state of flow, being fully engrossed in the experience. This can then lead to greater cognitive and behavioral impact from the experience.
Evolution of Virtual Live Streaming Technology
An example of 3D virtual live streaming technology using VR is seen below.
Coming back to current virtual live streaming, shoutcasting is now used as a method to provide extra commentary (an alternative to the player’s voice) and camera control for games which still don’t support virtual/live spectating and shoutcasters can be used as remote camera men and dialogue providers for in-game events. Action real-time strategy (ARTS) games such as Dota 2 and League of Legends have been a common target for this type of game casting.
The methods of streaming and shoutcasting were already detailed in a 2005 article on PlayOn about the Platform for Better Experience (PBE) and the way to set up a stream with Winamp and Shoutcast plugin to provide 2D viewing of LoL games. This method of streaming is still used by some eSports communities today to broadcast lower budget tournaments and still serves as a simple method of casting gameplay (Macleod, 2005).
The earliest forms of virtual live streaming began with the streaming of 2D audio and video which was (and still is) the output of many video games. At this point, it was known as shoutcasting. Shoutcasting is an old term used to describe the action of casting a video game from the player’s computer to a spectator’s client, using voice over internet protocol (VOIP) communication software, so that the client can hear and provide feedback on what is happening in the game.
Augmented reality and virtual reality technology has been growing at a rapid pace, changing the way we consume media or even product advertisements (Davenport, 2018). It is no surprise that virtual live streaming was introduced and evolved to include VR and AR elements.
Applications of 3D Virtual Live Streaming
The 3D visualization and animation technology has matured enough to produce compelling 3D content that has the potential to enhance user experiences in many application areas. 3D live streaming solutions are still in their infancy as this is somewhat of a niche area servicing users who demand the most compelling and highest quality 3D content, streamed in real-time to their browsers. One application of 3D live streaming is for broadcasting 3D TV channels over the Internet. A key challenge for traditional broadcasters of 3D TV has been finding a sufficiently large target audience with the appropriate TV set technology to justify maintaining a 3D TV channel. With the growing trend of ‘cord-cutting’, traditional pay-TV subscriptions are becoming less attractive to consumers who are now accustomed to on-demand content streamed over the Internet to their mobile devices and tablets. A 3D TV channel delivered over the Internet can circumvent these issues by targeting consumers who are accustomed to OTT content delivery. Transferring existing on-demand 3D video libraries into an appealing 3D virtual VoD (Video on Demand) service is another key market for 3D live streaming solutions. The work in [11] describes a system for creating a navigable 3D virtual world from a set of 3D videos by stitching them into a single 3D environment. High quality 3D content is also used in the video game industry to create cinematic sequences that are used to drive a game’s story and plot. An in-game 3D live streaming solution could leverage this type of content to bring TV-like viewing experiences to the game’s players. Considering the rapid rate of growth for the global games market [10], this is another potential niche for 3D live streaming solutions. High quality 3D content streamed in real-time can also enhance existing virtual reality (VR) systems and applications. Although it seems somewhat counter-intuitive to use a 3D live streaming solution with an already immersive 3D environment, many VR applications are built on a 3D game engine with content that is not of the same visual quality as the VR system itself. An on-demand content delivery system such as [12] can automatically replace lower quality game assets with equivalent higher quality assets as they become available, thus providing a potential context for a 3D live streaming solution.
Singapore’s Role in the Development of 3D Virtual Live Streaming
In the recent years mentioned before, IDA played the crucial role of researching and developing technologies and strategies enhancing the use of 3D virtual live streaming. It had developed a few prototype systems such as the 3D streaming server to allow users to have a feel of the immersive experience and also an early version of the IDA Java3D chat system to familiarize with the techniques of integrating the 3D content with the chat server and showcasing it to the user. These were a few examples of research areas carried out by IDA. These ideas lead to the development of DISTINCT. A serious game targeting National Education and geared towards the teaching of Secondary 3 students on the river civilizations subject in their Social Studies syllabus. DISTINCT makes use of custom created 3D models of the ancient civilizations and Java based voice chat to provide an interactive and immersive learning environment for the students. This was a huge success and it won the Gold award in the Web3D category at the 2008 Singapore IT awards. A research collaboration with a local start-up company called Gattai Games sparked off the interest in the area of 3D games leveraging on the MMO (Massive Multiplayer Online) technology. The target was to develop a fully functional 3D game prototype with basic MMO capabilities within a span of eight months. Gattai Games has been actively involved in the local game development industry in Singapore. This collaboration is expected to be the kick-start of more interactive games and games technology in Singapore’s future. Last but not least, there are other ongoing research activities and future plans that pave the way to the development of 3D games and virtual worlds in local institutions and the private sectors in Singapore.
Key Features and Technologies in 3D Live Streaming Services
To put it simply, 3D live streaming is a far more complex process than typical streaming or video call. It requires a whole new set of tools to create the immersive experience that the technology promises. For gaming and interactive media, this means interpreting the game world to produce the 3D environment, while also capturing player movements and other interactions. This is typically done using middleware to reduce development time, though more complex integration with the game may be required for unique games or if non-standard hardware is to be supported. For film, TV, and events in the real world, it involves the use of 3D cameras to use live action technology. Various rig and motion capture technologies may be used to capture actor performances and apply them directly to 3D models. In both cases, the biggest technical goal is a real-time 3D environment that feels as authentic as possible. This is achieved using one of two methods – offline rendering and real-time rendering. The method will depend on the project and budget; real-time rendering is ideal for interactive media, providing instant feedback on the quality and performance of the game or product. Offline rendering can take a considerable amount of time due to the high quality that is typically expected from 3D work, but this method can now be used effectively for pre-rendered video, ads, and cinematics in video games. For live events, the 3D data must be processed real-time, and this is where current 3D live streaming services are focusing their technology. An example of this is the rendering and live streaming of in-game esports events; something which is becoming increasingly popular.
Real-time Rendering and 3D Modeling
With traditional pre-rendered 3D graphics, whether they be cinematic or merely a still image, the time necessary to prepare an image can be arbitrarily long, as long as the information is prepared in advance of the time it is to be displayed. Because the content does not need to be displayed in real-time, it can be prepared and re-prepared until the desired visual quality is achieved. This can be done with very little direct user interaction, or it can be simulated within a more complex real-time structure, such as a physics engine or a behavioral AI. But the bottom line is, it does not need to be rendered at the same time it is being displayed. Because these methods and the tools and hardware used for them are very diverse, and expert knowledge on the part of the reader cannot be assumed, the following will primarily discuss the methods and technologies for real-time rendering, and the fashioning or preparation of 3D models or content to be displayed and rendered in a real-time context.
Rendering a 3D atmosphere from a few frames of reference, or from the simple models, might take a few minutes to many hours of computation. For time sensitive applications, such as interactive real-time simulations, it is infrequent to expect the user to spend inordinate amounts of time waiting for a single frame of a 3D model to be rendered. This is because the computational complexity of rendering a 3D scene is trivial compared to the complexity of real-time interaction with said scene. In the case of rendering, the primary goal is to prepare information for a visual image, and do so at the best possible level of visual quality, or realism, in the least amount of time. This way, the excess computational capacity can be spent on increasing the quality of the visuals, or to attain a higher level of performance, such as increased speed of interaction, or steadier frame rates for an interactive application.
Multi-camera Capture and Live Stitching
Camera systems are one of the most fundamental aspects of video production, providing the means to capture and represent events in the real world. In traditional video production, single camera systems have been the standard for many years, providing a simple and effective means to record events. In recent years, the development of 360-degree camera systems has provided an interesting alternative, allowing the capture of events in a fully panoramic manner. This can be particularly useful in the capture of events from a fixed point such as a sports event, where the viewer has the ability to freely look around the environment. The use of multiple standard or 360-degree cameras has the potential to provide a good representation of a 3D environment to be used in multi-camera 3D live streaming. Live stitching involves the use of software which can automatically piece together video feeds from multiple camera systems to provide a seamless panoramic video stream. This can provide a compelling alternative to 3D model-based live streaming as it can provide a much more realistic representation of the real world. Live stitching may be appealing to content providers who are interested in providing a 3D stream of an event or location in the real world, as it can be more cost-effective and less time-consuming than 3D modeling and also appeal to an audience who are already familiar with traditional video media.
Immersive Audio and Interactive Elements
Ambisonic audio was a unifying theme for the VR at GDC, as it’s adaptable to many speaker layouts, future-proofable, and provides an immersive experience. Dolby Atmos for VR also looks very promising. Both are the best audio formats that can be utilized by 3D live streaming services, where audiences can experience audio clarity and depth that matches the visual they can see, integrating seamlessly with any interactive elements triggered by the audio itself. This type of audio is what makes the difference between flat 2D media and full realistic immersion in a virtual environment.
Immersive audio experience that provides a surround sound effect can create a place illusion, taking audiences to virtual spaces where the sound was supposed to be heard, at the angle that sound would come from. Audio can be integrated into 3D live streaming environment and connect to certain events that trigger the sound to be played, in the form of ambient sound or a specific sound source. The mission to simulate the real world is also shown by the use of real-time 3D positioning of sound sources, binaural and HRTF (Head-Related Transfer Function) audio rendering.
Compatibility and Accessibility
Finally, data casting services such as this must often consider geographic location given restrictions on certain technologies in specific countries. Step ladder 3D has considered this and realized early that China is a key market for advanced visual techniques and holds major international events. Aiming for compatibility is crucial to the future success of these new technologies.
Compatibility also relies on the viewer’s end, and with a wide range of devices being used to view content, it is important that the streaming service is available to as many people as possible. This is now the time for mobile data casting and stepped 3D content, and this mobile phone or tablet viewing service is often being aimed at secondary VR experience in order to further place the viewer in the event, even without a headset. Step 3D content, we can imagine, is marked by increasing levels of immersion. Step 1 may be as simple as a 3D re-render of an existing 2D video. Step 2 could be multi-camera 3D content with free roam viewing. Step 3 could be as advanced as holographic projection. This type of 3D content will need to be progressively more accessible and compatible with any future VR devices. Primary to this is the integration of multi-camera 3D content with existing 2D live streams of modern sports and events.
Compatibility and accessibility are always important factors when considering any new technology. With the exponential increase of online streaming services, compatibility and accessibility only increase as a wider audience and broader locations are being reached. Prior to the event, or ‘location scouting’, 3D live services will visit the venue and ensure it is compatible with their live streaming scheme. This will often involve consulting with venue staff to ensure that technology such as VR cameras don’t interfere with the event itself or sport fixtures. Step ladder 360’s tennis match is a good example of where consulting the venue can avoid issues in later data casting. A pre-event site visit will also help produce a 3D model of the venue which can be used to aid camera planning and data cast event, and also used later on for VR content.
Key features and technologies in 3D live streaming services include a range of methods to digitize any location and convey it to the ‘viewing’ audience. From current affairs and sports to full VR experience, 3D Live offers a host of technologies to bring viewers closer to the action. With continuous advances in visual technology such as VR and AR, streaming services have had to adopt these 3D techniques to stay relevant and help place the viewer in the director’s chair.
Future Prospects and Challenges in 3D Live Streaming
To provide real-virtual world interaction, more complex VR and AR features need to be applied in 3D live streaming. An example is the synchronization of game control with a chat server, which can provide an output of the chat as a character voice dialogue with lip sync and a chat log on the game screen. The chat server is also able to recognize chat words that are related to commands to perform in-game actions. This complex feature is a challenge to make the VR or AR version look and feel similar to the non-VR or non-AR version of the same game. It obviously requires separate development and additional resources, but it will provide a new experience for gamers who are using VR or AR. From another perspective, 3D live streaming can also be a tool to create business simulations in a real-virtual world environment, using real-world data information. This could be a trend in the future for real-world companies or institutions that want to simulate scenarios before taking real actions, with obviously less cost and risk. The same also applies to online education, with an attractive virtual learning environment, which is currently a phase in today’s e-learning.
However, integrating 3D live streaming with VR or AR is not a trivial task. Basic 3D display and interactive features with simple head tracking are already supported in some online games. For non-streaming 3D games, the game is pre-rendered to produce separate images for the left and right eye, which are then displayed using the appropriate technologies. This approach doesn’t require extra resources when playing the game in the future. However, with 3D live streaming, the data should be close to the original game data, usually in a 2D format. Then, the original game is processed so that it can produce separate images for the left and right eye to be streamed to the client. This process consumes some resources for the broadcasters since there are two parallel streaming processes, and it requires specific hardware that is still rare. In another case, with highly hardware-computed game environments and objects, it is challenging to achieve equivalent visual and audio quality for both the streaming and non-streaming versions. This could result in additional development costs and may not provide an instant return on investment. However, in the long term, it will align with the development of VR and AR technologies. This can be facilitated by the high demand for VR and AR devices that has been discussed before, so it’s just a matter of time.
With the increasing awareness in using VR and AR technologies not only in gaming and applications, but also in a broader reality context, there would be more demand for 3D content, including 3D live streaming. The compatibility between 3D live streaming and VR or AR applications is quite natural because 3D live streaming acts as a bridge between the real world and the 3D virtual world. It is expected that in the near future, many VR and AR devices will be readily available in the market and more affordable for the general public. This would open up new possibilities and a wider audience for 3D live streaming services. The increased demand for VR and AR devices would also boost the development of 3D content and related services, including 3D live streaming, to meet the expectations of end users who are looking for higher quality and real-virtual world interaction in their VR or AR applications. Broadcasters and content providers would also have new opportunities to create more attractive content and open up new market segments in VR or AR based 3D content.
Advancements in virtual reality and augmented reality
Advancements in Virtual Reality and Augmented Reality
With the advent of 3D live streaming, there are a multitude of possible prospects that could come to fruition in the future. One of the most significant possibilities will be the integration of 3D live streams with VR and AR technology. By combining these two technologies, the capability for virtual experiences will be much more immersive and interactive than it is now. With VR technology, individuals could potentially enter a full 3D recreation of a live stream such as a sporting event. This would allow for almost perfect immersion into the environment with the ability to look around and truly take in the atmosphere of the event. Even for streams that are not recreated in an entirely 3D environment, the capability to view the stream in a virtual environment such as a movie theater could be an interesting alternative to watching it on a monitor. The environment could be highly customizable and the user could potentially invite friends to join them in the virtual theater. With AR technology, there could be a variety of interesting applications to enhance the experience of viewing a live stream in a 3D environment. One simple example could be the ability to anchor the virtual viewing environment to a physical object, such as a poster, so that the stream could be viewed by looking at the object through an AR device. More complex possibilities could involve integrating AR with AI technology to create live translation of events into the user’s native language with accurate subtitles.
Overcoming Bandwidth and Latency Issues
Currently, streaming of highly compressed data using adaptive bit rate streaming (HLS, Adobe HDS) is being conducted mainly in the form of on-demand video, but we can expect that it will take a while until it is officially put into practical use for 3D live streaming. For streaming of the same level of quality as 2D video, a network environment of 2-4 times the amount of data is necessary and it is believed that such an environment will become on par with the current network environments in about 10 years. Therefore, until the network environment catches up to the necessary level, streaming may be achieved by downloading the same day’s video on a server to a user’s terminal while simultaneously compressing it, and then playing back the local file. This method can also be considered one form of remote rendering.
As for the ideal amount of data, experiments have shown that the sense of depth in 3D video starts to become satisfactory at five times the amount of data in 2D video, but the sense of stereovision is lost if it is not at least 20 times the amount of data in 2D video. In other words, there is no difference to watching 2D video, and an appropriate amount of data is required in order to watch 3D video. With recent advances in video compression technology, up to 2D 4K/8K video can now be sufficiently processed. Even if that is the case, development of compression technology that makes it possible to store the same level of quality as 2D video in less than 1/5 the amount of data is required. Compression technology that can also work under low delay is desired for compressed video transmission over the internet, but for 3D live streaming we will assume that it is sufficient if it enables storage and playback of video data at the same level of quality as 2D video.
In 3D live streaming, it is inevitable that the size of video data will be larger than 2D video data due to the additional depth information. Therefore, one of the biggest challenges is the bandwidth issue. Users would need a high-speed broadband connection to watch 3D live streaming. However, according to a survey, most participants (70%) have an internet connection speed of 1.5Mbps or less at home. A rate of 1.5Mbps is actually not sufficient for smooth playback of 2D video, and it goes without saying that it is far from enough for 3D video. This means more than 70% of the population cannot currently receive 2D video in the ideal state. It is said that information is becoming increasingly biased due to the digital divide, and there is a concern that a gap will also form between 2D and 3D video.
Privacy and Security Concerns
With the deployment of a densely sensor-populated atmosphere in the virtual world, it is important to ensure the privacy of users and prevent tracking of their behavior beyond their awareness and control. Presently, a user’s behavior can be tracked to be recreated or analyzed later with or without the user’s consent. The goal of capturing and analyzing a user’s behavioral data in the virtual world is to be able to understand the user in order to serve them better. This includes the ability to understand the “way in which they think” as they interact with a sensor or virtual representation of a product, the ability to assess the value of the user to someone else such as a marketer, and the ability to document what the user did and why in order to recreate it. At one end of the tracking and analysis spectrum, it is valuable to enable someone to done something that a user liked so it can be recreated, to understand a disability or impairment in order to design better assistive technology, or to be able to know where a broken virtual object originally came from and who was using it. Coming into play in this aim to learn and understand, are privacy and security concerns that arise from the fact that behavioral data has varying sensitivity and there are generally strict policies and regulations regarding how data about people can be collected, analyzed, and stored. The concept of “informed consent” digs a deep ruruals the difference between the nature and extent of the information that can be collected and the control that the user has over that information and its use. Users may be weary that data about them will be shared to the users of a virtual environment.
