Intel® Collaboration Suite for WebRTC Simplifies Adding Real-Time Communication to Your Applications

Intel® Developer Zone offers tools and how-to information for cross-platform app development, platform and technology information, code samples, and peer expertise to help developers innovate and succeed. Join our communities for Android, Internet of Things, Intel® RealSense™ Technology, and Windows to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathon’s, contests, roadshows, and local events.

Overview

Web-based real-time communication (WebRTC) is an open standard proposed by both World Wide Web Consortium (W3C) and Internet Engineering Task Force (IETF) that allows browser-to-browser applications to support voice calling, video chat, and peer-to-peer (P2P) data transmission. End users can use their browsers for real-time communication without the need for any additional clients or plugins.

The WebRTC standard is gaining significant momentum and is currently fully supported by open standard browsers such as Google Chrome*, Mozilla Firefox*, and Opera*. Microsoft also announced its Edge* browser support on Object RTC (ORTC), which will be interoperable with WebRTC.

To ease adoption of this WebRTC technology and make it widely available to expand or create new applications, Intel has developed the end-to-end WebRTC solution, Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC). Intel CS for WebRTC is highly optimized for Intel® platforms, including Intel® Xeon® processor-based products such as Intel® Visual Compute Accelerator card, Intel® Core^TM processor-based desktop products, and Intel® Atom^TM processor-based mobile products.

You can download Intel CS for WebRTC from http://webrtc.intel.com at no charge. It includes the following main components:

• Intel CS for WebRTC Conference Server – enables not only P2P-style communication, but also efficient WebRTC-based video conferencing.
• Intel CS for WebRTC Gateway Server for SIP – provides the WebRTC connectivity into session initiation protocol (SIP) conferences.
• Intel CS for WebRTC Client SDK – allows you to develop WebRTC apps using JavaScript* APIs, Internet Explorer* plugin for WebRTC, Android* native apps using Java* APIs, iOS* native apps using Objective-C* APIs, or Windows* native apps using C++ APIs.
• Intel CS for WebRTC User Documentation – includes complete online documentation available on the WebRTC website http://webrtc.intel.com, with sample code, installation instructions, and API descriptions.

Problems with Existing WebRTC-Based RTC Solutions

WebRTC-based RTC solutions change the way people communicate, bringing real-time communication to the browser. However, as a new technology, WebRTC-based solutions require improvements in the following areas to be as complete as traditional RTC solutions.

• Mostly P2P communication based. The WebRTC standard itself as well as Google WebRTC open source reference implementation only focuses on peer-to-peer (P2P) communication, limiting most of the WebRTC-based solutions to two-party communication. Although some WebRTC solutions support multi-party chat, these solutions use mesh network topology, which is less efficient and can only support a few attendees for common client devices.
• Not fully accounting for client usage preferences. Although browsers are available for multiple platforms, not all users like browsers. That is, many mobile platform end-users prefer native apps, such as Android apps or iOS apps. Additionally, some commonly used browsers, such as Internet Explorer, still do not natively support WebRTC.
• Lack of flexibility on MCU server. Some WebRTC-based solutions support multipoint control unit (MCU) servers for multi-party communication. However, most of those MCU servers use a router/forward solution, which just forwards the publishers’ streams to the subscribers. Although this method fulfills part of the scenarios when clients have equivalent capabilities or SVC/Simulcast is supported, it becomes a high requirement for clients to easily meet. To work with a wide variety of devices, MCU servers must do some media-specific processing, such as transcoding and mixing.
• Limited deployment mode choices for customers. Most of the existing WebRTC-based RTC solutions work as a service model hosted by service providers. This style provides all the benefits of a cloud service, but is not useful for service providers and those who want to host the service themselves for data sensitive consideration.

Key differentiation of Intel® CS for WebRTC

Fully Functional WebRTC-Based Audio/Video Communication

Intel CS for WebRTC not only offers peer-to-peer WebRTC communication, but it also supports WebRTC-based multi-party video conferencing and provides the WebRTC client connectivity to other traditional video conferences, like SIP. For video conferencing, it provides router and mixer solutions simultaneously to handle complex customer scenarios. Additionally, it supports:

• H.264 and VP8 video codecs for input and output streams
• MCU multi-streaming
• Real-time streaming protocol (RTSP) stream input
• Customized video layout definition plus runtime control
• Voice activity detection (VAD) controlled video switching
• Flexible media recording

Easy to Deploy, Scale, and Integrate

Intel CS for WebRTC Conference and Gateway Servers provide pluggable integration modules as well as open APIs to work with existing enterprise systems. They easily scale to cluster mode and serve larger number of users with an increase of cluster node numbers. In addition, the Intel solution provides comprehensive client SDKs including JavaScript SDK, Android native SDK, iOS native SDK, and Windows native SDK to help customers quickly expand their client applications with video communication capabilities.

High-Performance Media Processing Capability

Intel CS for WebRTC MCU and Gateway servers are built on top of Intel® Media Server Studio, optimized for Intel® Core™ processors and Intel® Xeon® processor E3 family with Intel® Iris™ graphics, Intel® Iris™ Pro graphics, and Intel® HD graphics technology.

The client SDKs, including the Android native SDK and Windows C++ SDK, use the mobile and desktop platforms’ hardware media processing capabilities to improve the user experience. That is, the Android native SDK is optimized for Intel® Atom™ platforms (all Intel® Atom™ x3, x5, and x7 processor series) focusing on video power and performance, as well as end-to-end latency. The Windows C++ SDK also uses the media processing acceleration of the Intel® Core™ processor-based platforms (i3, i5, i7) for consistent HD video communication.

Secure, Intelligent, Reliable QoS Control Support

Intel CS for WebRTC solution ensures video communication data security through HTTPS, secure WebSocket, SRTP/DTLS, etc. Also the intelligent quality of service (QoS) control, e.g., NACK, FEC, and dynamic bitrate control, guarantees the communication quality between clients and servers against high packet loss and network bandwidth variance. Experiments listed in Figure 1 have shown that the Intel video engine handles up to 20% packet loss and 200ms delay.

Full Functional Video Communication with Intel CS for WebRTC Conference Servers

Flexible Communication Modes

Intel CS for WebRTC offers both peer-to-peer video call and MCU-based multi-party video conference communication modes.

A typical WebRTC usage scenario is direct peer-to-peer video call. After connecting to the signaling server, users can invite other parties for P2P video communication. All video, audio, and data streams are transported directly between each other. Meanwhile, the signaling messages for discovery and control go through the signaling server. As Figure 2 shows, Intel provides a reference signaling server implementation called Peer Server with source code included. Customers can construct their own signaling server based on this Peer Server or replace the whole Peer Server with an existing channel. The client SDK also provides the customization mechanism to let users implement their own signaling channel adapter.

Intel CS for WebRTC solution further offers the MCU-based multi-party video conference chat. All streams go through the MCU server the same as the signaling messages do as Figure 3 shows. This reduces the stream traffic and computing overhead on client devices compared to a mesh network solution.

Unlike most existing WebRTC MCUs, which usually work as a router to forward media streams for clients, Intel CS for WebRTC MCU server also handles the media processing and allows a wide range of devices to be used in the conference. Users can subscribe to either the forward streams or mixed streams from MCU server. Based on Intel Iris Pro graphics or Intel HD graphics technology, media processing on the MCU server can achieve excellent cost-performance ratio.

The Intel MCU provides more flexibility on mixed streams. You can generate multiple video resolution mixed streams to adapt to various client devices with different media processing capability and network bandwidth.

External Input for RTSP Streams

Intel CS for WebRTC allows bridging a wider range of devices into the conference by supporting external inputs from RTSP streams. This means almost all RTSP compatible devices, including IP cameras, can join the video conference. The IP camera support opens up usage scenarios and applications in security, remote education, remote healthcare, etc.

Mixed-Stream Layout Definition and Runtime Region Control

Through Intel CS for WebRTC video layout definition interface, which is an expanded version of RFC-5707 (MSML), you can define any rectangle-style video layout for conference, according to the runtime participant numbers. Figure 4 shows the video layout for one conference. The meeting contains 5 different layouts with 1, 2, 3, 4, or 5-6 participants.

Figure 5 describes the detailed layout regions for a maximum of 2 participants. The region with id number 1 is always the primary region of this layout.

Intel CS for WebRTC MCU also supports automatic voice-activated video switching through voice activity detection (VAD). The user most active on voice is switched to the primary region which is the yellow part of Figure 6.

You can also assign any stream to any region as needed during runtime for flexible video layout design of the conference.

Flexible Conference Recording

When recording in Intel CS for WebRTC, you can select any video feed and any audio feed. You not only can record switching across different streams that the conference room is offering (such as mixed and forward streams), but also select video and audio tracks separately from different streams. You can select the audio track from the mixed stream of participants and video track from the screen-sharing stream.

Scaling the Peer Server Reference Implementation

Although the Peer Server that Intel provides is a signaling server reference implementation for signal node, you can extend it to a distributed and large scale platform by refactoring the implementation. See Figure 7 for a scaling proposal.

Scaling the MCU Conference Server

The Intel CS for WebRTC MCU server was designed to be a distributed framework with separate components, including manager node, signaling nodes, accessing nodes, media processing nodes, etc. Those components are easy to scale and suitable for cloud deployment.

Figure 8 shows an example from the conference server user guide for deploying an MCU server cluster.

Interoperability with Intel CS for WebRTC Gateway

For legacy video conference solutions to adopt the WebRTC advantage on the client side, Intel CS for WebRTC provides the WebRTC gateway.

Key Functionality Offering

Intel CS for WebRTC gateway for SIP not only provides the basic signaling and protocol translation between WebRTC and SIP, it also provides the real-time media transcoding between VP8 and H.264 to address the video codec preference difference between them. In addition, the gateway keeps the sessions mapping between WebRTC and SIP to support bi-directional video calls. Figure 9 briefly shows how SIP devices can connect with WebRTC terminals through the Gateway Intel provided.

Validated SIP Environments

Note: See Intel CS for WebRTC Release Notes for current validated environments

Cloud Deployment

The Intel CS for WebRTC gateway instances are generally session-based. Each session is independent, so sessions are easily scalable to multiple instances for cloud deployment. You can make the gateway instance management a component of your existing conference system scheduling policy and achieve load balancing for the gateway.

Comprehensive Intel CS for WebRTC Client SDKs

The Intel CS for WebRTC also provides comprehensive client SDKs to help you easily implement all the functionalities that the server provides. The client SDKs allow client apps to communicate with remote clients or join conference meetings. Basic features include audio/video communication, data transmission, and screen sharing. P2P mode also supports a customized signaling channel that can be easily integrated into existed IT infrastructures.

Client SDKs include JavaScript SDK, Android SDK, iOS SDK, and Windows SDK. Current features are listed in Table 1.

Table 1. Client SDK Features

Customized Signaling Channel

In addition to the default Peer Server, Intel CS for WebRTC client SDK for P2P chat provides simple customizable interfaces to allow you to implement and integrate with your own signaling channel through the extensible messaging and presence protocol (XMPP) server channel. Figure 10 shows there is a separated signaling channel model in client SDK for P2P chat and allows user to customize.

Hardware Media Processing Acceleration

On Android platforms, VP8/H.264 decoding/encoding hardware acceleration is enabled if the underlying platform includes corresponding MediaCodec plugins. For Windows, H.264 decoding/encoding and VP8 decoding hardware acceleration is enabled with DXVA-based HMFT or Intel Media SDK. For iOS, H.264 encoding/decoding is hardware-accelerated through Video Toolbox framework. Table 2 below shows hardware acceleration for WebRTC on different platforms.

Table 2. Hardware Media Acceleration Status for Client SDKs

^#Conditional support: only enabled if the platform level enables VP8 hardware codec

NAT Traversal

Interactive Connectivity Establishment (ICE) helps devices connect to each other in various complicated Network Address Translation (NAT) conditions. The client SDKs support Session Traversal Utilities for NAT (STUN) and Traversal Using Relay NAT (TURN) servers. Figure 11 and Figure 12 show how client SDKs perform NAT traversal through STUN or TURN servers.

Fine-Grained Media & Network Parameter Control

Client SDKs further allow you to choose the video or audio source and its resolution and frame rate, the preferred video codec, and maximum bandwidth for video/audio streams.

Real-Time Connection Status Retrieval

Client SDKs provide APIs to retrieve real-time network and audio/video quality conditions. You can reduce the resolution or switch to an audio only stream if the network quality is not good, or adjust audio levels if audio quality is poor. Table 3 lists connection status information supported by client SDKs.

Table 3. Connection Status Information supported by Client SDKs

Conclusion

Based on WebRTC technology, Intel® Collaboration Suite for WebRTC builds an end-to-end solution, allowing you to enhance your applications with Internet video communication capabilities. The acceleration from Intel’s media processing platforms on the client and server sides, such as the Intel® Visual Compute Accelerator, improves the client user experience as well as the server side cost-effectiveness.

Additional Information

For more information, please visit the following web pages:

Intel Visual Compute Accelerator:
http://www.intel.com/content/www/us/en/servers/media-and-graphics/visual-compute-accelerator.html
http://www.intel.com/visualcloud

Intel Collaboration Suite for WebRTC:
http://webrtc.intel.com
https://software.intel.com/en-us/forums/webrtc
https://software.intel.com/zh-cn/forums/webrtc

The Internet Engineering Task Force (IETF) Working Group:
http://tools.ietf.org/wg/rtcweb/

W3C WebRTC Working Group:
http://www.w3.org/2011/04/webrtc/

WebRTC Open Project:
http://www.webrtc.org

Acknowledgements (alphabetical)

Elmer Amaya, Jianjun Zhu, Jianlin Qiu, Kreig DuBose, Qi Zhang, Shala Arshi, Shantanu Gupta, Yuqiang Xian

About the Author

Lei Zhai is the engineering manager in the Intel Software and Solutions Group (SSG), Systems Technologies & Optimizations (STO), Client Software Optimization (CSO). His engineering team focuses on Intel® Collaboration Suite of WebRTC product development and its optimization on IA platforms.