Data Stream Network

---

Data Stream Networks (DSN) are global networks that facilitate realtime communication between internet-enabled physical devices such as mobile phones, computers, vehicles, or other objects with embedded sensors. The DSN coordinates the streams of data coming from and going to each node.

Similar to how content delivery networks (CDNs) allow developers to serve content at scale by distributing it to servers closer to their users, DSNs assist developers whose applications produce or consume realtime data. Developers who use DSN are not required to create their own capabilities to handle the realtime streaming of data. The DSN uses distributed endpoints and orchestration to ensure the correct data is ingested from and/or pushed to each device, using the nearest available Point of Presence.

Process

A data stream (referred to as a channel or topic) is an abstraction that enables real-time, continuously shared information between multiple endpoints. Each endpoint connects to the channel, which enables the device to both publish (add information to the flow) and subscribe (have data pushed to it from the flow).

A DSN maintains, orchestrates and controls a functionally infinite number of channels. It ensures connected devices can locate and interact with the channels that they require to function. Typically served as a hosted network, a DSN is redundant, highly scalable and caches data close to its clients. These features ensure the connecting endpoints have access to all the data flowing through the DSN in near-real time, regardless of location or type.

• Client devices connect to the DSN using authentication to identify themselves.

• Client devices request access to channels.

• The network determines which data channels each device is allowed to access. Multiple clients and servers can connect to one or more channels.

• Data published on channels is pushed out in near real-time to all subscribers. Additionally, where a DSN spans multiple regions, federation of data is often required between channels in one region to another, allowing the same data to be fanned out to one or more regions and their subscribers in near real-time.

Applications for DSNs

DSNs are used with heterogeneous, loosely-coupled software, creating a delivery layer between applications, devices and people. Applications include location-tracking, visualization, the use of IoT sensor data, chat, in-app or push notifications, online multiplayer games, gambling, and live audio and video streams.

DSNs are used in cases where the value of data decreases as latency increases, such as in-play sports betting. .

In this scenario, data must be delivered within a very short time frame, typically a few milliseconds. A DSN, using modern communication protocols such as WebSockets, MQTT and Long Polling, can achieve this.

Technology and history

Historically, web applications request data from a server, which responds accordingly. This has required traditional methods to move data to its destination following an HTTP request/response cycle. Conversely, many modern applications use streaming data, which is produced continually. In these use cases, request/response cycles are notefficient in transporting data because of the overheads that result by setting up these cycles. These include increased latency, greater bandwidth use (caused by devices continually asking for new data) and higher server costs from large numbers of devices polling for new information.

Long Polling and WebSockets make the web more realtime by employing the same HTTP protocols as the Internet.

In Long Polling, a request can persist until the server has data to send in response. This isn’t viable in all scenarios, since it still produces successive request/response cycles. However, if the frequency of the request/response is reduced , the net effect compared to the constant ‘heartbeat’ required on a streaming data protocol is small.

With WebSockets, the HTTP connection is upgraded to a full duplex socket connection that enables both sides to push data in either direction. Once established, this is as efficient as other socket connections and has the advantage of being compatible with existing and widely-supported HTTP technology.

The WebSockets approach does not require the endpoint to continually request data from a server. Instead, the server must inform applications when new data is available.

DSNs use WebSockets or other protocols that enable a persisted duplex connection that remains open while an application is running. A persistent connection eliminates the delays caused by repeatedly setting up request/response cycles. It responds immediately when either side of the connection needs to send data, thereby reducing data delivery latency. Because it requires very low overhead to maintain, it is  energy- and bandwidth-efficient.

DSNs implement their own communication protocols on top of web protocols (HTTP and WebSockets). DSNs support other open protocols, including:

• MQTT

• XMPP

• Bayeux Protocol

• AMQP

• STOMP

Types of Data Stream Networks

DSNs are either single-location or globally distributed.

Single-location Data Stream Network

A single-location DSN operates from within a single data center that handles all traffic. Single-location DSNs don’t offer redundancy in the event of a problem at the data center. Moreover, the lack of a global footprint means single-location DSNs do not provide the benefits of data that is in close proximity to a worldwide userbase.

Examples of single-location DSNs are Pusher, Reappt by Push Technology, Fanout, Streamdata, PushFYI.

When operated at scale, open source solutions such as SocketCluster or Socket.io remain single-location DSNs, and consequently do not address availability or data federation across multiple regions.

Globally-distributed Data Stream Network

A globally-distributed DSN contains multiple geographically-distributed data centers. Each communication request from an endpoint is routed to the nearest data center, minimizing latency in in contrast to single-location DSNs. This arrangement also increases the reliability of the DSN through multi-data-center redundancy and constant synchronization, in the event of a regional failure.

Examples of globally distributed DSN providers are Ably Realtime and PubNub.

DSN service providers

• PubNub

• Ably Realtime

• Pusher

• Reappt by Push Technology

• Fanout

• Streamdata.io

• PushFYI

Further reading

• Content Delivery Network

• WebSocket

• Long Polling

• API Management

References

This article "Data Stream Network" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Data Stream Network. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

---