Stream for DolphinDB
Real-time stream processing handles a continuous "stream" of data as the data is published. The process includes real-time data collection, cleaning, analysis, warehousing, and displaying of results. The stream-based applications include trading, social networks, Internet of Things, system monitoring, and many other examples. DolphinDB's built-in stream processing framework is efficient and convenient, which supports stream publishing, subscription, data preprocessing, real-time in-memory calculation, complex window calculation, stream-table joins, anomaly detection, etc.
The advantages of Stream for DolphinDB over other streaming analytics systems are:
- High throughput, low latency, high availability
- One-stop solution that is seamlessly integrated with time-series database and data warehouse
- Natural support of stream-table duality and SQL statements
Stream for DolphinDB provides numerous convenient features, such as:
- Built-in time-series, cross-sectional, anomaly detection, and reactive state streaming engines.
- High frequency data replay
- Streaming data filtering
1. Flowchart and Related Concepts
Stream for DolphinDB uses the publish-subscribe model. Real-time data is ingested into the stream table and is then published to all subscribers. A data node, compute node or a third-party application can subscribe to and consume the streaming data through DolphinDB script or API. The DolphinDB streaming flowchart is as follows:
Real-time data is ingested into a stream table on a publisher node. The streaming data can be subscribed by various types of objects:
- Data warehouse for further analysis and reporting.
- Streaming engine to perform calculations and to output the results to another stream table. The results can be displayed in real time in a platform such as Grafana and can also serve as a data source for event processing by another subscription.
- API. For example, a third-party Java application can subscribe to streaming data with Java API.
1.1. Stream Table
Stream table is a special type of in-memory table to store and publish streaming data. Stream table supports concurrent read and write. A stream table can be appended to, but the records of a stream table cannot be modified or deleted. SQL statements can be used to query stream tables.
1.2. Publish-Subscribe
Stream for DolphinDB uses the classic publish-subscribe model. After new data is ingested to a stream table, all subscribers are notified to process the new data. A data node or compute node uses function subscribeTable to subscribe to the streaming data.
1.3. Streaming Engine
The streaming engine refers to a module dedicated to real-time calculation and analysis of streaming data. DolphinDB provides dozens of