Unified data processing with Spark Structured Streaming

Stream processing has progressed far beyond being an alternative to batch processing and is now considered a unified data processing system. This session will describe and review OSS stream processing systems such as those listed below, and will introduce Spark Structured Processing as a recent example of an unified data processing system.

• Lambda Architecture
• Kappa Architecture
• Unified Data Processing

Finally, I will discuss issues that we encountered while running Spark Structured Streaming in production, what it felt like to run, and caveats that you should keep in mind.

Session length

40 minutes

Language of the presentation

Japanese

Target audience

Intermediate: Requires a basic knowledge of the area

Who is your session intended to

People who want to know the difference between batch processing and stream processing
People who want to understand concepts around stream processing
People who want to know practical use cases of Spark Structured Streaming

Speaker

Sotaro Kimura (DWANGO Co., Ltd.)