Elastic and Fault-Tolerant Stream Processing in the Cloud

Add to your list(s) Download to your calendar using vCal

• Peter Pietzuch, Imperial College
• Thursday 16 May 2013, 16:00-17:00
• G29, Mech Eng .

If you have a question about this talk, please contact .

Host: Mirco Musolesi

Please note that coffee will be served at 3:15 at room 123 (School of Computer Science), followed by the one-hour seminar at 4pm.

As users of “big data” applications want fresh data processing results, we witness a new breed of stream processing systems that are designed to scale to large numbers of cloud-hosted machines. Such systems face new challenges: (i) to benefit from the “pay-as-you-go” model of cloud computing, they must scale out on demand; (ii) with deployments on hundreds of virtual machines (VMs), failures are common—systems must therefore be fault-tolerant with fast recovery times. An open question is how to achieve these two goals when stream queries include stateful operators whose state may depend on the complete history of the stream.

In this talk, I describe an integrated approach for dynamic scale out and recovery of stateful stream processing operators. The idea is to expose internal operator state explicitly to the stream processing system through a set of state management primitives. Externalised operator state is checkpointed periodically and backed up by the system. In addition, the system identifies operator bottlenecks and automatically scales them out by allocating new VMs. We evaluate this approach as part of the SEEP experimental stream processing system on the Amazon~EC2 cloud platform and show that it can scale automatically, while recovering quickly from failures.

This talk is part of the Computer Science Departmental Series series.

This talk is included in these lists:

• Computer Science Departmental Series
• Computer Science Distinguished Seminars
• G29, Mech Eng
• computer sience

Note that ex-directory lists are not shown.