Power of Events, The: An Introduction to Complex Event Processing in Distributed Enterprise Systems

Description

• Copyright 2002
• Edition: 1st

• Book
• ISBN-10: 0-201-72789-7
• ISBN-13: 978-0-201-72789-0

As global information systems become ever more powerful, complex, and intertwined, companies need new approaches to extracting information, transforming it into intelligence, and acting on it. David Luckham introduces a breakthrough solution that offers compelling benefits at every level and scale of the enterprise: Complex Event Processing (CEP). Luckham first identifies key challenges faced by today's enterprise information systems, and demonstrates the "event-driven" nature of management in the electronic enterprise. He then introduces CEP, showing how it can harness the power of events to automate management without compromising managers' control. Luckham illuminates fundamental concepts such as events, causality, event hierarchies, event patterns, and rules; then shows how these concepts can be used to solve key enterprise management problems. In Part III, he presents a realistic description of what it takes to build CEP applications that scale to real world problems, introducing the new RAPIDE event pattern language. The book concludes with detailed case studies that show Luckham's CEP tools at work in the enterprise.



Sample Content

Online Sample Chapters

The Global Information Society and the Need for New Technology

The RAPIDE Event Pattern Language

Downloadable Sample Chapter

Click below for Sample Chapter(s) related to this title:
Sample Chapter 1
Sample Chapter 8

Table of Contents

Preface.


Acknowledgments.

I. A SIMPLE INTRODUCTION TO COMPLEX EVENT PROCESSING.

1. The Global Information Society and the Need for New Technology.

Distributed Information Systems Everywhere.

The Global Communication Spaghetti Pot.

Event Causality.

Electronic Archeology: Layers upon Layers.

A Layered Enterprise System.

Vertical Causality: Tracking Events up and down the Layers.

Event Aggregation: Making High-Level Sense out of Low-Level Events.

The Gathering Storm of New Activities on the Web.

Global Electronic Trade.

Agile Systems.

Cyber Warfare and the Open Electronic Society.

Summary: Staying ahead of Chaos.

2. Managing the Electronic Enterprise in the Global Event Cloud.

How the Global Event Cloud Forms.

The Open Enterprise.

The Global Event Cloud.

The Electronic Enterprise.

Operating in the Global Event Cloud.

Going Beyond Workflow.

Parallel and Asynchronous Processes.

On-the-Fly Process Evolution.

Exceptions Must Be First-Class Citizens in Process Design.

Summary: Managing the Electronic Enterprise.

3. Viewing the Electronic Enterprise-Keeping the Human in Control.

Today's Event Monitoring Is Too Primitive.

System Monitoring Focuses on the Network Layer.

Network-Level Monitoring Doesn't Even Solve Network Problems.

An Example of Causal Tracking.

Information Gaps.

Examples of Information Gaps.

Problem-Relevant Information.

Viewing Enterprise Systems.

Creating and Coordinating Multiple Views.

Hierarchical Viewing.

An Example of Hierarchical Viewing.

Summary: Viewing the Electronic Enterprise.

4. Designing the Electronic Enterprise.

Process Architectures.

Roles of Architecture in the Process Lifecycle.

Constituents of Process Architectures.

Annotations.

Architectural Structure.

Interface Communication Architectures.

Architecture Diagrams.

Behavior Specification.

Design Constraints.

Examples of Informal Annotations.

Dynamic Process Architectures.

Diagrams for Dynamic Architectures?

Layered Architectures and Plug-and-Play.

Abstraction Principle.

Summary: Technology to Support Process Architecture.

5. Events, Timing, and Causality.

What Events Are.

How Events Are Created.

Time, Causality, and Aggregation.

The Cause-Time Axiom.

Genetic Parameters in Events.

Timestamps.

Causal Vectors.

Time.

Causality and Posets.

Causal Event Executions-Real-Time Posets.

Orderly Observation.

Observation and Uncertainty.

Summary.

6. Event Patterns, Rules, and Constraints.

Common Kinds of Pattern Searching.

Event Patterns.

A Strawman Pattern Language.

Pattern Matching.

Writing Patterns in STRAW-EPL.

Event Pattern Rules.

Constraints.

Summary.

7. Complex Events and Event Hierarchies.

Aggregation and Complex Events.

Creating Complex Events.

Event Abstraction Hierarchies.

Viewing a Fabrication Line.

Building Personalized Concept Abstraction Hierarchies.

Viewing Network Activity.

Viewing Stock-Trading Activity.

Summary.

II. BUILDING SOLUTIONS WITH CEP.

8. The RAPIDE Pattern Language.

Event Pattern Languages-Basic Requirements.

Features of Rapide.

Types.

Predefined Types.

Structured Types.

Event Types.

Execution Types.

Subtyping of Executions.

Attributes of Events.

Basic Event Patterns.

Placeholders and Pattern Matching.

Matching Basic Event Patterns.

Placeholder Bindings.

Notation to Aid in Writing Patterns.

Relational Operators and Complex Patterns.

Relational Operators.

Guarded Patterns.

Content-Based Pattern Matching.

Context-Based Pattern Matching.

Temporal Operators.

Repetitive Patterns.

Pattern Macros.

Summary.

9. CEP Rules and Agents.

Overview.

Event Pattern Rules.

Definition of Event Pattern Rules.

Rule Bodies.

Context and Visibility Laws.

Semantics of Event Pattern Rules.

Examples of Rules.

Event Processing Agents.

Definition of EPAs.

Semantics of EPAs.

Event Pattern Filters.

Definition of Filters.

Semantics of Filters.

Action Name Filters.

Content Filters

Context Filters.

Event Pattern Maps.

Definition of Maps.

Semantics of Maps.

Event Pattern Constraints.

Definition of Constraints.

Semantics of Constraints.

Examples of Constraints.

Other Classes of EPAs.

Summary.

10. Event Processing Networks.

Common Structures of EPNs.

Flexibility of Event Processing Networks.

Connecting Event Processing Agents.

Basic Connections.

Guarded Connections.

Multiple Basic Connections.

Dynamic Event Processing Networks.

Class Connections.

Creation and Termination Rules.

Connection Generators.

Architectures and Event Processing Networks.

Architecture Classes.

Semantics of Architecture Classes.

Examples of EPNs and Architectures.

Case Study: EPNs for Network Viewing.

Visual Tools for Constructing EPNs.

Security.

Scalability.

Summary.

11. Causal Models and Causal Maps.

Causality between Events, Revisited.

Why We Need Causal Models.

What Causal Models Are.

Defining a Causal Model and a Causal Map.

Using Pattern Pairs to Specify Causal Models.

Using Causal Rules.

Resolving Ambiguities.

Causal Maps.

A Small Example of a Causal Map.

A Second Example of a Causal Map.

Developing Accurate Causal Models.

Summary.

12. Case Study: Viewing Collaboration between Business Processes.

A Collaborative Business Agreement.

An Interface Communication Architecture.

Causal Model.

Causal Map.

Examples of Causal Rules.

Examples of Constraints.

Analysis of Examples of Posets.

Constraint Checking Becomes Part of the Collaboration.

13. Implementing Event Abstraction Hierarchies.

The Accessible Information Gap.

Event Abstraction Hierarchies, Revisited.

Induced Causality.

Abstraction Effect on Constraints.

Modifiability.

Bridging the Information Gaps.

Steps to Apply a Hierarchy to a Target System.

A Hierarchy for a Fabrication Process.

Personal Views.

Implementation.

Diagnostics.

14. Case Study: Viewing a Financial Trading System.

A Small Stock-Trading System.

The Information Gap for STS.

An Event Abstraction Hierarchy for STS.

Building the Event Abstraction Hierarchy.

Level 1.

Level 2.

Level 3.

Implementing Hierarchical Viewing for STS.

Three Steps toward Human Control.

Drill-Down Diagnostics.

Detecting Constraint Violations.

The Abstraction Effect.

Summary.

15. Infrastructure for Complex Event Processing.

Examples of Forms of Observed Events.

Interfacing CEP Infrastructure to Target Systems.

CEP Adapters.

CEP Runtime Infrastructure.

Infrastructure Interfaces and Components.

Functionality of the Interface.

Off-the-Shelf Infrastructure.

Event Pattern Languages.

Complex Event Pattern Matchers.

Quest for Scalability.

The Naive View of Pattern Matchers.

What Pattern Matchers Really Do.

Design Structure.

Rules Management.

Analysis Tools.

Summary.

Bibliography.
Index. 0201727897T04252002

Preface

Complex event processing (CEP) is a set of techniques and tools to help us understand and control event-driven information systems. And today, any kind of information system, from the Internet to a cell phone, is driven by events. What is a complex event? It is an event that could only happen if lots of other events happened.

For example, suppose you see a car you like at your favorite car dealership. That car is on the showroom floor only because a number of other events took place, events in the inventory control systems of the dealership and the manufacturer, shipping events, customs events at the port of entry, and so on. Of course, when you see exactly what you want in the showroom, you don't ask how or why! But if you don't see the model, make or color you want, and ask why not, then you'll get an explanation about allocation quotas, backlogs at the factory, or some other factors that affect events in the causal history leading up to the event you wanted.

This illustrates one of the ideas behind CEP. Events are related in various ways, by cause, by timing, and by membership. CEP applies to electronic information systems. It makes use of relationships between events to answer questions like, "is our system providing the correct level of service to our customers", "will our shipment arrive on time", and "is someone trying to steal our information". CEP adds a new dimension of event processing to what our event-driven information systems already do.

Why is there a need for CEP? Well, let's look at the situation, briefly.

Today's Information Society is founded upon gathering and sharing information. All of our organizations, commercial, government, and military, are dependent upon electronic information processing. Their foundational backbone is the kind of distributed computing system based on computer networks that is nowadays called the "information technology layer" (or IT layer) of the organization. The use of these systems has expanded rapidly over the past ten years to meet the increasing demands of automation, electronic commerce and the Internet explosion. Investment in technology has focussed on making IT systems faster, capable of handling larger and larger amounts of information, and able to collaborate with one another. We now live in the world of the open enterprise where commerce and information move across the boundaries of organizations, and nations. Our society has become dependent upon IT systems.

Less investment has been devoted to develop technology to solve the increasing problem of understanding what is happening in our IT systems. Whenever there is a crisis—a denial of service attack or a system failure—at first we don't understand what is going on, or how to fix it, and then in the aftermath we scramble for weeks to find out what caused it. We need to understand and control our critical information infrastructures better than that!

A lot of the information in IT systems is never recognized. Messages—or events—pass silently back and forth across our information systems as unrelated pieces of communication. They are a source of great power for when they are aggregated together, and correlated, and their relationships understood, they yield a wealth of information. A new technology is needed to harness the power of events in global information systems. This book is about such a technology.

A few words about CEP—what it is, and where it applies.

CEP consists of very simple techniques, a mix of old and new. Some of them are well known in other kinds of computer applications, such as rule-based systems in intelligent programs. Some of them are new techniques, such as tracking causal histories of events in large distributed computer systems. Or using patterns of events and event relationships, to recognize the presence of complex events that are signified by hundreds or thousands of simpler events in our IT systems. In CEP, new techniques are combined with well-known techniques in a unified framework.

An example of the kind of electronic complex event we are talking about is the completion of a financial transaction involving a bundle of financial contracts. Several merchant banks and brokerage houses may participate in the transaction. They use a global trading network. The event itself, the completion of the transaction, might be the result of hundreds of electronic messages and entries into several different databases around the world over a span of two or three days. These event don't necessarily happen in a nice linear order, one after the other. Some of them might happen simultaneously and independently of others, mixed in with events from other transactions. We can apply CEP to the trading network to recognize not only when that complex event happens, but more importantly, whether it is going to happen, or is getting off track and may not happen, and why!

CEP applies to a very broad spectrum of challenges in information systems. A short list includes (1) business process automation utilizing the Internet and electronic Marketplaces, (2) computer systems to automate the scheduling and control of anything from fabrication lines to air traffic, (3) network monitoring and performance prediction, and (4) detecting attempts to intrude into computer systems or attack them.

There is a fundamental reason for this broad applicability. It is simply because information systems are all driven by events. To be sure, each system, or application running on top of a system, depends upon different kinds of events. Network events are different from database events, which are different from financial trading events. But one of the major themes of CEP is that different kinds of events are related. CEP provides techniques for defining and utilizing relationships between events. CEP applies to any type of event that happens in a computer application or a network or an information system. In fact, one of its techniques lets you define your own events as patterns of the events in your computer system. CEP lets you see when your events happen. This is one way to understand what is going on in your system!

And that brings us to another point—flexibility. CEP allows users to specify the events that are of interest to them at any moment. Events of interest can be low level network monitoring alerts, or high-level enterprise management intelligence, depending upon the role and viewpoint of individual users. Different kinds of events can be specified and monitored simultaneously. And the specification of the events of interest, how they should be viewed and acted upon, can be changed on the fly, while the system is running.

The users of CEP can be human, or they can be automated processes. The processes that manage our enterprises are becoming more complex. Linear workflow processes that epitomize document processing in commercial transactions are not capable of managing the open electronic enterprise. In the future, enterprise management processes will be designed to incorporate complex event processing in order to get the kind of events they need to operate.

Now, a few words about the book itself and what the reader should expect. First of all, there are two parts to this book.

Part I is for a broad audience of people with an interest in various aspects of the Information Society, such as Electronic Commerce, the Internet, B2B collaboration, or generally, Electronic Information Processing. Part I deals with two questions about CEP. What it is for—that is, the kinds of problems in the Information Society that CEP can be applied to. And, what it is—a simplified view of CEP, the basic concepts and easy examples of applications.

Part I includes Chapters 1-7. The first four chapters describe the problems and issues in IT systems that CEP applies to. The next three chapters, 5-7, describe basic concepts of CEP such as what an "event" is, causal and timing relationships between events, patterns of events and event hierarchies, and how to apply them to solve the problems described earlier.

Part II consists of Chapter 8 onwards. It is intended for information systems specialists with some background in software. Part II presents how-to-build-it details and case studies of CEP applications. The goal of Part II is to describe what is needed to build applications of CEP that are capable of solving real world problems. It includes first of all, a detailed description of a complex event pattern language, reactive event pattern rules and event pattern constraints.

Secondly, Part II shows how to build solutions by using the event pattern rules and constraints to build event processing agents, and architectures of communicating agents. Part II includes case studies, as large and as detailed as one can fit in a chapter of a book.

The final chapter of this book deals with the question of how to develop an infrastructure for CEP. One can look around the event-driven applications being developed in the commercial world today, utilizing the power of distributed computing, the Internet and private networks. There is an almighty commercial struggle brewing for market share in the world of eMarketplaces and Electronic Commerce. It is quite predictable, considering the trends in middleware, the Java world, the .Net world, the security world, and so on, that CEP will be developed as a competitive advantage. This chapter deals with leveraging these developments to build infrastructure for CEP—now and quickly!

A word about references. This area of Internet technology is changing so quickly that any attempt to give comprehensive references would be outdated in six months. Not only that, but any less than complete set of references would be unfair to some. I assume that any reader has access to the Internet. So, they can search for current references to, e.g., "middleware" or "application server". So I have tended to include only a few references, either general references to Web sites or citations to seminal research papers that are not easily found.

At this current time in our society, any technology that attempts to view and control IT systems may be seen by some as conflicting with issues concerning privacy. In fact, CEP may provide a foundation for resolving some possible conflicts. However, I cannot deal with this topic here, and I do not.

Just a little history. CEP has grown out of a research project at Stanford on event -based simulation called the Rapide project. This research took place between 1990 and 2000. Out of Rapide came some early experiments in CEP applied to viewing small communicating systems built on commercial middleware, or applied to recognizing security threats in progress on the IT layer of a large university, where hackers love to play! These projects are documented on two Web sites:

http://pavg.stanford.edu/rapide/

http://pavg.stanford.edu/cep/

—David Luckham, Professor Emeritus,

Electrical Engineering, Stanford University

0201727897P10262001

Index

Click below to download the Index file related to this title:
Index



Updates

Submit Errata

