- Documenting the Need for New Technology
- Distributed Information Systems Everywhere
- The Global Communication Spaghetti Pot
- Electronic Archeology: Layers upon Layers
- The Gathering Storm of New Activities on the Web
- Global Electronic Trade
- Agile Systems
- Cyber Warfare and the Open Electronic Society
- Summary: Staying Ahead of Chao
This chapter is from the book
This chapter is from the book
This chapter is from the book
1.3 Electronic Archeology: Layers upon Layers
Enterprise systems are distributed, event-driven systems. But they all have another property in common. They are layered systems. Layering is a design technique for controlling complexity. But it is also a side e'ect of unplanned or uncontrolled growth. In some systems the situation bears analogy with the multiple layers of civilizations of di'erent historical periods, one on top of another, that the archeologist Heinrich Schliemann discovered at Troy, back in the 1870s.
Layered IT systems present another dimension in the search for new ways to understand the events that happen in them.
1.3.1 A Layered Enterprise System
Figure 1.4 shows a common layered structure of an enterprise system.
)
Figure 1.4
Typical layers in an enterprise system
Applications and Users at the Top
The top layer is called the business level or strategic planning level because it is the layer at which an enterprise plans and transacts its business. It contains the applications that people use to transact business and do their work. This layer contains user interfaces to the enterprise's services, planning and forecasting applications, inventory and accounting applications, spreadsheets, calendars, Web browsers, document preparation tools, e-mail, communication devices such as cell phones, system administration tools, and so on. The contents of this layer are expanding rapidly. Di'erent kinds of applications are appearing all the time, each requiring di'erent types of information.
The application layer is the level at which human users work and conceptualize their goals—for example, "I want to send e-mail to so-and-so," or "I want to initiate a process to purchase item X from the cheapest supplier." Events that signify activities at this level are the ones humans can understand most easily. These events result from using applications.
The high-level events that are of greatest significance are not explicitly generated by users using applications, but they are consequences of sets of other high-level events. For example, a single business-level event could consist of a sequence of events indicating uses of various applications. It could signify a completed supply chain transaction with business partners, negotiated by messages across the Internet. It is a virtual event signifying a very real activity with contractual and economic implications. These inferred events are aggregated from sets of other high-level events and are the events of interest to the business level or strategic planning level of the enterprise.
Another example of a complex business level event is a stockbroker, or a group of brokers, in a financial trading enterprise trading stocks on many di'erent accounts, including those owned by the trading house itself. Some patterns of trading activities on various accounts might violate Securities and Exchange Commission (SEC) regulations. But high-level events signifying trading violations have to be inferred from the explicit trade events generated by the brokers when they make trades. It depends on an ability to recognize complex patterns of events involving timing and relationships between the events. We understand a "violation event" when it is pointed out to us—no problem! But it isn't actually generated. That's the problem! It is a virtual event signifying a very real activity. It has to be recognized from the morass of actual events.
There are myriads of similar situations going on in enterprise systems all the time that have the potential for similar kinds of violations of the enterprise's policies—for example, in online Internet-based auction houses and in "dot-com" retailing, where violations of the regulations of various states may happen.
Conceptually, the events of interest to the highest layer of an enterprise, or a system of multiple cooperating enterprises, are virtual events comprising sets of application layer events. The strategic planning layer must be fed aggregations of application-level events.
But application-level events are not subjected to a battery of monitoring and analysis tools as the network-level events below it are. This is unfortunate but understandable, because such tools would need a new kind of technology that deals with complex patterns of events to be e'ective.
So, in a layered system, understanding at the highest level what is going on in the enterprise, at every layer, is one problem. But there are more problems to come because causality between events crosses layers.
Collaboration and Enabling Layer
Next in this figure comes the collaboration layer, which contains components that help make applications available to users. Here we find components that we have come to expect, such as databases and servers for e-mail groups, news, and chat rooms. We might even go so far as to include the operating systems, such as Windows and Linux, in this layer, although it could be argued that they belong in layers below. The collaboration layer now contains a growing number of products that enable more sophisticated uses of distributed information processing. For example, Web sites that service applications (so-called application service providers, or ASPs). And business rules engines that enable trading agreements and correlation of information. The collaboration layer contains more and more tools that enable users and applications to interact intelligently on complex projects, like building software systems or putting together multistep trading deals. This layer is the target of a growing industry in business-to-business products and services (often called B2B).
The dividing line between the collaboration layer and the application layer is not a definite, rigorous line. Sometimes, for example, a database would be more appropriately viewed as an application rather than as an enabler of applications, in which case we would put it in the layer above.
Middleware
The next layer down is the middleware layer, which lets all the stu' above communicate. Included are Object Request Brokers (ORBs) and messaging systems such as information buses. This level of communications sits on top of the basic networks and lets all the applications and application servers talk to one another. It contains the system components that are often categorized as Enterprise Application Integration (EAI) products. It is called "integration" because it helps link business-level applications by letting them communicate. And it is called the middleware layer because its components are viewed as lying "in the middle" between applications and networks.
Before 1990 this layer and the products in it didn't exist. It came about by design, to make it easier to get applications to communicate over networks. Middleware is a go-between, translating the message communication between applications into low-level messages called packets that networks are designed to carry.
Communications middleware has become an industry in itself. Middle-ware for distributed applications includes ORBs and various kinds of message-oriented middleware based upon message queues, publish/subscribe (pub/sub) services, and so on. These products provide a layer of communication protocols, together with Application Program Interfaces (APIs), that the enterprise business applications use to communicate. At present there are several widely used commercial middleware products that form the basis for the EAI industry.
The Under-the-Hood Layer
At the bottom is the network layer—the essential plumbing for transporting information from one point to another, both within an enterprise and across enterprises. The kind of networks we are talking about here first appeared in the 1970s. Before that, applications had no way to communicate with one another.
The network layer is the core of the information technology layer (IT layer) in an enterprise system. Newspaper articles refer to it as the "under-the-hood" part of an enterprise's IT system, or the Internet for that matter. It is generally looked upon as something the common man should not know about and certainly not tinker with—it is a source of many system problems. And when it collapses in one of many well-known or not so well-known ways, the whole system grinds to a halt. We often hear "The network is down." Network crashes can become a critical concern to the higher-level echelons in a distributed enterprise. And they certainly a'ect business. Spectacular network crashes make newspaper headlines—for example, when online stock-trading systems clog up and stop the customers from trading, or when hackers orchestrate a denial-of-service attack on Yahoo and other popular Web sites.
So the network layer has become the domain of a powerful new kind of expert, the specialist in network and IT systems management. Often this person has a title, like IT director. The job is to keep the network layer running at all costs—otherwise, the enterprise is out of business. It's the first point of pain for a distributed enterprise.
Consequently, network management products have proliferated. There are all sorts of tools for the IT director to use. They track resource consumption (CPU use and memory or disk space use) on every machine in the network and provide statistics and summaries in every color in the rainbow. They log alerts or warnings from network components such as routers and servers, slice and dice them by the minute or hour, raise the alarm, and page the IT director out of bed at any hour of the day or night.
But keeping the network running is not the IT director's only problem. Additionally, there are issues of security and privacy. And cyber warfare is an increasing headache for all enterprises. Everyone knows it is going to increase. Plenty of tools and products try to help the IT director solve cyber warfare problems, but again, they don't do too well.
The IT director's problem is making sense of all this network-level information. The importance of this task is well understood by enterprises today. They invest heavily in whatever technology is available to help. For example, the network operations room in a large global banking enterprise represents a billion-dollar investment these days.
1.3.2 Vertical Causality: Tracking Events up and down the Layers
Although there are no general layering standards for categorizing enterprise systems, the picture in Figure 1.4 is typical and captures the general idea.
The applications that humans use and understand (sometimes) are at the top.
Network plumbing that carries the messages and information is at the bottom.
Several layers are in between, depending upon the complexity of the system.
Activity at each layer is translated into activities at the layers below and conversely. Those lower-level activities must complete successfully in order for the higher-level activities to also complete successfully.
So, we have a general principle of layered enterprise systems: An activity at the top causes activities at successively lower levels, which in turn cause other activities to happen at the top.
Layering adds another dimension to understanding events. Discovering the causal relationships across layers, between high- and low-level activities, especially in real time, is another outstanding issue in enterprise systems. We call this vertical causality.
An ability to track vertical causality in a layered system has at least two important uses in managing our enterprises. First, knowing how a business-level event is broken down by your system into sets of lower-level events is very helpful in understanding properties of that high-level event, such as its timing—for example, why it took so long or failed to complete—and how to improve performance at the business level. Sometimes we may only need to understand the breakdown of the business-level event into calls to applications, while at other times, we may need a breakdown all the way to the network level. This is the kind of information that might have helped the IT manager answer the CEO's question about what caused customers in the New York area to complain about slow service.
Second, if we were able to track vertical causality, we could use it to group events at lower levels according to the high-level events they signify. That would help us make sense out of the mountains of low-level events that network monitoring tools give us.
Tracking vertical causality is not a simple problem because there is almost as much dynamism in the layers of our enterprise systems as there is in the communication between them across the Internet. For example, the ability of a service provider to load-balance incoming requests may depend on the loads of lower-level servers that are continually changing. A high-level complex event, such as defaulting on a service-level agreement, can happen in many di'erent ways. It may result from lots of di'erent sets of lower-level events. Moreover, our enterprise systems are dynamic in their composition and structure. The sets of components at each level are changing all the time, as are the kinds of activities the system is being employed to do. So the kinds of events that can happen at every level are changing too. Keeping up with what our systems do is not an easy task.
1.3.3 Event Aggregation: Making High-Level Sense out of Low-Level Events
What about those strategic-level complex virtual events that we talked about earlier? Virtual events are not actually generated in the system—nobody sends a message saying, "I violated law XYZ." The activities sig-nified by high-level virtual events have to be recognized from among sets of events at the application layer or below. These policy and contractual events are top-level events of immediate interest to the corporate level of the enterprise.
We have just discussed the potential usefulness of an ability to track vertical causality. But there needs to be a high-level event in order to track its causes. Often, all there is at the highest levels is a perception, a suspicion, a list of complaints. But the business and strategic implications are very real.
The complementary operation to downward tracking of vertical causality is the aggregation of sets or groups of lower-level events into a single higher-level event that expresses the meaning of the lower-level events, taken together. These are some examples:
A group of stock-trading events, related by accounts, timing and other data, taken together constitute a violation of a policy or regulation.
A large set of network-level operations, originating from the same machine and repeating similar accesses on di'erent target machines, may signify an attempt to intrude into a network.
In an application service provider, groups of events signifying accesses to distributed applications, such as those illustrated in Figure 1.3, together may imply a violation of a service-level agreement with a customer.
Recognizing or detecting a significant group of lower-level events from among all the enterprise event tra=c, and creating a single event that summarizes in its data their significance, is called event aggregation. The aggregate event will appear at a higher level in the enterprise's operations.
Event aggregation is a capability that needs to be developed. It is a very powerful technology for monitoring and managing our enterprises. It will in turn depend upon technology for recognizing patterns of events in large amounts of lower-level event tra=c, in real time. And it depends first on an ability to express patterns consisting of multiple events together with their common data and timing. If event aggregation is implemented properly, it can give us the ability to track the lower-level events that were aggregated to create a high-level event—so-called drill-down diagnostics for tracking vertical causality.