How Does Microsoft Operations Manager Work?

This chapter is from the book 

Business Logic Layer

The real intelligence of MOM lies in the business logic layer and includes a number of components. Within this layer, rules are set that govern what the business wants to monitor, to be alerted to, to report on, and other myriad details. The business logic layer (shown in Figure 3.9) is where the knowledge of how platforms such as Windows Server 2003 and applications such as Exchange 2003 should be configured and operate are integrated into MOM's framework.

The business logic layer is the most complicated and has the most components of all the layers. The components of this layer are organized into three major groups:

• Core functionality
• Complex responses
• Connecting to external systems

Collecting information from a wide variety of sources is a key characteristic of MOM. This handling of information includes storing it, correlating it to other information, and using it to form alerts and other actions. MOM 2005 collects, handles, analyzes, and responds to operational information using the following components:

• Managed computers and applications
• MOM Service
• Data Access Service (DAS)

The business logic behind a response or task is often more complex or nuanced than even a sophisticated user interface such as MOM's can facilitate. Those cases make it critical that the management application provides a mechanism allowing complex scripts or programs to execute the appropriate business logic. This is accomplished programmatically using the following scripting components:

• Scripting library
• Script responses
• Managed response class library
• Managed code responses

Additionally, the capability to connect to external systems and expose the functionality of MOM 2005 is important to the enterprise scalability and interoperability of MOM. Most large organizations have other management systems or trouble-ticketing systems that are vital to the organizations' operations, and MOM 2005 can interoperate with them. The following components deliver this functionality:

• MOM Connector Framework (MCF)
• Connector applications (local)
• MOM Web Service
• Connector applications (remote)
• Management Service Class Library (MCL)
• Custom applications

We will cover each of the components that make up the functionality of the business logic layer in the next sections. Chapter 19, "Interoperability," discusses the implementation of many of these components.

Managed Computers and Applications—Agent-Based

The method MOM uses for delivering and executing business logic is critical to its success. MOM's agent-based technology allows it to push the work of executing the business logic down to the managed servers. Intelligent local agents are one of the keys to MOM's success. These local agents operate independently, allowing them to respond quickly to changing conditions. The agents are functional even if they cannot contact their management server due to a network outage.

MOM typically manages systems using an installed agent on those systems, although MOM 2005 also includes the capability for agentless monitoring of a small number of systems. The agent runs as a service named the MOM Service and collects information as directed by the business logic. The agent is in effect the foot soldier of the MOM system, following the orders dictated by the business logic. The collected information is stored in a buffer locally on the monitored system and then forwarded to a management server. Forwarded information is compressed and encrypted to reduce the footprint on the network and to ensure confidentiality of the management data, allowing MOM to work across slow links and within insecure environments.

Agents can also be configured to look for more than one management server for redundancy and separation of data. Agents can report to a maximum of four management servers, the first of which is assigned automatically by the management service doing the agent install. The other management servers are listed in the agent configuration for automatic fault tolerance. You can see this represented in Figure 3.10, where the agent has a solid line to its primary management server (Monarch) and a dotted line to the other management server (Keystone). The database server is running on a server named Fountain, and the reporting server is Silverthorne.

Normally, the agents are distributed between management servers to provide load balancing as well. This is indicated in Figure 3.11, where Agent 1 reports to Monarch as its primary with Keystone as the backup. However, Agent 2 reports to Keystone as its primary with Monarch as the backup. This balances the management load across the management servers.

Agents can report to multiple management groups as well, which might have different rule sets. This is illustrated in Figure 3.12, where the agent is reporting to both Group1 and Group2. In both management groups, the Monarch management server is primary, and the other management server provides fault tolerance. When reporting to two or more management groups, the agent knows which rules were deployed by which management group and sends the information collected to the appropriate management server. These agents are known as multihomed. Chapter 9, "Installing and Configuring Agents," discusses the installation and configuration of agents, including multihomed agents.

The agent does not just store and forward the information it gathers to the management groups it reports to. If the business logic dictates, the agent can also evaluate and respond to the information. The responses can be generating an alert, running a script, sending a Simple Network Management Protocol (SNMP) trap, and so on. Having the agents respond locally to events ensures that the system will continue to be monitored and respond to events even when it cannot reach the rest of the MOM infrastructure.

Agents periodically heartbeat to their configured management server. The heartbeat information is used to ascertain whether the agent is still alive and save the management server from having to poll for the status of all its agents. The heartbeat information also contains the latest agent configuration version information. Right after a heartbeat, the agent also uploads any queued data to the management server. The management server then uses the configuration version information to send the latest configuration information to the agent, ensuring that the agent has the latest business logic applied to it. More information on agent heartbeats is available in Appendix A.

Agent Processes—MOM Service and MOM Host

Although we have been referring to the agent, things are actually a bit more complicated than that. The MOM agent actually uses two processes to achieve its objectives: the MOM Service and MOM Host processes, shown in Table 3.3. The MOM Service process handles the internal workings of the agent and communications with the management servers. The MOM Host process handles the information gathering and responses that the business logic dictates. There may be multiple instances of the MOM Host process on any given managed computer. The agent runs multiple MOM Host processes to ensure that the response and providers are isolated. If the process locks running a script or retrieving data from a provider, it will not affect the function of the overall agent or the function of any other MOM Host process.

Table 3.3. MOM Agent Processes and Tasks

Processes	Executable	Tasks
MOM Service	MOMService.exe	Communication with management server(s) Applications event log—Read/Write Security event log—Read/Write WMI event provider—Read File transfer—Send/Receive
MOM Host	MOMHost.exe	Monitors and collects Windows event log data Monitors and collects Windows performance counter data Monitors and collects WMI data Monitors and collects Application log data Runs script and batch responses Runs managed code responses

Even with gathering all this information and taking the appropriate actions, the footprint on the monitored system is light. On a typical managed system, agent activities consume less than 1% of processor time. The memory requirements will vary according to number of business logic rules applied to the system, but a rough estimate is between 20 and 60 megabytes. Even the agent on the management servers, where the agent service is handling events from a number of systems, uses less than 1% processor time as an average. Depending on the number of systems managed by a particular management server, the processing time scales with the number of events forwarded and can grow to 5% to 10% of processor time.

MOM can actually tell us what the overhead is. In Figure 3.13, you can see the results for a typical MOM installation monitoring approximately 20 systems. For clarity and simplicity, we selected only three systems for graphing. The graph measures the percentage processor time over the last seven days for the MOM Service process on three systems, including the management server (Monarch). What might not jump out at you is that the scale of the graph is 0% to 1% (the axis values scale automatically based on the values generated). You can see that for most servers the time is less than one-tenth of a percent (.1%). It is interesting to note that in the graph you can see that each of the managed computers has a consistent load, which is proportional to what is being monitored on the server. The exception is the management server itself, which is hovering around 0.25% with spikes of up to about 0.75%. The spike represents the extra work MOM does every day at 2:00 AM to look for new computers. There are also several spikes on 7/12/2006 at about 11 AM and 8 PM, which could bear some investigation. However, even during the spikes the management server load is less than 1% of average processor utilization.

MOM Service

The memory requirements of the MOM Service process are much more variable, being proportional to what is monitored or the number of rules deployed to the managed computer. Figure 3.14 shows the results of the agent memory utilization over a seven-day period. The roles of the servers are definitely relevant here. The Monarch computer is the MOM 2005 management server, Dillon is a computer running mainly IIS and antivirus/antispam software, and the Aurora server is running Exchange 2003. The agent on Dillon monitors relatively basic and static functions, so the memory utilization holds steady over the entire seven days at a bit above 6.6 million bytes or about 7MB. The agent on Aurora shows even less variability in the memory utilization but uses more memory and hovers just at under 20MB. This is an Exchange 2003 server and is one of the more managed roles within MOM 2005, hence the higher memory requirements. The agent on Monarch does a variety of other tasks that we will cover in the next section ("MOM Host"), so the memory utilization is much higher and grows from 59MB to just under 66MB.

MOM Host

The other process, the MOM Host, has its own processor and memory utilization patterns. Its processor and memory requirements are separate and in addition to those of the MOM Service process. In Figure 3.15 you can see the processor utilization for the MOM Host processes on the same three computers. The first thing to notice is that there are two instances of the process (MOMHost, MOMHost#1) for each managed computer and in fact there could be more in some cases depending on the different responses and providers that the agent is monitoring. However, you can see that over the course of the seven days of monitoring the requirements for the servers are quite light. For all the servers, the MOMHost instance is close to zero processor utilization, and the MOMHost#1 instance is less than 0.25%. This holds true even for the management server itself.

MOM Host memory utilization tends to be more activity driven than the processor utilization and so exhibits a bit more variability, at least for the second instance of the process (MOMHost#1). The memory requirements again vary by the class of the server and thus by the monitoring requirements. In the case of MOM Host, this is more directly proportional to the level of monitoring. As expected by this, you can see in Figure 3.16 that for the Aurora Exchange server the first instance, the MOMHost process holds steady at just under 10.6MB (or about 11MB), and that the second instance (MOMHost#1) is at just under 21MB at the end of the monitoring period. For the Dillon managed computer, the MOMHost instance is steady at about 7MB, and the MOMHost#1 instance varies between 4MB and 5MB. Finally, the Monarch management server MOMHost instance is steady with a jump midway in the monitoring period at about 8MB, and the second instance, MOMHost#1, hovers right at about 11MB (though there is a brief drop in the middle of the monitoring period). That jump in MOMHost reflects the addition of an agentless managed computer to the management server, which we will discuss in the next section of this chapter.

With all that detailed information, a good question to ask is how much processor and memory resources will be utilized by the agent process overall? As was shown in the previous discussion, it varies by the level of monitoring and the class of server. Our sample computers fall into the categories of a lightly monitored web server, a heavily monitored Exchange server, and a management server. Tables 3.4 and 3.5 show the total processor and memory utilization for the various agent processes in this typical environment.

Table 3.4. Typical Agent Memory Utilization

Class	Computer	MOMService	MOMHost	MOMHost#1	Total Memory
Web Server	DILLON	7MB	7MB	5MB	19MB
Exchange Server	AURORA	20MB	11MB	21MB	52MB
Management Server	MONARCH	66MB	8MB	11MB	85MB

Table 3.5. Typical Agent Processor Utilization

Class	Computer	MOMService	MOMHost	MOMHost#1	Total Processor
Web Server	DILLON	0.10%	0%	0.25%	0.35%
Exchange Server	AURORA	0.10%	0%	0.25%	0.35%
Management Server	MONARCH	0.75%	0%	0.25%	1.00%

These numbers are not significant, given current memory standards for servers. Even for a web server with only 256MB of RAM, the total agent memory utilization of about 19MB is about 7% of total memory. For a typical Exchange server with 1GB of RAM, the 52MB memory utilization is about 5% of total memory. Even for a typical management server with 1GB of RAM, the 85MB memory utilization is less than 8% of total memory. Far more is used by the SQL Server service! The processor numbers speak for themselves. Table 3.6 summarizes the typical resource utilization on managed computers and management servers.

Table 3.6. Typical Utilization Summary

Class	Memory Profile	Processor Utilization	Memory Utilization
Web Server	256MB	0.35%	7%
Exchange Server	1GB	0.35%	5%
Management Server	1GB	1.00%	8%

Managed Computers and Applications—Agentless

In addition to the primary method of managing computers using agent-based technology, MOM 2005 is capable of managing computers without an agent. In this mode, the management server agent takes on the role of the agent for the agentless managed computer. The information is gathered remotely using RPC and DCOM.

The information gathered is equivalent to that for agent-based managed computers. However, agentless monitoring has the following limitations:

• Scalability—The work of monitoring agentless managed computers is done by their management server and more specifically by the agent on the management server. This load is significant and can severely impact the performance of the management server, so the agentless mode is limited to only 10 agentless managed computers per management server and a total of 60 agentless managed computers per management group. This is a major limitation to the scalability of the agentless mode of operation.
• Tasks—With no local agent on the managed computer, tasks cannot be executed locally on the managed computer. Tasks that run on the management server can be executed against the agentless managed computer, such as the Ping task.
• Event log descriptions—Event descriptions are not gathered as part of the agentless monitoring, so the event log descriptions are not available unless the management server has the same event log messages .DLL file. An awkward workaround is to install the same software on the management server as is on the agentless managed computer.
• Firewall support—Given that the management server uses RPC and DCOM to monitor the agentless managed computer, agentless managed computers are not supported across a firewall. Opening up RPC and DCOM across a firewall cannot really be done securely, given the nature of the protocols.
• Management pack limitations—Management packs presuppose agent-based managed computers and may not fully operate on agentless managed computers. Most of the monitoring functions will work without any problems, but many of the more sophisticated responses will not function correctly.

Given these limitations, the agentless monitoring features are not suitable for many tasks, and this is definitely not the method to manage the majority of computers. However, in some cases such as those where there is concern about the installation of software on an application server, this will be a solution allowing a reasonable level of monitoring with a limited level of impact to the managed computer. It helps capture those one-off computers that normally resist management.

MOM Service Component

The MOM Service component (also known as the MOM Server component) is different from the MOM management server. The MOM Service is a component of a management server that runs as a set of services and handles key functionality, which we discuss in this section. The MOM management server refers to a role within the MOM 2005 infrastructure where there are a collection of components on a given computer, including the MOM Service, the DAS, and so on.

The MOM Service performs the following functions for the management server:

• Manages agent installation
• Manages agent configuration
• Monitors managed computer availability
• Consolidates data
• Monitors server-side responses
• Self monitoring
• Monitors agentless managed computers

Using computer discovery rules, the MOM Service component scans the directory for computers that match the computer discovery rules. After discovering computers, the MOM Service component can initiate an agent install and update for the soon-to-be-managed computers. This installation can be configured to take place automatically, or it can require administrative authorization before proceeding. The default is for the system to require approval for agent installations and to wait for 48 hours before removing agents from a system that falls out of the managed computer rules.

In addition, the managed computer attributes are scanned to discover what applications are installed on a managed computer and what roles they play. The attribute scan process is done via a task, which is executed by the agent rather than remotely by the management server. However, the MOM Service component initiates that task and receives the results back from the agents.

Attribute information is used to assign the computer to computer groups within MOM. The MOM Service component is also responsible for scanning computer group memberships. Membership in computer groups is based on attributes the agent discovers, as well as explicit assignments made at the console. Additional information on computer attributes and group membership formulas is available in Chapter 13, "Administering Management Packs."

Automatic rule deployment and view selection take place after the computer is placed in the appropriate computer groups. Based on the computer group membership, the MOM Service component delivers the appropriate rules to the agents on the managed computers. In this regard, it is the sergeant of a MOM system, passing on the orders for the agents to follow. It is also natural for the business logic and thus the rules that represent that logic to change. These rule updates are automatically distributed to the appropriate managed computers by the MOM Service component.

The agents periodically check in, or heartbeat, to their management server. The MOM Service component receives that heartbeat and also detects when a managed computer has not generated a heartbeat. It is during this heartbeat process that agents check to see whether there are new rules or rule updates that they need to receive from the MOM Service component.

The MOM Service component receives operational data from the agents and passes it on to the DAS component, in effect operating as a proxy between managed computers and DAS. The MOM Service component not only proxies the operational data but also processes the operational data and executes responses indicated by the business logic.

The MOM Service component also performs the agent functions for the management server that it runs on, so you will not find a separate service for the agent on a management server. Finally, the MOM Service component acts as the agent for agentless managed computers. This includes polling of agentless managed computers, collecting the operational data remotely, and running responses (where possible).

Data Access Service Component

The Data Access Service (DAS), also known as the Data Access Server, handles both data insertions and data requests to the MOM database. It handles all insertions to the operations database. It handles most of the requests for data as well. The most important exception to this is the reporting subsystem, which bypasses the DAS and uses DTS to transfer data from the operations database to the reporting database. However, the DTS process only copies the data and does not remove data from the operations database.

The DAS is a server-based Component Object Model Plus (COM+) application hosted by the DLLHOST process. The DAS exposes a set of DCOM objects and communicates that control access to the MOM database. The DCOM interfaces are associated with COM+ roles and provide authentication and authorization of the identities that access the interfaces.

Rather than being installed as a service, the DAS is installed as components in Component Services for improved performance by using object pooling. More than 100 different components are within the DAS. The DAS provides common centralized database access logic, centralized query logic, shared cache, and pooled connections to the operations database server. This translates to improved performance for the database server in reduced connections and duplicate requests, and also for the other components in reduced latency when retrieving cached information. Due to the centralized access and query logic, there is also less likelihood of data being entered incorrectly by wayward components. All components see the same consistent view of the data and operate under a consistent security model.

Some critical services the DAS provides are maintaining data consistency and logging. Whenever a change is made, such as the updating of an alert, the DAS records the change and the credentials of the user making the change, storing it in the operations database as well. This is important for auditing and security.

Programmatic Response Components

The scripting capabilities of MOM 2005 allow for customized monitoring and responses to events, alerts, and performance data.

There are two major types of programmatic responses in MOM 2005:

• Script responses—These allow you to extend the capabilities of the basic rules. Scripting responses are flexible and easy to use. MOM supports its own scripting interface with VBScript or JScript, or you can use custom scripting languages such as PerlScript. The scripts are stored within the operations database and are visible and editable from the MOM Administration console.

  Another advantage of script responses is that the code and any updates delivered to the managed computer by the management server and agent use the rule delivery process. Scripts can be executed on either the agent or the management server, as appropriate.

• Managed code responses—This response type can call a method within a .NET Framework assembly. These assemblies can be developed in any .NET Framework-compliant language such as Visual C# .NET, Visual Basic .NET, Visual C++ .NET, Visual J# .NET, and so on.

  The assemblies are not delivered to the managed computer by the MOM infrastructure, so they require manual distribution and updating.

  A big advantage of the managed code response type is that it can call practically any .NET Framework assembly. These calls can be made either on the managed computer or the management server.

The programmatic responses are supported by two libraries:

• Scripting library—The COM-based scripting class library contains various runtime scripting objects, such as the Alert object, Event object, PerfData object, and ScriptContext object. These objects allow response scripts to interact with alerts, events, and performance data. See Chapter 22, "Using and Developing Scripts," for more information.

• Managed response class library—The managed code response class library is a .NET Framework class library and is equivalent to the COM-based scripting objects in the scripting library.

  The Microsoft.EnterpriseManagement.Mom.Runtime namespace contains classes and other types for creating MOM-managed code responses. The items in this namespace are defined in the MOM.Context (MOM.Context.dll) assembly.

Table 3.8 compares the capabilities of the two programmatic response types. Overall, script responses are generally easier to use and are better integrated into the MOM 2005 infrastructure. Managed code responses require more effort to create and deploy but perform faster and have a wider array of application interoperability.

Table 3.8. Comparison of Programmatic Responses

Feature	Script Responses	Managed Code Responses
Programmatic access to the response context.	Yes	Yes
Capability to create a new alert.	Yes	Yes
Capability to create a new state monitoring alert.	Yes	Yes
Capability to create a new MOM event.	Yes	No
Capability to create a new MOM performance data item.	Yes	No
Capability to create and submit computer discovery data.	Yes	No
Supported programming languages.	All COM-compatible scripting languages such as VBScript and JScript, including third-party extensions such as PerlScript	All .NET Framework languages, including third-party extensions
The programming language of the response must be explicitly specified.	Yes	No
Capability to store response as native code.	No	Yes
Stored in MOM database.	Yes	No
Capability to directly invoke an application component from a MOM rule.	No	Yes
Distribution mechanism.	Management Packs	Manual
Deployment mechanism.	MOM agents	Manual
Update mechanism.	MOM agent updates	Manual
Source code can be viewed in the MOM User Interface (UI).	Yes	No
Source code can be edited in the MOM UI.	Yes	No
Capability to call COM components.	Yes (Limited)	Yes
Capability to call .NET assemblies.	Yes (Limited)	Yes

We will look at script responses in more detail in Chapter 22.

Connecting to Other Management Platforms

MOM 2005 is not alone in the enterprise. In a typical enterprise Information Technology (IT) ecosystem, there might be several other management applications ranging from trouble ticket systems to management frameworks. MOM 2005 is designed to integrate with those systems. The integration is fundamentally at the alert level, which supports the following functionality:

• Sending new MOM alerts to external applications
• Sending MOM alert updates to external applications
• Receiving alert updates from external applications to the MOM system
• Receiving new alerts from external applications to the MOM system

In other words, the alert flow is bidirectional. MOM alerts can be forwarded to the other management application and kept in sync on both platforms as the alert changes. Alerts generated in the external management application can be inserted into the MOM database and also kept in sync. For example, if an alert is forwarded to a trouble ticket application, resolving the alert in either the MOM console or the trouble ticket console results in the alert being resolved in both consoles.

In fact, this is the method that MOM 2005 uses to communicate between management groups in a complex MOM hierarchy.

The components that provide the functionality listed previously are

• MOM Connector Framework (MCF)
• MOM Web Service
• Connector applications (local)
• Connector applications (remote)

The MOM Connector Framework is a managed .NET class library that provides an infrastructure for developing connector applications. The MCF manages the communication of alerts and alert updates between MOM 2005 and the connector application. The MCF provides business logic to support the development of custom connectors between MOM and other management applications. The MCF is accessible as both a standalone class library and a web service. Connector applications running locally on the management server can access the class library, and connector applications running remotely need to use the web service. The MCF provides support for connector applications running on non-Windows platforms such as UNIX through the MCF Web Service.

Although similar functionality can be achieved through developing custom applications using the Management Service Class Library (MCL) discussed in the next section of this chapter, there are several advantages to developing connector applications using the MCF. These advantages include the following:

• Tracking alerts—The MCF handles the details of tracking which alerts have been forwarded and which ones require updating. This tracking means that the connector application does not need to include the code and logic for those functions, which simplifies the development effort.
• Crossing firewalls—The MCF Web service uses port 80 and can easily cross firewalls if needed. The SSL protocol can be used to increase security, in which case port 443 is used, which also crosses firewalls.
• Alert knowledge—The MCF also provides easy access to the alert's product knowledge content, if that needs to be forwarded with the alert.
• Bidirectional logic—The MCF has built-in logic to handle bidirectional synchronization. The MCF makes it easy to develop two-way connectors, which keep alerts synchronized.

The MCF also allows alert suppression and other logic to be handled using the standard MOM rules. The connector application will not need to perform these tasks, providing better integration into the MOM 2005 infrastructure with less development effort. The rules are stored in management packs and are easily configured by MOM administrators with no need for development skills or controls.

Two namespaces are defined within the MCF. The Microsoft.EnterpriseManagement.Mom.Connector is included for backward compatibility with MOM 2000 SP1. The Microsoft.EnterpriseManagement.Mom.Connector.V2 is the more feature-rich version for MOM 2005 and can also be exposed via a web service on the management server. See the MOM 2005 Software Development Kit (SDK) at http://go.microsoft.com/fwlink/?LinkId=50272 for more information.

Management Service Class Library (MCL) and Custom Applications

The MOM Management Server Class Library (MCL) is used to develop custom applications that access the MOM 2005 operations database programmatically using Visual Studio .NET. The MCL is a .NET Framework class library that exposes MOM operations data, configuration information, and information about the rule hierarchy. This class library is only available on management servers, meaning that the custom applications that use the class library must be developed, tested, and run on the management servers.

The Microsoft.EnterpriseManagement.Mom namespace defines the general-purpose classes and types for accessing MOM operations data, rules, and computers. Items in this namespace are defined in two separate assemblies:

• The Microsoft.Mom.SDK Assembly (in Microsoft.Mom.Sdk.dll)
• The MOM.Context Assembly (in MOM.Context.dll)

Custom applications that use classes in this namespace should reference both assemblies in the Visual Studio .NET project. These assemblies can be found in the SDK Bin folder of the MOM program files folder and in the management server's Global Assembly Cache.

For more information on the Management Server Class Library and its usage, see the MOM 2005 Software Development Kit (SDK), available at the link referenced previously.