This chapter is from the book
This chapter is from the book
This chapter is from the book
JMX and MBeans
Built into Cassandra and the JVM is the capability to use the JMX, or Java Management Extensions. In other words, using JMX gives you the capability to manage your servers remotely or check into settings programmatically, including the memory, CPU, threads, Gossip, or any other part of the system that has been instrumented in JMX. Instrumentation is what enables the application to provide application-specific information to be collected by external tools. JMX also gives you the ability to control certain aspects of this information.
JConsole
MBeans, or Managed Beans, are a special type of JavaBean that makes a resource inside the application or the JVM available externally. The standard tool that ships with Java for managing the MBeans is JConsole.
The most common use case for accessing a Cassandra server is that the server will be remote and you probably won’t have console access to it. It is also highly recommended that you run JConsole remotely as it is a heavy user of resources on a machine and can steal those resources away from the Cassandra node. If this is the case, you can use SSH tunneling to bring up JConsole. When you SSH, be sure to use the –X switch to ensure that X11 forwarding is on. This is what enables you to use JConsole over the network. After SSHing into the machine running Cassandra and finding the JConsole binary, just execute it as you would any normal binary. Assuming everything is configured correctly, you will get a JConsole login window as shown in Figure 8.1.
)
Figure 8.1 JConsole login window when logging in via localhost
Click on the radio button labeled Remote Process and type localhost:7199. If you have Cassandra set up with authentication, you will need to put in the username and password as well. Port 7199 is the default JMX port for Cassandra. The first thing you will notice once a connection has been established is that there are multiple tabs that contain information for you to look through. These tabs are Overview, Memory, Threads, Classes, VM Summary, and MBeans.
The Overview, Memory, and Threads tabs are sets of graphs that provide insight into the current state of the system. The Classes graph is simply a graph of how many classes are loaded into the JVM at the current time (or a different time range that you choose). The VM Summary is an overview of what the current view of the Java Virtual Machine is.
Memory
The Memory tab consists of graphs about the current state of the memory (see Figure 8.2). One of the most important memory stats that you want to be aware of is your current heap usage. It is also the default graph that comes up on the Memory tab. As with everything else in your system, it is helpful to know what a good baseline is for heap usage during normal system operations. If you have a 10GB heap set and you find during most of your operations that you are using only 3GB, you can likely reduce your JVM maximum heap size. This will enable you to reduce your memory footprint on the system and possibly even speed up your GCs.
)
Figure 8.2 JConsole Memory tab displaying heap usage graphs
Cassandra also does a lot of work off-heap. Things like bloom filters in version 1.2 and forward are off-heap now. Keeping tabs on what the off-heap memory usage looks like is also very important.
Garbage collection is also one of the metrics that can be viewed from the Memory tab. If needed (though typically not recommended unless you know what you are doing), you can even force a GC from the Memory tab in JConsole. In Cassandra 1.1 and later, there are even helpful bars that display the total memory used versus memory available both on-heap and off-heap.
Threads
The Threads tab in JConsole is dedicated to showing the current and peak usage patterns of various thread stages in Cassandra (see Figure 8.3). These include everything that you would normally see in the logs from things like the CommitLog handler and compaction scheduler all the way to garbage collection, Gossip, and streaming. It is also helpful here to see how many threads your system uses normally as well as under load.
)
Figure 8.3 JConsole Threads tab displaying the number of threads Cassandra is currently using and general information on the main Cassandra thread
MBeans
The final tab in JConsole is the tab for MBeans (see Figure 8.4). There are a lot of MBeans that are useful for assessing the state of your Cassandra node and your Cassandra cluster. You will notice that there are a few groupings here that can be expanded. Other than the standard Java MBeans that are available to every agent, there are several groupings specific to Cassandra. All of their class paths start with org.apache.cassandra.
)
Figure 8.4 JConsole MBeans tab view with the Cassandra DB and Internal trees opened
There are a lot of MBeans provided by an application as complex as Cassandra. There is no need to cover all of them as you can easily explore them on your own using the JConsole interface.
As a high-level overview, they are broken down into the following categories:
- DB. The MBeans stored in the DB section cover everything about the actual data storage part of Cassandra. You can view information about the cache and CommitLogs, or even information about the individual ColumnFamilys that you have created in each of the keyspaces. HintedHandoffManager, EndPointSnitchInfo, and CompactionManager information can also be found here.
- Internal. In the Internal section, there are MBeans that cover the state and statistics around the staged architecture. More specifically, you can find information about the state of Gossip and HintedHandoff as opposed to just finding information about the managers as in the DB section.
- Metrics. The metrics available in this section are ClientRequestMetrics. These are things like read and write timeouts and Unavailable errors.
- Net. The network section houses information about internode communication. This includes information about the FailureDetector, Gossiper, MessagingService, and StreamingService.
- Request. The MBeans in the Request section are about tasks related to read, write, and replication.
When you select any MBean in the tree, its MBeanInfo and MBean descriptor are displayed on the right-hand side of the window. If any additional attributes, operations, or notifications are available, they will appear in the tree as well, below the selected MBean.
Each one of these sections of MBeans provides access to a large amount of information, giving you insight into both the system as a whole and the individual nodes. Familiarizing yourself with what is available to you as an admin will help you when it comes time to instrument JMX-level checks from within your monitoring system.