Summary
The key to monitoring the performance on a system is to establish a baseline against which to compare your system's activity data. Start gathering activity data before the system goes into production. This is a proactive step and is the recommended approach.
Unless you truly enjoy analyzing reams of data, a number of worthy tools are available today that can be used to gather and analyze activity data in a heterogeneous environment. These tools exist and should be used. Here are some typical excuses:
They cost too much.
They are difficult to use.
We don't have the time to look over the data they report.
System administrators always seem to find time to fight fires, but it's much more efficient to plan ahead and be prepared than to react to problems after they occur. That is why it's important to be proactive with system performance monitoring.
If your budget is tight, use sar and some of the other utilities available in Solaris to gather system activity information. If your budget allows it, I suggest that you purchase either Sun's SyMON or one of the third-party performance-management software packages that are available. A single package can be used to manage all of your UNIX servers (as well as other types of systems) from one central location. Packages like Hewlett-Packard's Measureware and ViewPoint by Datametrics run on most major computing platforms and do a great deal to simplify the task of gathering system-performance metrics, generating reports, and performing trend analysis. These packages take up few system resources and run constantly to gather data on every part of your system, including applications and the network.
When it comes to tracking down and fixing a performance bottleneck, take a systematic approach. Make one change at a time and watch what affect it has on system performance. After all, the problem is probably an accumulation of issues and won't be fixed with one single step.
There are few complete references on system performance tuning. I recommend Adrian Cockcroft's the Sun Performance and Tuning: Java and the Internet for additional information about this topic.