Things to Watch Out For
TLB sizes vary between UltraSPARC versions. The UltraSPARC I and II microprocessors (143 megahertz480 megahertz) data TLB has 64 entries that supports all four page sizes. User applications can use any of the four page sizes available.
However, the 750-megahertz UltraSPARC III microprocessor includes several small TLBs and one large 512-entry TLB that supports only 8-kilobyte entries. Thus, use of large pages is typically not beneficial on 750 megahertz UltraSPARC III systems.
The 900-megahertz UltraSPARC III onwards has two large 512 entry TLBs, one of which is configured automatically based on the most common page sizes in a process address space. A process using one large page size in addition to the base page size (8 kilobytes) will have one of its large TLBs automatically programmed to enable the large page size when there are eight or more pages using the larger page size within the process. Thus, large pages can be used very effectively on UltraSPARC III. However, since the large TLBs can be configured only for one page size per process at a time, only two pages sizes should be used concurrently (typically 8 kilobytes plus one other size).
Another point of interest is large page fragmentation. Large pages require contiguous physical memory, and if smaller pages fragment physical memory, a request for larger pages might not be fulfilled. When the system boots, a sizeable pool of large pages is available, but if a significant number of smaller page sizes are allocated and locked down, an application requesting larger page sizes might not have its request fulfilled with the larger page size. The application's request for memory is still satisfied, it just might not be allocated with its preferred larger page size. In this case, it will still run, but its mappings will be backed by smaller page sizes. For this reason, it is advisable to use pmap -xs to ensure that larger page sizes are indeed allocated.
There is, however, a way to minimize the amount of fragmentation. The Solaris 9 kernel will attempt to relocate pages in order to create the required contiguous memory; this works for all but locked pages. By enabling the kernel cage, you can vastly improve this situation because the kernel will be allocated from a small contiguous range of memory, thus minimizing the fragmentation of other pages within the system.
The kernel cage is enabled on Sun EnterpriseTM 10K and Sun FireTM 15K systems, but not on other systems. You can enable it by setting the kernel_cage_enable in /etc/system, as follows:
set kernel_cage_enable=1