Techniques for Optimizing Applications: High Performance Computing

Description

• Copyright 2002
• Edition: 1st

• Book
• ISBN-10: 0-13-093476-3
• ISBN-13: 978-0-13-093476-5

This book is a practical guide to performance optimization of computationally intensive programs on Sun UltraSPARC platforms. It is primarily intended for developers of technical or high performance computing (HPC) applications for the Solaris(tm) operating environment. This audience includes both independent software vendor (ISV) developers and noncommercial developers. It can also be used by end-users of HPC applications to help them better understand how applications utilize system resources.

The book presents information so that it follows logical stages of the process for application development and optimization. We pay special attention to issues related to parallel applications and to using appropriate performance measurement tools. Wherever applicable, sections are illustrated with code examples that show benefits of methods described.

Unless otherwise noted, topics in this book are not limited to a particular programming language, parallelization method, software version, or hardware product. However, emphasis is on techniques relevant to applications written in Fortran 77, Fortran 90, and C, because these languages are most commonly used in HPC and technical applications. Most topics can be applied to C++ programs; however, we do not address performance optimization issues specific to object-oriented programming.



Extras

Related Articles

Application Troubleshooting: Alternate Methods of Debugging

Metropolitan Area Sun Ray Services

Server Virtualization Using Trusted Solaris 8 Operating Environment

Solaris Application Performance Optimization



Sample Content

Table of Contents

Acknowledgments.


Preface.


Who Should Read This Book.


How This Book Is Organized.


Additional Resources.


Code Examples.


Typographical Conventions.

I. GETTING STARTED.

1. Introduction.

Performance Components. Hardware. Software. Optimization Process Overview. Serial Optimization. Parallel Optimization.

2. Overview of Sun UltraSPARC Solaris Platforms.

UltraSPARC-Based Desktop and Server Product Line. UltraSPARC-Based Workstations. UltraSPARC-Based Servers. Sun Technical Compute Farm. Solaris Operating Environment. Sun WorkShop and Forte Developer Tools. HPC ClusterTools Software. Summary.

3. Application Development on Solaris.

Development Basics. Standards Conformance. Binary Compatibility. Source Code Verification Tools. Checking C Programs. Checking Fortran Programs. Additional Source Code Analysis Tools. 64-bit Development and Porting. Fortran Porting. Language Interoperability. Fortran 95 and Fortran 77. C and Fortran. Linking Mixed Languages. Summary.

II. OPTIMIZING SERIAL APPLICATIONS.

4. Measuring Program Performance.

Measurement Methodology. Benchmarking Guidelines. Measurement Tools. Program Timing Tools. Timing Entire Program. Timing Program Portions. Fine-Grained Timing Measurement. Program Profiling Tools. Profiling With prof and gprof. Profiling With tcov. Profiling Tools in Forte Developer 6. Process and System Monitoring Tools. /proc Tools. Process Tracing Tools. System Monitoring Tools. Hardware Counter Measurements. Monitoring Tools. Hardware Counter Overflow Profiling. Code Instrumentation With libcpc Calls. Summary.

5. Basic Compiler Optimizations.

Compilation Overview. Structure of Sun Compilers. Using Sun Compilers. -fast and -xtarget Options. Basic Guidelines. -xarch. Specifying Target Architecture. Generation of Conditional Move Instructions. Creating 64-bit Binaries. -xchip. -xO Optimization Level. -xinline, -xcrossfile. -xdepend. - xvector. -xsfpconst. -xprofile=collect, use. -xprefetch. Summary.

6. Advanced Compiler Optimizations.

IEEE Floating-Point Arithmetic. Binary Storage Format. Trap Handling and -ftrap. Gradual Underflow and -fns. -fsimple. -dalign. -xsafe= mem. Pointer Alias Analysis Options. -xrestrict. -xalias_level. -stackvar. Compiler Directives and Pragmas. pragma pipeloop. pragma opt. pragma prefetch. pragma pack. pragma align. Pointer Alias Analysis Pragmas. Summary.

7. Linker and Libraries in Performance Optimization.

Linking Overview. Static and Dynamic Linking. Structure of an ELF Binary. Solaris Linker Usage. Linking Static and Dynamic Libraries. Weak Symbol Binding. Linker Mapfiles. Linking Optimized Math Libraries. Creating Architecture-Specific Libraries. $PLATFORM and $ISALIST Linker Tokens. $ORIGIN Token. Runtime Linker in Profiling and Debugging. Interposing Libraries. Using LD_PROFILE and LD_DEBUG. Summary.

8. Source Code Optimization.

Overview of Memory Hierarchy. Memory Levels. Memory Organization of UltraSPARC-Based Systems. Memory Hierarchy Optimizations. Cache Blocking. Reducing Cache Conflicts. Reducing TLB Misses. Page-Coloring Effects. Memory Bank Interleaving. Inlining Assembly Templates. Optimal Data Alignment. Restructuring for Better Data Alignment. Double-Word Load and Store Generation. Cache Line Alignment. Preventing Register Window Overflow. Aliasing Optimizations. Aliasing in Fortran Programs. Pointer Aliasing in C Programs. Summary.

9. Loop Optimization.

Loop Unrolling and Tiling. Loop Interchange. Loop Fusion. Loop Fission. Loop Peeling. Loops With Conditionals. Strength Reduction in Loops. Division Replacement. Operations on Complex and Real Operands. Summary.

III. OPTIMIZING PARALLEL APPLICATIONS.

10. Parallel Processing Models on Solaris.

Parallelization Overview. Parallel Scalability Concepts. Parallel Architectural Models. Parallel Programming Models. Multithreading Models. Compiler Auto-Parallelization. OpenMP Compiler Directives. Explicit Multithreading Using P-threads. Multiprocessing Models. UNIX fork/exec Model. MPI Message-Passing Model. Hybrid Models. Summary.

11. Parallel Performance Measurement Tools.

Measurement Methodology. Timing a Parallel Program and Its Portions. Parallel Performance Monitoring With Forte Developer 6 Tools. Trace Normal Form Utilities. Analyzing and Profiling MPI Programs With the Prism Environment. Parallel System Monitoring Tools. Binding a Program to a Set of Processors. Measuring Performance on a Per-CPU Basis. Monitoring Kernel Lock Statistics. Hardware Counter Tools for Parallel Performance Monitoring. cpustat and cputrack Tools. busstat Tool. Summary.

12. Optimization of Explicitly Threaded Programs.

Programming Models for Multithreading. Master-Slave Model. Worker-Crew Model. Pipeline Model. Multithreading in the Solaris Operating Environment. Thread Models. Compiling Threaded Applications. True and False Data Sharing. Synchronization and Locking. Thread Stack Size. Thread Creation Issues. Pool of Threads. Pool of Threads With Spin Locks. Summary.

13. Optimization of Programs Using Compiler Parallelization.

Parallelization Support in Sun Compilers. Parallelization Model. Runtime Settings. Automatic Parallelization. Explicit Parallelization. OpenMP Support in Fortran 95 Compiler. OpenMP Programming Styles. Section Parallel Style. Single Program Multiple Data (SPMD) Style. OpenMP Performance Considerations. Synchronization Issues. Data Scoping. Memory Bandwidth Requirement. OpenMP and P-threads. Parallel Sun Performance Library. Linking the Library. Runtime Issues. 64-bit Integer Arguments. Fortran SUNPERF Module. Summary.

14. Optimization of Message-Passing Programs.

Programming Models and Performance Considerations. Workload Distribution. Pipeline Method. Loop Parallelization Methods. Communication Metrics. Sun MPI Implementation. Building and Running MPI Programs. Dynamic Process Management. MPI I/O. Sun MPI Environment Variables. Diagnostic Information. Dedicated and Timeshared System Execution. Optimized Collectives. Point-to-Point Communication. General Performance. Sun Scalable Scientific Subroutine Library. MPI, OpenMP, and Hybrid Approaches. MPI and OpenMP Approaches. Hybrid Approach. Summary.

IV. APPENDICES.

A. Commands That Identify System Configuration Parameters.

Hardware Parameters. System Configuration. Parameters of Installed Software and Hardware. Summary of Commands.

B. Architecture of UltraSPARC Microprocessor Family.

UltraSPARC I and II Processors. UltraSPARC III Processor. UltraSPARC IIi Processor. UltraSPARC IIe Processor.

C. Architecture of UltraSPARC Interconnect Family.

Ultra Port Architecture Interconnect. Gigaplane Interconnect. Gigaplane XB Crossbar Interconnect. Fireplane Interconnect.

D. Hardware Counter Performance Metrics.

CPU Counters. System ASIC Counters.

E. Interval Arithmetic Support in Forte Developer 6 Fortran 95 Compiler.

Interval Arithmetic Basics. Solution of Nonlinear Problems.

F. Differences in I/O Performance.

Reading a File with read/lseek. Reading a File with fread/fseek. Mapping a File to Memory.

References.
Index.

Preface

Preface

This book is a practical guide to optimizing performance of computationally intensive applications on Sun UltraSPARC platforms. It offers techniques for improving performance of applications that are predominantly compute-intensive or CPU-bound.

We wrote this book with a general enough scope so that it would be useful to as many developers of technical applications on Sun platforms as possible. Also, we made the material practical by showing developers how to use each optimization method.

For information on related topics such as system configuration and tuning, or improving the I/O and network performance, we refer readers to other resources.

This book differs from other books and technical documents written aboutperformance optimization of high performance computing (HPC) applications. In many cases, other resources either give a detailed description of a product or provide general recommendations that are sometimes difficult to apply to practical tasks. In addition, some older resources are not as useful because of changes in technology.

Though many of the techniques we offer apply to other platforms, we limited the scope of this guide to Sun compilers and UltraSPARC-based Solaris systems. We address new features in Sun compilers and in the Solaris Operating Environment, and we show readers how to use these products to get maximum performance on Sun hardware.

Who Should Read This Book

This guide is primarily for developers of technical or HPC applications for Solaris. This audience includes both independent software vendor (ISV) developers and non-commercial developers.

Developers creating or optimizing applications in the following fields may benefit from reading this book:

• Mechanical computer-aided engineering (MCAE)
• Electronic design automation (EDA)
• Computational chemistry
• Bioinformatics
• Operations research
• Financial modeling
• Reservoir simulation and seismic modeling
• Mechanical computer-aided design (MCAD) modeling
• Graphics rendering and imaging
• Climate and weather modeling

This book may also be helpful to technical application end-users in understanding the principles of HPC and how an application utilizes system resources.

We assume the reader has:

• familiarity with development basics in UNIX environments
• a working knowledge of programming in C and Fortran languages
• familiarity with computer architecture
• experience in parallel programming
• a basic knowledge of SPARC assembly (desirable)

Unless otherwise noted, topics in this book are not limited to a programming language, parallelization method, or software version. However, emphasis is on techniques relevant to applications written in Fortran 77, Fortran 90, and C, because these languages are most commonly used in HPC and technical applications.

Most topics can be applied to C++ programs; however, we do not address performance optimization issues for object oriented programming. We refer readers to other resources.

How This Book Is Organized

This book presents information so that it follows logical stages of the process for application development and optimization. We pay special attention to issues related to parallel applications and to using appropriate performance monitoring tools.

Wherever applicable, sections are illustrated with code examples that show benefits of methods described.

Part I - Getting Started

Chapter 1 "Introduction," introduces optimization for HPC applications. We describe the basics of the optimization process and illustrate it with flow charts for serial and parallel optimization.

Chapter 2 "Overview of Sun UltraSPARC Solaris Platforms," describes the available "tools of trade" for HPC developers using Solaris platforms. It gives an overview of Sun hardware and software products for technical computing. Also, the chapter introduces software development tools.

Chapter 3 "Application Development on Solaris," considers development and porting issues on Sun platforms. It includes sections on binary compatibility between platforms, standards conformance, code verification tools, language interoperability, and 64-bit porting issues.

Part II - Optimizing Serial Applications

Chapter 4 "Measuring Program Performance," focuses on tools that measure application performance. Accurate measurement of performance is crucial in tuning. We describe accurate timers available on Solaris, profiling tools, Forte Developer 6 Performance Analyzer, hardware performance counter access tools on UltraSPARC processors, and other system monitoring tools.

Chapter 5 "Basic Compiler Optimizations," introduces basic compiler optimizations and how to use compiler flags correctly. Options covered in this chapter are safe and generally can be applied without knowledge of any specifics of the application. The impact of using these flags is illustrated with examples, and analysis of the generated code with and without the options is presented.

Chapter 6 "Advanced Compiler Optimizations," extends Chapter 5 and gives an overview of techniques that enable aggressive compiler optimizations. These often result in additional performance gains but may also lead to incorrect answers or spurious side-effects. Also, we cover performance related compiler pragmas and directives, which can be inserted in a program. Information about a program can be passed to the compiler, allowing additional optimizations.

Chapter 7 "Linker and Libraries in Performance Optimization," highlights optimized libraries and features of the Solaris linker that can be used for application optimization. We describe the platform-specific optimized math libraries whose use can result in significant performance gains. We show linker techniques that allow linking of these platform-specific libraries in a portable fashion.

Chapter 8 "Source Code Optimization," provides an overview of tuning techniques at the source code level. The techniques were selected from the point of view of better utilizing the underlying architectural features of UltraSPARC systems. We pay special attention to memory hierarchy utilization such as cache blocking and reducing the translation lookaside buffer (TLB) misses. We present ways of simplifying the code to allow better compiler optimizations, such as alias disambiguation in C programs, to take place.

Chapter 9 "Loop Optimization," focuses on optimizing loops, one of the most commonly used constructs in scientific and HPC programs. We discuss ways in which developers can help the compiler control loop fusion and fission, as well as perform loop peeling. We show examples of register-tiling and consider loops with branches.

Part III - Optimizing Parallel Applications

Chapter 10 "Parallel Processing Models on Solaris," introduces concepts of parallel programming and different parallelization models available on Solaris/SPARC systems: automatic compiler parallelization, directives-based parallelism, explicit multithreading, UNIX fork/exec, message passing model, and hybrid programming (combined directives and message-passing).

Chapter 11 "Parallel Performance Measurement Tools," details the tools for performance measurement and monitoring of parallel programs. Similar to Chapter 4, we focus on accurate timers for timing parallel programs, tools for measuring synchronization and communication overheads, tools for measuring hardware counters, and tools for multiprocessor system monitoring.

Chapter 12 "Optimization of Explicitly Threaded Programs," provides an overview of explicit multithreading of programs using P-threads and Solaris threads. An overview of thread scheduling models in Solaris and their relevance to HPC programs is given and techniques for decreasing synchronization overheads are described.

Chapter 13 "Optimization of Programs Using Compiler Parallelization," covers support and optimization techniques for automatic and directive-based parallelization in Sun compilers. Special emphasis is given to tuning OpenMP programs using the Fortran 95 compiler. OpenMP programming styles and data-scoping issues are illustrated with examples. Comparisons between OpenMP and P-threads approaches are presented.

Chapter 14 "Optimization of Message-Passing Programs," describes message-passing models and how to tune MPI programs. We present an overview of message-passing programming models, compiling and linking programs using Sun MPI, and using Sun MPI environment variables. This chapter describes approaches for optimizing point-to-point and global communication with Sun MPI, using the S3L scientific library and using a hybrid OpenMP/MPI model.

Part IV - Appendices

Appendix A "Commands That Identify System Configuration Parameters," lists useful Solaris commands that identify system configuration parameters.

Appendix B "Architecture of UltraSPARC Microprocessor Family," gives an overview of architectural features of the UltraSPARC microprocessor family.

Appendix C "Architecture of UltraSPARC Interconnect Family," describes the architecture of interconnect technologies for UltraSPARC systems.

Appendix D "Hardware Counter Performance Metrics," shares some useful performance metrics that can be derived from hardware counters on UltraSPARC systems.

Appendix E "Interval Arithmetic Support in Forte Developer 6 Fortran 95 Compiler," gives an overview of interval arithmetic support in the Forte Developer 6 Fortran 95 compiler.

Appendix F "Differences in I/O Performance," considers the performance of different I/O techniques.

Additional Resources

To keep the scope of this book manageable, we intentionally omitted many subjects related to performance optimization. Our criteria was to omit subjects that were not applicable to a wide range of applications. Many of these subjects are presented in other documentation for Sun products. The following is a list of publications you may find useful for more narrowly focused subjects:

• Numerical Computation Guide
• Fortran Programming Guide
• Fortran User's Guide
• Analyzing Program Performance with Sun Workshop
• C User's Guide
• Forte 6 update 1 C User's Guide Supplement
• Linker and Libraries Guide
• Sun Performance Library Reference
• Multithreaded Programming Guide
• Programming Utilities Guide
• 64-bit Developer's Guide
• Solaris Tunable Parameters Reference Manual
• Sun HPC ClusterTools 3.1 Performance Guide
• Sun MPI 4.1 Programming and Reference Guide
• Prism 6.1 User's Guide
• Sun HPC ClusterTools 3.1 Installation Guide

All these publications are available online at http://docs.sun.com. We strongly recommend that developers visit this site, because nearly all published Sun documentation is available there. Printed versions are available from Sun Documentation Center at Fatbrain:

http://www1.fatbrain.com/documentation/sun

The following publications are related to UltraSPARC microprocessors:

• UltraSPARC I and II User's Manual
• UltraSPARC IIi User's Manual

These publications are available at:

http://www.sun.com/microelectronics/manuals

Other sites of great use for developers are http://www.sun.com/developers/ and http://soldc.sun.com/, which contains current information for the Sun developer community.

A description of Sun product lines is available at http://www.sun.com/desktop and http://www.sun.com/servers. Sun products and solutions for the HPC are listed at http://www.sun.com/hpc.

For specialized books and additional theoretical information on application optimization, we refer readers to other sources. There are many excellent books on topics such as optimizing compilers, software tuning techniques, and efficient parallelization. The following are some helpful resources:

• J. Hennessy, D. Patterson - Computer Architecture: A Quantitative Approach, Second Edition; Morgan Kaufmann Publishing, 1996
• K. Dowd, Ch. Severance - High Performance Computing, Second Edition; O'Reilly & Associates, 1998
• D. E. Culler, J. P. Singh, A. Gupta - Parallel Computer Architecture: A Hardware Software Approach, Morgan Kaufmann Publishing, 1999
• S. S. Muchnick - Advanced Compiler Design and Implementation; Morgan Kaufmann Publishing, 1997.
• S. Kleinman, D. Shah, B. Smaalders - Programming with Threads; SunSoft Press, A Prentice Hall Title, 1996.
• W. Gropp, E. Lusk, A. Skjellum - Using MPI: Portable Parallel Programming with Message-Passing Interface (Scientific and Engineering Computation Series); Second Edition, MIT Press, 1999.
• R. Chandra, L. Dagum, D. Kohr, D. Maydan, J. McDonald, R. Menon - Parallel Programming in OpenMP; Morgan Kaufmann Publishing, 2000.

More resources are listed in the References section at the end of this book.

Readers interested in "hands on" training should check with the Sun Educational Services to determine if a class is scheduled and enrollment is open.

Code Examples

Many sections in this book are illustrated with code examples that show benefits of optimization techniques and coding practices. The code examples can be downloaded from the Sun BluePrints site:

http://www.sun.com/blueprints/tools

A makefile is provided for each chapter so that the examples can be run with a single make command.

Unless otherwise noted, all examples and results presented in this book use Forte Developer 6 compilers and the HPC 3.1 ClusterTools release. The results for serial runs were obtained on Sun Ultra 60, Sun Ultra 80, and Sun Blade 1000 systems. The results for parallel runs were performed on Sun Enterprise 4500, Sun Enterprise 10000 servers, and a Sun technical compute farm. The system parameters are listed in TABLE P-1 and TABLE P-2, respectively. More information about Sun platforms is in Chapter 2.



Updates

Submit Errata

