Software Build Systems: Principles and Experience

Description

• Copyright 2011
• Dimensions: 7" x 9-1/8"
• Edition: 1st

• eBook
• ISBN-10: 0-321-76944-9
• ISBN-13: 978-0-321-76944-2

“This book represents a thorough and extensive treatment of the software build process including the choices, benefits, and challenges of a well designed build process. I recommend it not only to all software build engineers but to all software developers since a well designed build process is key to an effective software development process.”

—Kevin Bodie, Director Software Development, Pitney Bowes Inc.

“An excellent and detailed explanation of build systems, an important but often overlooked part of software development projects. The discussion of productivity as related to build systems is, alone, well worth the time spent reading this book.”

—John M. Pantone, Objectech Corporation, VP, IT Educator and Course Developer

“Peter Smith provides an interesting and accessible look into the world of software build systems, distilling years of experience and covering virtually every type of tool in the build engineer’s toolbox. Well organized, well written, and very thorough; I would recommend this book to anyone with a build system under their responsibility.”

—Jeff Overbey, Project Co-Lead, Photran

“Software Build Systems teaches how to think about building software. It surveys the tools and techniques for building software products and the ways things go wrong. This book will appeal to those new to build systems as well as experienced build system engineers.”

—Monte Davidoff, Software Development Consultant, Alluvial Software, Inc.

Inadequate build systems can dramatically impact developer productivity. Bad dependencies, false compile errors, failed software images, slow compilation, and time-wasting manual processes are just some of the byproducts of a subpar build system. In Software Build Systems, software productivity expert Peter Smith shows you how to implement build systems that overcome all these problems, so you can deliver reliable software more rapidly, at lower cost.

Smith explains the core principles underlying highly efficient build systems, surveying both system features and usage scenarios. Next, he encapsulates years of experience in creating and maintaining diverse build systems–helping you make well-informed choices about tools and practices, and avoid common traps and pitfalls. Throughout, he shares a wide range of practical examples and lessons from multiple environments, including Java, C++, C, and C#. Coverage includes

• Mastering build system concepts, including source trees, build tools, and compilation tools

• Comparing five leading build tools: GNU Make, Ant, SCons, CMake, and the Eclipse IDE’s integrated build features

• Ensuring accurate dependency checking and efficient incremental compilation

• Using metadata to assist debugging, profiling, and source code documentation

• Packaging software for installation on your target machine

• Best practices for managing complex version-control systems, build machines, and compilation tools

If you’re a developer, this book will illuminate the issues involved in building and maintaining the build system that’s best for your team. If you’re a manager, you’ll discover how to evaluate your team’s build system and improve its effectiveness. And if you’re a build “guru,” you’ll learn how to optimize the performance and scalability of your build system, no matter how demanding your requirements are.



Sample Content

Table of Contents

PREFACE     xxi

Why Do Build Systems Become Complex?     xxiii

The True Cost of a Build System     xxiv

The Focus of This Book     xxvii

Who Should Read This Book?     xxviii

How This Book Is Organized     xxix

Summary     xxxi

ACKNOWLEDGMENTS     xxxiii

ABOUT THE AUTHOR     xxxv

PART I THE BASICS     1

Chapter 1 BUILD SYSTEM OVERVIEW     3

What Is a Build System?     3

Compiled Languages     4

Interpreted Languages     6

Web-Based Applications     6

Unit Testing     7

Static Analysis     8

Documentation Generation     9

Components of a Build System     10

Version-Control Tools     10

Source and Object Trees     11

Compilation Tools and Build Tools     13

Build Machines     14

Release Packaging and Target Machines     15

The Build Process and Build Description     16

How a Build System Is Used     18

Build-Management Tools     19

Build System Quality     21

Summary     21

Chapter 2 A MAKE-BASED BUILD SYSTEM     23

Calculator Example     23

Creating a Simple Makefile     26

Simplifying the Makefile     28

Additional Build Targets     29

Using a Framework     31

Summary     33

Chapter 3 THE RUNTIME VIEW OF A PROGRAM     35

Executable Programs     36

Native Machine Code     36

Monolithic System Images     37

Full Program Interpretation     38

Interpreted Byte Codes     39

Libraries     40

Static Linking     41

Dynamic Linking     42

Configuration and Data Files     43

Distributed Programs     44

Summary     46

Chapter 4 FILE TYPES AND COMPILATION TOOLS     47

C/C++     48

Compilation Tools     49

Source Files     50

Assembly Language Files     52

Object Files     53

Executable Programs     56

Static Libraries     57

Dynamic Libraries     58

C++ Compilation     59

Java     60

Compilation Tools     61

Source Files     62

Object Files     63

Executable Programs     65

Libraries     67

C#     68

Compilation Tools     68

Source Files     69

Executable Programs     71

Libraries     74

Other File Types     76

UML-Based Code Generation     77

Graphic Images     79

XML Configuration Files     81

Internationalization and Resource Bundles     81

Summary     82

Chapter 5 SUBTARGETS AND BUILD VARIANTS     83

Building Subtargets     84

Building Different Editions of the Software     86

Specifying the Build Variant     87

Varying the Code     90

Building Different Target Architectures     94

Multiple Compilers     94

Platform-Specific Files/Functions     95

Multiple Object Trees     96

Summary     98

PART II THE BUILD TOOLS     99

Chapter 6 MAKE     107

The GNU Make Programming Language     108

Makefile Rules to Construct the Dependency Graph     109

Makefile Rule Types     110

Makefile Variables     112

Built-In Variables and Rules     114

Data Structures and Functions     116

Understanding Program Flow     119

Further Reading     122

Real-World Build System Scenarios     123

Scenario 1: Source Code in a Single Directory     123

Scenario 2(a): Source Code in Multiple Directories     125

Scenario 2(b): Recursive Make over Multiple Directories     126

Scenario 2(c): Inclusive Make over Multiple Directories     130

Scenario 3: Defining New Compilation Tools     137

Scenario 4: Building with Multiple Variants     138

Scenario 5: Cleaning a Build Tree     140

Scenario 6: Debugging Incorrect Builds     142

Praise and Criticism     144

Praise     144

Criticism     146

Evaluation     148

Similar Tools     149

Berkeley Make     149

NMake     150

ElectricAccelerator and SparkBuild     151

Summary     153

Chapter 7 ANT     155

The Ant Programming Language     156

A Little More Than “Hello World”     157

Defining and Using Targets     158

Ant’s Flow of Control     161

Defining Properties     162

Built-In and Optional Tasks     164

Selecting Multiple Files and Directories     168

Conditions     170

Extending the Ant Language     172

Further Reading     173

Real-World Build System Scenarios     174

Scenario 1: Source Code in a Single Directory     174

Scenario 2(a): Source Code in Multiple Directories     175

Scenario 2(b): Many Directories, with Multiple build.xml Files     175

Scenario 3: Defining New Compilation Tools     179

Scenario 4: Building with Multiple Variants     183

Scenario 5: Cleaning a Build Tree     188

Scenario 6: Debugging Incorrect Builds     188

Praise and Criticism     191

Praise     191

Criticism     191

Evaluation     193

Similar Tools     193

NAnt     194

MSBuild     194

Summary     196

Chapter 8 SCons     197

The SCons Programming Language     198

The Python Programming Language     199

Simple Compiling     202

Managing Build Environments     206

Program Flow and Dependency Analysis     210

Deciding When to Rebuild     212

Extending the Language     214

Other Interesting Features     218

Further Reading     219

Real-World Build System Scenarios     219

Scenario 1: Source Code in a Single Directory     219

Scenario 2(a): Source Code in Multiple Directories     219

Scenario 2(b): Multiple SConstruct Files     220

Scenario 3: Defining New Compilation Tools     222

Scenario 4: Building with Multiple Variants     224

Scenario 5: Cleaning a Build Tree     226

Scenario 6: Debugging Incorrect Builds     226

Praise and Criticism     229

Praise     230

Criticism     231

Evaluation     231

Similar Tools     232

Cons     232

Rake     233

Summary     235

Chapter 9 CMAKE     237

The CMake Programming Language     238

CMake Language Basics     239

Building Executable Programs and Libraries     240

Control Flow     243

Cross-Platform Support     246

Generating a Native Build System     248

Other Interesting Features and Further Reading     254

Real-World Build System Scenarios     255

Scenario 1: Source Code in a Single Directory     255

Scenario 2: Source Code in Multiple Directories     256

Scenario 3: Defining New Compilation Tools     257

Scenario 4: Building with Multiple Variants     259

Scenario 5: Cleaning a Build Tree     260

Scenario 6: Debugging Incorrect Builds     260

Praise and Criticism     261

Praise     261

Criticism     262

Evaluation     262

Similar Build Tools     263

Automake     263

Qmake     264

Summary     264

Chapter 10 ECLIPSE     267

The Eclipse Concepts and GUI     268

Creating Projects     269

Building a Project     276

Running a Project     282

Using the Internal Project Model     285

Other Build Features     286

Further Reading     288

Real-World Build System Scenarios     288

Scenario 1: Source Code in a Single Directory     288

Scenario 2: Source Code in Multiple Directories     290

Scenario 3: Defining New Compilation Tools     291

Scenario 4: Building with Multiple Variants     292

Scenario 5: Cleaning a Build Tree     295

Scenario 6: Debugging Incorrect Builds     296

Praise and Criticism     296

Praise     297

Criticism     297

Evaluation     298

Similar Build Tools     299

CDT for Eclipse, C/C++ Development Tooling     299

Summary     301

PART III ADVANCED TOPICS     303

Chapter 11 DEPENDENCIES     305

The Dependency Graph     307

Incremental Compilation     307

Full, Incremental, and Subtarget Builds     308

The Problem with Bad Dependencies     310

Problem: Missing Dependencies Causing a Runtime Error     310

Problem: Missing Dependencies Causing a Compile Error     311

Problem: Unwanted Dependencies Causing Excess Rebuilding     312

Problem: Unwanted Dependencies Causing Failed Dependency Analysis     312

Problem: Circular Dependencies     313

Problem: Implicit Sequencing As a Substitute for Dependencies     314

Problem: The Clean Target Doesn’t Clean Everything     315

Step 1: Computing the Dependency Graph     315

Gathering Exact Dependencies     316

Caching the Dependency Graph     319

Updating the Cached Dependency Graph     320

Step 2: Determining Which Files Are Out-of-Date     324

Time Stamp-Based Methods     324

Checksum-Based Methods     326

Flag Comparison     328

Advanced Methods     329

Step 3: Sequencing the Compilation Steps     330

Summary     333

Chapter 12 BUILDING WITH METADATA     335

Debugging Support     336

Profiling Support     338

Coverage Support     340

Source Code Documentation     341

Unit Testing     344

Static Analysis     348

Adding Metadata to a Build System     349

Summary     350

Chapter 13 SOFTWARE PACKAGING AND INSTALLATION     351

Archive Files     352

Packaging Scripts     353

Other Archive Formats     356

Improvements     356

Package-Management Tools     359

The RPM Package Manager Format     360

The rpmbuild Process     361

An Example RPM Spec File     363

Creating the RPM File from the Spec File     369

Installing the RPM Example     371

Custom-Built GUI Installation Tools     373

The Nullsoft Scriptable Install System (NSIS)     374

The Installer Script     376

Defining the Pages     379

The License Page     380

Directory Selection     381

The Main Component     381

The Optional Components     383

Defining a Custom Page     385

The Installation Page and the Uninstaller     387

Summary     388

Chapter 14 VERSION MANAGEMENT     391

What Should Be Version-Controlled     392

Build Description Files     393

References to Tools     395

Large Binary Files     400

Source Tree Configurations     401

What Should Not Be in the Source Tree     402

Generated Files in the Source Tree     402

Generated Files Under Version Control     404

Build-Management Scripts     405

Version Numbering     406

Version-Numbering Systems     406

Coordinating and Updating the Version Number     407

Storing and Retrieving the Version Number     410

Summary     411

Chapter 15 BUILD MACHINES     413

Native and Cross-Compilation     414

Native Compilation     414

Cross-Compilation     415

Hybrid Environments     416

Centralized Development Environments     416

Why Build Machines Differ     418

Managing Multiple Build Machines     421

Open-Source Development Environments     424

GNU Autoconf     428

The High-Level Workflow     428

An Autoconf Example     430

Running autoheader and autoconf     434

Running the configure Script on the Build Machine     435

Using the Configuration Information     437

Summary     438

Chapter 16 TOOL MANAGEMENT     441

Rules for Managing Tools     442

Tool Rule #1: Take Notes     442

Tool Rule #2: Use Version Control for the Source Code     443

Tool Rule #3: Periodically Upgrade Tools     444

Tool Rule #4: Use Version Control for the Tool Binaries     445

Breaking the Rules     448

Writing Your Own Compilation Tools     449

Custom-Written Tools with Lex and Yacc     450

Summary     453

PART IV SCALING UP     455

Chapter 17 REDUCING COMPLEXITY FOR END USERS     457

Build Frameworks     458

Developer-Facing Portion of the Build Description     459

Framework Portion of the Build Description     460

Convention over Configuration     461

Maven: An Example Build Tool     462

Reasons to Avoid Supporting Multiple Variants     463

You’ll Have More Variants to Test     463

Source Code Becomes Messy     465

Build Times Can Increase     465

Higher Disk Space Requirements     466

Various Ways to Reduce Complexity     466

Use a Modern Build Tool     466

Automatically Detect Dependencies     467

Keep Generated Files out of the Source Tree     467

Ensure That Cleaning a Build Tree Works Correctly     468

Abort the Build After the First Error     468

Provide Meaningful Error Messages     470

Validate Input Parameters     470

Don’t Overengineer Build Scripts     471

Avoid Using Cryptic Language Features     471

Don’t Use Environment Variables to Control the Build Process     472

Ensure That Release and Debug Builds Are Similar     473

Display the Exact Command Being Executed     474

Version-Control References to Tools     475

Version-Control the Build Instructions     475

Automatically Detect Changes in Compilation Flags     475

Don’t Invoke the Version-Control Tool from the Build System     476

Use Continuous Integration as Often as Possible     476

Standardize on a Single Type of Build Machine     477

Standardize on a Single Compiler     477

Avoid Littering Code with #ifdefs     477

Use Meaningful Symbol Names     478

Remove Stale Code     478

Don’t Duplicate Source Files     479

Use a Consistent Build System     480

Scheduling and Staffing Build System Changes     480

Summary     482

Chapter 18 MANAGING BUILD SIZE     485

The Problem with Monolithic Builds     486

Component-Based Software     488

Advantages of Using Components     491

What Exactly Is a Component?     493

Integrating Components into a Single Product     498

People and Process Management     502

Development Team Structure     503

Component Line-Up Management     505

Managing the Component Cache     507

Coordinating New Software Features     509

Apache Ivy     512

Chapter 19 FASTER BUILDS     515

Measuring Build System Performance     516

Measuring Performance in the Start-Up Phase     516

Measuring Performance in the Compilation Phase     526

Performance-Measurement Tools     531

Fixing the Problem: Improving Performance     534

Build Avoidance: Eliminating Unnecessary Rebuilds     535

Object File Caching     536

Smart Dependencies     539

Other Build-Avoidance Techniques     544

Parallelism     545

Build Clusters/Clouds     546

Parallel Build Tools     546

Limitations of Scalability     547

Reducing Disk Usage     548

Summary     551

REFERENCES     553

INDEX     559



Updates

Submit Errata

