This chapter is from the book
This chapter is from the book
This chapter is from the book
Where Does AV Foundation Fit?
One of the first steps to learning AV Foundation is to get a clear understanding of where it fits within Apple’s overall media landscape. Mac OS X and iOS provide developers with a number of high-level and low-level frameworks for working with timed media. Figure 1.1 shows how AV Foundation fits into the overall picture.
)
Figure 1.1 Mac OS X and iOS media environment
Both platforms offer a number of high-level solutions for working with media. On iOS, the UIKit framework makes it easy to incorporate basic still image and video capture into your applications. Both Mac OS X and iOS can make use of the HTML5 <audio> and <video> tags inside either a WebView or UIWebView to play audio and video content. Both platforms additionally provide the AVKit framework, which simplifies building modern video playback applications. All these solutions are convenient and easy to use and should be considered when adding media functionality into your applications. However, although these solutions are convenient, they often lack the flexibility and control needed by more advanced applications.
At the other end of the spectrum are several lower-level frameworks that provide supporting functionality used by all the higher-level solutions. Most of these are low-level, procedural C-based frameworks that are incredibly powerful and performant, but are complex to learn and use and require a strong understanding of how media is processed at a hardware level. Let’s look at some of the key supporting frameworks and the functionality each provides.
Core Audio
Core Audio handles all audio processing on OS X and iOS. Core Audio is a suite of frameworks providing interfaces for the recording, playback, and processing of audio and MIDI content. Core Audio provides both higher-level interfaces, such as those provided by the Audio Queue Services framework, which can be used for basic audio playback and recording needs. It also provides very low-level interfaces, specifically Audio Units, which provide complete control over the audio signal and enable you to build sophisticated audio processing features like those used by tools such as Apple’s Logic Pro X and Avid’s Pro Tools. For an excellent overview of this topic, I highly recommend reading Learning Core Audio by Chris Adamson and Kevin Avila (2012, Boston: Addison-Wesley).
Core Video
Core Video provides a pipeline model for digital video on OS X and iOS. It provides image buffer and buffer-pool support to its counterpart, Core Media, providing it an interface for accessing the individual frames in a digital video. It simplifies working with this data by translating between pixel formats and managing video synchronization concerns.
Core Media
Core Media is part of the low-level media pipeline used by AV Foundation. It provides the low-level data types and interfaces needed for working with audio samples and video frames. Core Media additionally provides the timing model used by AV Foundation based around the CMTime data type. CMTime, and its associated data types, are used when working with time-based operations in AV Foundation.
Core Animation
Core Animation is the compositing and animation framework provided on OS X and iOS. The behavior it provides is essential to the beautiful, fluid animations seen on Apple’s platforms. It offers a simple, declarative programming model providing an Objective-C wrapper over functionality enabled by OpenGL and OpenGL ES. Using Core Animation, AV Foundation provides hardware-accelerated rendering of video content in both playback and video capture scenarios. AV Foundation additionally makes use of Core Animation, enabling you to add animated titling and image effects in video editing and playback scenarios.
Sitting between the high-level and low-level frameworks is AV Foundation. The positioning of AV Foundation within the overall media landscape is significant. It offers much of the power and performance of the lower-level frameworks, but in a much simpler Objective-C interface. It can work seamlessly with higher-level frameworks, such as Media Player and the Assets Library, making use of the services they provide, and at the same time it can interact directly with Core Media and Core Audio when more advanced needs arise. Additionally, because AV Foundation sits below the UIKit and AppKit layers, it also means you have a single media framework to use on both platforms. There is only one framework to learn, providing you the opportunity to port not only your code, but also your knowledge and experience to either platform.