- Moving to Real-Time Business Integration: An Example
- Application Integration Approaches
- Application Integration: Clearly the Future
This chapter is from the book
This chapter is from the book
This chapter is from the book
Application Integration Approaches
As we've come to appreciate, application integration is a combination of problems. Each organization and trading community has its own set of integration issues that must be addressed. Because of this, it is next to impossible to find a single technological solution set that can be applied universally. Therefore, each application integration solution will generally require different approaches. At this time, and in the foreseeable future, one-stop shopping is simply not an application integration reality.
Although approaches to application integration vary considerably, it is possible to create some general categories, which include
-
Information-oriented
-
Business process integration-oriented
-
Service-oriented
-
Portal-oriented
Information-Oriented
Technologists who promote the information-oriented approach to application integration argue that integration should occur between the databases (or proprietary APIs that produce information, such as BAPI)—that is, databases or information-producing APIs should be viewed as the primary points of integration (see Figure 1.1). Within Information-Oriented Application Integration (IOAI), there are many approaches. Information-oriented solutions can be grouped into three categories: data replication, data federation, and interface processing.
)
Figure 1.1. Information-Oriented Application Integration deals with the simple exchange of information between two or more systems.
Data Replication
Data replication is simply moving data between two or more databases. These databases can come from the same vendor, or from many vendors (see Figure 1.2). They can even be databases that employ different models. The fundamental requirement of database replication is that it accounts for the differences between database models and database schemas by providing the infrastructure to exchange data. Solutions that provide for such infrastructures are plentiful and inexpensive.
)
Figure 1.2. Database replication is the simple exchange of information between databases.
Many database-oriented middleware solutions currently on the market provide database replication services, as well. Replication services are accomplished by placing a layer of software between two or more databases. On one side, the data is extracted from the source database or databases, and on the other side, the data is placed in the target database or databases. Many of these solutions provide transformation services, as well—the ability to adjust the schemas and the content so they make sense to the target database.
The advantages of database replication are simplicity and low cost. Database replication is easy to implement, and the technology is cheap to purchase and install. Unfortunately, these advantages are quickly lost if methods need to be bound to the data, or if methods are shared along with the data. If these requirements exist, service-based solutions must be considered.
Data Federation
Database federation is the integration of multiple databases and database models into a single, unified view of the databases (see Figure 1.3). Put another way, database federations are virtual enterprise databases that are comprised of many real, physical databases. While database federation has been around for some time, the solution set has been perfected only recently.
)
Figure 1.3. Data federation allows many databases to appear as a single database.
Database federation software places a layer of software (middleware) between the physical distributed databases and the applications that view the data. This layer connects to the back-end databases using available interfaces and maps the physical databases to a virtual database model that exists only in the software. The application uses this virtual database to access the required information. The database federation handles the collection and distribution of the data, as needed, to the physical databases.
The advantage of using this software is that it can bind many different data types into a unified model that supports information exchange.
Database federation allows access to any connected database in the enterprise through a single, well-defined interface. This is the most elegant solution to the data-oriented application integration problem. Unlike replication, this solution does not require changes to the source or target applications. Still, changes do have to be made at the application-oriented level to support federated database software. This is due to the fact that different interfaces are being used to access a different database model (the virtual database).
Interface Processing
Interface processing solutions use well-defined application interfaces to focus on the integration of both packaged and custom applications (see Figure 1.4). Currently, interest in integrating popular Enterprise Resource Planning (ERP) applications (e.g., SAP, PeopleSoft, and Oracle) has made this the most exciting application integration sector.
)
Figure 1.4. Interface processing externalizes information out of packaged applications through a well-defined API.
Integration brokers support application interface processing solutions by providing adapters to connect to as many custom or packaged applications as possible, externalizing information out of those applications through their open or, more often than not, proprietary interfaces. They also connect to technology solutions that include middleware and screen scrapers as points of integration.
The efficient integration of many different types of applications defines the primary advantage of using application integration-oriented products. In just days, it is possible to connect a SAP R/3 application to an Oracle application, with the application interface processing solution accounting for differences between schema, content, and application semantics by translating on the fly the information moving between the systems.
The downside to using application interface-oriented products is that there is little regard for business logic and methods within the source or target systems—logic and methods that may be relevant to a particular integration effort. In such a case, service-based solutions probably make the better choice. Ultimately, application interface processing technology will learn to share methods as well as information, perhaps by joining forces with service-based approaches.
Business Process Integration-Oriented
Simply put, business process integration-oriented products layer a set of easily defined and centrally managed processes on top of existing sets of processes contained within a set of enterprise applications (see Figure 1.5).
)
Figure 1.5. Business process integration allows application integration architects to place a well-defined business process as the controlling
entity, able to access both information and processes encapsulated in remote systems.
Packaged Application Interfaces: Information versus Services
While packaged application interfaces are primarily information oriented, there are a few that provide access to application services as well. These are hybrids. The best example of this is Business Application Programming Interfaces (BAPI) from SAP, but a few others also exist.
These interfaces allow you to invoke remote application services, such as processing a credit check or calculating shipping costs—processes that are more service than information oriented.
Packaged application interfaces, as you'll discover in Chapter 2, provide very different approaches to access both information and services. There are no standards for packaged application integration, even though the J2EE Connector Architect (JCA) is attempting to set some.
Business process integration (BPI) is the science and mechanism of managing the movement of data, and the invocation of processes in the correct and proper order to support the management and execution of common processes that exist in and between applications. Business Process Integration-Oriented Application Integration (BPIOAI) provides another layer of easily defined and centrally managed processes that exist on top of an existing set of processes and data contained within a set of applications.
The goal is to bring together relevant processes found in an enterprise or trading community to obtain the maximum amount of value, while supporting the flow of information and control logic between these processes. These products view the middleware, or the "plumbing," as a commodity and provide easy-to-use visual interfaces for binding these processes together.
In reality, business process integration is another layer of value resting upon existing application integration solutions, solutions that include integration servers, application servers, distributed objects, and other middleware layers. Business process integration offers a mechanism to bind disparate processes together and to create process-to-process solutions that automate tasks once performed manually.
However, by diminishing the importance of the plumbing, it is too easy to lose sight of the larger picture. In reality, no single application integration vendor has solved the plumbing issues. Ultimately, the solution to these issues will be delivered by a combination of business process integration and middleware vendors. That being the case, it is clear that the binding of middleware and process automation tools represents the future of application integration.
Business process integration is a strategy, as much as technology, which strengthens your organization's ability to interact with disparate applications by integrating entire business processes, both within and between enterprises. Indeed, business process integration delivers application integration by dealing with several organizations using various metadata, platforms, and processes. Thus, business process integration technology must be flexible, providing a translation layer between the source and target systems, and the business process integration engine.
There are many differences between more traditional application integration and business process integration.
-
A single instance of business process integration typically spans many instances of traditional application integration.
-
Application integration typically means the exchange of information between two or more systems without visibility into internal processes.
-
Business process integration leads with a process model and moves information between applications in support of that model.
-
Application integration is typically a tactical solution, motivated by the requirement for two or more applications to communicate.
-
Business process integration is strategic, leveraging business rules to determine how systems should interact and better leverage the business value from each system through a common abstract business model.
BPIOAI views middleware, or the plumbing, as a commodity, with the ability to leverage both message-oriented and transactional middleware as points of integration into any number of source or target systems. In fact, most integration servers and application servers are beginning to offer business process integration tools that support their middleware technology. Indeed, business process integration generally provides easy-to-use visual interfaces for binding these processes together and, along the way, creates visual BPIOAI.
While some may question the relevance of Business Process Integration-Oriented Application Integration, and even of application integration itself, I would argue that BPIOAI is the ultimate destination of application integration (acknowledging that we still have a long way to go to perfect the middleware). Despite current shortcomings, many application integration vendors are aggressively promoting BPIOAI as a vital component of their application integration technology package. In doing so, their strategy is clear—they are anxious to join the world of high-end, BPIOAI modeling tools. They hope that their application integration-enabled middleware, such as integration servers and application servers, will accomplish just that.
BPIOAI is best defined as applying appropriate rules, in an agreed-upon logical sequence, in order to pass information between participating systems, as well as visualize and share application-level processes, including the creation of a common abstract process that spans both internal and external systems. This definition holds true regardless of whether the business processes are automated or not. For example, processing an insurance claim and delivering a car to a customer are business events that can be automated with BPIOAI.
To this end, there are three main services that business process integration provides: the visualization of processes contained within all trading partner systems, interface abstraction, and the real-time measurement of business process performance.
By visualizing enterprise and cross-enterprise processes contained within trading partners, business managers are able to become involved in enterprise integration. The use of graphics and diagrams provides a powerful tool for communication and consensus building. Moreover, this approach provides a business-oriented view of the integration scenarios, with real-time integration with the enabling middleware or points of integration. This provides business analysts with the ability to make changes to the process model, implement it within the trading community, and typically not involve the respective IT departments.
Interface abstraction refers to the mapping of the business process integration model to physical system interfaces and the abstraction of both connectivity and system integration solutions from the business analyst. Business process integration exists at the uppermost level in the application integration middleware stack. Those who use business process integration tools are able to view the world at a logical business level and are not limited by physical integration flows, interfaces, or adapters. What's more, the middleware mechanisms employed are also abstracted and are not a concern of the business process analyst, as long as the common process model is interacting correctly with all source and target systems that exist within all companies.
BPI by Example
There are three companies that participate in a trading community: Companies A, B, and C. Company A produces parts for skateboards, while Company B assembles and tests the skateboards, and finally, Company C sells the skateboards. Each has its own set of processes that are native to the respective company and its internal systems: a production system, an assembly system, and a sales system, respectively. Until now, automated integration has been nonexistent, and mail and fax serve communication needs between companies.
In order to integrate these applications, the trading community has decided to implement BPIOAI, defining a common process model that spans all companies and internal systems. This process model defines a sequence and logical order of events from the realization of consumer demand, the purchase of raw materials, the creation of the parts, the assembly of parts into a product, the testing of the product, and finally, the sale of the product to the ultimate consumer. This common model integrates with local systems by having visibility into their internal application processes, if possible, or perhaps through more primitive layers such as the database or application interface. What's important is that the common process model is able to produce events that are understood by the systems participating in the process, as well as react to events that the applications communicate back to the business process integration engine.
The use of a common process model that spans multiple companies for application integration provides many advantages, including:
-
The ability to create a common, agreed-upon process between companies automating the integration of all information systems to react to business events such as increased consumer demand, material shortages, and quality problems in real time.
-
The ability to monitor all aspects of the business and trading community to determine the current state of the process in real time.
-
The ability to redefine the process at any given time in support of the business, and thus makes the process more efficient.
-
The ability to hide the complexities of the local applications from the business users and to have the business user work with a common set of business semantics.
Walking Through a Process
Although each business process integration tool and project may take a slightly different approach, the internal process of interacting with the physical systems typically consists of the following set of events:
-
The source system that exists inside of a company posts an event to the business process integration engine; for example, a skateboard is sold.
-
The event is transformed, if required, so the event adheres to a standard set of business semantics and information processing mechanisms (synchronous versus asynchronous). This is going to be engine dependent, but there always has to be a common set of process semantics and information processing mechanisms defined at the engine level so the analyst can make sense of a business process that spans many types of applications, platforms, and databases.
-
The business process integration engine reacts to the event, once transformed, invoking other processes in other systems to support the execution of the common process model. For example, if a skateboard is sold, then send an order to the skateboard assembler, posting an event from the process engine to the assembler's target system (typically over the Internet).
-
Based on receiving that event, the local system reacts as per its internal processes and posts an event back to the process engines (say, when the skateboard is assembled).
-
The common process model sequences the master process, sending and receiving other events in support of the common process model. This is an ongoing activity, with information moving up to the process engine from the local systems, transformed if required, and down from the process engine to the local systems in support of the execution of the process model.
Another way to view the process of creating a business process integration model is defining the hierarchy of processes within the trading community. This means that smaller subprocesses can be linked at the lower tier of integration or are native to the source or target systems. Building up from the lower-level processes to the higher-level processes, you may link the subprocesses into higher-level processes within the domain of the trading community.
The measurement of business process performance provides the business process integration with the ability to analyze a business in real time. By leveraging tight integration with the process model and the middleware, business analysts are able to gather business statistics in real time from the trading community; for example, the performance of a supplier in shipping goods to the plant, and the plant's ability to turn those raw materials into product.
Moreover, business process integration enables the technology user to track and direct each instance of a business process; for example, processing individual orders or medical insurance claims through a life cycle that may consume seconds, minutes, hours, days, or weeks. Finally, we need to measure and maintain contextual information for the duration of a process instance that spans many individual activities.
Indeed, the goal of BPIOAI, and of application integration in general, is to automate the data movement and process flow so that another layer of BPIOAI will exist over and above the processes encapsulated in existing systems. In other words, BPIOAI completes application integration, allowing the integration of systems, not only by sharing information readily, but also by managing the sharing of that information with easy-to-use tools.
In general, business process integration logic addresses only process flow and integration. It is not a traditional programming logic, such as processing a user interface, updating a database, or executing a transaction. Indeed, in most BPIOAI scenarios, the process logic is separated from the application logic. It functions solely to coordinate, or manage, the information flow between many source and target applications that exist within organizations.
Service-Oriented
Service-Oriented Application Integration (SOAI) allows applications to share common business logic or methods. This is accomplished either by defining methods that can be shared, and therefore integrated, or by providing the infrastructure for such method sharing such as Web services (see Figure 1.6). Methods may be shared either by being hosted on a central server, by accessing them interapplication (e.g., distributed objects), or through standard Web services mechanisms, such as .NET.
)
Figure 1.6. Service-Oriented Application Integration provides mechanisms to create composite applications, leveraging services found in
many remote systems.
Attempts to share common processes have a long history, one that began more than ten years ago with the multitiered client/server—a set of shared services on a common server that provided the enterprise with the infrastructure for reuse and, now, for integration—and the distributed object movement. "Reusability" is a valuable objective. A common set of methods among enterprise applications invites reusability and, as a result, significantly reduces the need for redundant methods and/or applications.
While most methods exist for single-organization use, we are learning that there are times when it makes sense to share between organizations. In a new twist on the longstanding practice of reusability, we are now hoping to expand this sharing beyond intraenterprise—to trading partners, as well; for example, sharing a common logic to process credit requests from customers or to calculate shipping costs using a set of Web services.
Unfortunately, absolute reuse has yet to be achieved on the enterprise level. It is an even more distant goal between trading partners. The reasons for this failure are primarily political. They range from internal politics to the inability to select a consistent technology set. In most cases, the actual limit on reuse results directly from a lack of enterprise architecture and central control.
Utilizing the tools and techniques of application integration gives us the opportunity to learn how to share common methods. More than that, these tools and techniques create the infrastructure that can make such sharing a reality. By taking advantage of this opportunity, we are integrating applications so that information can be shared, even as we provide the infrastructure for the reuse of business logic.
Sounds great, doesn't it? The downside might give you pause, however. This "great-sounding" application integration solution also confronts us with the most invasive level of application integration, thus the most costly. This is no small matter if you're considering Web services, distributed objects, or transactional frameworks.
While IOAI generally does not require changes to either the source or target applications, SOAI requires that most, if not all, enterprise applications be changed in order to take advantage of the paradigm. Clearly, this downside makes SOAI a tough sell. However, it is applicable in many problem domains. You just need to make sure you leverage SOAI only when you need it.
Changing applications is a very expensive proposition. In addition to changing application logic, there is the need to test, integrate, and redeploy the application within the enterprise—a process that often causes costs to spiral upward. This seems to be the case, no matter if you're approaching SOAI with older technologies such as Common Object Request Broker Architecture (CORBA), or new technologies such as .NET, the latest service-based architecture to come down the road.
Before embracing the invasiveness and expense of SOAI, enterprises must clearly understand both its opportunities and its risks. Only then can its value be evaluated objectively. The opportunity to share application services that are common to many applications—and therefore making it possible to integrate those applications—represents a tremendous benefit. However, that benefit comes with the very real risk that the expense of implementing SOAI will outpace its value.
Portal-Oriented
Portal-Oriented Application Integration (POAI) allows us to view a multitude of systems—both internal enterprise systems and external trading community systems—through a single-user interface or application. POAI benefits us by avoiding the back-end integration problem altogether; it adapts the user interface of each system to a common user interface (aggregated user interface)—most often a Web browser (see Figure 1.7). As a result, it integrates all participating systems through the browser, although the applications are not directly integrated within or between the enterprises.
)
Figure 1.7. Portal-Oriented Application Integration.
While the other types of application integration are focused on the real-time exchange of information (or adherence to a common process model) between systems and companies, POAI is concerned with externalizing information out of a multitude of enterprise systems to a single application and interface. That's clearly an approach that goes against the notions of the other types of application integration, which are more real-time- and event-driven-oriented, and the inclusion in this book of POAI was somewhat of a judgment call.
However, application integration, while typically referring to the automated movement of information or the binding of processes between two or more applications, without the assistance of an end user, can clearly also occur at the user interface. Indeed, most examples of B2B information exchange today are also examples of POAI, with digital exchanges leading the way. Therefore, it's different, but it still belongs within the discussion of application integration.