This chapter is from the book
This chapter is from the book
This chapter is from the book
Adapter Analysis
As with any other software development project, analysis is very important for adapter development and maintenance. Understanding the complete integration scenario is fundamental to any adapter analysis methodology or technique. Knowing the objectives of the integration scenario is useful in determining the specific role of the adapters.
Where should a software analyst begin when developing adapters? The analysis is a four-step process, beginning with the business objectives and ending with an analysis model—complete with integration use cases, analysis of application architectures, data models, and API. Following is a series of individual steps and tasks that can be used as a guideline for adapter analysis in general:
Understanding the business objective.
Integrate applications to eliminate duplicate data entry and ensuring data integrity.
Enable Web-based, service-oriented access to application functions. The services can be used by customers, partners, and internal staff.
Enhance applications to support workflow and process automation.
Document the business objectives that are driving the integration requirements. It will be useful in calculating the ROI as well as justifying future integration requirements.
Analyze the end-to-end integration scenario.
Identify all the applications participating in the integration scenario.
If applicable, identify master or controlling applications. Typically, these applications manage master databases and provide unique services.
In a peer-to-peer integration model, identify the initiator of integration scenarios. If the initiators are different, based on business rules, a table stating the association between business rules and which application is the initiator when the business rule is asserted.
Identify the mismatch between individual data models of all participating applications. This information will be useful for defining the data structures to be exchanged by applications within an integration scenario. Data transformation and validation requirements are derived from this analysis.
Document the analysis results, either in a text document with as much detail as possible or using a UML-based development tool. A use case analysis model captures the results of analysis in text form. Use case models identify the relationships between use cases, system boundaries, and entities interacting with the use cases. An example of a use case capturing a simple end-to-end integration scenario is shown in the section on "Documenting Integration Scenario".
Analyze the individual application architecture.
Identify the database structures (tables, objects, and so on) of each application required to complete the integration scenario.
Identify any available API, staging database tables, and other types of interfaces in the application. These interfaces could be used as points of integration for adapters. If no such interfaces are available, it is a good indication of the effort required to develop integration capabilities for the application.
Identify third-party middleware technologies used in multitier applications. Generally, distributed applications are easier to integrate because they have more points of integration tied together by middleware platforms and tools.
One of the most important design features that needs research is the applications support for synchronous and asynchronous transactions. Knowing which applications support either or both types of transactions helps in the design of a high-quality adapter. It is quite common for adapters to be burdened with additional work required to handle asynchronous interfaces in a synchronous transaction and vice versa.
Document the available points of integration (PIN) and the type of PIN (database table, stored procedure, API, message, and so on). Also, document the business function exposed by the individual PIN. Document the sync-async mismatch between PINs because it will drive the individual adapter designs later in the process.
Identify programming constraints.
Many times, integration project teams face the difficult task of integrating old business applications. Not only are the programming language and supporting tools an issue, but also knowledge about its architecture and design. Identifying the constraints, especially with respect to the non-availability of appropriate development tools, needs to be documented very early in the process. Project managers need to treat these as potential risks and try to develop alternate strategies.
Differences in programming languages of the business applications should be taken into consideration when identifying constraints. Not all programming languages are easier to deal with when developing adapters. Java provides a JNI interface to call code developed in other languages. However, it is not easy, and not all languages are supported by JNI.
The constraints identified by the adapter analysis teams are very useful for project managers in estimating the adapter development efforts as well as setting the right expectations.
This four-step process is very basic and is not supposed to be a full-fledged analysis process. Individual project teams and project managers need to verify whether their analysis process handles some of things stated above.
Documenting the Integration Scenario
Integration scenarios can be simple point-to-point data exchange or complex distributed data synchronization. Other types of integration scenarios include accessing services over the Web or other proprietary platforms. Increasingly, process modeling and automation are seen as means to capture business processes and link them to applications using workflow and other tools.
Each type of integration scenario has unique requirements and can be implemented using different design patterns. But before you know which design patterns to use, it is important to document the integration scenario and identify all the participants as well as the different integration points. A common practice in software analysis today is defining the requirements in use cases. A use case is part of UML specifications, and is more often used to capture user requirements. However, use cases can also be used to capture integration requirements. Requirements of a simple integration scenario are shown in the example use case in Figure 7.4.
Figure 7.4 Example integration use case.){kind=link}
It may look odd at first that the diagram shows two system boundaries. Most business analysts using UML are perhaps used to a single system boundary that defines the business application for which the user requirements are captured. This diagram shows the difference between normal requirements analysis and integration analysis. More complex integration scenarios will have more system boundaries (systems) involved in the use case model.
Another difference is the identification of data structures that need to be exchanged between systems. In Figure 7.4, the data structures are shown as text notes outside the system boundaries. It is important to understand these requirements early on because they may require different project teams to spend significant time later on in supporting the integration projects.
From a project management perspective, a use case model is very useful for understanding the complexity of the integration scenario, the different systems involved, and the points of integration between the systems at a high level without any technical details. Project managers can then begin to estimate the effort and coordinate with other project managers the work required from members of other teams in successfully implementing the integration solution.
Determining Hub and Spoke Scenarios
Sometimes, integration scenarios can be defined as a hub and spoke type of integration scenario. This is usually the case when one application is the master or controlling application, and all other applications are reacting to the state changes of the controlling application. Controlling application could be legacy applications, third-party packages, and even J2EE applications.
In a hub and spoke integration scenario, the controlling application's action determines the effect on all other applications in the scenario. If the controlling application (hub) changes any of its data, other applications (spokes) need to be notified of the change so that they can synchronize their database states as required. Changes to data in other applications besides the controlling application are not part of the integration scenario; they are considered to be local changes.
The benefits of identifying hub and spoke scenarios are evident when selecting integration technologies implementing the solution. However, from a project management perspective, it is quite likely that the project managers responsible for the controlling application (hub) need to take the leadership role and put in extra efforts to coordinate the integration efforts from other teams. Without a defined flow of project management activities, it is difficult to understand and manage the integration project.
Determining Peer-to-peer Scenarios
A different type of integration scenario occurs when all participants have equal responsibilities. In a peer-to-peer integration scenario, the participants synchronize data between their databases by processing data updates from all applications. An update to one application is broadcast to all other applications in the scenario. There is no concept of a master application controlling the chain of events.
Peer-to-peer integration scenarios are more likely to emerge in the context of process automation rather then data integration. Most data-integration scenarios have one application as the data owner, and only this application triggers data updates. The hub and spoke scenario discussed earlier is more suitable for data integration. Examples of peer-to-peer scenarios include workflow and process automation-related integration scenarios in which participating applications are expected to generate events reflecting internal state changes, and to process incoming events by taking appropriate actions. In a workflow automation environment, all applications generate events or they process events and then trigger the associated actions. The coordination and propagation of the events are not controlled by any one application, but depend on a higher-level abstraction of the finite state machine.
In some special cases, one or more of the peers in a peer-to-peer integration scenario is a composite application. A composite application is an aggregation of other applications built to support specific integration requirements. The individual applications that are part of the composite application may have a hub-and-spoke or peer-to-peer relationship with other applications; but for analysis, the composite application can be considered as supporting the peer-to-peer scenario. Composite applications may not be common today, but over time they will be found in most enterprises.
It is always better to separate the peer-to-peer integration scenarios from hub and spoke scenarios. Most adapters only handle data integration. Event processing is not usually considered an adapter function. Sometimes, it is handled by a workflow automation tool, which maintains the state changes on the application's behalf. By identifying the peer-to-peer scenarios project, managers can choose to either enhance the workflow tool if one exists or include the functionality in adapters. Without proper support in the adapter frameworks, however, it is very difficult to have both data integration and event processing implemented as one adapter.
Analyzing Application Architectures
Adapters are part of business applications, so one of the challenges of building adapters for existing applications is having the knowledge of application internals or the architecture. Very often, the application architecture and design are not explicitly documented or modeled. As a result, adapter developers need to spend a significant amount of time understanding the application internals. Integration teams often spend most of their time modeling the database structure and business rules—something that should have been done earlier by the application development teams that maintain the application. Project managers need to consider the impact of this when estimating the integration efforts. Depending on the size of the application, available documentation, and access to resources knowledgeable about the application, the architecture analysis effort can be reasonable or frustrating.
Project managers should not skip this activity because the result is a longer design and development cycle. In other words, the architecture analysis work cannot be avoided. It can be done as part of the integration analysis process or deferred to the development teams. Putting it off to the development teams usually results in a less-than-desirable integration solution.
Analyzing Application Data Models
Data impedance or differences in data models between applications is the classic application integration problem. These issues are complicated by business rules attached to data models. For example, it is quite likely that the structure of an account number column in the accounting database is actually a composite of three business data elements: Customer ID, Geographical Area code, and Account type. These types of data fields and data models pose significant integration challenges during the development and implementation phase. It is not surprising to see integration projects fail due to lengthy implementation cycles as a result of hidden data impedance. Project managers need to understand the significance of these seemingly small technical issues. It is not just about transforming data from one format to another; it involves applying and verifying business rules, as well.
Experienced project managers tend to estimate additional time for data model analysis because generally it is one of those areas that comes back later as a major underestimated factor. The lack of any documentation on the business rules attached with data models means coordination and help from business analysts. Either way, this is a very important task of adapter development, especially when focused on data integration.
Analyzing Application APIs
Sometimes we get lucky, and one such instance in the life of an adapter developer is when applications have a well-defined API. Adapter developers eventually develop a sense of respect for applications with well-documented APIs. However, not all APIs are useful for adapter development. Many times, low-level APIs that work with technical objects instead of higher-level business objects tend to be more work than they're worth. This is not to say that lower-level APIs are not useful, but in the context of adapter development, accessing technical objects such as data rows in an RDBMS table, or data objects in a ODBMS is faster and easier with the native interfaces such as SQL and OQL. Application-level APIs that encapsulate business logic are more useful for adapter development.
In the absence of a higher-level API, a decision needs to be made very early on in the project on the relative use of lower-level APIs in adapter development. If the lower-level APIs are expected to save time and encapsulate the adapter developers from many technical details, there is still value in using the APIs. However, if the difference between native interfaces and the lower-level APIs is not much and the value added by the APIs is more cosmetic, a better strategy would be to build a higher-level API as part of the adapter development project.
In the long term, adapter development time and maintenance time can be reduced by higher-level, business-oriented application APIs. Project managers need to consider and weigh the long-term benefits versus short-term time constraints.
Analyzing Buy versus Build Options
Having a good understanding of the end-to-end integration scenario, individual application architectures, data impedance between applications, availability of clearly defined APIs, and so on should enable project managers to contribute to the buy versus build decision-making process. Many integration solutions are available in the market. Some specialize in data transformation, and others provide integration brokers and application servers capable of simplifying integration.
This book is not about the various decision-making processes involved in selecting integration solutions, but the information generated during the analysis phase outlined in the chapter should enable project managers to contribute to that process. It is always a good strategy to engage application integration vendors and consultants and get the benefit of their expertise in managing and deploying integration solutions. The number of adapter vendors is increasing, and the availability of open standards such as JCA makes adapter development a better-understood technology. It will be easier to buy a JCA-compatible resource adapter and maintain it in-house than build a completely proprietary adapter technology. However, Chapter 10, "Overview of JCA," identifies some areas in which JCA is still missing critical specifications. It is very likely that adapter vendors will fill this gap in JCA specifications with their own proprietary solutions.
Building adapters in-house makes sense, especially when proprietary technology and applications are considered. No one understands the proprietary technologies better than the team that originally developed it. Some adapter vendors will probably provide tools that make JCA resource adapter development easier.
Estimating Adapter Development
Software estimation is always a challenge, and adapter development projects have more variables to consider. The primary variables that affect adapter development include the following:
The number of application interfaces (points of interface) to be included in the adapter. A good design principle is to implement not more than three to five interfaces in one adapter. If there are more then five interfaces, it is better to group them by business or technical functions, and implement one adapter for each group. You implement fewer interfaces in one adapter to keep the memory footprint as small as possible. Adapters are expected to scale a lot more than the applications. Hence, the smaller the adapter, the larger the number of instances the JVM can manage. It may not be possible in all cases to break down the adapter functionality into groups for better implementation. The optimum number of interfaces per adapter depends on the specific integration scenario and the technical complexities.
The number of data models and databases, differences between them, and resulting data transformation requirements.
The differences between the adapter development environment and the application development environment. For example, a JCA adapter developer may need to interface with a C or C++ API to access data objects stored in an object database. These paradigm differences require more time to understand, design, and develop the adapter.
Internationalization requirements resulting in DBCS support in adapters and the resulting conversions always create unwanted surprises during development and deployment.
Other factors affect the overall estimation effort, but these depend on the individual team structures, IT environments, and the company culture. The objective of this chapter is to identify areas of concern, not define project management solutions.