- Dedicated WebSphere Application Server Engineering Support Organization
- LOB-Based Support Model
- WebSphere Organization with Separate Engineering and Service Delivery Functions
- Building a Global WebSphere Workforce
- Engineering Support with a Three-Level Approach
- Technical Support for Very Large IT Projects with Multigenerational Plans
- WebSphere Center of Excellence
- Summary
Engineering Support with a Three-Level Approach
A technical support model with a three-level structure has been popular with application development and support organizations. This technical support organization model has also been adopted by some technical teams supporting IT infrastructure. For example, some operating system support organizations may adopt a three-level resource pool support structure. However, what makes sense for some IT infrastructure engineering groups may not make sense for the WebSphere organization. This is because WebSphere technology is a part of JEE specifications. This relationship ties the WebSphere Application Server system tightly with the JEE applications that execute within WebSphere JEE containers. This close interdependency between the WebSphere Application Server system and WebSphere applications10 makes it necessary for the WebSphere team to carefully consider the impact of its support structure and organization choice over the application teams, especially the application support teams.11
The shared responsibility of the WebSphere team and the application support team for supporting the WebSphere application and the initial uncertainty of the team responsible for the root cause of a production problem requires that the two teams work well together.
Level 1 Technical Support
The application support team plays a pivotal role in both production application support and system stability. The application support team is the first line of defense against system stability issues. It has in-depth knowledge of the complex mission-critical application for which the WebSphere team provides a production execution environment. It also has limited WebSphere Application Server system privileges to expedite application production support chores.
For example, the application support team may have operator system privileges for the WebSphere Application Server. As a practical consideration, the WebSphere team most likely would want to grant operator privileges to the application support team because this reduces the WebSphere team participation in production issues that the production support team12 is fully capable of managing. (It is not unusual to see the WebSphere team provide or share WebSphere technical training opportunities with the application support teams. Chapter 11, “Critical Work Relationships,” discusses this topic.)
In this situation, the application support team calls in the WebSphere team when it has a difficult problem that demands expert-level WebSphere Application Server knowledge and skills, and the WebSphere team plays the role of Subject Matter Expert (SME), consultant, and senior-level technical support.
If the application support team is already playing the limited function of WebSphere technical support, installing a WebSphere Level 1 technical support role most likely creates a parallel or overlapping function with the application production team.
First, the WebSphere Level 1 support team may not have the necessary application knowledge to tell whether there is an application issue or a WebSphere system problem.
Second, when there is a difficult WebSphere system problem for which the production support team needs to call in WebSphere technical support, the WebSphere Level 1 team support may not be able to provide the senior-level WebSphere technical expertise needed. The WebSphere Level 1 support team may have to pass it on to the next level without adding value.
A third concern is the speed with which the right kind of WebSphere technical assistance is reached. When the company suffers a production issue, the application production support team does not have the patience to call in WebSphere Level 1 technical support when the only assistance provided may be sending out a page to WebSphere Level 2 support, who are capable of providing the kind of WebSphere technical assistance needed. When the WebSphere team is called upon to help, it is usually a serious and difficult problem that needs immediate attention. During a production emergency, it may be irritating to all the technical teams involved if there are any delays engaging the right WebSphere expert.
A large company may have thousands of large and complex WebSphere Application Server systems from different business divisions. It takes time to learn only a subset of these systems and become effective in providing support. A large IT organization is seldom a perfect world of ideal process optimization and flawless system consistency. Instead, large and complex WebSphere Application Server systems have a long history of evolution through mergers, acquisitions, business changes, and organization dynamics, and as a result, system differences and process inconsistencies exist. The WebSphere team must work in such reality. Therefore, the fourth area of concern is the difficulty in learning a large number of complex systems. In such a challenging environment, it is difficult for each engineer to provide WebSphere technical support for all the WebSphere systems and applications of a large company. This approach has pronounced technical training difficulties. It may not be possible for one WebSphere engineer to learn a large number of complex WebSphere Application Server systems.
Level 2 Technical Support
WebSphere Level 2 support has a solid set of WebSphere Application Server engineering tasks to perform, such as, deep dive performance tuning, troubleshooting, and high-impact and high-visibility production problem resolution. This job can cover the complete category of WebSphere Application Server engineering and it is a stable, visible, and desirable position.
If you have a WebSphere Level 2 technical support role, the best practice needs to be building a technical Level 2 support team with team members at each of the major geographical locations. This allows you to build a truly global WebSphere team that can provide the best WebSphere support around the clock. For example, you have the best engineering support around the clock if you have WebSphere Level 2 support around the world rather than concentrating on one geographic location. In addition, even global talent distribution makes it easier to build a local career path and mentoring opportunities for the WebSphere organization.
Level 3 Technical Support
The WebSphere Level 3 technical support role may have an overlapping role in terms of its function, especially for difficult production problems with several teams, and poses additional management issues. For example, if the WebSphere Level 3 support team works on application code to isolate defects, is that a service defined in a service level agreement (SLA) and, if not, what is the financial arrangement between organizations? If the WebSphere Level 3 support team provides application code improvements to fix a problem, who should be responsible for production instability introduced by the defects in the code improvements provided?
Another case is how to escalate and engage the IBM WebSphere Technical Support Organization when you have production problems and WebSphere Level 3 technical support. If a WebSphere Level 3 technical support role is set up, it is necessary to define the work relationship of the Level 3 support with IBM technical support, and how to engage IBM for technical support. The following exercise may help determine whether a WebSphere Level 3 technical support role is required.
As an established practice, IBM WebSphere Support and Java Support Organizations usually play the role of advanced WebSphere technical support. As with obtaining any technical support, there is an overhead cost in engaging IBM WebSphere and Java support. An IBM Problem Management Request (PMR) has to be opened with a detailed description of the technical problem that is occurring. Then, depending on the severity and difficulty of the problem, many technical discussions with IBM Level 1 or Level 2 WebSphere support or IBM Java support (or both) will likely take place.
If you have three levels of WebSphere technical support, it is an interesting question as to when and who needs to open an IBM PMR to secure IBM technical support. Let’s assume that the Level 1 WebSphere support won’t bypass Level 2 and Level 3 WebSphere support and therefore the Level 1 WebSphere support does not open an IBM PMR to seek IBM technical support. Let’s say that the Level 2 WebSphere support engineer looks into the problem, but determines that more advanced WebSphere technical support is needed. Then, what does the Level 2 WebSphere support engineer do for an urgent production problem? Escalate to the WebSphere Level 3 support or open a PMR and seek technical support from IBM? Say that the technical support process stipulates that your Level 2 WebSphere support must escalate to the WebSphere Level 3 support, and the Level 2 support follows the process and escalates the problem to Level 3. In that case, the urgent production issue has already traversed, at this point, through three layers of technical support within a WebSphere organization. With each layer of technical support hand-off, there is a time-consuming burden of communicating a complex technical problem. Note that this is at a moment when the WebSphere engineers need to focus on solving a tough technical problem under the pressure to stabilize the WebSphere Application Server system and restore production operation. If the Level 3 WebSphere support decides to seek IBM WebSphere support, the WebSphere Level 3 support has to spend time both opening a PMR and discussing the problem with IBM Level 1 or Level 2 technical support. Figure 1.4 describes this support model.
Figure 1.4 Three-level structure of WebSphere technical support
Of course, to avoid a WebSphere technical support request traveling through five layers of engagement and communication process in order to loop in all WebSphere experts, and more importantly, to solve the production problem as soon as possible, the WebSphere Level 1 support and the application production support can immediately engage both the WebSphere Level 2 and IBM WebSphere support, as well as the WebSphere Level 3 support. Then, should this structure of three levels of WebSphere technical support be flattened? Does the WebSphere Level 3 support function overlap with IBM WebSphere support, the same as the WebSphere Level 1 support function overlaps with that of the application production support?
In the tough world of WebSphere engineering, where production problems urgently demand many technical teams promptly working together for a timely solution, an organization structure with many layers of laborious support escalation process is cumbersome and impractical.
JEE experts in the application development organization need to also be considered. Because WebSphere technology is an implementation of a subset of JEE specifications, it is helpful to have assistance from senior JEE experts. This is especially true when there are production emergencies and there is a need to better understand the JEE application. WebSphere Level 3 support must have solid JEE experience. However, senior JEE developers and application architects in the application development organization can and should play the role of JEE application consulting during a production emergency.
The JEE experts on the WebSphere team are focused on the infrastructure side of the middleware work while the JEE developers and architects spend more time on the JEE applications. The developers and architects of the JEE application certainly know more of their application design philosophy and the technical details of their application code.
In addition, in terms of the overall IT organization, it is more efficient and cheaper to build a temporary task force to solve a serious technical problem, rather than training, retaining, and paying for WebSphere Level 3 support engineers for their advanced JEE application development expertise, which is not heavily used.
WebSphere plan engineers and process engineers are usually senior WebSphere professionals. They can be mobilized to participate in difficult technical problem resolution as part of a task force during a high-severity production incident. This is a more resourceful organization approach than keeping a pool of senior WebSphere engineers as Level 3 WebSphere support.
WebSphere technical support with three dedicated levels of expertise may look elegant and sound logical. It may look like a good way to organize because it has clear-cut divisions according to the skill levels. It could appear to be a logical way to separate WebSphere production support work based on its perceived efficiency of using the right talent for the right job. However, this three-level resource pool support model will most likely be unable to endure the test of time and engineering practice to provide unified support for a large company with a sizable WebSphere Application Server installation base that holds numerous complex WebSphere applications:
- There are technical training difficulties with learning a large number of complex WebSphere systems, practically all the WebSphere applications and infrastructure that a company has. This is especially true for a large company with a large number of WebSphere Application Server systems.
- Possible overlapping responsibilities and redundant roles with production support and IBM technical organization when the roles and responsibility, as well as engagement model, are not well defined.
- Reduced speed in production service restoration and production problem resolution, especially when the engagement and escalation process is not optimized.
It helps to have a flattened team structure with normal on-call rotation for every WebSphere engineer. This allows all the technical talents, including IBM, the WebSphere engineers, and the application development team, work shoulder to shoulder as an effective technical force to resolve tough problems together.
Nevertheless, a carefully defined process and a clear division of roles and responsibilities may help mitigate the risk of a layered support model. For example, problems can be classified according to difficulty and severity. For low-severity and relatively less difficult problems, Level 1 support can work independently. For high-severity problems, Level 2 and Level 3 need to be informed of the problem and provide leadership and guidance to Level 1, or participate directly in the problem resolution when appropriate. Therefore, the Level 1 WebSphere support performs the usual support chores, such as recycling the server, collecting and uploading data for IBM, opening a PMR, documenting the problem, and providing a report to senior management. The Level 2 and Level 3 support can play the leadership role in problem resolution and work as consultants and mentors to Level 1 WebSphere support. A flexible tiered structure of a virtual team and rotation may also work. For example, senior WebSphere planning engineers and process engineers can serve as Level 2 and Level 3 support when needed. For example, when the WebSphere team fights through a major production emergency, senior WebSphere engineers can be pulled in to help Level 1 WebSphere support engineers and work together as a virtual team.
A leveled resource pool model may be suitable for supporting a moderate number of WebSphere Application Server systems because it is possible for the WebSphere team members to learn a reasonable number of large systems, such as, supporting the WebSphere Application Server systems of one business division.
This model works well if the WebSphere Application Systems are standardized and the WebSphere applications are consistent in terms of architecture and operations. For example, a shared environment that holds many WebSphere applications belongs to one business division.