This chapter is from the book
This chapter is from the book
This chapter is from the book
Getting Results
You can get good results from Oracle Support. It is your job to drive the process and know what results you want to obtain. It is time-consuming, but the following techniques usually produce a solution.
Research the Problem
Know how to obtain the version numbers of the program that is causing the problem. On a Unix system, you can issue the following command to find out the module revision number of the badly behaving executable:
strings -a [filename] | grep Header
To find the version information on the NT operating system, open a command window, change to the directory where the executable program is stored, and do a find "Header" [filename].
NOTE
To get the filename for a report or process that runs through the concurrent manager, open the log file and look at line three.
NOTE
You should change directories to the location where the executable is stored, or [filename] should be the full path to the executable.
Give the Analyst Full Information
Go slow enough for the analyst to type the details into the TAR record. Make sure everything you know about the problem is in the TAR record. Make sure you have the right database and applications version numbers.
NOTE
Filling a TAR always requires the version of the operating system, database, the apps, the form or module, and all recent patches applied. Your DBA or system administrator can supply the operating system level, but you can find the recent version of the database by clicking About Oracle Applications in the Help menu. The version of the database and the apps is under the Database Server section. The points after the first number are very important (that is 8.1.1.6.0 and 8.1.3.0 are very different versions). If you need support for one of the forms, the information needed for forms can also be found in the About Oracle Applications under the Current Form section. This section gives the form name (very important information for filing a TAR), the version, and the last updated date. If you are creating an iTAR, you can paste this into the TAR record.
TIP
When you create an iTAR, you will be asked if you have tried recompiling the form/report. Do so, or get an administrator to do it for you, and then retest for your problem before creating the TAR. That way you can answer yes to the question about recompiling.
Be Able to Create the Problem on Demand
Demonstrating a problem on demand and showing in the documentation why the program is misbehaving are the two keys to getting fast results.
If you have customizations, make sure you can create the problem on Oracle's version of the program. Make sure it is an Oracle problem and not something you did locally.
Consider getting a copy of PC Anywhere so that the support analyst can see issues that might exist only at your site. Remember, good support is all about good communication with your support analyst, and you can use these products to give a demonstration of what is happening to your analyst.
TIP
While re-creating your problem, take screen shots of the results; especially the error messages displayed. Also, take screen shots of any setups that might affect the problem you are having. Paste all the screen shots into a word processing file, and then you can upload the file to Oracle Support.
Log Only One Issue per TAR
Try to keep the problem simple, focused, and well-defined. Each unrelated problem should be reported and resolved on its own TAR. Oracle Support analysts are specialized by Oracle Application or technology. When you split apart unrelated problems, you can direct each TAR to the correct support group.
Research Upgrade Issues, Bugs, and Bulletins Proactively
The MetaLink pages on Oracle's Web site have a wealth of white papers, certified combinations, and search tools to assist you with administration of the applications. Why wait until the third week in January to discover that your version of the 1099 reporting program doesn't work the way you expect? Before conference room pilot activity, implementation project teams should research issues. Production system administrators should research issues before applying major patches and before starting upgrade projects.
Research the Documentation
The published documentation tells you how the program was designed and how the process is supposed to work. The documentation is right almost all of the time. If you find a function is performing differently from the description in the documentation, you are on the shortcut route to locating the patch you need or defining a bug.
NOTE
Once, I spent about a half-day diagnosing a problem on the customer open interface. The client had an outdated version of the manuals and Support was able to provide photocopied manual pages rapidly to resolve the confusion.
Use the Debug Profile Options Where Available
Many applications have profile options that can be set to produce extra output in the concurrent request logs or in special directories. For the Oracle Applications discussed in this book, the following Release 11i profile options can help generate additional information about a problem:
CST: Cost update debug level
MRP: Debug Mode
OE: Debug
OE: Debug Level
OE: Trans. Manager Debug Level
OE: Debug Trace
OM: Debug Level
PO: Set Debug Concurrent ON
PO: Set Debug Workflow ON
AR: Debug Level for PostBatch
Tax: Debug File Directory
Tax: Debug Flag
AR: Enable Debug Message Output
Account Generator:Run in Debug Mode
FND: Debug Log Enabled
FND: Debug Log Filename
FND: Debug Log Level
FND: Debug Log Module
GL: Debug Log Directory
INV: Debug Trace
INV: Debug file (including the complete path)
INV:Debug Level
MRP: i2 Debug Mode
MRP:Debug Mode
PA: Debug Log Directory
PA: Debug Mode
FA: Print Debug
HR: FastFormula debug level.
Concurrent: Debug Flags
TIP
Make sure the profile option FND: Debug Log Filename is not null, or some of the previous profile options will not work. Also, you should set the option FND: Debug Log Level and FND: Debug Log Module. The Log Module controls which program is being debugged, and if you leave it blank, your debug file will contain stuff from everything on your system.
If the problem is with a form, you can use the trace function built into the Help Diagnostics to get a diagnostic trace file to upload to Oracle Support. The trace function requires that you enable the profile options Utilities: Diagnostics and Utilities: SQL Trace for the user performing the trace. Then re-create the problem, and after the error has occurred, turn trace off and upload the raw and TKPROF versions of the file. You will so amaze your support analyst with your knowledge of working with Support that he will be eager to work on your TAR.
NOTE
Since trace and TKPROF are often classified as programmer's tools at many companies, you may be restricted from using these functions. If you need them, get your system administrator involved with the TAR.
To turn on the trace function for reports, find the report filename from the log file and select Concurrent, Program, Define using the System Administrator responsibility. Query the report filename and click the Enable Trace box in the lower-left corner of the window. Then run the report again. The DBA can locate the trace file for you. Once the problem is fixed, remember to turn off the Enable Trace flag.
Control the Number of People from Your Company Who Contact Support
Because working with Support is an acquired skill, Oracle customers and Support often control the number of people who can initiate a TAR. This technique also helps to ensure that the people working the problem have the Unix, database skills, and system security access to investigate the technical aspects of the problem. Unfortunately, this arrangement does not ensure the person contacting Support will have the business or applications skills to define the problem and re-create the error on demand.
TIP
If the points of contact with Oracle Support are technical people, make sure the end user who first identifies the problem participates in the initial support call when the TAR is opened and defined. This technique saves everyone a lot of time and establishes end-user participation in the solution.
Make Sure the Support Analyst Can Easily Contact You
Often, the analyst initiates a call to you when she needs more information or has a resolution. Make sure she can get in touch with you. Both you and Oracle can log contact information in the TAR definition, so make the following access methods available:
Phone numbers (your desk, cell, fax, alternate)
Voice mail
Pagers
E-mail addresses
TIP
Let the support analyst know the hours you will be available and that you want to continue working on the problem. You can leave this information on the voice mail attached to the TAR number.
Update the TAR When New Developments Arise
If you discover something new about the problem, update the TAR on MetaLink or call the support number and continue working on the existing TAR. The chances are high that your support analyst will not be available when you call. That's a good reason to log it on MetaLink. However, if you leave a voice mail or update MetaLink with the new information, you can get back in the queue for service, and he will call you back. Try to avoid messages that don't add something new to the problem definition or that don't move the issue forward. Also, if the problem goes away, close the TAR so everyone can focus on problems that are still open and important.
Establish an Internet Connection
A relatively high-speed Internet connection is important to working with Support. When a patch becomes available, the analyst can post it to the Oracle FTP site under your TAR number, and you can download it that same day. This technique is far superior to waiting for delivery of a tape by an express courier.
NOTE
A high-speed connection is desirable. Patches of more than 60MB are common, and you don't want to download that at 28.8KBps.
Establish Your Dial-In Procedures in Advance
Occasionally, the support analyst wants to dial in to your system to run some ad hoc SQL queries on your data. If you are the first to discover a bug and Support is not able to reproduce it on an Oracle system, Oracle Development requires dial-in access to your database in most cases. If your company policy permits remote access by nonemployees, you can save time by defining the connection process in advance.
Create a document of standard connection procedures for Support to use in advance. Test it by connecting to your system from the outside to make sure the modems and guest accounts are behaving correctly. Have the document ready to FAX or e-mail so that Support will have an efficient method to connect to your system. If you enter the TAR electronically, do not paste the dial-in procedures into the body of the TAR because that is a security problem when the TAR is posted on MetaLink. Instead, have the support analyst enter the connection procedures and mark them to be unpublished.
Use Experienced People
Because working with Support is an acquired skill, training of contact people is worth the effort. Education and training improves diagnostic capability and speeds problem resolution. When you have a badly behaving application, it is helpful if you know what the correct behavior should be before you call Support. An inexperienced support analyst and a nontechnical, novice user have low odds of solving a problem. In a Unix environment, someone at your site should have the following operating system and database skills:
You need the ability to navigate through the Unix file system. Understand where the Oracle programs are stored under the $APPL_TOP directory structure.
You should be able to issue simple commands from the Unix command-line prompt.
You should be able to look at the data in the database tables with simple Structured Query Language (SQL) commands.
You should be able to use the Unix visual editor (vi).
TIP
If you don't have these skills, consider asking Support to dial in to your system early in the diagnosis of the problem.
Escalate Issues When Appropriate
There are two ways to affect the priority of your TAR. First, Oracle assigns severity levels based on business impact to each TAR. Second, if work is not progressing, you can escalate the TAR through the management hierarchy of Oracle Support.
Each TAR is classified by severity, and Support acts according to specified procedures:
Severity 1—This TAR is characterized by a complete loss of service and the mission-critical process is interrupted. The situation is a full emergency and usually has one or several of the following characteristics: data corruption, a critical function is not available, the system hangs indefinitely (causing unacceptable delays), or the system crashes repeatedly after restart attempts. Oracle Support is committed to work on these TARs continuously (twenty-four–seven) until the issue is resolved or as long as useful progress is being made. You must also be available to work on the problem on a continuous basis.
Severity 2—This problem is caused by a severe loss of service, and no acceptable workaround procedure is available. Oracle Support works on severity 2 TARs during normal business hours at the originating support center. If Oracle cannot duplicate the problem on its system, the analyst might request access to your system. Most TARs can be classified at this level.
Severity 3—This classification is used when there is a minor loss of service. There is some inconvenience, but a workaround procedure can compensate to restore functionality. The support analyst works on this classification of problems during normal business hours.
Severity 4—This is the lowest level of classification. This level results in no loss of service, and the problem is a minor error that does not impede the operation of the system. Historically, the severity 4 classification has been used for making enhancement requests, but now enhancement requests are entered through the ERS system on MetaLink. ERS is superior to using a TAR because you can query some public enhancement requests already in the system.
Oracle has established a four-level escalation process for severe TARs. Level 1 is the initial contact and problem definition between you and the support analyst. You negotiate the severity level of the TAR with supporting business justification. You can request TAR reassignment, callback, or status of issues. If insufficient progress is being made, you can request to speak to a Duty Manager (level 2).
The second level of support is between you and the Duty Manager at the Oracle Support center. The analyst updates the escalation level field on the TAR to the Duty Manager. The status of the TAR is set to Immediate Response Required. The analyst pages or personally hands off the TAR to the Duty Manager.
When the escalation has happened, the Duty Manager calls you within 30 minutes and determines with you an acceptable action plan. The Duty Manager documents the conversation and the plan in the TAR log. The Duty Manager follows up to ensure that the plan is followed and/or resets your expectations. The Duty Manager can assign more experienced analysts to the problem.
Level 3 support is between your project team leader and the Support Senior Manager. If the level 1 and level 2 escalation processes fail, your project team leader might call and escalate the issue to the Support Product Manager. These two managers discuss and agree on an action plan to move the issue forward.
NOTE
We have resolved hundreds of support issues, and never have we had to raise a TAR to support level 3. The Duty Managers are very responsive. Descriptions of the procedures at levels 3 and 4 are as provided by Oracle Support management. Oracle says that 99% of all TARs are resolved at or below the duty-manager level.
After normal escalation at levels 1, 2, and 3, your project manager might escalate to the Applications Support Director. Presumably, there is a problem with the way the two organizations are working together, and these two managers must resolve those issues before work can proceed on the TAR.
Place Full Documentation in the TAR
Include all relevant documents, reports, log files, and so forth, in the body of the TAR. If you are escalating, new people become involved. If you are discovering a new bug, Development people become involved. If you have a severity 1 problem, other support centers might need to understand your comments to work continuously on the problem.
When Dealing with Bugs, Give Details to the Developer
When you are working with Development to diagnose and repair an official bug, make sure you understand the program specifications. At this point, you are helping to define the logic of the program. Try to determine how your problem fits within the original design of the system. Because Development is interested in making the program conform to the original design, you are not allowed to extend the original specifications or sneak in an enhancement request with this procedure.
Allow Time for Testing
Perform the appropriate unit and integration tests on any materials you receive from Oracle Support. You should test module patches, Family Packs, and Maintenance Packs before applying them to the production system. Most Oracle Application sites maintain at least one test database for this purpose. You should remain in control of your systems, and you should know the expected result of your tests. Don't compromise your production data, and know your recovery plan.
Use Experienced People When in a Rapid Implementation Scenario
Working on a TAR and dealing with Oracle Support can be time-consuming. For an experienced user of support services, the shortest problem can easily take two hours to resolve:
Half hour to detect the problem.
Half hour to define the problem and create the iTAR.
Three-quarters of an hour to work with Support to document and diagnose the problem and find a solution.
One-quarter of an hour to implement the solution.
That two hours is about as good as it gets, and an inexperienced project team member can take two or three times that much time.
More complex support issues can easily take 2–10 times the effort of the easy problems. Because a typical implementation for the core financial, distribution, and manufacturing applications might generate 50 to 100 TARs or more, this kind of activity can easily add more than a man-month of effort to a project.
Be Reasonable
Often, your issue with Support will not be resolved because Support cannot change the original design of the Applications. You might have to close your TAR by accepting an enhancement request when necessary. You have to know when to draw the line and be graceful about it. However, don't compromise if the documentation or common sense indicates that the software is misbehaving.
Be Professional
Your professionalism will make working with Support much easier. Most of the analysts are genuinely interested in helping you solve your software problems, and they have powerful tools to diagnose what is happening and find solutions. Work with the support analyst as a partner.
You might be under a lot of stress during the support call, but it is your job to relieve stress and move the TAR forward. You might be mad at Oracle Corporation in general for shipping you buggy software, but don't transfer that anger to the support analyst who is the one person who can solve the problem.
I have seen analysts verbally abused many times, and it never accomplished anything. Stick to the diagnostic facts, and be constructive and polite. You will be rewarded with a cooperative and concerned analyst. If you don't get the results you deserve, you can easily get a new and more experienced analyst by escalating the problem to the Duty Manager.
Track Progress and Close the TAR
You must drive the TAR resolution process. If you can't solve a problem in the first phone call, make sure your issues are being worked every day. Establish a log and update the status of open TARs after each phone call. Follow up on each open TAR every day. You can use the Internet to update the iTAR, voice mail, or e-mail. Oracle Support can track the amount of time each TAR is assigned to a support analyst, you, or Oracle Development. When you update your TAR, you change the assignment status back to Oracle and that starts its clock ticking again.
When you finish work on a TAR, make sure it is closed so all will understand the status of the issue. Soft-closing a TAR with a future date is available if you just need to test a solution.
Document Your System
Keep track of the patches, changes, and revision levels of your system. If you have multiple instances at various stages of testing, training, development, and production, you will want to know the state of each implementation. You need to have this information for several reasons:
The support analyst needs to know the state of your system, and Oracle can't keep track of the patches and changes you apply.
You can use this information again on an upgrade.
Sometimes patches don't fix problems. It is helpful to know which ones didn't work so you won't waste time on them on other instances.
Rarely, but unfortunately, a patch can make a problem worse or change the problem to something else. You need to know the history to unwind the problem.