File system gets filled due to Report cache in Oracle apps R12

In Oracle Apps R12 the report server is started by the concurrent manager whenever there is a need to run a report and it is an in-process report server which is started as per the need.

During this process the oracle reports writes to a temporary cache directory. The path and cache size is defined in the rwbuilder.conf file.

For example:

<cache class="oracle.reports.cache.RWCache">
<property name="cacheSize" value="50"/>
<property name="cacheDir" value="full path for $INST_TOP/logs/ora/10.1.2/reports/cache"/>
<!--property name="maxCacheFileNumber" value="max number of cache files"/-->
<!--property name="ignoreParameters" value="parameter names to be ignored in constructing cache key, separated by comma ','"/-->
</cache>

This cache directory doesn’t have any control on its growth and purging, though its temporary in nature. The Purge Concurrent Request and/or Manager Data" (FNDCPPUR) program also doesn’t purge these files.

Hence as more oracle reports are run, more temporary files gets created and space gets filled up under:

 $INST_TOP/logs/ora/10.1.2/reports/cache.

Solution: 

1. Manually delete the files under:

$INST_TOP/logs/ora/10.1.2/reports/cache

2. Change the cache size parameter in rwbuilder.conf to zero:

<property name="cacheSize" value="50"/> to
<property name="cacheSize" value="0"/>

OACORE fails to start after increasing the JVM processes in Oracle R12

Issue: OACORE processes are not starting.

Recent changes: Increased the JVM processes in the context file.

Error:

adoacorectl.sh: exiting with status 206

Solution:

1)  Stop the middle tier processes:

adopmnctl.sh stop

2) Kill any runaway/zombie processes.

3) Remove the following jms lock files:

$ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/forms_default_group_1/jms.state.lock,

$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/jms.state.lock

$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/oafm_default_group_1jms.state.lock

4) Restart middle tier:

adopmnctl.sh start

Flow cycle of a web browser connection with Oracle Web server

Let’s consider Oracle Apps 11i/R12. 

 How does the web browser make connection with the http server?

 Below is the flow:

* An attempt is made to obtain an IP address using the IPname of the website/URL.

   * The Domain Name servers (DNS) provides this lookup (conversion of IP name to IP address).

   * Open an IP socket connection to that IP address.

   * Then the socket connection is used to write an HTTP data stream to the web server.

 
   * Receive an HTTP data stream back from the Web/HTTP server in response. This data stream contains status codes whose values are determined by the HTTP protocol.  

Issue starting 10g Forms and reports server

Error:

 Ping URL for proc: 887947305 failed while pinging url:/reports/rwservlet/pingserver?start=auto

12/11/01 08:18:46 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_BI_Forms~default_island~1 (887947305:8511)

12/11/01 08:19:10 [2] [libopmnoc4j] AJP13: Receiving response code: 500 while configured success code is: 200

12/11/01 08:19:10 [2] [libopmnoc4j] Ping URL for proc: 887947305 failed while pinging url:/reports/rwservlet/pingserver?start=auto

Search String: 

REP-56105

tail –f OC4J~OC4J_BI_Forms~default_island~1

ConfigFileName: /u01/oracle/product/ais10g/forms90/server/forms/server/formsweb.cfg

REP-56105: Engine rwEng-0 died with error

SOLUTION: 

Bug no: 3557723

Work Around:

Disable pingserver command in $ORACLE_HOME/opmn/conf/opmn.xml

- Take the backup of opmn.xml file

@   cd $ORACLE_HOME/opmn/conf

@  cp opmn.xml opmn.xml.PINGAUTO

@ - Open file opmn.xml

@ - Take out the following lines from opmn.xml

@   < category id="urlping-parameters">

@      < data id="/reports/rwservlet/pingserver?start=auto" value="200"/>

@   < /category>

@ - Bounce the opmn

@   cd $ORACLE_HOME/opmn/bin

@  ./opmnctl stopall     ;        ./opmnctl startall

Error: CACHE mismatching

SOLUTION: 

Also Check for the domain name in /u01/oracle/product/ais10g/webcache

Webcache.xml ----change the domain name.

Unable to view new responsibilities added to an User.

Issue: unable to view new responsibilities as added to existing users or re-activate end dated responsibilities. 

Recent Changes: Applied a new Patch using adpatch.

Solution:

1. 429852.1 - Reactivated User Does Not See Responsibilities - This step is where we applied patch 6008417.

2. Workaround

2.1. Stop and restart the "Workflow Java Deferred Agent Listener" in OAM

2.2. This will clear the events in Ready status. To verify this, run:

select

decode(state,

0, '0 (Ready)',

1, '1 (Delayed)',

2, '2 (Retained)',

3, '3 (Exception)',

to_char(state|| ' (Other)')) state,

count(*) how_many

from wf_java_deferred

group by state

The number of 'Ready' needs to get to 0 before you continue.

2.3. Login to E-Business as SYSADMIN and navigate to Workflow Administrator Web Application, Business Event

Search for oracle.apps.fnd.wf.ds.userRole.updated.

Then click on clock symbol under the Subscription column.

Disable the AP_WEB_PROXY_ASSIGN_PKG.proxy_assignments subscription

2.4. Stop and restart the Workflow Agent Listener Service

3. Follow the steps on note 464445.1 - specifically the step to Run the Directory Services Validate User/Role concurrent request.

Issue running adcfgclone.pl on Database Oracle home.

 While running adcfgclone.pl as oracle user, the session just hangs without giving any error in the log file. The issue was that it wasn’t able to connect to the database as:
sqlplus "/as sysdba"

Deep dive:

Most generally this can occur due to improper shutdown of the database. While in the course of a database shutdown, if any of the oracle process is not completed killed and it holds a large shared memory segment, such issues can occur which would prevent from normal database operation.

So next time when an instance tries to start it is not able to grab a large chunk of memory to get started.

This can be checked as follows:

After the database has been shut down , there shouldn’t be any processes running as oracle on the server level:

ipcs -m | grep <oracle user>

eg:

ipcs -m | grep oracle

key              shmid          owner   perms         bytes        nattch     status
0x185ee230 884763     oracle     640        789785472      0

Here in the above example, a single oracle process holds 789785472 bytes of shared memory, even after shutdown. If the database instance is started, then we would expect abnormal behaviors on the database side, like the login issue that we are discussing now.

This can cleaned as mentioned below:

ipcrm -m < shmid>

eg:

ipcrm -m 884763 

This would free up the memory held.

This resolved the issue.

Automatic Diagnostic Repository Command Interpreter: ADRCI in Oracle 11g

 We go to bdump and access the alert.log file whenever we have to analyze an issue in the database.

However In 11g, RDBMS diagnostic data has been reorganized and is stored inside a common directory structure, the Automatic Diagnostic Repository (ADR). An ADR is a centralized directory structure where one can find trace files, alert messages, incident dumps, core files, etc. This parameter is set by DIAGNOSTIC_DEST initialization parameter.

You will find the following directories under ADR HOME -

Alert : The alert directory contains the XML alert log

Cdump : core dumps are stored in this directory

Trace : Process trace files and Alert.log are stored in the trace directory. 'Background_dump_dest' and 'user_dump_dest' are now ignored in 11g. Now all the trace files will be generated in 'trace' directory.

Incident : The incident directory stores dump files created when critical errors are encountered.
Each occurrence of a critical error( incident ) is given its own incident directory, with the incident ID used to form the directory name.

Metadata : The metadata directory stores a series of files that contain diagnostic metadata.

HM : The hm directory stores reports for health checks

Incpkg, ir, lck, sweep : These directories contain internal diagnosability framework state.

ADRCI is the command line utility using which ADR is accessed.

----  To view the alert log file:

Example

$ adrci

ADRCI: Release 11.2.0.2.0 - Production on Wed Aug 1 12:02

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved

ADR base = “/d01/app/oracle”

ADRCI > show alert –tail –f

---- ADRCI can also be used to track a problem and an incident:

Example: Whenever there is a problem recorded in the alert log file, we also get a message stating: Use ADRCI or Support Workbench to package the incident. Adrci can be used to know more about the problem and the incident:

ADRCI > show problem

ADRCI > show incident

ADRCI > show incident -mode detail -p "incident_id="

---- ADRCI can also be used for :

-       Incident Packaging Service (IPS):  To send details of the problem to oracle support through a package or a zip file.

-       Purge trace files. Example: ADRCI> purge –age 1440 –type trace (1440 is minutes, ie, one day. So all trace files older than 1 day will be purged).

Issue: Concurrent request running from a very long time

.
Application version: 11.5.10.2 multi-node (4 nodes)
Database: 11g RAC


Scenario:

Business critical concurrent request is running from more than 2 hours which was supposed to complete within 1 hour.
The standard manager and the manager responsible to execute the critical concurrent request (custom manager) were overloaded with many requests. The server on which these managers run had very high server load and all CPU’s were utilized.

After diagnosis, we had to kill few requests which were running from more than 2 hours and were creating load on the server. These requests were in fact not performing anything and had gone zombie.

The server load came down and CPU’s available were set free to execute new requests/processes.

However the business critical request was still running and wasn’t moving ahead at all. The database session associated was also not executing anything.

Challenge:

Since this being a critical business request, we couldn’t terminate the request coz we had already lost few hours running it. At the same time the database session also wasn’t performing any execution and the whole request was not going anywhere even though it showed running normal status.

Trick:

Not recommended, however can be a life saver at times.

---- Forcibly change the status of the request to pending normal.
---- Kill the database session of the request.

Use the below scripts:

update applsys.fnd_concurrent_requests set phase_code = 'P',
                                           status_code = 'I',
                                           actual_start_date = null,
                                           crm_release_date = null,
                                           controlling_manager=null,
                                           logfile_name = null,
                                           logfile_node_name = null,
                                           outfile_name = null,
                                           outfile_node_name = null,
                                           crm_tstmp = null
where request_id in ('request-id');


             alter system kill session 'sid, serial#, @instance-id';


Hence a new database session will get created as soon as the old one is killed and the request starts running. Since the server on which the request is supposed to get processed also had no issues now, the request went through very fast without any issues ;)