Thursday, June 2, 2011

Workflow Mailer Automatic Bounce script

Application:- 11.5.10.2
Database:- 10.2.0.4
OS:- Solaris 10

One of the task for Apps DBA is to montior mailer and bring it up through OAM.

How about automating this task?

Below script will be helpful if you want to automate task for bringing up your mailer automatically through scripting.

Before starting please collect value of COMPONENT_ID from FND_SVC_COMPONENTS
against 'Workflow Notification Mailer' and place it in below script.

select COMPONENT_ID from fnd_svc_components where COMPONENT_NAME = 'Workflow Notification Mailer';

#!/bin/ksh
#set -x
################################################################################
#
# Script Name: Mailer_Monitor.ksh
#
# Directory:
#
# Description:
#
#
#
# Arguments:
#
# Output/log:
#
# called by:
#
# called scripts:
#
# Created By: Deepak Jha
#
# Modification History:
#
# DATE: DBA: CR#/SR#: Modification Description
# __________ ___________ __________ ________________________
#
#
################################################################################
export ORACLE_SID=TEST
export ORACLE_HOME=/opt/oracle/${ORACLE_SID}/db/10.2
HOSTNAME=`hostname`
DBREP=/tmp/mailerdown.rep
EMAIL_ADDRESS=abc@xyz.com
rm -f $DBREP 2>/dev/null
${ORACLE_HOME}/bin/sqlplus -s 'apps/apps_pass'<set heading off
set pages 0
set feedback off
spool /tmp/mailerstats.rep
SELECT COMPONENT_STATUS from FND_SVC_COMPONENTS
where COMPONENT_ID= '';
spool off
EOF

MONITOR=`grep -v "SQL>" /tmp/mailerstats.rep`
if [ $MONITOR != "RUNNING" ]
then
echo "Mailer is not running and will be started" >> $DBREP
cat $DBREP | mailx -s "$HOSTNAME:$ORACLE_SID: MAILER DOWN" "$EMAIL_ADDRESS"
${ORACLE_HOME}/bin/sqlplus -s 'apps/apps_pass'<update fnd_concurrent_queues
set control_code = 'A'
where concurrent_queue_name = 'WFMLRSVC';
commit;
EOF
else
exit 0;
fi
#exit


Note:- Above script has scope of enhancement.
Thanks,
Deepak

Monday, April 11, 2011

11i:- Issues after applying ATG RUP7 -- Part 3


  • Issue:- After applying ATG RUP7 end-dated users were not able to login after reactivating them.



  • Error:- "oracle.apps.fnd.framework.OAException: java.lang.NullPointerExceptionat oracle.apps.fnd.framework.OAException.wrapperException(OAException.java:891)at oracle.apps.fnd.framework.webui.OAWebBeanHelper.processRequest(OAWebBeanHelper.java:603)"



  • Cause:- The issue occurs if the two tables wf_local_user_roles and wf_user_role_assignments are not is synch. But we are bit worried as why these table are not getting updated as soon as user end date is removed. This was working fine in RUP6 instance.



  • Temporary workaround :- is below but this needs to be done after every occurunce of user end date removal and that was real pain:-

1. Run the concurrent program below in 2 passes and then bounce Apache:


1.1 "Workflow Directory Services User/Role Validation" Parameters : 100000, Yes, No, No.


1.2 "Workflow Directory Services User/Role Validation" Parameters : 100000, No, Yes, No"


2. Bounce Apache (This step is not ALWAYS required. You may want to test after completing 1.1 and 1.2 to see if the problem is fixed before bouncing.)



  • Permanent Resolution:-


  1. Go to Workflow Administrator Web Applications

  2. Go to Administrator Workflow --> Business Events

  3. Search for oracle.apps.global.user.change subcription event

  4. Click on Subscription.

  5. Click on Pencil button against wf_sso.user_create_rf to disable it.

  6. Save changes and replicate the issue.

11i:- Issues after applying ATG RUP7 -- Part 2


  • Issue:- After applying ATG RUP7 Users were not able to approve/reject workflows via email response.

  • Cause:- Inbound mailer was unable to process user mails. The ATG RUP7 version of Workflow Mailer module has been having several issues concerning to the email responses. There were found multiple problems. I checked workflow logs to check as what is the cause behind it. Error i found in workflow log was:-

"[Apr 2, 2011 8:58:07 AM BST]:1301731087867:Thread[inboundThreadGroup1,5,inboundThreadGroup]:0:-1:XXXX:-1:-1:UNEXPECTED:[SVC-GSM-WFMLRSVC-109454-10004 : oracle.apps.fnd.wf.mailer.IMAPInboundProcessor.open()]:Unable to connect to mail store"


Error somewhat suggested as connectivity issue between mailer node and IMAP server. We checked the connectivity but their was no . Then we enabled workflow logs at STATEMENT level to reach error stack:- use below steps for doing same:-



  1. Enable debug for Mailer Service Component :In OAM : Site Map / Notification Mailer / Edit / Advanced / Go to Step 2 / Change "Log Level" to "Statement"Apply change.

  2. Bounce Mailer Container to force generation of a fresh log :In OAM : Site Map / Notification Mailer / Click on "Workflow Mailer Service" in "Container" columnOnce in containers page, select the "Workflow Mailer Service", then Stop. Refresh page until status becomes Deactivated. Then Start. Refresh page until status becomes Activated.

  3. Use below script to identify log file name/path.:-

SELECT fcp.logfile_nameFROM fnd_concurrent_queues fcq, fnd_concurrent_processes fcp, fnd_lookups flkupWHERE concurrent_queue_name in ('WFMLRSVC')AND fcq.concurrent_queue_id = fcp.concurrent_queue_idAND fcq.application_id = fcp.queue_application_idAND flkup.lookup_code=fcp.process_status_codeAND lookup_type ='CP_PROCESS_STATUS_CODE'AND meaning='Active';.



On close investigation of log file we saw error like :- " [Apr 2, 2011 1:52:51 PM BST]:1301748771561:Thread[inboundThreadGroup1,5,inboundThreadGroup]:0:-1::XXXXX:-1:-1:UNEXPECTED:[SVC-GSM-WFMLRSVC-109510-10004 : oracle.apps.fnd.wf.mailer.IMAPResponseHandler.handleUnsolicited(EmailParser)]:Problem encounted when handling an unsolicited email -> java.io.IOException: Error in encoded stream, got 1".




  • Resolution:- Searched error code on metalink and got a hit for noteid:- 974088.1 and their i found refrence for patch 9383048. Issue has been resolved by above patch.

11i:- Issues after applying ATG RUP7 -- Part 1

As part of pre-req to 2010-2011 payroll patches oracle suggested to apply HRMS Family Pack K Rollup 5 and one of the pre-req for rollup 5 is ATG RUP7. We faced lot of issues after applying RUP7 and some of them are still undocumented in known issues document for this patch. I am highlighting only those issues which we faced after applying it and could not found it in known issue document for this patch.

  • Issue:- After applying ATG RUP7 system priv 'SELECT ANY DICTIONARY' was missing from apps user. This issue can be because of AD minipack I.7.

  • Resolution:- Grant 'SELECT ANY DICTIONARY' to apps using sysdba.

Thursday, March 31, 2011

11i Responsibility not visible to the user in the list of responsibilities assigned to the user

Issue:- User not able to view responsibility under following scenarios:-

  1. User end date has been removed.

  2. Newly added responsibilty to exsisting user is not visible.
Cause:- Oracle not says so but its bug on top of RUP6 and also not resolved by RUP7. Resolution:- Its really hit and try kind of approch, sometime one works other time don't. Ideally approach 2 should work all the time but start testing your luck with 1 as its easiest. Option (1) After adding responsibility wait for couple of seconds and then end date that resp. save , then again wait for couple of seconds and remove end date, save. Ask user to login and try. if not then bounce apache with clear cache and ask user to try. In case of removing end date do above for all the resp, bouhc apache with clear cache and try. Option(2)

  1. Check if the responsibility is not end dated.

  2. Run the request " Syn responsibility role data into the WF table".

  3. Run the request " Synchronize WF LOCAL tables" enter the parameters- Orig System: ALL- Parallel Process:0- Logging Mode: LOGGING (Logging Activated)- Temporary Tablespace : Blank- Riase Errors: Yes

  4. Run the request "Workflow Directory Services User/Role Validation" enter the parameters- p_BatchSize:10000- Fix dangling user/roles : Yes- Add missing user/role assignments : Yes- Update WHO columns in WF tables: Yes

  5. Log off and login back and verify that the responsibility appears.

  6. If not then bounce apache with clear cache option.

Wednesday, March 23, 2011

Workflow Notification Mailer not sending mails.

Issue:- Users reported not getting mail from oracle. OAM was showing notification mailer as up and running.


Error:- Useful error in FNDCPG* log was ' 421 4.3.0 collect: Cannot write ./dfp2ACSkLM020213 (bfcommit, uid=0, gid=25): No such file or directory^M'.

Troubleshooting:- Mailer was up at application level but user were not getting mails. We tested sending test mails from application but it never went through, on checking wf_notification table for mail with open/mail status.
Use below query to confirm it:-

select notification_id, status, mail_status, begin_date,
TRUNC(((sysdate-begin_date)/(1/24))*60) exec_time
from WF_NOTIFICATIONS
where status = 'OPEN' and mail_status = 'MAIL';


if outcome of the below query continue to give output for sometime then it means your notification mailer is facing some issue

We found out chunk of mails waiting their. I thought to check SMTP mailer on the box.Mail sent directly through server were reaching user mailbox but not through E-BIZ, bit strange. On further analysis, reason can be when you start mailer, it start from count 1, i.e. it will try to send 1st mail and then goes on. Mailer try to send mails for count 1-5(or some other figure) before going down. /var/spool/mqueue is the directory where sendmail(unix utility used by SMTP) stores the deffered messages. I am not sure but something was stopping sendmail to check /var/spool/mqueue directory and hence mailer was not able to deliver messages that were their. Timestamp of directory should also help you to know about it. Actually it should be updated quite frequently(depending on mailer usage).

Resolution:- Their might be some other resolution also but for non-live box easiest was to bounce the OS hosting database. And moreover it falls in unix admin patio so I didn’t thought much.

usdsop cannot create a new process while running adadmin

Issue:- While trying to compile apps schema using adadmin we are getting below error after selecting number of parallel process:-
"aiosp2() Error: failure in usdspn()Contents of error buffer are:"usdsop cannot create a new process
Cause: usdsop encountered an error creating a new process.
[Reason].Action: Check that your system had enough resources to start a new process. Contact your system administrator to obtain more resou (RE"

Troubleshooting:- At first instance its seems to be clear case of memory limitation on the server. This issue was occuring for every tool invloving multiple worker.
We tried running adadmin using less number of worker and that worked, this confirmed issue on the OS memory on server.

Resolution:- Asked our OS admin to enlarge the swap space to 1.5 times the current size and it resolved the issue.

RCA:- System was almost out of swap space and increasing swap was only option to move further. You can check swap using vmstat, top or swap -l command.