Thursday, March 31, 2011

11i Responsibility not visible to the user in the list of responsibilities assigned to the user

Issue:- User not able to view responsibility under following scenarios:-

  1. User end date has been removed.

  2. Newly added responsibilty to exsisting user is not visible.
Cause:- Oracle not says so but its bug on top of RUP6 and also not resolved by RUP7. Resolution:- Its really hit and try kind of approch, sometime one works other time don't. Ideally approach 2 should work all the time but start testing your luck with 1 as its easiest. Option (1) After adding responsibility wait for couple of seconds and then end date that resp. save , then again wait for couple of seconds and remove end date, save. Ask user to login and try. if not then bounce apache with clear cache and ask user to try. In case of removing end date do above for all the resp, bouhc apache with clear cache and try. Option(2)

  1. Check if the responsibility is not end dated.

  2. Run the request " Syn responsibility role data into the WF table".

  3. Run the request " Synchronize WF LOCAL tables" enter the parameters- Orig System: ALL- Parallel Process:0- Logging Mode: LOGGING (Logging Activated)- Temporary Tablespace : Blank- Riase Errors: Yes

  4. Run the request "Workflow Directory Services User/Role Validation" enter the parameters- p_BatchSize:10000- Fix dangling user/roles : Yes- Add missing user/role assignments : Yes- Update WHO columns in WF tables: Yes

  5. Log off and login back and verify that the responsibility appears.

  6. If not then bounce apache with clear cache option.

Wednesday, March 23, 2011

Workflow Notification Mailer not sending mails.

Issue:- Users reported not getting mail from oracle. OAM was showing notification mailer as up and running.


Error:- Useful error in FNDCPG* log was ' 421 4.3.0 collect: Cannot write ./dfp2ACSkLM020213 (bfcommit, uid=0, gid=25): No such file or directory^M'.

Troubleshooting:- Mailer was up at application level but user were not getting mails. We tested sending test mails from application but it never went through, on checking wf_notification table for mail with open/mail status.
Use below query to confirm it:-

select notification_id, status, mail_status, begin_date,
TRUNC(((sysdate-begin_date)/(1/24))*60) exec_time
from WF_NOTIFICATIONS
where status = 'OPEN' and mail_status = 'MAIL';


if outcome of the below query continue to give output for sometime then it means your notification mailer is facing some issue

We found out chunk of mails waiting their. I thought to check SMTP mailer on the box.Mail sent directly through server were reaching user mailbox but not through E-BIZ, bit strange. On further analysis, reason can be when you start mailer, it start from count 1, i.e. it will try to send 1st mail and then goes on. Mailer try to send mails for count 1-5(or some other figure) before going down. /var/spool/mqueue is the directory where sendmail(unix utility used by SMTP) stores the deffered messages. I am not sure but something was stopping sendmail to check /var/spool/mqueue directory and hence mailer was not able to deliver messages that were their. Timestamp of directory should also help you to know about it. Actually it should be updated quite frequently(depending on mailer usage).

Resolution:- Their might be some other resolution also but for non-live box easiest was to bounce the OS hosting database. And moreover it falls in unix admin patio so I didn’t thought much.

usdsop cannot create a new process while running adadmin

Issue:- While trying to compile apps schema using adadmin we are getting below error after selecting number of parallel process:-
"aiosp2() Error: failure in usdspn()Contents of error buffer are:"usdsop cannot create a new process
Cause: usdsop encountered an error creating a new process.
[Reason].Action: Check that your system had enough resources to start a new process. Contact your system administrator to obtain more resou (RE"

Troubleshooting:- At first instance its seems to be clear case of memory limitation on the server. This issue was occuring for every tool invloving multiple worker.
We tried running adadmin using less number of worker and that worked, this confirmed issue on the OS memory on server.

Resolution:- Asked our OS admin to enlarge the swap space to 1.5 times the current size and it resolved the issue.

RCA:- System was almost out of swap space and increasing swap was only option to move further. You can check swap using vmstat, top or swap -l command.