New Relic PHP Daemon Installation Issues on RHEL7.x

Yesterday I was trying to install the PHP monitor from New Relic. If you don’t know New Relic, and you need something quick to get you started with monitoring for free, go check them out.

They have a general system monitor for hardware resources (with alerts) but more interestingly, you can monitor your PHP applications with a special daemon you install as a PHP module on top of Apache.

The PHP monitoring daemon is compromised of 2 parts – the PHP extension and the New Relic daemon which operates like a proxy between PHP and the New Relic servers.

Problem was I couldn’t get the PHP extension  to talk with the New Relic daemon. After turning on verbose debugging, log file was showing the following error:

verbosedebug: RINIT processing started
verbosedebug: daemon connect(fd=13 uds=/tmp/.newrelic.sock) returned -1 errno=ENOENT
warning: daemon connect(fd=13 uds=/tmp/.newrelic.sock) returned -1 errno=ENOENT. Failed to connect to the newrelic-daemon. Please make sure that there is a properly configured newrelic-daemon running. For additional assistance, please see: https://newrelic.com/docs/php/newrelic-daemon-startup-modes
debug: unable to begin transaction: no daemon connection

After spending some considerable amount of time on googling this error, playing with file permissions, switching from UNIX file socket to TCP ports and restarting the Apache webserver I realized that the New Relic daemon (the proxy) has 2 modes of startup operation:

  1. Started by Apache
  2. Started by init script

I didn’t want to switch from the Apache startup method to init script in order not to complicate the environment so I switched again to TCP port, stopped Apache completely and manually killed the New Relic proxy daemon. (kill -9) Viola! problem solved.

Turns out that when I restarted Apache (apachectl restart) It didn’t killed the New Relic process, which probably caused this issue.

Advertisements

How to solve “E_ACCESS_DENIED / Forbidden” Error When Adding Library Widget to IBM Connections Community (After changing connectionsAdmin user)

I’d like to share a very painful experience during one of my last customer visits where I had to switch from using a local fileregistry websphere administrator (wasadmin) to an LDAP based administrator. This had to be done in order to change the connectionsAdmin user and activate SPNEGO SSO in IBM Connections 5.0 CR2.

I followed all the regular and overly-complicated mixture of weird documentation in IBM’s knowledge center (someday someone at IBM’s board of directors will be held accountable for all this mess) mainly using this topic and its sub-topics.

At the end of all the configurations and settings, the library component of Connections (CCM) was not functioning so I started looking at the logs and digging and googling and blah blah blah, eventually I realized I’m facing another one of IBM famous bugs as per described in this excellent link. I’m not going to get into all the details (again, because of Julius Schwarzweller excellent blog post) but in essence this bug happens when  you migrate your environment from 4.5 to 5.0 and than change the filenetAdmin password and/or username. I followed all the steps described in Juilius’ blog to fix the error and indeed the problem has been solved! I could see libraries loading up in Communities and everything looked OK…

…Until I tried to add a new library widget to a community:

2015-10-25 20_53_49-2015-10-22 11_09_32-Overview - test2.jpg - Windows Photo Viewer

“403 forbidden”

Connections log was showing this error:

P8ErrorHandle E com.ibm.ecm.qkr.services.p8.P8ErrorHandler handleError 403 AccessDenied CQL5953: You do not have access to the requested object. Contact the owner of the object for assistance in changing your access rights.
 com.filenet.api.exception.EngineRuntimeException: FNRCE0001E: E_ACCESS_DENIED: The requester has insufficient access rights to perform the requested operation. failedBatchItem=0 errorStack={
 at com.filenet.engine.persist.IndependentPersister.checkObjectPropertyAccess(IndependentPersister.java:2502)
 at com.filenet.engine.persist.IndependentPersister.checkPropertyAccess(IndependentPersister.java:2311)
 at com.filenet.engine.persist.IndependentPersister.checkCreatePermissions(IndependentPersister.java:1984)
 at com.filenet.engine.persist.IndependentPersister.preCreate(IndependentPersister.java:1170)
 at com.filenet.engine.persist.IndependentPersister.preExecuteChange(IndependentPersister.java:607)
...
...

And also this:

EventPropagat E com.ibm.lconn.widgets.service.EventPropagater postRemoteEvent CLFWZ0004E: Event 'widget.added' sent to remote lifecycle handler at https://connections.XXXXX.co.il:9443/dm/atom/communities/feed returned bad response: 403 - Forbidden
AddWidgetActi E com.ibm.lconn.widgets.actions.AddWidgetAction execute CLFWZ0004E: Event 'widget.added' sent to remote lifecycle handler at https://connections.XXXXXX.co.il:9443/dm/atom/communities/feed returned bad response: 403 - Forbidden
 com.ibm.lconn.widgets.model.LifecycleStatusCodeException: CLFWZ0004E: Event 'widget.added' sent to remote lifecycle handler at https://connections.XXXXXX.co.il:9443/dm/atom/communities/feed returned bad response: 403 - Forbidden
 at com.ibm.lconn.widgets.service.EventPropagater.postRemoteEvent(EventPropagater.java:569)
 at com.ibm.lconn.widgets.service.EventPropagater.addWidget(EventPropagater.java:753)
 at com.ibm.lconn.widgets.service.WidgetInfoService.addWidgetPropagateInternal(WidgetInfoService.java:285)
 at com.ibm.lconn.widgets.service.WidgetInfoService.addWidget(WidgetInfoService.java:376)
 at com.ibm.lconn.widgets.actions.AddWidgetAction.execute(AddWidgetAction.java:70)
 at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
...
...

I immediately turned to look at the FileNet engine log file (p8_error) which showed basically the same thing:

FNRCE0001E - ERROR method name: checkObjectPropertyAccess principal name: wpsadmin Global Transaction: true User Transaction: false Exception Info: The requester has insufficient access rights to perform the requested operation. Not granted required access on target object.
com.filenet.api.exception.EngineRuntimeException: FNRCE0001E: E_ACCESS_DENIED: The requester has insufficient access rights to perform the requested operation. Not granted required access on target object. failedBatchItem=0
 at com.filenet.engine.persist.IndependentPersister.checkObjectPropertyAccess(IndependentPersister.java:2502)
 at com.filenet.engine.persist.IndependentPersister.checkPropertyAccess(IndependentPersister.java:2311)
 at com.filenet.engine.persist.IndependentPersister.checkCreatePermissions(IndependentPersister.java:1984)
 at com.filenet.engine.persist.IndependentPersister.preCreate(IndependentPersister.java:1170)
 at com.filenet.engine.persist.IndependentPersister.preExecuteChange(IndependentPersister.java:607)
 at com.filenet.engine.persist.SubscribablePersister.preExecuteChange(SubscribablePersister.java:227)
 at com.filenet.engine.persist.ReplicablePersister.preExecuteChange(ReplicablePersister.java:125)
 at com.filenet.engine.persist.ContainablePersister.preExecuteChange(ContainablePersister.java:89)
 at com.filenet.engine.persist.FolderPersister.preExecuteChange(FolderPersister.java:207)
 at com.filenet.engine.persist.IndependentPersister.executeChangeWork(IndependentPersister.java:505)
 at com.filenet.engine.persist.IndependentPersister.executeChange(IndependentPersister.java:339)
 at com.filenet.engine.persist.SubscribablePersister.executeChange

Obviously I have a security issue at the FileNet level. “wpsadmin” is the login name of the new filenetAdmin (and connectionsAdmin) and for some reason it does not have the required permissions for the Connections object store. I verified that the account is listed in the FileNet object store (And domain) as an administrator with full permissions (including child object so that all objects will inherit the credentials):

2015-10-25 21_06_19-2015-10-22 11_13_12-Administration Console for Content Platform Engine 2015-10-25 21_05_10-2015-10-22 11_14_12-Administration Console for Content Platform Engine

In the above screenshots, you can clearly see that “wpsadmin” has full control on the domain level and also on the object store level with these permissions:

2015-10-22 11_13_40-Administration Console for Content Platform Engine

So you can understand why I was puzzled and quite frankly out of ideas. After fiddling around with the FileNet settings and inspecting permissions on different object types I’ve noticed that the permissions are not being passed in inheritance to child objects. I suspect that during my attempts to solve this problem, I’ve deleted and recreated the old + new admin user from the Connections profile database (using the API as described here and here). The solution was to run the “Security Script Wizard” in ACCE options of the object store as per described here and here. I selected the new FileNet admin user as an Object Store administrator and viola, problem solved!

Button line is, if you ever need to change the connectionsAdmin user, keep in mind that on some circumstances, the new account will not be granted proper permissions on the Connections Object Store. This is where the Security Script Wizard comes in very handy. According to the documentation, it does not modify existing permissions, only adding new users which  sounds safe to me.

If anyone can offer a better explanation to what happened or has a less invasive solution to this problem – please share your thoughts below.

IBM Connections CCM 5.0 Libraries Blank Screen (“500 internal server error”) after upgrading to CR2 when the Files app is not installed

For a while now, I am pondering what should be my first professional blog post as a SySAdmin. Finally I have come to the conclusion that nothing can be more fun (and useful to others) than sharing an installation problem whose root cause is a software defect  (AKA: bug)

I always hate it when I can’t find help to prevalent problems. On the other hand when I do manage to find a solution, it saves so much time – so here I’m starting to give something back. Also I almost never give out change to homeless people so this how I fix my conscience. (No offence everyone, I’m not really comparing you to homeless people :-))

I was at a customer’s site doing an upgrade of IBM Connections from 5.0 to 5.0 CR2 (yes, I know CR3 is out, the issue starte a long time ago). I used this relatively excellent (and rare) step-by-step documentation from IBM. All went fine (except some very annoying glitches in the FileNet upgrade scripts) but when I came to test the environment I found that the Libraries were not working. The page simply said “Loading…” and nothing was happening:

noname

The console in Chrome Developer Tools showed “500 internal server error” (but again, there were no errors in the application log, which is so stupid)

I checked all CCM settings in ACCE, I tried to manually redeploy applications and ran the Config tasks again and again. SystemOut log files weren’t very helpful and P8 logs were shouting some random E_OBJECT_NOT_FOUND error that prove to be completely unrelated.

Eventually I opened a PMR with IBM and as it turns out, the new version of the Libraries app requires that the Files app will be installed. However this particular customer did not install the Files application (due to licensing issues), which created a situation where required JAR files were not installed in the connections provision directory!

I had to manually copy these 2 files:

com.ibm.lconn.communityfiles.web.resources_3.0.0.20150216-1530.jar
com.ibm.lconn.files.web.resources_3.0.0.20150216-1530.jar

From the IBM Connections 5.0 CR2 installation media to:

CONNECTIONS_PROVISION_PATH/webresources

Then I had to shutdown the Connections JVM server, delete the content of temp and wstemp folders in the “AppSrv01” folder and finally restart Connections.  PROBLEM SOLVED!

If you’re planning to install 5.0 CR2/3 please be aware that until IBM issues a fix (or an official technote) to this problem, you will have to manually copy the appropriate files depending on your version.

This concludes my first ever blog post, hope you enjoyed it, ’cause I sure did. yee haa.

 

UPDATE #1

After searching for the missing jar filenames in google, I found this technote. It contains some more info on this issue.