Related Error Messages:
0000004c LocalExceptio E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "publishEvent" on bean "BeanId(Wikis#lc.events.publish.jar#EventPublisher, null)". Exception data: com.ibm.lconn.events.internal.invoke.async.AsyncEventInvokerException: CLFWY0148E: Could not obtain an event connection
at com.ibm.lconn.events.internal.publish.impl.JMSPublisher.publishEvent(JMSPublisher.java:199)
at com.ibm.lconn.events.internal.publish.impl.JMSPublisher.publishEvent(JMSPublisher.java:163)
at com.ibm.lconn.events.internal.publish.impl.JMSPublisherBean.publishEvent(JMSPublisherBean.java:87)
at com.ibm.lconn.events.internal.publish.impl.EJSLocalStatelessEventPublisher_e13147f4.publishEvent(Unknown Source)
at com.ibm.lconn.events.internal.publish.impl.EJBBasedJMSPublisher.publishEvent(EJBBasedJMSPublisher.java:137)
at com.ibm.lconn.events.internal.impl.Events.invokeAsync(Events.java:475)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleAuditHelper.publishEvent(UserLifeCycleAuditHelper.java:124)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleCmdsImpl.publishAuditEvent(UserLifeCycleCmdsImpl.java:2288)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleCmdsImpl.access$500(UserLifeCycleCmdsImpl.java:53)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleCmdsImpl$IUserCallbackImpl.onPerson(UserLifeCycleCmdsImpl.java:1456)
at com.ibm.lconn.share.platform.userlifecycle.UserLifeCycleSPIImpl.updateAllUsers(UserLifeCycleSPIImpl.java:381)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleCmdsImpl.syncAllMembersByExtIdAsync(UserLifeCycleCmdsImpl.java:713)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleCmdsImpl.access$100(UserLifeCycleCmdsImpl.java:53)
at com.ibm.lconn.core.web.util.admin.UserLifeCycleCmdsImpl$MemberIdUpdateAllJob.run(UserLifeCycleCmdsImpl.java:608)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:784)
Caused by: javax.jms.JMSException: CWSIA0241E: An exception was received during the call to the method JmsManagedConnectionFactoryImpl.createConnection: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus ConnectionsBus running. Additional failure information: CWSIT0103E: No messaging engine was found that matched the following parameters: bus=ConnectionsBus, targetGroup=null, targetType=BusMember, targetSignificance=Preferred, transportChain=InboundSecureMessaging, proximity=Bus..
at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:198)
at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:138)
at com.ibm.lconn.events.internal.publish.impl.JMSPublisher.publishEvent(JMSPublisher.java:193)
... 16 more
Caused by: com.ibm.websphere.sib.exception.SIResourceException: CWSIT0088E: There are currently no messaging engines in bus ConnectionsBus running. Additional failure information: CWSIT0103E: No messaging engine was found that matched the following parameters: bus=ConnectionsBus, targetGroup=null, targetType=BusMember, targetSignificance=Preferred, transportChain=InboundSecureMessaging, proximity=Bus.
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.localBootstrap(TrmSICoreConnectionFactoryImpl.java:363)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:304)
at com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.createConnection(TrmSICoreConnectionFactoryImpl.java:222)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaManagedConnectionFactoryImpl.createManagedConnection(JmsJcaManagedConnectionFactoryImpl.java:473)
at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:2107)
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1785)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3272)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2610)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1500)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1012)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaConnectionFactoryImpl.createConnection(JmsJcaConnectionFactoryImpl.java:289)
at com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.createConnection(JmsManagedConnectionFactoryImpl.java:165)
... 18 more
Network File System (NFS) Recommendations for WebSphere Application Server to AVOID DATA LOSS
Source: http://www-01.ibm.com/support/docview.wss?uid=swg21645066
Abstract
This article details some known issues with the
Network File System (NFS) implementations that affect the reliability of
data stored on such file systems by the WebSphere Application Server
transaction, activity and compenstation services and the Service
Integration Bus filestore. When reliability is compromised file
corruption is possible. In most cases this results in DATA LOSS. It also
makes recommendations for configuring both NFS and WebSphere
Application Server based on known configuration problems and customer
experiences.
Content
NFS is a convenient Network Attached Storage (NAS) technology.
NFS v4 is commonly used as a reliable storage for the following components of WebSphere Application Server:
- The recovery log service - the recovery log service is used by the transaction, activity and compensation services for data logging purposes. In the case of the transaction service, NFS is used to enable highly available transaction logs for peer recovery.
- The default messaging provider for JMS (Service Integration Bus) - to allow messaging engines configured to use a filestore to failover between servers
As with any file system, NFS must provide a
behaviour and reliability that is consistent with the design of both the
transaction service and filestore. Two principles that both of these
server components rely on are:
- Lease based locking - both components may lock files and rely on
the lock to exclude other processes attempting to lock the same file on
either the same machine or some other machine. Should the process that
owns the lock hang or end for whatever reason, the lock should be
released, allowing other processes to acquire the lock. Once another
process has acquired the lock, if the original process recovers from a
hang, it should no longer be able to write to the file; it has lost the
lock. A basic test for lease based locking is available here:
http://www-01.ibm.com/support/docview.wss?uid=swg24010222
- File writes may be forced, which means that all writes to the file up to that point must be safely stored before the force operation completes. Safely stored means that in the event of a crash in any part of the system, when those files are next read by any process, the data written before the last successful force is accurate and complete. Testing this is difficult.
Required Linux NFS Patch to AVOID DATA LOSS
SuSe Linux Bug Number 828236.
Redhat Linux Bug Number 963785.
Vendor support should be contacted to acquire an NFS patch or kernel for these respective bug numbers.
These bugs address an issue in NFS where an NFS
client (in the WebSphere Application Server case either the transaction
service or a messaging engine) may lose a file lock, but without
reacquiring the lock is able to continue writing the file. There is no
notification that the lock is lost.
For the transaction service this means
that when peer recovery is configured, a peer server might attempt to
resolve the logs of a server perceived to be failed. If both servers
write the logs at the same time due to this bug, the transaction logs
will become corrupted and transactions may rollback prematurely.
For a messaging engine this means two
instances of the same messaging engine could run in two different
servers concurrently. This will result in filestore corruption and
consequently data loss.
NFS on Linux implements advisory locking which means that files can be read and written without actually acquiring a lock.
Real world scenarios that have resulted in data
loss because of this problem involve a network partition and a virtual
image hang. Both conditions resulted in the loss of file lock but the
process owning the lock is still able to write the file without the lock
and is not notified in any way.
Other Recommended NFS Patches by Implementation
The following table lists known issues by implementation:
| Implementation | Details |
| Solaris 10 (SPARC) | Patch 147440-13 is recommended. |
| Solaris 10 (x86-64) | Patch 147441-13 is recommended. |
| SLES V10 Update 3 | Suspected problem introduced by Update 3, resolved in kernel level 2.6.16.60-0.60.1. |
| AIX 5.3 TL6 to TL9 | APAR IZ29559 is recommended. APAR IV35811 is recommended. APAR IV46850 is recommended. |
Required NFS Configuration
Mount Options
The following table lists mount options that are required. They are not exclusive but other options should not negate these.
| Option | Description |
| -t nfs4 | Forces NFS v4 to prevent any possibility of falling back to NFS v3. |
| -o hard,intr | Soft mounts can lead to file corruption so hard mounts are required. intr allows a user to interrupt from the keyboard. |
For example:
mount -t nfs4 -o hard,intr server:/logs
mount -t nfs4 -o hard,intr server:/logs
Export Options
The following table lists export options for Linux only. They are not exclusive but other options should not negate these.
| Options | Description |
| rw | Read/Write. |
| sync | Ensures the default behaviour which is reply AFTER changes committed to stable storage. This avoids the possibility that a failure can lead to loss of data because it has not been persisted. |
| no_wdelay | Do not attempt to delay in order to batch writes. This is a potential performance optimization, however, the transaction service and filestore wait for the write to complete and are the only writers, so no batching will ever be possible. |
Required Java Configuration for Recovery Log Service on Windows
On IBM Windows JVM versions 1.5 and 1.6, a JVM
custom property must be set on all servers accessing recovery log files,
except when memory mapping is disabled (see When to Disable Recovery
Log Service Memory Mapping on Windows below). The name of the custom
property is:
com.ibm.mappedByteBufferForce (true/false)
The default is false. This JVM custom property must be set to 'true'.
com.ibm.mappedByteBufferForce (true/false)
The default is false. This JVM custom property must be set to 'true'.
When to Disable Recovery Log Service Memory Mapping on z/OS
The transaction, activity and compensation
services can use memory mapping to read and write the transaction,
activity and compensation logs. This is the default in all current
versions of WebSphere Application Server, except on Windows when peer
recovery is enabled for the transaction service and except on zOS
version 8.5 and above.
The reason that the default changed on zOS at
version 8.5 is because memory mapped files cannot be expanded on zOS.
This problem is documented in APAR PM58494.
On zOS, for all versions of WebSphere Application
Server it is recommended that memory mapped files is disabled (note
version 8.5 and above memory mapped files is disabled by default).
When to Disable Recovery Log Service Memory Mapping on Windows
WebSphere Application Server automatically
disables memory mapping if the platform is Windows AND peer recovery is
enabled. CIFS is most likely to used when peer recovery is enabled but
can of course be used even if peer recovery is not enabled. If the
transaction, activity or compensation logs reside on CIFS it is required
to disable memory mapped files. An IOException results if memory
mapping is not disabled.
Therefore, if the transaction, activity or
compensation logs reside on CIFS, memory mapping must be disabled if
peer recovery is enabled (which automatically disables memory mapping).
To disable memory mapped files set the JVM custom property:
com.ibm.ws.recoverylog.spi.NoMemoryMappedFiles (true/false)
to 'true' on any server that accesses recovery log files.
If peer recovery is enabled but the recovery logs
do not reside on CIFS, memory mapping be explicitly be enabled by
setting the property to 'false'.
How To Set JVM Custom Properties
The following Infocenter article describes how to set JVM custom properties for application servers:
No comments:
Post a Comment