WS-AT Transaction Affinity Enablement on Distributed Transactions System

Published: May 13, 2013 by Zhenghui Lee

• Categories:
• webservices 1

• Tags:
• 2PC 1
• XA 1
• jax-rpc 1
• jax-ws 1
• transaction 1

Table of Contents

• Table of Contents
• Introduction
• In practice
  • Prerequisite:
  • Installation
  • Develop a WS-AT enabled application
  • Deploy the sample application
  • Launch the application via the welcome jsp page
  • WebSphere Proxy Server Configuration
  • Create/configure a proxy cluster
  • Rerun the sample

Move my earlier post which created on 13 May 2013) to this new techblog

Last week, One of our customer ask for onsite support for a production issue for a webservice invocation with WS-AT enablement. As the owner of SCA for WebSphere Enterprise Service Bus and WebSphere Process Server, I was onsite support on this cirtical issue. That finally result is proved being a usage error, but I am thinking it is still worth writing an article here in case anyone else come cross the same issue, and this post will help on him/her.

The objective of this article is to provide you a detailed steps of how to configure the WebSphere proxy server to enable transaction affinity in a WS-AT enabled application within a distributed system.

Introduction

WS-AT is a standard and technology that provides transactional support for web services, it allows the distributed web service application, and the resources they use, to take part in distributed global transactions. In WebSphere Application Server, we support the transaction that applies to relational database, message queue manager and other resource managers which support XA standard. Application server use the two-phase commits to cooperate the participated resource manangers, actually, WebSphere is typically the coordinator.

If you are developing a enterprise application in this kind of distributed system, it may be worth taking account of the transaction affinity in some special cases, e.g: Imaging you have a scenario like below:

Deadlock scenario

the application deployed on different servers but need to access/update same data stored in DB2 database within a global transaction. Normally, this will run into the deadlock situation if we are running it on WAS version earlier/equal than/to v7 or not turn on the feature of DB2 lock sharing between trasaction branches on the resource reference for the datasource we are using in application, because the scope of database lock is physicial JDBC connection. In above case, the EJB-A update one row in database, and then call to the method in EJB-B which attemp to update the same row within same global transaction, then the deadlock occurs. Of course, The deadlock can be avoided if we deploy both EJBs to same server, since both EJBs can share the same DB2 connection. But in a real-life application deployment environment, we can't always assume this can be done. Actually, the WebSphere Proxy Server provide transaction affinity function, which is the target we are looking for to resolve the issue mentioned above.

In practice

Prerequisite:

• WebSphere Process Server v7 as well as the latest fixpack
• DB2 Instance
• IHS server for WebSphere Application Server v7.0.0.x

Installation

• Install DB2
• Install a single cluster environment
• Configure IHS server and make sure it sits in front of the app cluster

Develop a WS-AT enabled application

Develop a WS-AT enabled application which use to verify the transaction affinity result.

• I'm using SCA JAX-WS binding as the sample just because I'm familiar with it, you can create the application in RAD via JAX-WS standard way. This is not key. To simplify the application deployment steps, I developed both web service client/provider in one SCA Module, below is the assembley view for the SCA Module:

The assembly view for the sample application

I created two import bindings in this case, just for easily reproduce the deadlock situation with hard-code endpoint address which point to the different cluster memebers separately.

• In the client code, we make sure the invocations grouped by global transaction, the code snippet is below:

    private static String doInvokeWithTX(ActionWithTX action) {
  		UserTransaction userTx = null;
  		try {
  			userTx = getUserTx();
  			userTx.begin();
  			System.out.println("[Client] Begin global transaction -  " + TXUtil.dumpTxId());
    			
  			String result = action.action();
    			
  			System.out.println("[Client] End global transaction -  " + TXUtil.dumpTxId());
  			userTx.commit();
    			
  			return result;
  		} catch (Exception error) {
  			error.printStackTrace();
  			try {
  				userTx.rollback();
  			} catch (Exception e) {
  				e.printStackTrace();
  			} 
  		}
    		
  		 return null;
  	}
  

• The application entry is a JSP welcome page, we can input invocation group count there, for each invocation group, it will take to the web service endpoints twice within one global transaction, if we do the web service call through the IHS server, the reuqest will be randomly dispatched to the cluster memeber, then we will see the deadlock occurs.

• Modify the endpoint address in the two improt bindings file to comply with the real onfiguration info for the IHS http port

• The whole PI and ear files can be download there:

  • PI: link

  • ear: link

Deploy the sample application

• Create an application policy set, and make sure it contain the WS-Transaction policy

Application policySet creation for WS-Transaction

• Deploy application to app cluster and configure the policy set on both import binding and export binding to enable WS-AT

Attach policySet to web service provider

Attach policySet to web service client

Launch the application via the welcome jsp page

Launch the welcome jsp page via link like: link:http://localhost:80/ProcessOperatorWeb/index.jsp[index.jsp] and give a enough large number for the invocation group, click query button, then obsever the system out log file,we will see the timeout exception occurs.

WebSphere Proxy Server Configuration

As I mentioned early, the IHS doesn't proivde the transaction affinity functionality. To allow the transaction affinity for WS-AT in WebSphere application server, that means, to delegate all(subsequent) requests belongs to one transaction to the same target server, it is mandatory to use a Websphere Proxy Server(cluster) in front of the application server cluster. The WebSphere Proxy Server interprets the information sent in the SOAP headers (especially wscoor:Identifier and wsa:ReferenceParameters) using a special WebSphere Application Server WS-Tx/AT workload management plugin. Therefore, it knows which server the request needs to be directed to. After this change, the topology is like below:

Typical topology of HTTP server in conjunction with a Proxy Server for IBM WebSphere Application Server

Create/configure a proxy cluster

• Stop all of clusters(servers) if they are running.

• Start a blowser for dministrative console, the link maybe something like: link:http://localhost:9060/ibm/console[admin console]

• Create a proxy cluster

Proxy Cluster creation - 1

• Input a cluster name

Proxy Cluster creation - 2

• Input the first cluster memeber name and select the proxy server template

Proxy Cluster creation - 3

• Add additional cluster member on appropriate node

Proxy Cluster creation - 4

• Save the creation

Proxy Cluster creation - 5

• Go to the http proxy server setting and click the link of proxy setting

Proxy Cluster creation - 6

• Make sure the check box is checked for enable web service support

Proxy Cluster creation - 7

• Generate the plugin configuration via specify a appropriate scope value

Proxy Cluster creation - 8

In order to configure the IHS to route the requests to proxy server via HTTP plugin, we need to configure the proxy server to generate the plugin configuration file, we can find the genrated plugin configure file in /etc/ directory. It is optional to specify a plugin config change script, we can leave it emtpy if we do the plugin config change manually.

• Select appropriate load balance algorithm for proxy cluster

Select load balance algorithm

• Go to the details of each proxy cluster memeber

Proxy Cluster creation - 9

• Check out the ports used by the proxy server and make sure they are added to appropriate virutal host definition. e.g:

The port info for the 1st cluster memeber

The apropriate virtual host definition should cover the host alias

Make sure do the check for each cluster member

• (optional) Go to the application cluster memeber(note, not proxy cluster memeber) to configure the transaction service if needs

Select corresponding app cluster memeber

Go to transaction service configuration

Fill in the WS-Transaction http(s) URL prefix with the value of the proxy server ports

In the distributed transaction environment, many communications will be estabilshed in undercover for the coordination purpose between the transaction coordinator and participators, in particular to WS-AT, these communications are transfered via transport protocol – http(s). So if the transaction manangers are running in different network domains,can not be reached directly, we need to configure this transaction property to make sure the coordination communication go through via the proxy server. The value we specify here will be reflected to the WS-Transaction headers for the system endpoint of TX coordinator or participator. If the intermediary node is not a Proxy Server, the prefix must be unique for each server.If you are using a Proxy Server, prefixes can be the same for each server in a cluster, because the Proxy Server determines dynamically which server to forward the request to.

The WS-AT with intermediary nodes

• Start proxy cluster and we can see plugin.xml generated in /etc directory

Plugin xml generation for proxy cluster

• Go to plugin config directory in IHS insttallation directory, modify the plugin-cfg.xml in IHS to comply with the proxy cluster info, and then IHS can route requests to proxy servers accordingly. Below is the sample change in my test environment

Add the host alias info to the virtual host

Add the new cluster section for the proxy cluster based on the information we get from the generated plugin-cfg file for proxy server

Add a new UriGroup for transaction affinity needed application, if the application has been added in other UriGroup, remove them from that group firstly image::./proxyClusterCfg-18.jpg[height=200,caption="",link="./proxyClusterCfg-18.jpg]

Add a new route rule

• Below is a sample plugin-cfg.xml file which is merged with proxy cluster configuration:

  plugin-cfg.xml: link

• Retart all of server processes, including Dmgr, Node agent, HTTP server and clusters

Rerun the sample

After restart all of server process, launch the welcome page and click the query button with a large enough number of invocation counts, then we can see all of inovcation can be processed without any exception and for invocation belongs to global transaction, they always be processed in the same app cluster member

For the different invocation group(run in different global transaction, they will be disptach to different cluster memeber for load balance.

Trace log for cluster memebers

• AppClusterMember-0

  |[5/13/13 9:16:37:791 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@ba80ba8#tid=3579 for TXGroup-0, batch-id 0
  [5/13/13 9:16:37:791 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-0')
  [5/13/13 9:16:37:791 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-0')
  [5/13/13 9:16:37:791 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-0')
  [5/13/13 9:16:37:791 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-0')
  [5/13/13 9:16:37:869 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@ba80ba8#tid=3579 for TXGroup-0, batch-id 1
  [5/13/13 9:16:37:869 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-0')
  [5/13/13 9:16:37:869 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-0')
  [5/13/13 9:16:37:869 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-0')
  [5/13/13 9:16:37:869 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-0')
  [5/13/13 9:16:37:901 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@ba80ba8#tid=3579 for TXGroup-0, batch-id 2
  [5/13/13 9:16:37:901 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-0')
  [5/13/13 9:16:37:901 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-0')
  [5/13/13 9:16:37:901 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-0')
  [5/13/13 9:16:37:901 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-0')
  [5/13/13 9:16:37:979 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@40dc40dc#tid=3580 for TXGroup-1, batch-id 0
  [5/13/13 9:16:37:979 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-1')
  [5/13/13 9:16:37:979 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-1')
  [5/13/13 9:16:37:979 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-1')
  [5/13/13 9:16:37:979 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-1')
  [5/13/13 9:16:38:026 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@40dc40dc#tid=3580 for TXGroup-1, batch-id 1
  [5/13/13 9:16:38:026 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-1')
  [5/13/13 9:16:38:026 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-1')
  [5/13/13 9:16:38:026 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-1')
  [5/13/13 9:16:38:026 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-1')
  [5/13/13 9:16:38:119 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@40dc40dc#tid=3580 for TXGroup-1, batch-id 2
  [5/13/13 9:16:38:119 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-1')
  [5/13/13 9:16:38:119 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-1')
  [5/13/13 9:16:38:119 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-1')
  [5/13/13 9:16:38:119 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-1')
  [5/13/13 9:16:38:182 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@26fb26fb#tid=3582 for TXGroup-2, batch-id 0
  [5/13/13 9:16:38:182 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-2')
  [5/13/13 9:16:38:182 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-2')
  [5/13/13 9:16:38:182 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-2')
  [5/13/13 9:16:38:182 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-2')
  [5/13/13 9:16:38:229 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@26fb26fb#tid=3582 for TXGroup-2, batch-id 1
  [5/13/13 9:16:38:229 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-2')
  [5/13/13 9:16:38:229 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-2')
  [5/13/13 9:16:38:229 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-2')
  [5/13/13 9:16:38:229 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-2')
  [5/13/13 9:16:38:276 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@26fb26fb#tid=3582 for TXGroup-2, batch-id 2
  [5/13/13 9:16:38:276 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-2')
  [5/13/13 9:16:38:276 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-2')
  [5/13/13 9:16:38:276 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-2')
  [5/13/13 9:16:38:276 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-2')
  [5/13/13 9:16:38:338 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@436b436b#tid=3583 for TXGroup-3, batch-id 0
  [5/13/13 9:16:38:338 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-3')
  [5/13/13 9:16:38:369 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-3')
  [5/13/13 9:16:38:369 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-3')
  [5/13/13 9:16:38:385 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-3')
  [5/13/13 9:16:38:416 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@436b436b#tid=3583 for TXGroup-3, batch-id 1
  [5/13/13 9:16:38:416 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-3')
  [5/13/13 9:16:38:416 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-3')
  [5/13/13 9:16:38:416 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-3')
  [5/13/13 9:16:38:416 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-3')
  [5/13/13 9:16:38:479 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@436b436b#tid=3583 for TXGroup-3, batch-id 2
  [5/13/13 9:16:38:479 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-3')
  [5/13/13 9:16:38:479 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-3')
  [5/13/13 9:16:38:479 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-3')
  [5/13/13 9:16:38:479 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-3')
  [5/13/13 9:16:38:557 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@164b164b#tid=3584 for TXGroup-4, batch-id 0
  [5/13/13 9:16:38:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-4')
  [5/13/13 9:16:38:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-4')
  [5/13/13 9:16:38:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-4')
  [5/13/13 9:16:38:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-4')
  [5/13/13 9:16:38:635 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@164b164b#tid=3584 for TXGroup-4, batch-id 1
  [5/13/13 9:16:38:635 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-4')
  [5/13/13 9:16:38:635 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-4')
  [5/13/13 9:16:38:635 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-4')
  [5/13/13 9:16:38:635 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-4')
  [5/13/13 9:16:38:713 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@164b164b#tid=3584 for TXGroup-4, batch-id 2
  [5/13/13 9:16:38:713 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-4')
  [5/13/13 9:16:38:713 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-4')
  [5/13/13 9:16:38:713 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-4')
  [5/13/13 9:16:38:713 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-4')
  [5/13/13 9:16:39:432 CST] 00000045 WSChannelFram A   CHFW0019I: The Transport Channel Service has started chain chain_3.
  [5/13/13 9:16:40:495 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@645f645f#tid=3588 for TXGroup-8, batch-id 0
  [5/13/13 9:16:40:495 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-8')
  [5/13/13 9:16:40:511 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-8')
  [5/13/13 9:16:40:511 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-8')
  [5/13/13 9:16:40:527 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-8')
  [5/13/13 9:16:40:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@645f645f#tid=3588 for TXGroup-8, batch-id 1
  [5/13/13 9:16:40:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-8')
  [5/13/13 9:16:40:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-8')
  [5/13/13 9:16:40:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-8')
  [5/13/13 9:16:40:573 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-8')
  [5/13/13 9:16:40:620 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@645f645f#tid=3588 for TXGroup-8, batch-id 2
  [5/13/13 9:16:40:620 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-8')
  [5/13/13 9:16:40:620 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-8')
  [5/13/13 9:16:40:620 CST] 00000046 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-8')
  [5/13/13 9:16:40:620 CST] 00000046 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-8') 
  

• AppClusterMember-1

  |[5/13/13 9:16:39:167 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@22462246#tid=3512 for TXGroup-5, batch-id 0
  [5/13/13 9:16:39:167 CST] 00000049 SystemOut     O Create DataSource Connection
  [5/13/13 9:16:39:167 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-5')
  [5/13/13 9:16:39:182 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-5')
  [5/13/13 9:16:39:182 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-5')
  [5/13/13 9:16:39:182 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-5')
  [5/13/13 9:16:39:260 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@22462246#tid=3512 for TXGroup-5, batch-id 1
  [5/13/13 9:16:39:260 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-5')
  [5/13/13 9:16:39:260 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-5')
  [5/13/13 9:16:39:260 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-5')
  [5/13/13 9:16:39:260 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-5')
  [5/13/13 9:16:39:307 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@22462246#tid=3512 for TXGroup-5, batch-id 2
  [5/13/13 9:16:39:307 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-5')
  [5/13/13 9:16:39:307 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-5')
  [5/13/13 9:16:39:307 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-5')
  [5/13/13 9:16:39:307 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-5')
  [5/13/13 9:16:39:683 CST] 00000049 WSChannelFram A   CHFW0019I: The Transport Channel Service has started chain chain_2.
  [5/13/13 9:16:39:886 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@a130a13#tid=3514 for TXGroup-6, batch-id 0
  [5/13/13 9:16:39:886 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-6')
  [5/13/13 9:16:39:901 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-6')
  [5/13/13 9:16:39:901 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-6')
  [5/13/13 9:16:39:901 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-6')
  [5/13/13 9:16:39:995 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@a130a13#tid=3514 for TXGroup-6, batch-id 1
  [5/13/13 9:16:39:995 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-6')
  [5/13/13 9:16:40:011 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-6')
  [5/13/13 9:16:40:011 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-6')
  [5/13/13 9:16:40:011 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-6')
  [5/13/13 9:16:40:058 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@a130a13#tid=3514 for TXGroup-6, batch-id 2
  [5/13/13 9:16:40:058 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-6')
  [5/13/13 9:16:40:058 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-6')
  [5/13/13 9:16:40:058 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-6')
  [5/13/13 9:16:40:089 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-6')
  [5/13/13 9:16:40:261 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@563f563f#tid=3515 for TXGroup-7, batch-id 0
  [5/13/13 9:16:40:261 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-7')
  [5/13/13 9:16:40:276 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-7')
  [5/13/13 9:16:40:276 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-7')
  [5/13/13 9:16:40:292 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-7')
  [5/13/13 9:16:40:339 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@563f563f#tid=3515 for TXGroup-7, batch-id 1
  [5/13/13 9:16:40:339 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-7')
  [5/13/13 9:16:40:339 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-7')
  [5/13/13 9:16:40:339 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-7')
  [5/13/13 9:16:40:339 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-7')
  [5/13/13 9:16:40:386 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@563f563f#tid=3515 for TXGroup-7, batch-id 2
  [5/13/13 9:16:40:386 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-7')
  [5/13/13 9:16:40:386 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-7')
  [5/13/13 9:16:40:386 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-7')
  [5/13/13 9:16:40:386 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-7')
  [5/13/13 9:16:40:714 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@13d013d#tid=3516 for TXGroup-9, batch-id 0
  [5/13/13 9:16:40:714 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-9')
  [5/13/13 9:16:40:714 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-9')
  [5/13/13 9:16:40:714 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-9')
  [5/13/13 9:16:40:714 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-9')
  [5/13/13 9:16:40:761 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@13d013d#tid=3516 for TXGroup-9, batch-id 1
  [5/13/13 9:16:40:761 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-9')
  [5/13/13 9:16:40:761 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-9')
  [5/13/13 9:16:40:761 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-9')
  [5/13/13 9:16:40:761 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-9')
  [5/13/13 9:16:40:808 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Running in global transaction -  com.ibm.ws.tx.jta.TransactionImpl@13d013d#tid=3516 for TXGroup-9, batch-id 2
  [5/13/13 9:16:40:808 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-9')
  [5/13/13 9:16:40:808 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - INSERT INTO WPRBE00.TEST_T_FOR_HUAWEI (ID, NAME ) VALUES ('12345','TXGroup-9')
  [5/13/13 9:16:40:808 CST] 00000049 SystemOut     O [ProcessOperatorImpl] Begin to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-9')
  [5/13/13 9:16:40:808 CST] 00000049 SystemOut     O [ProcessOperatorImpl] End to execute sql - DELETE FROM WPRBE00.TEST_T_FOR_HUAWEI WHERE (ID='12345') AND (NAME='TXGroup-9')