Having said that:
a) you need to check whether the TX enqueues are mode 4 or mode 6, it could make a big difference to diagnosis. Since you are presumably licensed for the performance pack you could query ASH (v$active_session_history, or the dba_ equivalent) to check in more detail. If the waits are numerous and long-term you could also start a trace on a session for a few minutes to see exactly what it is doing.

b) TX enqueues are generally not related to “hot blocks”, they are about data competition, although there can be some odd index leaf block split effects. If you have any evidence that the problem relates to index blocks being delayed as they split and move about a key RAC-related detail is to check for the sequence CACHE size that you are using for your monotonically increasing value. If the problem relates to RAC and high-value leaf blocks then a large sequence cache size (in the order of 1,000 to 10,000) may be sufficient to resolve the problem.

c) The phrase “Over 90% of AWR report on both nodes show Enq Tx.” is not very clear. If it means that 90% of your DB Time is spent in Enq TX and it’s on this PK index then why do you think there is “a lot of contention on the table as well”. If you do raise your problem on the OTN database forum, make sure you give some quantitive information, rather than vague indicators – e.g. Load Profile and Top Timed Events from both nodes. You’ll also end up being asked to supply some “Segments by …” sections as well, probably.

d) The single table hash cluster is very limited in it applicability. It is not appropriate for continuously growing data, only for data of a fixed (or strictly bounded) size. Since you’re talking about “line items” with a sequence based key I suspect you have a data set that is, nominally, growing without a bound.

I have a current issue with large waits Enq Tx-row lock contention in a RAC system. The primary wait “appears” to be on a table of line items, about 85 columns of non-pk with a two column pk, both number. The application code seems to do an insert of the two columns of the primary key. Then it does another (or two or three based on dba_tab_modifications) updates of all 85 non-pk columns. Over 90% of AWR report on both nodes show Enq Tx. The line item table also has eight indexes, and 20 columns in the table go to fk’s in small(er) code tables.

My initial thought would be to reduce hot blocks. The primary key is, by the way, a monotonically increasing id, plus a few line numbers, usually less than 10. I am assuming that the primary key is introducing the hot blocks, and using either a id with instance_id/sessionid/sequence_no, or a hash clustered primary key would reduce some of the contention.

In this case, I believe there is a lot of contention on the table itself. This leads me to look at single table hash clustered design as a potential resolution.
My problem is in properly designing the cluster hashkeys and size parameters if needed. I am not sure if I size based on avg. row size / block size, or by total number estimated rows ever stored in the table.

The multiple total-row update pattern is alarming to me, but I am stuck with it for the moment.
My current SQL is :

create cluster quote_line_cluster 
( quote_id         number,
  quote_line_no    number
)
single table
hashkeys 1000;
CREATE TABLE "QUOTE_LINE_CL"
  (
    "QUOTE_ID"                       NUMBER NOT NULL ENABLE,
    "QUOTE_LINE_NO"                  NUMBER NOT NULL ENABLE,....
/*snipped column definitions--about 85*/
)
CLUSTER quote_line_cluster ( quote_id,quote_line_no );

And for the sequence:

create sequence rac_seq start with 1 cache 10000
/
create or replace package rac_seq_pkg
is 
function get_seq return number;
end;
/
create or replace package body rac_seq_pkg
is
function get_seq return number
is
l_id number;
begin
select to_number(TO_CHAR(USERENV('INSTANCE'))||TO_CHAR(USERENV('SESSIONID'))||to_char(rac_seq.nextval)) into l_id from dual;
return l_id;
end get_seq;
end;
/

I think the sequence will help resolve the initial primary key insert hot spots. But will a hash cluster potentially resolve the table hot spots, or am I off track in pursuing this approach?

Any advice would be appreciated.

Thanks,

Dana

I didn’t phrase my comment very well – the “waiter that couldn’t get back” was probably waiting in the common queue for a server to become available, not “on a server waiting and then quitting” (although the client did have some ‘select for update … skipped locked’ statements – which have some very funny side effects on enqueue behaviour.

If I recall corrrectly the wait times were in the order of a very small number of seconds. Trying to solve the exact cause of the problem is a non-issue since it went away when the system was configured properly – and since the suggested change had the predicted (as in “and this might be the cause of your locking problem”) result I never got past the “hand-waving” explanation of why it was an appropriate solution and on to look at the exact mechanism.

Oracle releases a shared server process back to other sessions only when the database call has finished. Even if a session is hung waiting for a lock (or sleeping due dbms_lock.sleep, waiting for IO, etc), the same server process will be exclusively associated with a single session.

This rules out issues like “not being able to get a process to complete the database call” as a process is always associated with the session during a call.

If it had been a real hang (circular wait chain) then deadlock detection would have rolled someone’s call back.. but if you didn’t see those it’s probably just that a lot of sessions ended up waiting (indirectly) for the same blocker and this issue may have been amplified with other symptoms like CPU starvation due many shared server process startups.

And that leads to the question, how long stalls did you see? Was it seconds instead of milliseconds or minutes instead of seconds or more?

Or as you said it could have been some loadrunner load profile timing issue..

There’s no “select for update”. The only statements in the library cache are:

insert into tabX
update tabX where pk =
delete from tabX where pk =

I think the underyling problem is one of confusion in the connection pooling mechanism – and the apparent resolution from change the number of shared servers is simply a timing benefit.

One point I haven’t mentioned, by the way, is that the thing is a benchmark type of thing being controlled by LoadRunner. It’s quite possible that there’s a flaw in the driving data, or in the way LoadRunner has been configured, that allows “business transactions” to interfere with each other.

Your enqueue wait is for a TX lock in mode 6 – which should be a simple data collision on a heap table.

Your users are queueuing because they want to update (delete) the same rows (or row) in the table.

It looks like you need to find a way to stop doing this, or find a way of commiting the change as quickly as possible.

DELETE FROM APP_USER_FUNCTION WHERE DOMAIN_ID = 1000000 AND USER_ID = 9999999

then in this case, I think, increasing the value of the initrans would not help you.
When a session have to delete a row, it place a lock on it and the other(s) session(s) must wait until a commit or rollback.

Initrans can help you if you have a lot of insert.

To tune this…mumble…I would place a commit after the delete IF AND ONLY IF this don’t change the flow of the program.

I found a bit strange that 550 concurrent session issue the same delete.

I am facing a similar problem at a client site .
There are around 500 Sessions waiting for enq: TX – row lock contention .

The SQL being fired is DELETE FROM APP_USER_FUNCTION WHERE DOMAIN_ID = :B2 AND USER_ID = :B1 from 550 different sessions .

Following were my observations

SID EVENT P1 P2 SECONDS_IN_WAIT STATE
—- —————————— ———- ———- ————— ———-
1016 enq: TX – row lock contention 1415053318 1835049 0 WAITING
1017 1415053318 1835049 0 WAITING
.

The table in question was APP_USER_FUNCTION .
This table doesnt have a primary key or a foreign key on it.Also index is present on domain_id and user_id column (Con-catinated index) .

Can you tell me by increasing the inittrans value of this table from 1 to say 10 helpme in any ways !!

Or any other way to tune this type of contention !!

Regards

Amit bhube