Wednesday, April 24, 2013

DBLink odd behavior in Oracle database

Hello all,
recently I came across an odd behavior in the oracle database in the production environment. I am not sure if this is the correct behavior.
I managed to reproduce the behavior in test environment.

my scenario:
1)I have a local database, a remote database and a DBLink in the local database pointing to the remote one.
2) read a value from the remote database.
3) insert or update values in a table in the local database.
4)disconnect the local database from the remote database(either a-unplug the network cable from the remote database b- unplug the network cable from the local database).
5) commit the changes on the local database.

for whatever strange reason it's the database decided to start a 2PC commit.

what I don't get is that I only read from the remote database, so why is the database trying to start a 2PC commit.

I could not find a viable solution. so had to implement the following workaround:
1) change the read operation into calling a procedure.
2) have the procedure call the remote database.
3) defined the procedure as having an autonomous transaction.
4) after reading the value , rollback the autonomous transaction.

it seems since the read from the remote database was not in the main transaction, when I try to commit it , it is not changed into a 2PC commit.

Weblogic monitoring

For A while now, We were getting some performance issues with our ADF application on weblogic server. So when we tried to  trace the issue we had to follow several steps.

  •  Thread dump analysis:
to get a thread dump you do :
  1. log in to the weblogic admin console.
  2. in the domain structure pane on the left side of the screen click on the domain_name link .
  3. in the settings for the domain_name pane click on the monitoring tab then click on the health sub-tab.
  4. click on the server in the domain that you want to take the thread dump to.

     5.  in the opened window click on the monitoring tab the click on the threads subtab
     6.  press Dump thread stack button .

    7. you will get the following:

    8. copy the dump to text editor and save it.
    9. refresh the page to get a new dump.
   10. repeat steps(8,9) several times to get a number of thread dumps.

ok, now that you have a number of thread dumps, you can start to analyze them .

you can simply read them in a text editor of your choice. though I don't recommend it fearing for your mental health. 
What I do recommend though is to use one of the thread analysis tools available on the web.
I tried :

personally , I prefer threadLogic since it's more user friendly and makes it easier to trace the dump files.

after tracing several dumps we noticed that there were a lot of threads waiting on a Database read,
after going through the database logs and some other steps we managed to solve the problem.

  • Monitoring the Weblogic resources:
there is a good extension in weblogic that could be helpful in monitoring some of the resources of the weblogic server.
you can access it through:   <weblogic_machine_host_name>:<admin_port>/console/dashboard

you can monitor JDBC connections,Thread count, Heap size and it will be presented graphically.

I know that theses values can be fetched from several pages in the weblogic console, but the through the dashboard you can have several values in the same chart in graphical presentation.