WebLogic Server Performance Monitoring Best Practices



WebLogic Server Performance Monitoring Best Practices

Performance Monitoring:

The following are some of the adverse conditions within the Weblogic Server to have performance degradation.

Poorly Tuned Application:

Applications that are poorly tuned typically manifest themselves via high CPU and I/O utilization on the Weblogic Server. This is due to the large number of sort and read operations that must be performed by the Weblogic Server. Poorly tuned Weblogic Servers typically manifest themselves via high I/O operations. This is due to a lack of pre-allocated memory for caching and other critical internal processes. Increasing the amount of memory allocated to the Weblogic Server will typically solve most Weblogic Server tuning issues. This, however, can actually decrease performance if there is a lack of memory resources on the server because the Weblogic cache space will be swapped in and out of memory. Therefore, it is also critical to review the CPU, memory, and I/O utilization of the server as a whole. Tuning of server resources is not addressed in this document.

 The following are the key areas:

  1. Heap Memory Utilization
  2. CPU Utilization
  3. Garbage Collection
  4. Idle Thread Count
  5. Socket Connection
  6. Thread Dumps
  7. Users Session Locked in the Weblogic Server
  8. Multicasting

Lock Conflicts:

Poorly tuned applications tend to have components that lock objects or resources in the Weblogic Server for long periods. Although Weblogic Server functions correctly under these conditions, the users may experience poor response times or even application unavailability. It is important to detect the processes that are preventing others from retrieving information. Tuning the Weblogic Server executed by that process will improve the response time of the application. More importantly, the number of users are being blocked is an indication of the level of application unavailability.

Note: Frequent lock conflicts typically indicate that there is an application tuning issue.

Poor Response Time:

Applications occasionally have problems that cause the user to experience poor response times. It is important to measure the response time of the Weblogic Server so that poor response time periods can be detected. Once such a period is detected, further research and analysis can be performed to find the root cause of the response time issues.

Device Errors:

Occasional disk errors typically have no adverse effect on the Weblogic Server. Frequent disk device errors have the potential to cause Weblogic Server corruption and slow performance. Therefore, monitoring the disk errors encountered by Weblogic Server is important. Network errors cause frequent retransmits having an adverse effect on Weblogic Server performance. It is a good idea to monitor the traffic between your client applications and the server. An application designed and tuned for a slow network performs great on a fast network; however, the opposite is not true.

CPU Over load:

CPU time is the amount of time that the Weblogic Server spent processing the business methods that are deployed in the Weblogic Server. If this is the main timed event, tuning business methods that are deployed and/or increasing server CPU resources will provide the greatest performance improvement.

Weblogic Server Availability Monitoring Best Examples



Weblogic Server Availability Monitoring Best Examples

Availability Monitoring:

The following adverse conditions with the Weblogic environment can cause the Weblogic Servers to become unavailable. 
  1. Weblogic Server Instance Unavailable
  2. Weblogic Server Internal Errors
  3. Critical File Systems Full
  4. Capacity Limits Reached
  5. Weblogic Session-Replication Issues

 Weblogic Server Instance Unavailable:

Weblogic Server Instances run as processes on the system. Occasionally they may terminate abnormally causing Weblogic Server to become unavailable. Partial outages also exist when Weblogic Server is functioning normally for the currently connected users but will not allow any new connections. Therefore, Weblogic Server must be monitored from both the process and connection perspectives.
 If an instance is, down and is 24/ 7, alarm the Weblogic Administrators through email or SMS.

Weblogic Server Internal Errors:

Weblogic Server writes internal errors to the Standard Output Log file for the Instance. Although many of the errors are not critical, all errors should be addressed in a timely manner. Errors containing critical error codes must be detected and addressed immediately.

Critical File Systems Full:

One of the most critical Weblogic Server availability issues occurs when the Weblogic Server is unable to process transactions because the file system allocated for log space is full. Log space is used by transactions to store Weblogic Server updates and it is sized by the Weblogic Administrator according to the number and size of the transactions for that Serevr. Weblogic Server writes error and dump information to file systems designated by the Weblogic Administrator. The error and dump files systems are important because Weblogic Server will not be able to log critical internal errors when error and dump file systems become full. Internal errors usually lead to unavailability of the Weblogic Server.

Weblogic Session-Replication Issues:

Weblogic Server has the capability to replicate all Sessions from one Instance to another. Although Weblogic Server functions correctly when it is unable to replicate data, applications using the replicated data may become unavailable. To a user, application unavailability and database unavailability are the same. 

Tracking HTTP Session Replication Failures In Weblogic Cluster



Tracking HTTP Session Replication Failures In Weblogic Cluster

Today we will learn how to trace HTTP session replication failures in Weblogic cluster. Sometimes, HTTP session states are not replicated from primary server to secondary server.There are several reasons for session repllication failures like application using HTTP session does not function properly as designed and leads to loss of session data. We may see lot of HTTP failure error ans warning in server log files. There are several ways to debug the HTTP session failures and some of them are below ..
If Apache is used as load balancer, enable the debug perameter on in httpd.conf file and logs will generated in/tmp/wlproxy.log
Use the weblogic.Admin command-line utility to dynamically turn the debug options on and off as below :
java weblogic.Admin -url t3://localhost:7001 -username weblogic -password weblogic SET -type ServerDebug -property DebugCluster true
Alternatively, we can edit the config.xml and the perameter each server that you want to debug and set it to “true” to enable or “false” to disable. Then restart the Admin Server. Managed Servers will reconnect to the Admin Server and the debug flags will then dynamically take effect.
 For example:
At the end, with all the flags set, in your config.xml the ServerDebug tag would look like:
DebugClusterAnnouncements=”true” DebugFailOver=”true”
DebugReplication=”true” DebugReplicationDetails=”true” Name=”MS1″/>
Make sure the stdOutSeverity level of the server is INFO and StdoutDebugEnabled is set to “true”. The debug information will be logged into the server log as well as to the standard out.
Now we will learn how to understand a debug log file, in MS1 log file
<000000>
<000000>
10.18.24.98: [8088,8088, -1, -1,8088, -1, -1]: TD1: MS2>
It means that the primary server is MS1 and a secondary has been created on MS2. A message like the one shown below will be logged on MS2 to confirm that.
ExecuteThread: ’1′ for queue: ‘Replication’> <>
<000000>
####
’1′ for queue: ‘Replication’>
<> <000000>
If you check the JSESSIONID it looks like:
JSESSIONID=1E9Xwn7nLYfOsc1obgRZIwW5s72an7HPPvSD7iaWHMXzpHga5cQj
!-1587356342!-1587878545
JSESSIONID is the default name of the cookie, which can be changed to anything in weblogic.xml. The format of JSESSIONID is
SessionId!PrimaryServer JVM Hash!SecondaryServer JVMHash
Every time data is changed (either set/get or removed) in the session you’ll see the logging message.
On MS1:
<000000>
On MS2:
####
’1′ for queue: ‘Replication’> <> <000000>
If for any reason session replication fails, you will see the message below in the MS1 log:
<000000>
<000000>
10.18.24.98: [8088,8088, -1, -1,8088, -1, -1]:TD1:MS2>
And, the JSESSIONID would appear as
JSESSIONID=1E9Xwn7nLYfOsc1obgRZIwW5s72an7HPPvSD7iaWHMXzpHga5cQj!-1587356342!NONE
The secondary server hash would become NONE and  we can configrm session data is lost and HTTP session replication failure.

Weblogic Admin Tasks Using Command Line Utility



Weblogic Admin Tasks Using Command Line Utility

Today we will learn how to use weblogic command line utilities to perform serveral administration tasks. Weblogic server provides lot of command line utilities to create,configure,manage all the weblogic resources like servers,clusters,machines,connection pools,machines etc.
Advantages include, we can integrate this command line utilities in batch files,ant, shell scripts for execution. For example if we need to replicate the same production/dev environment in several machines, this scritps are very useful and just we need to change some variables according to environment.
Now we will use “weblogic.Admin” command line utility (user friendly utility in weblogic). First we will set the environment by executing ”setDomainEnv.cmd” .
First we will create a managed server on port 8201 using weblogic.Admin utility.
 C:OracleWeblogicuser_projectsdomainsTD1>java weblogic.Admin -url localhost:
7001 -username weblogic -password weblogic1 CREATE -mbean
“TD1:Name=TestMS1,Type=Server”

By executing the above command , TestMS1 has been created but on admin port. To change the listen port execute the below command.
C:OracleWeblogicuser_projectsdomainsTD1>java weblogic.Admin -url localhost:
7001 -username weblogic -password weblogic1 SET -mbean
“TD1:Name=TestMS2,Type=Server” -property ListenPort 8201

Follow the above steps to create one more managed server with name “TestMS2″.
Now create a cluster and assign multicast address using below commands :
C:OracleWeblogicuser_projectsdomainsTD1>java weblogic.Admin -url localhost:
7001 -username weblogic -password weblogic1 CREATE -mbean
“TD1:Name=TestCluster,Type=Cluster”

C:OracleWeblogicuser_projectsdomainsTD1>java weblogic.Admin -url localhost:
7001 -username weblogic -password weblogic1 SET -mbean
“TD1:Name=TestCluster,Type=Cluster” -property “MulticastAddress” “224.198.0.1″

We will assign the managed server to cluster as below:
C:OracleWeblogicuser_projectsdomainsTD1>java weblogic.Admin -url localhost:
7001 -username weblogic -password weblogic1 SET -mbean
“TD1:Name=TestMS1,Type =Server” -property Cluster “TD1:Name=TestCluster,Type=Cluster”

Repeat the above execution for adding TestMS2 in cluster.
To view configuration attributes of a particular Mbean, use weblogic.Admin GET:
C:OracleWeblogicuser_projectsdomainsTD1bin>java weblogic.Admin -url localh
ost:7001 -username weblogic -password weblogic1 -pretty GET -type ServerConfig
where ConfigMBean could be ServerConfig, ExecuteQueueConfig, DomainConfig, and so on.