How threads work in Weblogic.

1. A client contacts the ListenThread, the entry point into WebLogic Server, which accepts the connection. It then registers the socket with a WLS component known as the SocketMuxer for further processing.

2. The SocketMuxer is responsible for reading and dispatching all client requests to the proper WLS container. It then adds this socket to an internal data structure for processing and makes a request of an ExecuteThreadManager to create a new SocketReaderRequest. This request is then dispatched by the manager to an ExecuteThread

3. As a result, the ExecuteThread becomes a SocketReader thread - it will continually run the SocketMuxer’s processSockets and checks the muxer’s queue to determine if there is work to be done. If an entry exists, it pulls it off the queue and processes it.

4. The SocketReader thread reads the client request and determines the protocol type of this client request, to create a new protocol specific MuxableSocket.

5. The MuxableSocketDiscrminator stores a new MuxableSocket with the implementation matching the protocol of the client. It also returns true to the SocketReader to notify it that the message is complete and it can be dispatched.

6. MuxableSocketDiscriminator re-registers the protocol specific version of the MuxableSocket that was created earlier. The net result is that “Step 2” is repeated, and a new protocol specific MuxableSocket is placed in the SocketMuxer’s queue for processing.

7. A socket reader will get the new protocol specific MuxableSocket off the queue and read it. It will then checks to see if the message is complete, based on the protocol. If it is, it will invoke the protocol specific MuxableSocketDiscriminator

8. Before the work requested by the client can be performed, there may be many iterations of “step 7”. This is determined by the protocol – for example, t3 will read a portion of the message, dispatch it so it can act upon the portion of the protocol read thus far.

9. The subsystem will create an ExecuteRequest and send it to an ExecuteThreadManager for processing. The request is dispatched to an ExecuteThread, and the result is returned to the client.


From a high level overview’s perspective, the SocketMuxer can be explained as follows. Each and every socket connection that comes into WebLogic Server is “registered” with the SocketMuxer - which then maintains a list of these connections, each represented by a form of the MuxableSocket interface. It then becomes the responsibility of the SocketMuxer to read and dispatch each client request to the appropriate subsystem. This is a fairly elaborate process, which is illustrated by steps 2 through 8 above.

There are only a few key things to know about the SocketMuxer:

First, it has a data structure in which it stores a socket entry for each client connected to WebLogic Server.
Second, a “socket reader” is the main component of the SocketMuxer - which really is just an execute thread that is running the SocketMuxer’s processSockets() method.

Third, the SocketMuxer does most of its work through the only interface it knows how to operate on – the MuxableSocket interface.

Socket Reader:

A SocketReaderRequest is merely an implementation of an ExecuteRequest, which is sent to the ExecuteThreadManger by the invocation of the registerSocket(). When the ExecuteThread invokes the execute() method of the SocketReaderRequest, the SocketMuxer’s processSockets() method is invoked.
So, a socket reader thread is simply a normal execute thread which runs the main processing method of the SocketMuxer, processSockets().


The acceptBacklog parameter of Weblogic server is passed to ServerSocket. The value of acceptBacklog means "The maximum queue length for incoming connection indications (a request to connect) is set to the backlog parameter. If a connection indication arrives when the queue is full, the connection is refused. "

Thus if too many connects come on the server at the same time, the server would queue this connects and process them one at a time. The value does not mean that only that many clients can connect to the server.

It does not limit the number of connections made. It limits the number of potential connections that can lie in the backlog queue. So for e.g.: AcceptBacklog is 2. If hundreds of connections were made to the server and the server has one thread to accept new connections.

This thread accepts the new connection and dispatches it to a new thread and then goes back to listening to new connections. Sample code is

while (true) {

Socket sock = serversocket.accept(); // Line 1
new MyThread(sock).run(); // Line 2

}

Here the thread accepts a new connection at line1. Dispatches to new thread in line 2. Evaluates the while expression and goes back to line 1. So in between the time it takes for it to get back to line 1(say T1) many new connections requests are made by the clients. These new connections lie on the accept backlog queue and this queue length is controlled by the accept backlog parameter.

If the queue length is 2, and between this time T1 several hundred connections are made to the server only 2 would get accepted and rest of them rejected. For rejection there must be too many simultaneous requests to the server, if it’s not simultaneous then the chances of queue getting full is less.

Standalone client to view the JNDI objects

As part of weblogic admin job, many times developers approach us asking us to show JNDI tree to if there objects are binded to the server. Generally we use "server --> View JNDI Tree" but this pages renders very slowly or it doesn't open as it happened to me today on one particular server. So I had written this basic stand alone client to find out objects.

import javax.naming.*;
import java.util.Hashtable;

public class ListJNDIObjects
{
static Hashtable ht = null;

public ListJNDIObjects()
{
ht = new Hashtable();
ht.put(Context.INITIAL_CONTEXT_FACTORY,"weblogic.jndi.WLInitialContextFactory");
ht.put(Context.PROVIDER_URL,"t3://vasserver:9080");
ht.put(Context.SECURITY_PRINCIPAL,"weblogic");
ht.put(Context.SECURITY_CREDENTIALS,"weblogic");

}

public static void main(String args[])
{
try
{
String jndiContext = "";
ListJNDIObjects listJndi = new ListJNDIObjects();
Context context = new InitialContext(ht);

if(args.length != 0)
{
jndiContext = args[0];
}

NamingEnumeration jndiList = context.list(jndiContext);

while(jndiList.hasMore())
{
NameClassPair ncp = (NameClassPair)jndiList.next();
System.out.println(ncp);
}

context.close();


}
catch(NamingException ne)
{
System.out.println("JNDI List failed : " + ne);
}
}
}

Troubleshooting OutofMemory

Most java.lang.OutOfMemoryErrors are the result of a program simply creating and using more objects than can fit in the maximum allowable heap space. The most common resolution to this type of error is:

1. Increasing the maximum heap size using the appropriate JVM command line option (e.g. -mx512m).

Beyond this, you will need to take steps to learn more about your JVM's heap usage. The easiest and best way to do this is with the verbose gc option (Usually specified -verbosegc but sometimes as -verbose:gc or -Xverbose:gc). This setting will usually output a single line for each major and minor garbage collection that takes place. The format is specific to each JVM, but generally each line shows "heap in use", "amount freed", and "time spent in garbage collection".

Some SUN JVM users have received the OutOfMemoryError as the result of permanent generation limitations. The java heap is comprised of several segments; the permanent generation being one of those. Therefore, if the java.lang.OutOfMemoryError is issued when the java heap has not been completely used (as shown with the verbose gc option), the most common resolution will be:

2. Increasing the size of the permanent generation space using the appropriate JVM command option.

e.g. BEA Solaris platform recommendation:
If you have problems with OutOfMemory errors and the JVM crashing with
JDK 1.3, try setting: -XX:MaxPermSize=128m.

There is currently an open bug on Sun's bug parade that describes this problem. See,
http://developer.java.sun.com/developer/bugParade/bugs/4390238.html

When the above conditions and remedies do not help, the problem is often thought to be a memory leak. Real leaks are actually rare because Java Applications are not responsible for freeing memory; the JVM is. Still, when an application allocates java objects but never releases (de-references) them, this condition is very similar to a traditional memory leak seen commonly in C and C++ applications. This type of memory issue generally appears in the verbose GC output as a slow and steady loss of free heap space. Eventually the JVM's Full or Major Garbage Collection task runs more and more frequently trying to reclaim heap space. Eventually, it will not keep up with demand and the java.lang.OutOfMemoryError message is output. At this point the JVM is unable to execute java code and any subsequent results are unpredictable.

To diagnose and resolve this type of problem, you will likely need to obtain a Java Heap Profiling tool. The following procedure should help:

3a. First make sure that you have conducted your tests without JVM JIT optimization. This can be done by adding
"-Djava.compiler=none" to your JVM startup command. This test should be done to avoid any JVM bugs which may exist with the optimization of your
java code. This step is also required for use of the available JVM debugging tools and it will be helpful to establish that the problem you are trying to locate is not a JVM bug but is instead created
by java code.
3b. Use the JProbe (http://www.jprobe.com or http://www.sitraka.com/software/jprobe) utility to inspect your JVM heap in order to determine which object class instances are being accumulated.

3c. Another similar product is OptimizeIt (http://www.optimizeit.com or http://www.borland.com/optimizeit)

3d. The JVM itself has the ability to dump its heap contents upon process termination. Therefore, you may find it helpful to supply the following JVM options:

-Xrunhprof:heap=dump,format=a (Use java -Xrunhprof:help for details)

and invoke the following code within your JVM when you wish to inspect the current heap contents:

System.gc(); // Request a Full Garbage Collection
Thread.sleep(5000); // Wait for completion
System.exit(); // Terminate the JVM process

The resulting java.hprof.txt file can be inspected to determine if any application changes can or should be made to
reduce the number of active objects. More information on this Java API can be found at:

http://java.sun.com/j2se/1.3/docs/guide/jvmpi/jvmpi.html#hprof-heap

There is still one more possibility or concept to explore. The amount of heap being used is often driven by the multi-threaded nature of a server JVM (such as WebLogic). If you allocate 100 threads for handling server requests, then it is possible for all 100 to be running in parallel. Under load, this configuration can use approximately 10 times the amount of Java Heap Space as one with only 10 threads. Therefore, if your verbose gc output shows a rapid or sudden climb in heap usage until none is available (free), you may simply have too many simultaneous activities for the amount of available heap. In this case, the resolution will be to:

4. Make your java heap as large as is possible (for the physical machine configuration) and then reduce server thread counts until your server application can stay within it limits.

The above suggestions should adequately address most Java Heap related memory problems. However, it is still possible to encounter system limitations with memory and/or JVM memory leaks.

Most UNIX operating systems allow limits to be placed on various process resources. Such limits may prevent creating Java Threads and other objects which need to allocate native process components such as stacks which are part of the total process size.

5. Carefully inspect the reported error message to make sure that an operating system process limitation is not at play.

JVM memory leaks are very rare but still possible. Therefore:

6. If your process size continues to grow until system resources are exhausted or limits exceeded, you may wish to use native O/S tools to determine which process segments are responsible. Continuous growth in the JVMs native components should be reported to the JVM vendor. Remember, when the JVM heap size reaches its maximum (-mx), the process size will not grow as a result of Java Heap allocation. Therefore, escalating process size is generally a result of native code (e.g. The JVM or JNI libraries).

End of Life for weblogic versions

Here is a link which gives you the details of the End of life of all the weblogic versions; you can find the EOL information on the site

http://www.oracle.com/support/library/brochure/lifetime-support-technology.pdf

Modify administration console

1.) Copy the console.jar file from the path= \weblogic92\server\lib\consoleapp\webapp\WEB-INF\lib”

2.) Extract the jar file using following command ”jar xf ”

3.) Modify the values in global.properties.

4.) Then again create a jar file using ‘jar –cvf “abc.jar” .

5.) Replace the console.jar file.

6.) Re-start the server.

Custom env variables for Managed Server.

If you want to set custom variable for a managed server instance though administration console such as for example,

Manager Server 1

VAR1=aaaa

VAR2=1234

You could specifies these on the remote start tab as a argument for each of the servers

-D=value

Continous HTTP access error logged when using Weblogic Cluster

Weblogic Cluster with one Administration Server and up to 12 Managed Servers. Our Weblogic version is 9.2.

SSL has been enabled for all Servers and the HTTP Listen Ports are all turned off. The cluster and all applications are working fine and communicating over HTTP as expected, however the logs grow 3MB+ per hour due to an error that repeatedly logged. This error appears to be an attempt by an internal Weblogic process to access some resource over HTTP.

Here are the various messages received in the logs. There are 3 messages logged per error occurred.

This first log message is from the domain log (I replaced the hostname/IP info with ):

#### <\[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <1218456526962> BEA-090475 Plaintext data for protocol HTTP was received from peer instead of an SSL handshake.>

This second log message is from the ".out" log from the Administration Server:

#### <\[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <1218456538658> <000000> - - \[11/Aug/2008:12:08:58 +0000] "GET /bea_wls_internal/HTTPClntLogin/a.tun?wl-login=https+dummy+WLREQS+9.2.2.0+dummy+%0A&rand=8038349418070109915&AS=2048&HL=19 HTTP/1.1" 200 1337

I have searched all over trying to find a solution but with no luck thus far. I have seen some posts about turning on HTTPTunneling and this does seem to prevent the error from occurring, but this is not a solution for us.

I could really use some help in understanding whats going on and trying to resolve this issue. Our client will make a big deal out of this and will want a solution. Any help would be greatly appreciated.

Message was edited by:
user651444

Please Open an SR with Oracle-BEA and ask for CR344429

Steps to change the language displayed in Admin Console

 ·          Before server startup

First you need set the preference languages specified in the browser 

·          To set preference language in browser

Right click the internet explore icon >>>>select >>>properties>>>general tab>>languages >>>add encoding  

After server start up 

      In admin console on right side panel click on 

                        Console>>>>preference >>>language>>>select desired language from  

                          drop down menu

      It is necessary to click the refresh button on your browser to see the changes 

Would CP1252 coding can recognize the codings for ASCII, UTF-7 and UTF-8?

1.            Weblogic server 8.1 is a Java application program. All strings are handled internally as Unicode strings. 

2.            Each encoding name has certain specific charsets which are used in HTML pages.

3.            The encoding conversion between Unicode and the HTML charsets is performed using the Java encoding converter when handling HTML data.

4.            Various charsets are used in HTML pages.

5.            It is recommended that you use the same particular encoding throughout your application.

Example:

res.setContentType("text/html;charset=Shift_JIS");

I referred below links and jotted down above points for more information refer below link

http://edocs.beasys.co.jp/e-docs/wls61/jconfig/wls61jconfig.html#1013050

http://edocs.beasys.co.jp/e-docs/wls/docs81/en/relnotes_ja.html

Work Manager

Default Work Manager

1. What is the max number of threads which can be spawned when using the default Work Manager (self tuned thread pool)

There is no limit on max number of threads can be spawned with self tuning thread. It adjusts its thread count based on an algorithm with a goal to achieve best possible throughput.

2. What if we do not use the default work manager then will the thread pool not self tuned

If you do not use default work manager, thread pool will be still self tuned to meet your custom work manager requirements.

WLS10:Please enable the DomainRuntimeMBean Server and the Edit MBean Server in this domain's configuration

Weblogic server10 “Please enable the DomainRuntimeMBean Server and the Edit MBean Server in this domain's configuration”

After placing the user credentials to Weblogic console it throws “Please enable the DomainRuntimeMBean Server and the Edit MBean Server in this domain's configuration” Error.

A required MBean Server is disabled which prevents the proper operation of the Weblogic Administration Console.

Please enable the DomainRuntimeMBean Server and the Edit MBean Server in this domain's configuration.

Suggestion :-

-> CAUSE:

The pending configuration in the pending directory became corrupt

-> ACTION

Delete the files in pending directory present in the domain home, restart the server

How to debug windows services (weblogic)

You can debug windows service:

1.) Go to “C:\bea921\weblogic92\server\bin”

2.) beasvc –debug “BEA Products NodeManager (C_bea921_weblogic92)”

It will print the messages related to windows services

Weblogic Utility commands (weblogic.admin)

This script will help you in identifying the server is reachable.

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic CONNECT 10”

This will return the state of the server:

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic GETSTATE”

This will return the state of manage server:

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic GETSTATE MS1”

This will provide you information about the license file:

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic LICENSES”

This will ping the server:

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic PING 10”

Tis will generate the thread dump and append in to server.out file:

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic THREAD_DUMP”

This will help you in creating the JDBC Connection pool.

“java weblogic.Admin -url localhost:7001 -username weblogic -password weblogic CREATE -name myPool -type JDBCConnectionPool”

Thread Dump: How to take thread dumps

There are many ways to take a thread dump depending on the operating system you are using.

Solaris or Unix box:

Kill -3

Windows box:

Run setWLSEnv.cmd.
(%WL_HOME%\server\bin\setWLSEnv.cmd)

Then execute:
java weblogic.Admin -url <> -username weblogic -password weblogic
THREAD_DUMP

Windows Service:

WL_HOME\bin\beasvc -dump -svcname:service-name >>

Where WL_HOME is the directory in which you installed WebLogic Server and
service-name is the Windows service that is running a server instance.

Thread dump will be re-directed to server log file.

For more details you can go through the link below,

http://edocs.bea.com/wls/docs81/adminguide/winservice.html