[Solved] My Java application fails to connect to my Oracle database

Problem

Yesterday we faced a very weird issue: we have a Java application connecting to an Oracle DB via ojdbc6.

In these days it stopped working, magically: no relevant changes in the system configuration or source code.

The application couldn’t connect to the database: on client side, debugging the stack traces shown it was stuck in the logon process; on server side, Oracle was killing the connection for a timeout waiting for the client:

(alert.log)


***********************************************************************

Fatal NI connect error 12170.

VERSION INFORMATION:
 TNS for Linux: Version 11.2.0.2.0 - Production
 Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.2.0 - Production
 TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.2.0 - Production
 Time: 28-DEC-2015 09:07:56
 Tracing not turned on.
 Tns error struct:
 ns main err code: 12535

TNS-12535: TNS:operation timed out
 ns secondary err code: 12606
 nt main err code: 0
 nt secondary err code: 0
 nt OS err code: 0
 Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=47264))
WARNING: inbound connection timed out (ORA-3136)

What’s weird is that we could successfully connect to the same DB using the same credentials of our application via SqlPlus, SQL Developer. And even the same application worked fine from another machine using the same JDK.

 

To make sure it wasn’t our application, we tested with the following small java class (please forgive the style):

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

public class OraclePwdTest {
  static {
    try {
      Class.forName("oracle.jdbc.OracleDriver");
    } catch (ClassNotFoundException e) {
      e.printStackTrace();
    }
  }

  public static void main(String[] args) {
    String ip = args[0];
    String sid = "mySid";
    int port = Integer.valueOf(args[1]);
    String user = "myUser";
    String pwd = "myPass";
    Connection conn = null;
    //jdbc:oracle:thin:@[HOST][:PORT]:SID
    String url = "jdbc:oracle:thin:" + ip + ":" + port + ":" + sid;
    java.util.Properties prop = new java.util.Properties();

    prop.put("user", user);
    prop.put("password", pwd);
    //prop.put ("internal_logon", "sysdba");
    try {
      System.out.println("Connecting to " + url);
      conn = DriverManager.getConnection(url, prop);

      System.out.println("Connected");
    } catch (SQLException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
}

This class shown the same problems of our application but it turned out to be very convenient because it made our iterations much faster than they would have been with our application.

Another test we did was swapping ojdbc6 with ojdbc7: didn’t help, while ojdbc14 worked fine.

Solution

After all of these tests, Google helped us to put the pieces together, and we stepped into this page where the reply number 6 finally clarified the problem:

ojdbc6 relies on /dev/random which isn’t always fast:

“When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.”

(/dev/random man page)

We substituted /dev/random with /dev/urandom, which doesn’t block, and this worked just fine.

We therefore explained the issue with the following chain of events:

End of year, colleagues in vacation, no entropy on servers, /dev/random blocking, ojdbc6 slow, Oracle DB cutting the connection.