SOA@Oracle SCA, BPEL, BPM & Service Bus: 3/1/11

Wednesday, March 23, 2011

Disaster Recovery with SOA Suite and OSB and other products

In a production environment we did a disaster recovery test. The production environment contains of two separate locations. Location #1 will be up and running, while we isolated location #2 from #1. We assume that location #1 is down. The goal is to bring up location #2 as soon as possible.

I want to share you experience during this exercise. All the applications servers are bases on Oracle technology:

Oracle SOA Suite 10g (10.1.3.4)
Oracle SOA Suite 11g, ps2
Oracle Service Bus 11g, ps3
Oracle WebCenter 11g, ps2
Oracle Weblogic + ADF 11g, ps3

All the application servers are installed in a cluster, form 2-node to 8-node clusters.

The database are running in active/standby, this means that Oracle Real Application Clusters is not used. On location #1 the active databases is running, on location #2 the databases are running in standby mode.

The trick is how do we get the application clusters up and running, while location #1 is not available. When location #1 is not available, the application servers on location #2 can not reach the database.

Meanwhile the DBA are working to bring up the database on location #2 from standby to active.

We were able to run the application servers on node #2 without restarting the servers when the databases are brought up to active. Only the Oracle SOA 10g application servers need to be restarted.

What did we do? We configured all the hostnames in the datasource to a generic hostname in the DNS server.

Instead of having a hostname to the active database on location #1, we are using a general hostname.

old: soaprod.node1.database.nl
new: soaprod.cluster.database.nl

In the period that the database on location #2 is brought up to active, the DNS entry of the *.cluster.database.nl to location #2.

When the database is active we saw:

Oracle SOA 10g did not run succesfully. A restarted was needed.
Oracle SOA 11g, did run, but some datasource are in suspend mode, resuming the datasource fixed the issue.
Oracle WebCenter 11g, did run, but only on location #1. It was installed on a replication volume, we brought up this server from the replication volume succesfully.
Oracle Web Service Manager 10g, did not run, a restarted was needed.
Oracle Weblogic + ADF, did run succesfully.

Notes:

All the adminstration servers are installed on a seperate server on a replication volume. This makes it easy to bring up the administartion server on the other location.

The log files will grow, while the 'Cluster' periodlivly checks if the other node is available.

Using virtualisation to run your application servers saves a lot of time in case of a disaster.

Some Reference links:

Saturday, March 05, 2011

OSB: Automatic update of Service Acounts

When you use Service Accounts in the Oracle Service Bus (OSB), you can not customize them during deployment. Until OSB 10g, there was a non documented feature to change Service Accounts via Weblogic Scripting Language (WSLT). From OSB 11g you can not use ServiceAccountConfigurationMBean to change the Service Accounts (http://download.oracle.com/docs/cd/E14571_01/apirefs.1111/e15033/index.html ).

Oracle said this is a security issue and they will not customize Service Accounts during deployment to different environments. The recommended way is to customize accounts manually using service bus console.

My opion is that the owner of the OSB application should decide if this is a security issue or not. Oracle should give a proper solution to deploy Service Accounts to different environments.

The workaround the use different Service Accounts for multiple envirnoments is using the following approach:

Create your Service Accounts in Oracle Workshop based on ${variables}
Create an export file with your OSB aritfacts.
Unzip the OSB export file
Find the all the *.ServiceAcount files
Replace the ${variables} with the username/passwords for the environment you want to deploy.
Zip all the OSB artifacts back into the OSB export file
Use a customization file for your environment (dev/test/accept/production)
Deploy it!

Here is an example of my solution based on ANT:

Property file wit two Service Accounts; osb_test.properties:

ServiceAccount.MySecuredLDAPUsername.value=cn=LDAP,ou=Users,o=Services
  ServiceAccount.MySecuredLDAPPassword.value=welcome1

  ServiceAccount.MySecuredHTTPServiceUsername.value=oc4j_admin
  ServiceAccount.MySecuredHTTPServicePassword.value=welcome1

The Service Account file from Oracle Workshop (Eclipse), .sa file:

<?xml version="1.0" encoding="UTF-8"?>
<ser:service-account xsi:type="ser:StaticServiceAccount" 
   xmlns:ser="http://www.bea.com/wli/sb/services"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns:con="http://www.bea.com/wli/sb/resources/config">
  <ser:static-account>
    <con:username>${ServiceAccount.MySecuredLDAPUsername.value}</con:username>
    <con:password>${ServiceAccount.MySecuredLDAPPassword.value}</con:password>
  </ser:static-account>
</ser:service-account>

The Ant script the extracts and replace the Service Account tags.

<target name="_ReplaceTokens">
    <!-- replace ${} tokens into <param1>.new file -->
    <echo message="Replace Tokens in ${param1}"/>
    <copy file="${param1}" tofile="${param1}.new" overwrite="true">
      <filterchain>
        <replaceregex pattern="\$\{" replace="{"/>
        <filterreader classname="org.apache.tools.ant.filters.ReplaceTokens">
          <param type="propertiesfile" value="${param2}"/>
          <param type="tokenchar" name="begintoken" value="{"/>
          <param type="tokenchar" name="endtoken" value="}"/>
        </filterreader>
      </filterchain>
    </copy>
  </target>
  
    <target name="_replaceServiceAccounts" description="Replace ServiceAccounts with correct passwords">
    <tstamp prefix="Start _replaceServiceAccounts"/>

    <property name="sa.injection.dir" value="${project.derived.dir}/${osbs.env}_${osbs.svntype}"/>
    <delete dir="${sa.injection.dir}" verbose="${OSBVerbose}" failonerror="false" includeemptydirs="true"/>
    <mkdir dir="${sa.injection.dir}"/>
    <unzip src="${config.jar}" dest="${sa.injection.dir}"/>

    <echo message="Parse service accounts"/>

    <for param="sa.file">
      <path>
        <fileset dir="${sa.injection.dir}" includes="**/*.ServiceAccount"/>
      </path>
      <sequential>
        <echo message="Parse service accounts file: @{sa.file}"/>

        <antcall target="_ReplaceTokens" inheritAll="No">
          <param name="param1" value="@{sa.file}"/>
          <param name="param2" value="${project.source.dir}/release/Customization/${osbs.env}/sla.properties"/>
        </antcall>
        <copy file="@{sa.file}.new" tofile="@{sa.file}"/>
        <delete file="@{sa.file}.new" quiet="true"/>

      </sequential>
    </for>
    <zip destfile="${config.jar}" basedir="${sa.injection.dir}" update="false"/>

    <tstamp prefix="finished _replaceServiceAccounts"/>
  </target>

References