Configuring High Availability Servers

High Availability (HA) Servers maximize system availability and eliminate any single point of failure. 

HA works in API Management Server and API Gateway Server as follows:

AMS HA: If there is a failure in the Active Server while it was deploying an application, the Secondary Server assumes the Active role and starts deploying the API Projects again.
AGS HA: If there is a failure in the Active Server while it was processing the messages from the components, the Secondary Server assumes the Active role and continues to process the messages.

Fiorano provides complete flexibility to administrators with the below options to implement High Availability on AMS HA and AGS HA:

• Database Replication: From active to passive serverIn scenarios where it is not possible to share the database, administrators can still use Fiorano's High Availability Enterprise Systems using the in-built replication support.
• Shared Database: Between active and passive server

HA Locking Mechanism

HA locking mechanism is employed by the servers in Replicated as well as Shared modes to determine the server state in case a server of the pair is unavailable or if the network fails. This mechanism works by hosting a file on a machine and sharing it with the servers. This file, provided with Read and Write permissions, is termed as "LockFile" and the machine hosting the LockFile is referred as the Gateway Machine.
A server can switch to Active state only if it holds a lock over the LockFile. For example, if two servers are configured with a shared LockFile, whichever server gets the first access to the lockfile becomes the Active Server. This mechanism makes the state-switching of an HA server more deterministic.

• Before configuring the servers for High Availability by means of any of the two methods (Replicated or Shared), LockFile needs to be created.

• LockFile can be of any format as the servers just look for the file but not the content.

• The lock file on a machine having UNIX/Solaris operating system should be shared by using the NFS protocol - version 4. If the lock file is present on Windows it should be shared using the Samba Protocol (it can be shared on Windows using NFS – version 4, if it supports it.)

• Make sure that the operating system hosting the server supports the protocol used for sharing the lock file.

• The directory containing the LockFile should have read/write permissions set. On the Operating Systems other than Windows, one can verify the permissions using the ls -l command.

• On Windows Operating System, the directory on the gateway machine containing the LockFile should be mapped to a network drive.

Example1:
If the directory containing the lock file on Windows, is shared using the samba protocol, this directory should be mapped to a network drive on the Windows machine hosting the server. Let us say the shared directory is mapped to a drive letter 'Z:/' and the lock file is AMS.txt, the lock file path would be 'Z:/AMS.txt'. This path should be used to set the LockFile parameter while configuring the profile via eStudio. 

On Non-Windows Operating System, the directory on the Gateway Machine containing the LockFile should be mounted on the machine hosting the server.

Example2:
If the lock file is present in a windows samba share and is mounted at /home/user/db on the machine hosting the server, then the lock file path on the machine hosting the server would be /home/user/db/AMS.txt. This absolute path should be used to set the LockFile parameter while configuring the profile via Studio.

Working with a LockFile

Creating and Sharing the LockFile

To share a LockFile on a separate system:

1. Create a folder with an identifiable name in the designated system.
2. Right-click the folder and do either one of the below options:
   • Go to Share with and click Specific people.
   • Click Properties, go to Sharing tab in the dialog box, and click the Share button.
3. In the text field, type the name of the system where the file has to be mounted and click Add; the name appears in the list below.
4. Click the system name and select Read/Write under Permission Level
5. In this shared folder, create a file with an identifiable name. Examples: AMS.txt, AGS.txt, AMSShared.txt.

Mounting the LockFile

To mount the shared LockFile on the system where operations are carried out:

1. Right-click My Computer and click Map network drive.
2. In the Map network drive dialog box:
   • Choose any drive from the Drive drop-down.
   • Click Browse and choose the location of the shared drive orIn the Folder text field, provide the IP address of the Gateway Machine and the shared folder name in the following format:
     IP Address\Shared Folder
   • Select Reconnect at logon check box to avoid mounting the file after every log in.
     
     

     

Database Replication

The central concept of backchannel replication is that the Active Server (the server which is in the Active State) replicates its data store and state to the Passive Server, thus keeping both servers in sync. This replication channel is supported on a private network dedicated to the synchronization of the broker state and messaging data.

If servers are running in Replicated HA mode, it becomes essential to have a reliable network for database synchronization calls. 

The secondary server accepts no client connection while in its hot-standby (passive) role, but is prepared to immediately transition to the Active role as soon as it detects that the Active Server is unavailable. If the primary fails, all Fiorano applications fail over from the primary and reconnect to the designated secondary backup broker. This Hot-failover process is immediate and is completely transparent to all client applications. The Secondary Server in the active role is sensitive to re-establishment of the replication channel. Once the primary comes up again, it assumes the role of the Secondary Server (since the original Secondary Server assumes to be the Primary Server).

The primary and secondary broker-pair use the replication channel to routinely seek the heartbeat of the other and watch for any interruption in the data flow or connection to switch states. A locking mechanism (explained below) is employed to determine the state of the servers.

Default Replication Server Profiles

The default AMS and AGS profiles for Primary and Secondary Servers are mentioned in the below table.

Configuring Replication Servers

On a broad perspective, the steps involved in the configuration is:

• Setting up the LockFile
• Configuring the profile

Setting up LockFile

Refer (13.0) Configuring High Availability Servers#Working with a LockFile section to set up the LockFile.

Configuring AMS HA Servers

1. To load Primary AMS, open Profile Management perspective, right-click Profiles node and navigate to: Load Profile > Fiorano API/ESB > haserver1 > primary > AMS. 
   
   

   

   
   
   
2. Load Secondary AMS by navigating to: Load Profile > Fiorano API/ESB > haserver1 > secondary > AMS
   
   

   

   
   
   
3. Primary AMS HA Server appears as APIMANAGER_AMS_HA_Primary and Secondary AMS HA Server appears as APIMANAGER_AMS_HA_Secondary under Profiles node.
   
   

   

   
   
   
4. Under Profiles node, select APIMANAGER_AMS_HA_Primary and APIMANAGER_AMS_HA_Secondary (use CTRL or SHIFT); right-click and select AMS Replicated HA
   
   

   

   
   
   
5. In the AMS Replicated HA properties dialog box, set the basic configuration for the parameters as below:
   • BackupServerIp: Specifies the backup server IP. Secondary Server acts as the backup server for Primary Server and vice-versa and hence provide the respective IP addresses.
   • BackupHAIPAddress: IP Address of backup peer server in HA mode.This parameter is mandatory to run HA.
   • LockFile: Full path of the file which will be used for determining the HA states. This file should be present in a different machine which is always available to the HA servers with all permissions.
   • GatewayServerIPAddress: IPAddress used to detect network failures. The IP specified must be of a machine that is always available on the network and this machine must be used to save the LockFile (above). Note: The IP addresses of all the above properties need to be changed from 'localhost' to the appropriate one accordingly.
     
     

     

     
     
     
6. Click Close and then save the profile to apply the changes.

Configuring AGS HA Servers

1. Deploy Primary AGS and Secondary AGS from the default profile: haprofile1 by following the same path as mentioned in Configuring AMS HA Servers section (choose "AGS" in place of "AMS"). Both the servers appear as "HA_APIGATEWAY_AGS".
   
   

   

   
   
   
2. Under Profiles node, select both the profiles (use CTRL or SHIFT); right-click and select AGS Replicated HA
   
   

   

   
   
   
3. In the AGS Replicated HA properties dialog box, set the basic configuration for the parameters as below:
   • BackupServerIp: Specifies the backup server IP. Secondary Server acts as the backup server for Primary Server and vice-versa and hence provide the respective IP addresses.
   • BackupHAIPAddress: IP Address of backup peer server in HA mode.This parameter is mandatory to run HA.
   • LockFile: Full path of the file which will be used for determining the HA states. This file should be present in a different machine which is always available to the HA servers with all permissions.
   • GatewayServerIPAddress: IPAddress used to detect network failures. The IP specified must be of a machine that is always available on the network and this machine must be used to save the LockFile (above).
   • PrimaryURL: URL of Enterprise server from which configuration should be loaded.
   • BackupURLs: Semi-colon separated backup URLs which are used when server at PrimaryURL is not available.
     Note: The IP addresses of all the above properties need to be changed from 'localhost' to the appropriate one accordingly.
     
     

     

     
     
     
4. Click Close and then save the profile to apply the changes.

Starting Server with the Replicated AMS/AGS Server Profile

To load the server on one of the profiles mentioned in Default Replicated Server Profiles section, use the following command at the location: "$Fiorano_home/esb/server/bin":

• AMS

Windows: server.bat -mode AMS -profile <ProfileName>/<Primary or Secondary>. Linux: ./server.sh -mode AMS -nobackground -profile <ProfileName>/<Primary or Secondary> Example:./server.sh -mode AMS -nobackground -profile haprofile1/primary (Linux)

• AGS

Windows: server.bat -profile <ProfileName>/<Primary or Secondary>. Linux: ./server.sh -nobackground -profile <ProfileName>/<Primary or Secondary> Example:./server.sh -nobackground -profile haprofile1/primary (Linux)

"-nobackground" in the Linux script helps to display the status on the console rather than running in the background.

Server State

Possible States of the Replicated HA Servers as displayed in the Command Prompt:

• Active
• Passive
• StandaloneSame as active. Indicates that the backup server is down or not present in network.
• DeadIndicates that the server is down or not present in network.
• ActiveSync/Active Transition
• PassiveSync/Passive Transition
• WaitingSame as passive. Indicates that the databases of the Active and Passive Servers need to be synchronized.

Shared Database

In this mode of High availability, the primary-secondary broker pair shares a common database and do not replicate data over the network. If the primary fails, all Fiorano applications fail over from the primary and reconnect to the designated secondary backup broker. The primary and secondary broker-pair use the network channel between them to routinely seek the heartbeat of the other and watch for any break in connection to switch States.
Locking Mechanism is employed to determine the state of the servers. The database which is common to both the servers is referred as Shared Database.

• The shared database is critical for the servers to function, as the servers store the entire data (API Projects will be created only in the Shared location) in it.

• It is mandatory for the Shared Database to be always accessible to the servers. Unavailability of the shared database could lead to data loss and data corruption.

• Data will be present only in the Shared Database unlike in Replicated HA where a copy is always created on two different locations

Default Shared Server Profiles

The default AMS and AGS profiles for Primary and Secondary Servers are mentioned in the below table.

Configuring Shared HA Servers

On a broad perspective, the steps involved in the configuration is:

• Setting up the LockFile
• Setting up the Shared Database
• Configuring the profile

Setting up the LockFile

Refer (13.0) Configuring High Availability Servers#Working with a LockFile section to set up the LockFile.

Setting up the Shared Database

Follow the same method as mentioned in (13.0) Configuring High Availability Servers#Working with a LockFile section (ignoring the LockFile part) to set up and mount a Database directory.

Configuring AMS Shared HA Servers

1. To load Primary AMS, open Profile Management perspective, right-click Profiles node and navigate to: Load Profile > Fiorano ESB > haprofile_shared > primary > AMS.
   
   

   

   
   
   
2. Load Secondary AMS by navigating to: Load Profile > Fiorano ESB > haprofile_shared > secondary > AMS
   
   

   

   
   
   
3. Primary AMS Shared HA Server appears as AMS_SharedHAPrimary and Secondary AMS Shared HA Server appears as AMS_SharedHASecondary under Profiles node.
   
   

   

   
   
   
4. Under Profiles node, select AMS_SharedHAPrimary and AMS_SharedHASecondary (use CTRL or SHIFT); right-click and select AMS Shared HA
   
   

   

   
   
   
5. In the AMS Shared HA properties dialog box, set values for the parameters: BackupServerIp, BackupHAIPAddress, LockFile and GatewayServerIPAddress. (Refer Step#5 in Configuring AMS HA Servers section for description of these properties).
   
   

   

   
   
   
6. Click Close and then save the profile to apply the changes.

Configuring AGS Shared HA Servers

1. Deploy Primary AGS and Secondary AGS from the default profile: haprofile_shared by following the steps in Configuring AMS Shared HA Servers section (choose "AGS" in place of "AMS"). Both the servers appear as "haAGS_shared".
   
   

   

   
   
   
2. Under Profiles node, select both haAGS_shared profiles (use CTRL or SHIFT); right-click and select AGS Replicated HA
   
   

   

   
   
   
3. In the AGS Shared HA properties dialog box, set values for the parameters: BackupServerIp, BackupHAIPAddress, LockFile, GatewayServerIPAddress, PrimaryURL and BackupURLs. (Refer Step#5 in Configuring AGS HA Servers section for description of these properties).
   
   

   

   
   
   
4. Click Close and then save the profile to apply the changes.

Starting Server with the Shared AMS/AGS Server Profile

Since the shared HA pair use a common database, the location of the database has to specified while starting up each server. The "-dbPath" command line option is used for specifying the location of the shared database.
To load the server on one of the above profiles (after configuring for HA), use the following command at the location: "$Fiorano_home/esb/server/bin":

• AMS

Windows: server.bat -mode AMS -profile <ProfileName>/<Primary or Secondary> -dbPath <Shared DB location>.

Linux: ./server.sh -mode AMS -nobackground -profile <ProfileName>/<Primary or Secondary> -dbPath <Shared DB location>

Example: ./server.sh -mode AMS -nobackground -profile haprofile_shared/primary -dbPath /root/shareddb10/AMS (Linux Primary AMS Server)

• AGS

Windows: server.bat -profile <Primary or Secondary> -dbPath <Shared DB location>.

Linux: ./server.sh -nobackground -profile <Primary or Secondary> -dbPath <Shared DB location>

Example: ./server.sh -nobackground -profile haprofile_shared/primary -dbPath /root/shareddb10/AGS (Linux Primary AGS Server)

• "-nobackground" in the Linux script helps to display the status on the console rather than running in the background.

• While running the servers on the same machine, both the HA servers should have their databases pointing to the same physical directory.

Server State

Possible States of a server in Shared Database as displayed in the Command Prompt:

• Active
• Passive
• ActivatingServer switches to Activating State as soon as it acquires the lock. Once all its services are activated, it switches to Active or Passive State accordingly.