Recovering from primary CONTROL-M/Server failure (Server failover)

These procedures assume that a failover server has already been installed and initialized on a secondary computer. Two procedures are used during failover situations:

1. Verify that:
• Both primary and secondary CONTROL-M/Server installations and CONTROL-M/Server Configuration Agent are not running.
• Both the primary and mirror database servers are running.
2. Log on to the secondary computer as the CONTROL-M/Server owner.
3. Specify the ctm_menu command to open the CONTROL-M Main Menu.
4. Type 4 to select Database Mirroring and press <Enter>.
5. From the Database Mirroring menu, type 7 to select Start Failover and press <Enter>.

WARNING


Do not start the secondary CONTROL-M/Server while the primary CONTROL-M/Server is running, or before the Start Failover procedure finishes executing. If the secondary CONTROL-M/Server is started too early, the following message may be displayed: 

** Unable to start CONTROL-M due to path specified for PROCLIB directory $PROCLIB. This path is invalid **

6. The following message is displayed:

This operation disables CONTROL-M database mirroring, and starts FailOver CONTROL-M !!!

Confirm that CONTROL-M in the primary data center is not running.Continue [y/n] :

7. Type y and press <Enter> to continue with Failover.

The following message is displayed:
CONTROL-M/Server database mirroring 

Mirroring of the CONTROL-M/Server database allows rapid recovery of CONTROL-M/Server functionality in the event of a failure of the CONTROL-M/Server database. You can implement mirroring during or after CONTROL-M/Server installation.

To implement CONTROL-M/Server database mirroring

1. Review the database requirements for mirroring as described in the CONTROL-M/Server for UNIX and Microsoft Windows Installation Guide. 


2. Verify that CONTROL-M/Server, CONTROL-M/Server Configuration Agent, and CONTROL-M/Server utilities are not running. Ensure that the SQL database is running.


3. Choose Database Mirroring => Initialize Mirroring from the CONTROL-M Main Menu.


4. At the conclusion of the procedure, restart CONTROL-M/Server and the CONTROL-M/Server Configuration Agent

Read more »

Recovering from Database Failure

If the primary database fails, two options exist for restoring CONTROL-M/Server operation:

1) Operate CONTROL-M/Server using the mirror database instead of the primary database. The primary CONTROL-M/Server database can be rebuilt at a later stage.

This method resumes CONTROL-M/Server operation in the minimum time possible. However, mirroring cannot be reactivated until after the primary database is repaired.

2) Restore the CONTROL-M primary database from the mirror database and start CONTROL-M/Server again.

This method allows the CONTROL-M/Server database to continue to operate with mirroring. However, you will not be able to start CONTROL-M/Server until you have completed restoring the CONTROL-M/Server database - and this can result in significant downtime.

Read more »

Failover Server

A Secondary installation of the CONTROL-M/Server resides on a separate computer. Typically the database of the secondary CONTROL-M/Server, created as part of the CONTROL-M/Server installation, acts as a mirror database.

The secondary CONTROL-M/Server, including its database, acts as a backup to the primary CONTROL-M/Server. This secondary CONTROL-M/Server computer is sometimes called the Mirror Environment.

The secondary CONTROL-M/Server duplicates all of the primary CONTROL-M/Server’s links to CONTROL-M/Agents, and CONTROL-M/Server Configuration Agent. The secondary CONTROL-M/Server is defined in each agent’s list of authorized server hosts.

During normal operation, the secondary database is constantly updated to remain identical with the primary database. The secondary CONTROL-M/Server remains inactive.

If the primary CONTROL-M/Server fails due to a hardware/software problem on the primary computer, the administrator activates failover on the secondary CONTROL-M/Server. The secondary CONTROL-M/Server performs all normal CONTROL-M/Server functions until the primary CONTROL-M/Server is repaired.

Jobs submitted to agent computers before failover continue executing. The secondary CONTROL-M/Server polls the agent computers to determine the status of jobs listed in the Active Jobs file.

When the primary CONTROL-M/Server is stable, normal operation is restored. The failover server is shut down, the primary database is restored, and the primary CONTROL-M/Server is restarted. The secondary server returns to passive readiness.

Implementing a Failover Server

A failover server can be implemented any time after CONTROL-M/Server has been installed.

The secondary CONTROL-M/Server installation should be built on a separate computer and database server, independent of the computers that host the primary CONTROL-M/Server and its database.

Several administrative functions are used to initialize and use the failover server, and restore the primary computer. These functions are accessed from the Database Mirroring menu of the CONTROL-M/Server Main Menu.

Preparing a Failover server

Verify the following conditions for the secondary installation:

The Agent-to-Server Port Number and Server-to-Agent Port Number communication parameters must be the same on both CONTROL-M/Server computers. These ports must also be dedicated to the use of CONTROL-M/Server. You should also ensure that both computers are using the same date.

1. The failover server can be configured to use a dedicated or a non-dedicated SQL database server. However, it should not use the same instance of the database server used by the primary environment.


2. The Database (Data Portion) Size parameter should be assigned the same value as the current size of the primary database.


3. All agent computers that are defined on the primary CONTROL-M/Server should also be defined on the failover server. These definitions must be identical. If you installed CONTROL-M/Agent on the primary CONTROL-M/Server computer, it should also be installed on the failover server. The parameters should be the same on both CONTROL-M/Server installations.

The host name of the failover server must be added to the list of authorized hosts of each agent.

Read more »

Mirror Database

In addition to the primary database installed with CONTROL-M/Server, a secondary database is built on another Database Server computer. All updates to the primary CONTROL-M/Server database are written to the secondary mirror database.

If updates to the mirror database fails, the mirror database is considered damaged and writing to it is disabled. Normal CONTROL-M/Server operation is not affected. Mirroring can be re-initialized after you repair the communications link.

If the CONTROL-M/Server primary database server fails, or if the primary database loses integrity, CONTROL-M/Server operation can be restored using either of the following manual interventions:

You can direct CONTROL-M/Server to use the mirror database instead of the primary database. The primary CONTROL-M/Server database can be rebuilt when time allows.

This method ensures that CONTROL-M/Server functionality is resumed as soon as possible. However, mirroring is not resumed until after you repair the failed primary database and restore it from the mirror database.

You can repair and rebuild the CONTROL-M/Server primary database and restore it from the mirror database.

This method allows mirroring of the CONTROL-M/Server database to continue without interruption. However, depending on database size and the cause of the failure of the database, it may take some time to rebuild the CONTROL-M/Server database before CONTROL-M/Server can be restarted.

Implementing a Mirror Database

Database mirroring can be initialized any time after CONTROL-M/Server has been installed.

The mirror database should be built on a separate database server, independent of the database server that hosts the primary CONTROL-M/Server database.

Several administrative functions are used to initialize, enable, and restore CONTROL-M/Server database mirroring. These functions are accessed from the Database Mirroring menu of the CONTROL-M/Server Main Menu.

Read more »

Failover Planning

As CONTROL-M is integrated in the production environment of the data center, it becomes increasingly important to ensure that interruptions of CONTROL-M functionality are as short as possible.

A properly designed and executed failover plan ensures that CONTROL-M functions are resumed as soon as possible if a recovery is necessary.

Failover planning for CONTROL-M should provide for the following contingencies:

1. Loss of integrity in the CONTROL-M/Server primary database due to a failure of the SQL server that maintains the database


2. Failure of the computer on which CONTROL-M/Server runs (server computer) Ideally, your failover plan should include a complete backup for both the server computer and the CONTROL-M/Server database.

You can implement either one of the following levels of failover capability:

Mirror Database: a database on an independent database server mirrors the CONTROL-M/Server database. If this level of failover is implemented, only CONTROL-M/Server database is backed-up and will be available in case of loss. There is no failover support for any loss of information affecting the CONTROL-M/Server installation. Readmore...

Failover Server: a secondary CONTROL-M/Server installation on a different computer. The CONTROL-M/Server database of the secondary installation is considered the mirror database. If this level of failover is implemented, failover support is provided for loss of information affecting both the CONTROL-M/Server installation and the CONTROL-M/Server database. Readmore...

Read more »

The Control-M Data Center

The Control-M/Data Center is the true core of the Control-M system. The Data Center is comprised of a Control-M/Server, Database and Control M/Agents.

The Control-M/Server

The Control-M/Server is the central component of the Data Center. It controls interaction with the Enterprise Manager, Control-M/Agents and the database backend. The Control-M/Server (CTM/SERVER) is the component that maintains the schedule, job definitions, calendars, resources and conditions. CTM SERVER is the only component that can load or order jobs.

The Database

The database serves as a backend data repository for Control-M (Sybase, PGSQL, Oracle or MS SQL server can be used). The CTM/SERVER has its own data-space within an Oracle instance. This data-space contains the Master versions of all of the job-related data: job-definitions, scheduling tables, resources, calendars active jobs tables and shout tables, to name a few. It also houses runtime specific data for the CTM/SERVER such as process flags, user access profiles and agent status.

The Database is critical to the operation of CTM/SERVER & Control-M. If the database is not available, Control-M will not operate.

The Control-M/Agent

The Control-M/Agent (CTM AGENT) is a small-footprint application that is installed on each target server for jobs, as well as the CTM SERVER. It runs as a service on Windows-based systems and has a “hook” into “inetd” on UNIX systems. It is dormant until a request comes from the CTM SERVER or until a job-related message needs to be sent to CTM SERVER.

Read more »