Control-M EM Gateway "No Sync" Issue - Possible Causes and Solutions

Control-M EM Gateway "No Sync" Issue is a rare scenario but as per my experience once it occurs it reappears several times until you find a root cause for this. First lets see the quick workaround for this and then we will see the possible causes for this.

Workaround:

• shut_ca
• shut_ctm
• init_prflag
• start_ctm
• start_ca

 

Possible Causes/Preferred solutions:

1.      1) This might be a Java heap memory issue. Whenever you see "java.lang.OutOfMemoryError: Java heap space" in the CE log on Control-M server then you need to increase the heap memory based on the volume of the jobs. Also make sure your system supports the increased memory. Should be under <ctm Server>/data/config.dat file.

a.      CTMS_JVM_MAX_HEAP 512 to 2048 (per the jobs)

2.      2) If you are running more MFT jobs, this may cause the issue as well. Try not running more MFT jobs from server instead use the module installed on agents.

3.      3) Check if there are any jobs passing garbage/junk value which make the CE process crash.

4.      4) Preferred to use Update Control-M Server definitions during Check-In in the synchronization parameter of Control-M Server in CCM

5.      5) heck for any agents which are causing the issue. This can be checked in the CE log. Look for “Lock Table CMR_NODES in exclusive mode”. This may cause DB lock and eventually EM lost connection with server.

6.      6)  There might be another third party software grabbing control-m ports. Check that using the below command

a.      netstat -an | grep 2370

b.      netstat -an | grep 2369

c.      netstat -tulpn | grep 2370

d.      netstat -tulpn | grep 2369

7.     8) Check if there is any network scanner is running on the machine. If so make sure to exempt control-m application from that.

8)  9) If you have any obsolete unavailable agents either disable or delete them.

     10) Run the below command to see if there are any updates causing the issue. Observe the field “AgentAppUpdates” this column should be 0… if its keep growing then that’s definietely an issue caused by a MFT/AFT agent which is in “PERSISTENT” mode. Check on which agent there are more AFT jobs running and change that to “Transient” mode.

      ctmipc –dest ce –msgid ctl –data “CtmThreadPool”.

0

)

Also, Run ctms_data_collector and em_data_collector to collect the logs and open a case with BMC

Read more »

How to Add a Condition using SHOUT (POSTPROC)

As most of us know that we can add conditions using "Conditions" tab or through steps tab.  Normally we can add/remove conditions based on the below scenario's.

• When job Ended OK (using Conditions Tab)
• When the job Ended NOTOK (using Steps Tab)
• When a job failed with a specific return code (using Steps Tab)
• When you find a particular string in Log/Sysout (using Steps Tab)
• When Run Counter criteria met (using Steps Tab)

... there might be many other scenario's where we may need to add/remove conditions. few of them are When a job executing for longer time, or If the job not submitted by a particular time or If the job not completed by a certain time. In general we will trigger an ALERT in these scenario's using PostProc tab. Now I'm going to explain you how can we add a condition instead of generating an alert.

Like how we use LateTime/ExecTime/LateSub, we'll goto PostProc tab and select the particular event which you would like to check and add a condition.

A SHOUT WHEN EXECTIME > nnn (The range is 001 to 999 in minutes only) or 
A SHOUT WHEN LATE SUB > HHMM or 
A SHOUT WHEN LATE TIME > HHMM

If you want to Average time then use + and you can also use % (percentage) to limit above the average time.

Here is the example:

"condition" must be added in the shout destination table as a script i.e program as destination. You can name as you desire (something like add-condition.sh). The script should contain the below statement to add the condition.

ctmcontb –add $2

The ctmcontb utility performs different operations on prerequisite conditions in the Control-M/Server database (Like Add, Delete, List conditions).

In the same way you can also invoke any other utility depend upon the requirement.

Read more »

Control-M EM GUI

The CONTROL-M/Enterprise Manager application provides a single, centralized point of access and control of your batch processing environment. CONTROL-M/Enterprise Manager lets you to automate the scheduling and processing of your production jobs. Its main GUIs are CONTROL-M/Desktop and the CONTROL-M/Enterprise Manager window (simply called CONTROL-M/EM).

Logging into Control-M/EM:

Click on the Icon which is located on your Desktop.

Go to Start Menu\Programs\CONTROL-M Enterprise Manager 6.3.01\ Default\ CONTROL-M Enterprise Manager GUI

Read more »

ViewPoint Manager

Defining ViewPoints

A ViewPoint builds a view of the active environment by retrieving and displaying data according to the criteria defined in the ViewPoint’s collection, hierarchy, and global filter definitions.

■ Collections determine the jobs to be loaded into memory.

■ Global filters determine which jobs to display.

■ Hierarchies determine the hierarchical structure in which to display jobs.

CONTROL-M/EM is supplied with a number of predefined ViewPoints, but you can create or modify definitions.

To define a ViewPoint

1. In CONTROL-M/Enterprise Manager, choose Tools => ViewPoint Manager.





2. In the ViewPoint Manager dialog box, do one of the following:
1. To create a new ViewPoint definition, select the ViewPoints node and click New. Then, in the ViewPoint dialog box that opens, enter a name for the ViewPoint.



2. To edit an existing ViewPoint definition, select the definition under the ViewPoints node and click Edit.


3. In the ViewPoint Properties dialog box, do the following:
1. Enter a description for the ViewPoint (optional).
2. Select a collection for the ViewPoint.
3. Select a global filter for the ViewPoint.
4. Select a Hierarchy definition for the ViewPoint.
5. After selecting the hierarchy, select the Initial Level of the Hierarchy to display when the ViewPoint is opened.
6. By default, the highest hierarchy level nodes are displayed in when the ViewPoint is opened.
7. Click OK.

Read more »

Control-M/EM GUI Authorizations

To perform any action in Control-M we need Authorization/Permission. This will be setup using Authorizations window in Control-M/EM GUI. Using this we can create, modify or Delete user-id's and their privileges. This is the window Schedulers or Admin will use to create a User-Id or Reset Password or modifying privileges.. etc.

To access the Authorizations window Follow the below steps:

1. Log on to CONTROL-M/EM GUI or CONTROL-M/Desktop.
2.  Choose Tools => Authorizations. The Users tab is displayed.

The Authorization window has separate panels for users and groups. The Users panel enables you to add, modify, copy, or delete a user whose authorizations can be configured. The Groups panel enables you to add, modify, copy, or delete a group whose authorizations apply to members of the group.

Actions that are permitted or denied for each access level are described below. 

1. None - Denies access to the component.  
2. Browse - Can view and refresh the component.  
3. Update - Permits adding and modifying CONTROL-M information.  
4. Full - Permits adding, modifying, and deleting CONTROL-M information.  
5. Default - [Valid for user authorizations. Not valid for group authorizations.] Inherits the permissions from the group to which the user belongs. Use of this is, We can create a Group with list of Privileges

The panel will look like below.

Authorizations window – Users tab fields

• Filter: Select this check box to enable the User Name filter.
• User Name text box: Specify a user name or pattern (for example, adm*). Click Apply to apply the filter. Click Clear to erase the contents of the text box.
• Name: CONTROL-M/EM logon name of the user to whom the authorization refers. Names listed satisfy the requirements of the filter, if any.
• Full Name: Full name of the user.
• Description: Description of the user.

Authorizations window – Groups tab fields

• Filter: Select this check box to enable the Group Name filter.
• Group Name text box: Specify a group name or pattern (for example, adm*). Click Apply to apply the filter. Click Clear to erase the contents of the text box.
• Name Group name: Groups listed satisfy the filter requirements, if any.
• Description: Description of the group.

User Properties Window:

Authorizations window – User Properties fields

• General: Provides logon data about users (or lists members of a group).
• Member Of: [User Authorizations window] Matches users with groups.
• Jobs: Jobs a user (or member) can access and actions he can perform.
• Privileges: 
• Whether the CONTROL-M Configuration Manager can be accessed and used, Alerts and Archived Viewpoints can be viewed, and the user (or member) can use the Command Line Interface (cli).
• Actions that can be performed on Collections, Hierarchies, Filters, ViewPoints, and Authorization screens.
• Scheduling Tables: Scheduling tables a user (or member) can access in CONTROL-M/Desktop, authorization levels for the tables, and whether jobs or tables can be ordered or forced.
• Prerequisite Conditions: Prerequisite conditions that a user (or member) can access.
• Control Resources: Control resources that a user (or member) can access.
• Quantitative Resources: Quantitative resources that a user (or member) can access.
• Global Conditions: Global conditions that a user (or member) can access.
• Calendars: Calendars that a user (or member) can access.
• Owners: Jobs in scheduling tables that a user can access.

Read more »

Failover Planning

As CONTROL-M is integrated in the production environment of the data center, it becomes increasingly important to ensure that interruptions of CONTROL-M functionality are as short as possible.

A properly designed and executed failover plan ensures that CONTROL-M functions are resumed as soon as possible if a recovery is necessary.

Failover planning for CONTROL-M should provide for the following contingencies:

1. Loss of integrity in the CONTROL-M/Server primary database due to a failure of the SQL server that maintains the database


2. Failure of the computer on which CONTROL-M/Server runs (server computer) Ideally, your failover plan should include a complete backup for both the server computer and the CONTROL-M/Server database.

You can implement either one of the following levels of failover capability:

Mirror Database: a database on an independent database server mirrors the CONTROL-M/Server database. If this level of failover is implemented, only CONTROL-M/Server database is backed-up and will be available in case of loss. There is no failover support for any loss of information affecting the CONTROL-M/Server installation. Readmore...

Failover Server: a secondary CONTROL-M/Server installation on a different computer. The CONTROL-M/Server database of the secondary installation is considered the mirror database. If this level of failover is implemented, failover support is provided for loss of information affecting both the CONTROL-M/Server installation and the CONTROL-M/Server database. Readmore...

Read more »

Control-M New Day Process(NDP)

• At a set time each day (defined during installation as the start of day at the site), CONTROL-M performs New Day processing, during which:
• CONTROL-M performs a number of maintenance and cleanup functions that the operator would otherwise have to perform manually.
• Job scheduling definitions are selected from the scheduling tables (based on their basic scheduling criteria) and are placed in the Active Jobs file. These jobs can then be submitted and tracked by the CONTROL-M monitor.
• During New Day processing, jobs that have ended OK or whose retention period has expired according to job scheduling definition parameters are deleted from the Active Jobs file.
• In simple terms "this loads all of the jobs that Control-M knows should run that day. Control-M does not necessarily know all of the jobs to run. it only knows the ones scheduled to run that day. In process of cleanup Control-M will clear all the jobs (ended OK, Deleted Jobs, Maxwait expired jobs, Eligible Failed jobs) except the jobs which are in Hold and Jobs which are having MAXWAIT". 
• Note: The jobs which were ended OK will be retained for one day.

Read more »

Understanding the flow diagram in AJF and Workspace

In the flow diagram, components of the environment (definition environment or active environment) are represented by node boxes. The information displayed in a node varies depending on the type of node and on customization options

Read more »

AJF (Active Job File)

In Control-M EM GUI, where you can see the currently running jobs for today is called AJF (Active Job File).

The CONTROL-M/Enterprise Manager window (AJF) consists of the following parts:

Ø  Navigation Tree – Hierarchical view of jobs in the current environment.

Ø  Flow Diagram – Graphic representation of job production flow, based on job dependencies established by prerequisite conditions specified in job processing definitions. Components of the active environment are represented by boxes called nodes. The information displayed in a node varies depending on the type of node and on how certain customization options.

Ø  Net Overview – Miniature version of the flow diagram, indicating the part of the network currently displayed in the Flow Diagram view. By ragging the selected area in this view to a different part of the network, you can quickly navigate in a complex environment.

Read more »

Dynamic Filter in Control-M EM GUI

A dynamic filter is a filter that you define locally as you need it. You can then discard it or save it for reuse.

To define a dynamic filter:

1. If the Dynamic Filter Definition dialog box is not currently displayed, display it by performing one of the following tasks (whichever is appropriate):
• If the Select ViewPoint dialog box is currently displayed (that is, you are in the process of opening a ViewPoint), after you select the ViewPoint, click Dynamic Filter.
• If the ViewPoint is currently displayed, select <Dynamic Filter> in the Filter list box.
• The Dynamic Filter Definition dialog box displays the last-used dynamic filtering criteria.
2. To edit an existing filter, select it in the Filter Presets field and click OK. To create a new filter, specify a name for it in the Filter Presets field.
3. Fill in filtering criteria.
4. To save this definition for later reuse, click SAVE Button.

Consider the following when filling in filtering criteria:

• Most fields are self-explanatory. If you need a description of the fields that correspond to fields in the job editing form,
• You can use the LIKE operator for all text fields except the Odate and Time parameters. For Odate From and Time From, use the >= operator. For the Odate To and Time To fields, use the <= operator.
• For Odate From and Odate To, use the yymmdd format. For Time From and Time To, use the hh:mm format.
• Fields can contain several criteria separated by commas (for example, CONTROLM = A*,B*).
• If you specify values for Time From and To fields in the filter, the filter processes values in the Start Time and End Time fields in the Job Detail screen (not the From and Until fields in the Job Detail screen).
• The Job Status and TaskType fields each have a button, when clicked, opens an appropriate dialog box (with relevant check boxes):
• For BMC Batch Impact Manager users: Check the Included in Business Service checkbox to filter on jobs belong to a critical batch service.

Read more »

Find jobs in Control-M

To find jobs in the flow diagram that conform to specific criteria

1. In CONTROL-M/Desktop or CONTROL-M/EM, select Edit => Find Jobs.
2. In the Find Job dialog box, fill in the criteria (you can specify pattern-matching strings in certain fields).
3. In CONTROL-M/EM only, to save the specified Find criteria for future use, assign a name in the Find Presets field, and click Save.
4. Click Find, which selects all nodes matching the criteria, and displays the first node that fits the criteria.

To change the focus (navigate among multiple selected nodes)

Click the appropriate navigation button (First)  , (Previous) , (Next) , or (Last) ,  or use the View => Toggle Selection menu.

Read more »

Prerequisite Conditions

"Prerequisite Condition is a flag representing a user-specified situation or condition. Submission of a job for execution can be made dependent upon the existence of one or more conditions. Conditions are recorded in the Conditions table in Control-M EM Server database"

Job dependencies in the flow diagram are represented by lines and arrows connecting

job nodes, for example.These lines can even indicate dependencies between jobs in different CONTROL-Ms. The direction of arrows at the ends of the lines indicates the flow direction (predecessor and successor relationship) of the connected nodes.

Dotted lines between two jobs indicates a conditional link between the two jobs (for example, optional In conditions defined using OR logic).

An arrow without a connecting line at the top or bottom of a node indicates one of the following condition types:

Arrow at the top of the Job — a manual In prerequisite condition. This is a condition that does not get added automatically by another job, but rather must be manually added.

Arrow at the bottom of the Job — The Out prerequisite condition has no corresponding In condition or job dependent upon the condition.

In CONTROL-M/EM, the connecting lines between conditions are

■ Green —condition exists (active)
■ Black — condition does not exist yet (inactive)

Read more »

Control-M Job Actions

Actions that can be performed on jobs:

Action	Description
Confirm	Confirm the job.
Delete	Delete the job.
Documentation	View the job's documentation.
Edit JCL/Script	Edit the job's JCL or job script statements.
Enhanced Why	View the predecessor’s jobs and Group Scheduling table for a selected job or Group Scheduling table. It is selectable only from Flow Diagram.
Force OK	Set the job status to OK.
Free	Free the job.
Hold	Hold the job.
Kill	Terminate the job and its associated processes.
Log	View log messages for the job.
Reactivate	[OS/390] Reactivate post-processing for the selected job.
Rerun	Rerun the job.
Restart	[OS/390] Restart the job using CONTROL-R.
Statistics	View statistics of the job's latest runs.
Undelete	Undelete the deleted job.
View JCL/Script	View the job's JCL or job script statements.
Sysout	View the job's Sysouts.
Why	Determine why the job has not yet been executed.

Menu options can be disabled in specific situations.

For example:

•  The Why option is disabled when the job is executing or has completed.
•  The Hold option is disabled when the job is already held.
•  All options (excluding the Properties option) are disabled when communication is temporarily not synchronized.
•  When any one of these actions is selected, CONTROL-M/EM accesses and/or updates the relevant CONTROL-M installation's database.

Before updating a job's details, you must HOLD the job. After updating, you can FREE the job.

Read more »

Order / Force using Control-M / EM GUI

Order/Force a Job:

Click the Icon on the Toolbar or select Tools -> Order/Force a Job (Cntrl+R is the shortcut key)

You will get Order/Force dialog box. You have to select Control-M (Data center) and Table from the dropdown list to display all the jobs in that table. Then you can select the job which you want to order/force job.

Difference between Order and Force:

• Order: will bring the in to AJF if the job is in schedule.
• Force: will bring the job into AJF irrespective of the schedule.

Note: If you would like to bring the job with HOLD then you need to select ‘Order/Force with Hold’ option.

Read more »

Control-M Viewpoints

Viewpoints

Batch flows at most sites exist in a distributed environment spanning many computers of different types. CONTROL-M provides a powerful graphic user interface that allows you to view, monitor, and control all enterprise-wide batch flows from a single console, called the CONTROL-M/Enterprise Manager console (sometimes referred to as CONTROL-M/EM GUI or CONTROL-M/EM).

Using CONTROL-M/EM, you can see the progress of your batch flows, and identify problems and potential problems. CONTROL-M/EM provides a customizable, dynamic tool called ViewPoints to allow you to display only those jobs and job flows of interest. For example, you can view and monitor:

• Jobs on a particular CONTROL-M
• Jobs belonging to a specific user
• Jobs having a particular status, such as failed jobs

ViewPoints are constantly updated and show in real-time the execution status of the batch production. CONTROL-M provides a set of predefined ViewPoints. You can define other ViewPoints according to need.

Opening a Viewpoint:

Click on File/Open Viewpoint in the Control-M/Enterprise Manager window to load the list of viewpoints. You can have more then one viewpoint open at a time.

Each window can use a different Viewpoint to display the environment.

• Click on “All Active Jobs”/your desired viewpoint and then click the OK button.
• Click on “NOTOK Only”/Your desired viewpoint and then click the OK button

In this example two viewpoints have been opened. One viewpoint shows “All Active Jobs”. The other viewpoint shows “Ended NOTOK” jobs only.

You can open as many viewpoints as you wish and click on the Window/Tile and the Window/Cascade menu options to rearrange the view. You can simply click and drag the viewpoint windows around to the desired place. All viewpoints will default to show at the datacenter level.

The CONTROL-M/Enterprise Manager window consists of the following parts:

1. Navigation Tree – Hierarchical view of jobs in the current environment.


2. Flow Diagram – Graphic representation of job production flow, based on job dependencies established by prerequisite conditions specified in job processing definitions. Components of the active environment are represented by boxes called nodes. The information displayed in a node varies depending on the type of node and on how certain customization options.


3. Net Overview – Miniature version of the flow diagram, indicating the part of the network currently displayed in the Flow Diagram view. By ragging the selected area in this view to a different part of the network, you can quickly navigate in a complex environment.

Understanding the flow diagram:

In the flow diagram, components of the environment (definition environment or active environment) are represented by node boxes. The information displayed in a node varies depending on the type of node and on customization options.

Read more »

Control-M GUI's

GUI’s:

          The main 3 GUI’s are listed below.

ü  Control-M / EM

ü  Control-M Desktop

ü  Control-M Configuration Manager

Control-M / EM:

This is a GUI where we can view all the jobs which are running for today. The jobs which are active for today is called ‘Active Job File (AJF)’.

Control-M Desktop

Using this GUI you can make all permanent changes to scheduling tables and calendars. You can create new jobs, tables and calendars using Control-M desktop. The changes which you are performing through this will be permanently stored in control-M server database.

Control-M Configuration Manager

Through this GUI you can monitor all control-m components (Control-M server, agents, remote hosts… etc) health status. You can also implement additional security using this GUI. We can define agents, Remote Hosts through this GUI.  

Read more »