Control-M Latest Fixpacks for v8, v9 and v9.0.18

Thought of sharing this... because many people asked me about this.... Here are the latest Fixpacks available for different control-m versions as of Oct-2018.

Read more »

Control-M V9 Features

High Availability

Control-M high availability solution enables you to maximize your production environment uptime and prevent data loss in the event of hardware maintenance or failure.

In the case of hardware failure or if all Control-M processes are down unexpectedly, you can now perform either an automatic or manual failover for both Control-M/EM and Control-M/Server. After the problem is fixed, you can fallback to the primary host. This functionally applies to customers using an Oracle or MSSQL database for Control-M.

In the event of a hardware failure, when using Control-M/Server with PostgreSQL database, you can perform a manual failover to a secondary Control-M/Server, and later, when the problem is fixed, you can fallback to the primary Control-M/Server. In addition, this solution provides PostgreSQL database replication for Control-M/Server.

Automated Agent Deployment

Upgrade one or many agents to V9 and future versions including fix packs in just a few simple steps. You can choose between transferring the package to agents for later deployment, or automatically transferring and deploying in a single step.

Automated Client Deployment

The new Client Deployment utility replaces the WebLaunch tool in previous Control-M versions. Use the Client Deployment utility to easily distribute full client installation packages, fix packs and patches to users. You can set policies to force users to apply update packages by a specific date.

Promotion between environments

Promotion between environments is a new capability of the Control-M Workload Change Manager that enables you to move job definitions between environments. For example, from a development environment to a production environment. During the promotion process, job definitions are modified and validated according the Site Standards of the target environment.

Control-M configuration Manager (CCM) enhancements

The Control-M configuration Manager user interface has been updated to improve user experience. In addition, the following Control-M components are now managed via the tool: Control-M Database, Naming Server (NS), Configuration Management Server (CMS) as well as Control-M Workload Archiving.

Usage alerts

Keep an eye on your task usage with Usage Alerts, available from within the CCM. Define task usage alert thresholds across all of your EMs. Exceeding a threshold will send an email alert.

Security enhancements

Introducing the SSL certificate deployment capability that extends the packaging and deployment of SSL security certificates with the option to specify an external Certificate Authority.

Improved security algorithm.

SSL support for:

• AES suite ciphers (128 bit and 256 bit)

• SHA-2 hashing (on Enterprise Manager Client-Server communication)

• RSA keys with length 2048, 3072 and 4096

• Control-M Enterprise Manager password hashed using SHA-512

NFS support for Control-M/Server and Control-M Agent installation

You can now install Control-M/Server and Control-M/Agents on NFS.

IPv6 communication

IPv6 support, including support for mixed IPv4 and IPv6 environments, is now available for the following components:

• Control-M/Agent

• Control-M/Server

• Control-M Mainframe

• Control-M/Enterprise Management – Server, Clients

Note: Control-M Agentless is not supported at this configuration.

Sharing information between jobs

Ability to share information between jobs by extracting data from a job output and passing it to other jobs as a variable. Additional granularity of the variables is introduced as well, allowing to define variables in groups called “named pools” or as Smart folder variables that are unique per folder execution, in addition to global and local variables.

Runtime estimations

With Control-M Batch Impact Manager you can now see estimated runtimes of all the jobs in a regular viewpoint without them being defined as a part of a Batch Impact Manager service.

Multi-days job flows monitoring

Improved monitoring and troubleshooting for the job flows that span on multiple days. You can now configure the active environment retention period for a SMART folder so the SMART folder itself and its jobs are kept in the active environment until the end of the period.

Dynamic job ordering on a remote Control-M Server

Ability to order jobs on remote Control-M Servers by setting an action in a job to trigger another job on a different Control-M Server and also pass variables as a part of this order operation.

Control-M Event Management Rules in Workload Automation client

z/OS customers can now view, edit, and manage CMEM rules via the Control-M Workload Automation client.

What If Host simulation

New capability for Control-M Batch Impact Manager and Forecast to simulate the influence of the unavailability of a specific Control-M Agent or a group of Control-M Agents on your job flows, and prepare for planned shutdowns and maintenance windows.

Global calendars

You can now save time and reduce the efforts invested in calendar management by creating a single calendar in the Control-M/Enterprise Manager that will be distributed to all the Control-M/Servers.

Bookmarks

You can bookmark the jobs in a workspace or a viewpoint to analyze flows and easily locate your jobs. Bookmarks are supported in the Planning, Monitoring, Forecast, and History domains.

Export flow diagram to a file

Till now one of the biggest challenge is to export the graphical view. Its not anymore, now You can now export your flow diagram to a Microsoft Visio-compatible EMF format from any domain. Just click the “Save as” function in the application “File” menu. its that simple.

Enhanced search when loading jobs to a workspace

Now you can view more fine-tuned results when loading jobs into a workspace in the Planning domain by filtering folders and jobs with a larger variety of fields.

x

Read more »

Control-M Server Processes and their Duties/Activities

SU: The Supervisor (p_ctmsu):

• performs heartbeat monitoring of all processes 
• Before start of New Day, it suspends all the processes and then to itself 

SL: The Selector (p_ctmsl):

• Makes a selection loop to determine which job will be submitted for the execution to the OS 
• Performs partial jobs post-processing actions 
• Frees all resources occupied by jobs 

NS: The Network Server (p_ctmns):

• Manage the communication to all Control-M/Agents 
• When a message arrives from the Agent, NS sends a trigger to a process that will handle the request 
• Updates the Agent status file and notifies TR, SL, CD and CS processes by sending triggers 

TR: The Tracker (p_ctmtr):

• Updates jobs state when informed by the agent 
• Performs late execution shouts and DO STMT 

CO: Communication Gateway (p_ctmco):

• It handles two types requests- one from CD like Download and other from Control-M/EM, to be handled by CS like order, view 
• It analyzes requests and sends them to the appropriate destination 

CS: Communication Service (p_ctmcs):

• All requests to the CS pass through the CO process 
• Handles the requests that comes from Control-M/EM 
• Multiple number of CS processes can be set 

RT: Communication Router (p_ctmrt) 

• Manage the communication messages between all processes 
• Responsible for the messages to arrive in the correct order 

WD: The Watchdog (p_ctmwd):

• Monitors essential Control-M processes, resources and sends an appropriate alert when there is a problem 
• WD checks all the primary Control-M processes are functioning 

LG: The Logger Process:

• Logs all the details

CD: Communication Download Process (p_ctmcd):

• CD process checks if the New Day Time has arrived 
• It sends a message to SU to suspend all Control-M processes and the “STOP LINK” message to Control-M/EM 
• CD updates the IOALOG about New-Day Beginning and updates a Date in the Control-M server database 
• It sends the New Day message to all available agents for cleanup purpose 
• It cleans the jobs from Active Job File, log messages and statistics information 
• It orders jobs upon their scheduling criteria to the AJF 
• Once New day is completed, it sends the “START LINK” message to Control-M/EM and a message to CO to activate all the Control-M processes

Read more »

Tasks performed during New Day Process

CONTROL-M processes execute as background processes on the server computer and remain active at all times except during NDP. Once a day, at the time specified by the CONTROL-M system parameter Day Time (starting time of working day), CONTROL-M starts the New Day Procedure (formerly known as the “General Daily procedure”).

The New Day procedure performs automatic functions that start a new day under CONTROL-M. This procedure is used as a master scheduler for all CONTROL-M

activities.

The following actions were performed during NDP till v6.2.01, on later versions these were disabled and a new utility can be executed as a batch job during the day to perform the cleanup

-> Agents cleanup
-> Statistics cleanup

The New Day Procedure performs the following sequence of operations:

1. A new Odate (CONTROL-M date) is calculated (based on the system date and CONTROL-M system parameter Day Time).


2. A partial cleanup of the CONTROL-M/Server log is performed, based on system parameter Maximum Days Retained By CONTROL-M Log. 


3. A partial cleanup of job SYSOUT directories on agent computers is performed, based on system parameter Maximum Days to Retain Sysout Files.


4. A selective cleanup of the Active Jobs file is performed. Jobs that have already executed and ended OK, and jobs whose parameter Max Wait has been exceeded (and are not Held), are erased from the Active Jobs file.


5. A partial cleanup of the statistics in the CONTROL-M/Server database is performed. For each job, CONTROL-M retains statistical information regarding the last 20 successful executions. This value can be changed by defining the RUNINF_PURGE_LIMIT parameter in the CONTROL-M config.dat file. CONTROL-M must be shut down and restarted for this change to take effect.


6. A selective cleanup of prerequisite conditions is performed. Prerequisite conditions are deleted if their date reference (month and day) is equal to the new CONTROL-M date +1 (tomorrow). This prevents jobs from being triggered by prerequisite conditions remaining from the previous year.


7. Job orders are placed in the Active Jobs file according to job processing parameters contained in Scheduling tables (assigned to User Daily “SYSTEM”). These job orders can include the submission of User Daily jobs.


8. CONTROL-M begins downloading the new Active Jobs file to the CONTROL-M Configuration Manager. During download, CONTROL-M processes are suspended.

Read more »

What does it mean by "chmod 777 ." in Unix?

In general the below are the 5 values which we'll use for permissions  in unix.

• 0 - No Access
• 1 - Execute
• 2 - Write
• 4 - Read
• 7 - Full Access

Usually file permission field is a 3 digit number. First digit denotes the OWNER of the file/folder, second digit denotes GROUP, third digit denotes EVERYONE.

chmod 400 file - Read by owner 

chmod 040 file - Read by group 

chmod 004 file - Read by world 

chmod 200 file - Write by owner 

chmod 020 file - Write by group 

chmod 002 file - Write by world 

chmod 100 file - execute by owner 

chmod 010 file - execute by group 

chmod 001 file - execute by world 

chmod 111 file - Allow execute permission to owner and group and world 

chmod 222 file - Allow write  permission to owner and group and world 

chmod 444 file - Allow read permission to owner and group and world 

chmod 777 file - Allow everyone to read, write, and execute file 

You can also use them by adding them. For example if you would like to give Read & Write access then you can use 2+4 = 6. If you would like to give Read and Execute access then you can use 1+4 = 5.

Read more »

How to Add a Condition using SHOUT (POSTPROC)

As most of us know that we can add conditions using "Conditions" tab or through steps tab.  Normally we can add/remove conditions based on the below scenario's.

• When job Ended OK (using Conditions Tab)
• When the job Ended NOTOK (using Steps Tab)
• When a job failed with a specific return code (using Steps Tab)
• When you find a particular string in Log/Sysout (using Steps Tab)
• When Run Counter criteria met (using Steps Tab)

... there might be many other scenario's where we may need to add/remove conditions. few of them are When a job executing for longer time, or If the job not submitted by a particular time or If the job not completed by a certain time. In general we will trigger an ALERT in these scenario's using PostProc tab. Now I'm going to explain you how can we add a condition instead of generating an alert.

Like how we use LateTime/ExecTime/LateSub, we'll goto PostProc tab and select the particular event which you would like to check and add a condition.

A SHOUT WHEN EXECTIME > nnn (The range is 001 to 999 in minutes only) or 
A SHOUT WHEN LATE SUB > HHMM or 
A SHOUT WHEN LATE TIME > HHMM

If you want to Average time then use + and you can also use % (percentage) to limit above the average time.

Here is the example:

"condition" must be added in the shout destination table as a script i.e program as destination. You can name as you desire (something like add-condition.sh). The script should contain the below statement to add the condition.

ctmcontb –add $2

The ctmcontb utility performs different operations on prerequisite conditions in the Control-M/Server database (Like Add, Delete, List conditions).

In the same way you can also invoke any other utility depend upon the requirement.

Read more »

Control-M 8 Changes vs Earlier Versions

I've started working on BMC Control-M lastest version which is 8.0 recently. I had experience in Control-M scheduling for around 6.7 years in various versions (v6.1, v6.2, v6.3 and v6.4). Till now what I learnt is completely different in V8.0. It took couple of days for me to understand the new concepts and Envitonment. This is really awesome, very simple GUI interface and many options were simplified. I just thought of sharing few of my findings. so here is my 1st blog post on Control-m 8.

Parameter	Previously Known As
Host/Host Group	Node/Node group
Run as	Owner
Override Path	Override library
Variable	AutoEdit
Sub Application	Group
Created by	Author
Folder	Table
Order Method	User daily
Output	Sysout

Come back to this website for more info....

Read more »

Why statistics are not correctly shown in the Control-M/EM GUI?

"ctmjsa" is used to collect statistics of all Control-M jobs.

Generally calculation of statistics can be done with Jobname or Memname. 

This is determined by the Operational Parameter Statistics Mode.

When it is set to Memname the statistics won't be shown right.

Change Statistics Mode to Jobname, recycle Control-M, run ctmjsa again, and now view Statistics of any job from Control-M EM GUI. You can see the difference.

Read more »

How can you say if the agent for a particular job is available?

While there is a continuous connection between CONTROL-M EM and Control-M/EM Server, the communication between CONTROL-M/Server and CONTROL-M/Agent is not constant.

CONTROL-M/Server is doing the following:

1. It sends a TRACK_ALL message for each agent that has running jobs every 'Polling Interval' (the default is 15 minutes).

2. There is no indication that an agent becomes unavailable unless there was a request to that agent.

While the agent waits for its time window it does not check if the agent is available. This will be checked only when the job will be submitted.

3. You can check the Agent status using Control-M Configuration Manager.

What can I do if my agents become unavailable sometimes?

Solution: Run ctm_menu:

(5) Parameter Customization -> (4) Parameter Customization for Specific Agent Platform:

Communication Timeout (default 120)

Maximum Retries (default 12)

Increase the timeout parameters for the agent platforms. The same can be achieved using Control-M Configuration Manager. You can open Agent properties and change the parameters accordingly.

Read more »

How do I check communication between Server and Agent?

From the Agent run:

   ag_diag_comm

From the Server run:

   ctm_diag_comm <Agent host name>

Compare the results. Specially check the following:

1. Server-to-Agent Port Number is the same on both outputs
2. Agent -to-Server Port Number is the same on both outputs
3. Server-Agent Comm. Protocol is the same on both outputs
4. Unix ping to Agent Platform, ctm_ping to agent, and Unix Ping to Server Platform all Succeeded
5. Service "<agent service>" is Active
6. Server Host Name, in ag_diag_comm output, is correct.

Read more »