IT team achieves collaborative automation with Rundeck.

Before Rundeck

This Fortune 500 financial services group is a leading insurance, banking and investment and retirement solutions provider. They have offices in seven U.S. cities, three overseas locations, and employ more than 35,000 people worldwide.

Meet the Rundeck administrator and an Infrastructure Engineer Manager at this company. As they scale their operations, his team is on a quest to automate “all-of-the-things” at the IT department. 

The technical footprint this team helps support is vast, running on more than 20,000 Linux servers in different datacenters around the globe. They’re responsible for administering dev, test, and production environments, and developing the automation to manage these environments more efficiently.

Why Rundeck?

When their package installation tool, HP Server Automation, went out of support, the infrastructure team decided to look for a replacement and settled on Rundeck. Rundeck had the flexibility to implement their workflow for automatically installing differently configured packages to their Linux servers.

This is when this Infrastructure Engineer Manager first became acquainted with Rundeck -- when he took over responsibility for managing the package installation process.

Automation in a Highly Regulated Industry

As an insurance and financial services company, their operations are highly regulated, and include compliance requirements for IT operations. All change management actions must be approved by a due diligence team ahead of time, and are then executed by their IT services partner after business hours. 

Before automating with Rundeck, these actions were highly labor intensive. They were submitted to the due diligence team as a script of manual logins and tasks to be implemented by their service provider after approval.  Many of these procedures became automated workflows in Rundeck that the service provider could call to gather system information or execute system changes.  As a result, the infrastructure team also improved security as secrets such as login credentials and SSH keys do not need to be widely shared. In line with best security practices, access to the workflows in Rundeck are controlled by user roles, and all actions are logged by Rundeck in addition to server logs.

The Infrastructure team also automated workflows that validated the results of the work by the service partner. Before they would have to manually log into servers to access server logs and configuration information to see if maintenance operations were completed successfully.  Now, they can run Rundeck jobs that collect this data automatically, giving back an hour each day to the engineers.

Next the team turned their attention to automating responses to alerts with Rundeck.  The goal was to reduce distractions to operators by creating workflows that would automatically grab diagnostic system information in response to common alerts.  For example, they utilize Sensu as a monitoring tool which sends alerts to Rundeck.  Rundeck reports the alert to their ITSM tool, and in known cases, is able to execute a workflow to remediate the issue automatically without having to escalate to a human operator.  When the case does require intervention from an operator, their Rundeck-powered remediation playbook provides an arsenal of automation workflows that the engineer can invoke from the UI to resolve the issue.

Rundeck Usage Continues to Spread 

When the Infrastructure team began sharing the results they were seeing using Rundeck, they soon had requests from other teams to help improve their automation as well. In one case, another team had a need to enable mass configuration changes across their systems.  In the past, this would have been a labor intensive process completed after business hours.  By automating the change with Rundeck, it became possible to implement these configuration changes alongside regular maintenance automation, saving considerable engineering time and disruption.  The team was so pleased with the results, they also began automating other tasks with Rundeck.

The Infrastructure team continues to automate more processes with Rundeck.  Their next project is speeding up auditing of user activity in systems.