An Ansible Journey 15: Templates

My final post will walk through the process of using a template to deliver customized files that may vary across hosts, operating systems, applications and so on. The principle is as in the prior posts, not overly difficult. It consists of:

1. Creating the template file(s)
2. Creating the variables with changing values to insert in those template files
3. Adding the play to do the work
   
4. Restarting any services once the change is made.

Creating the Template Files

For this exercise, we will be customizing two sshd_config files, one on Rocky Linux and one on Ubuntu. The main difference is in the content of each file. The default Rocky Linux file will not work with Ubuntu, and vice versa. 

We start by creating a templates directory under playbooks/roles/base. In that directory, we need to drop a copy of both versions of the files and give them a name that indicates to which OS they belong:

[ansadm@rocky-1 ansible]$ ls -l playbooks/roles/base/templates/
total 12
-rw-rw-r--. 1 ansadm ansadm 4297 Apr 26 16:11 sshd_config_rocky.j2
-rw-rw-r--. 1 ansadm ansadm 3282 Apr 26 16:11 sshd_config_ubuntu.j2

Ansible uses the Jinja2 templating system to manage templates. Jinja2 files use a .j2 extension.

Next, we will add the same directive to both. The portion of the sshd_config file that we are editing is identical in both versions, so I'll show that just once here:

[...]
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

AllowUsers {{ ssh_users }}
[...]

We're adding the AllowUsers directive which expects a list of usernames, separated by spaces. Since this list could change from one host to another, we'll use a variable to store the individual user names.

Variables

In each of the host variable files, we'll add some additional variables to include a list of users, the name of the template files we created above, and the sshd service name:

Rocky Linux Files:

ssh_users: "ansadm rlohman"
ssh_template_file: sshd_config_rocky.j2
ssh_service_name: sshd

Ubuntu File:

ssh_users: "ansadm rlohman"
ssh_template_file: sshd_config_ubuntu.j2
ssh_service_name: ssh

Now, just prior to copying the files out, the placeholder variable in the templates will be swapped with the ssh_users string. The ssh_template_file variable will be used to select the appropriate file to use as a template.

Adding the Play

in the playbooks/roles/base/tasks/main.yml file, we'll add the following play to execute the task:

- name: Generate sshd_config File from Template
  tags: ssh
  template:
    src: "{{ ssh_template_file }}"
    dest: /etc/ssh/sshd_config
    owner: root
    group: root
    mode: 0644
  notify: restart_sshd

There's nothing here we have not already done. The src parameter accepts a variable in place of the actual template file to use. The ownership and permissions are set according to what the configuration file has in the /etc/ssh directory.

Service Restarts

There is one additional item that we'll need to account for. the service name between the two is also different. Ubuntu uses ssh.service, whereas Rocky uses sshd.service. That is handled by our last variable, ssh_service_name, in the restart_sshd handler. We again use the notify directive to call the handler once the change is registered.

And with that, our configuration for sshd is now templatized (read, quickly and easily customized) for our servers.

<PREV - CONTENTS

An Ansible Journney 14: Host Variables and Handlers

This post covers the use of host variables and handlers. these offer a way to reduce the lines of code used to specify an infrastructure, and to extend the functionality of a play or task.

Host Variables

Host variables are defined in a host_vars directory that resides in your playbooks directory. Each file must be named exactly as the hostname (or IP address/FQDN) in your inventory file. In my host_vars directory, I have three files:

[ansadm@rocky-1 ansible]$ ll playbooks/host_vars/
total 12
-rw-rw-r--. 1 ansadm ansadm 91 Apr 26 15:18 192.168.5.250.yml
-rw-rw-r--. 1 ansadm ansadm 87 Apr 26 12:48 192.168.5.251.yml
-rw-rw-r--. 1 ansadm ansadm 88 Apr 26 12:51 192.168.6.30.yml
[ansadm@rocky-1 ansible]$

Although these files all have a .yml extension, the use of "---" on the first line is optional. Within that file, we specify each variable using a key/value pair syntax:

[ansadm@rocky-1 ansible]$ cat playbooks/host_vars/192.168.5.250.yml
---
nginx_package_name: nginx
nginx_service_name: nginx
extra_package_name: pcp-pmda-nginx

Once your variables file is created, you can reference the individual variables using Ansible's variable syntax:

"{{ variable-name }}"

In our web_servers role, our first task instructs the dnf module to install two packages, nginx and pcp-pmda-nginx:

- name: Install Nginx Web Service
  tags: nginx,web
  dnf:
    name:
      - nginx
      - pcp-pmda-nginx
    state: latest
 

with the host variables file shown above, we can obtain the same result referencing the variable names:

- name: Install Nginx Web Service
  tags: nginx,web
  dnf:
    name:
      - "{{ nginx_package_name }}"
      - "{{ extra_package_name }}"
    state: latest

Upon execution, Ansible will replace "{{ nginx_service_name }}" with "nginx."

In a case where the same variable may have a different value across different operating systems, we could specify the same variable name with different values in each host variable file, and Ansible will perform the correct replacement. Thus, we've reduced the side of the play, and extended it to work in more situations.

Handlers

Handlers offer a way to reuse task functionality. In an earlier post, we built a restart_nginx play that checked variable, nginx_conf, to see if it registered a change. If so, then we called the restart Nginx play using a when clause:

when: nginx_conf.changed

This works well, but there may be other reasons to restart our web server that have nothing to do with the configuration file. Without a handler, we would need to write a second restart Nginx play that is called when a different variable registers the change.

To create a handler, we need to start with a directory under the role of our choice called handlers:

playbooks/
    roles/
        web_servers/
            handlers/
                main.yml

As you can see, our handler, like our task book is stored in a file called main.yml. The code for our restart Nginx handler is:

- name: restart_nginx
  service:
    name: "{{ nginx_service_name }}"
    state: restarted

Going back to the play that calls the handler, we'll replace the register: nginx_conf line with a notify directive that calls the handler by name:

 - name: Enable HTTPS in Nginx
  tags: nginx,web
  lineinfile:
    path: "/etc/nginx/nginx.conf"
    line: "        listen       *:443 ssl;"
    insertafter: 'listen       80'
    firstmatch: true
    backup: true
  notify: restart_nginx

The advantage here is that we can call the handler from any code in the web_server role.

 

<PREV - CONTENTS - NEXT>

An Ansible Journey 13: Roles

Roles provide the user will separating out different blocks of code for ease of reading. In this context can be thought of as the role that the target server plays in the environment. They allow you to move code into other files, keeping the playbook cleaner. Fortunately, using roles doesn't restrict you from categorizing work; you can still have, for example, a role called web_servers and still differentiate between say, a web server that is public-facing and a web server for internal use (which may have relaxed security requirements).

In one of the earlier videos, Jay split up his playbook into two two: the bootstrap.yml file and the site.yml. Roles takes this a step further.

Creating roles is very straight-forward, requiring only two general modifications to a playbook with all tasks in one file: directory structure changes and moving code blocks to a new file or files.

Directory Changes

Roles imposes a standard for using roles in your filesystem. In general, no directory is required unless there are files in that directory. Here is an example:

opt/
    ansible/
        inventory/
        playbooks/
            site.yml
            roles/
                base/
                files_servers/
                    tasks/
                        main.yml
                web_servers/
                    files/
                        landing_page.yml
                    tasks/
                        main.yml

Under playbooks we create a directory called roles. That contains a directory for each server role we are defining. The base role is one that would likely apply to all servers, containing such items as the implementation of organizational security policies, support team configurations, etc. Within each role, we have at a minimum, a tasks directory. We can optionally add additional directories, like files, if there are files that we will be transferring to our target servers. Finally, within each tasks directory, we have a file called main.yml. The directory corresponding to a role is required, as is the tasks directory, and the main.yml file.

Code Changes

In the previous model, we created roles by using the [role-name] construct in our playbook:

playbook.yml file
    [base]
        (plays intended for all hosts)
    [web_servers]
        (plays intended for all web servers)
    [db_servers]
        (plays intended for all database servers)
    [file_servers]
    ...

The playbook.yml file will have all plays removed, with each set of plays being defined in its respective main.yml file underneath tasks:

playbook.yml file:

- hosts: all
  become: true
  pre_tasks:
    (any plays that will be run on any server, with each run.*)

- hosts: all
  become: true
  roles:
    - <rolename>
...

main.yml file

- name: Install Nginx Web Service
  tags: nginx,ubuntu,web
  apt:
    name:
      - nginx-core
      - nginx-doc
    state: latest
  when: ansible_distribution == "Ubuntu"

- name: Enable HTTPS in Nginx - Rocky
  tags: nginx,rocky,web
  lineinfile:
    path: "/etc/nginx/nginx.conf"
    line: "        listen       *:443 ssl;"
    insertafter: 'listen       80'
    firstmatch: true
    backup: true
  when: ansible_distribution == "Rocky"
  register: nginx_conf
... 

The playbook.yml file retains each section it did before, but we replace the plays with:

  roles:
    - <rolename>

Then, we take the plays that were in the playbook.yml file and place them into the main.yml file. We'll retain the tags so we can continue to narrow the scope of each run during testing, quick patch activity, etc. One thing to note that is not clear in the example is that whereas the plays are indented two spaces in the playbook.yml file, they start on the left (column 1). This is due to the YAML requirement that all code begin on the left. Note that the first line of our playbook.yml file (---) is not required in the main.yml files.

The main.yml file (and any other collection of plays) in the tasks directory are referred to as "taskbooks."

 

<PREV - CONTENTS - NEXT>

An Ansible Journey 12: Managing Users

Starting Notes

First, I ran across a missing Ansible collection [of modules] while working through this video. Specifically, the authorized_key module was missing. After a short search, I was able to drop the collection on the server with the following command:

ansible-galaxy collection install ansible.posix

Once done, I could deploy the keys.

Next, the author added a line in this video:

changed_when: false

to two of the commands in the bootstrap.yml file. I received errors from Ansible when adding this line, so I removed it. By watching the video, you can see the impact of removing this line, which is rather trivial.

Finally, I've rewritten the entire access and permissions parts. The method employed by the author is good for setting up a small lab, but in my case, I tend to work from a large enterprise perspective, and the manual work required to prepare an Ansible user account simply does not scale well to hundreds or thousands of servers.

User Management

User management is actually very straight forward, and the plays are rather easy to understand. We will start by creating an application user account. It is customary to run applications with service accounts that have restricted permissions. In this case, we are creating a user account to run the billing application. The application and user are part of the accounting group, so we will first be sure that the accounting group exists.

- name: Ensure Group "accounting" Exists
  group:
    name: accounting
    state: present
    gid: 5006

- name: Ensure User svc_billing Exists on Servers
  tags: always
  user:
    name: svc_billing
    groups: accounting
    shell: /sbin/nologin
    state: present

The accounting group in this case is called, well, accounting. We want it to be present, and we manage group ids also, so we'll want to specify that gid.

Next we create our service account, svc_billing. We use groups to force that account to be a member of the accounting group. Since this is a service account, we don't expect anyone to actually use it for a login, so we disable logins for that account.

Now we'll move on to an administrative user account. That account will need extra authority to execute commands. Since permissions are maintained in our fictitious company at the group level, we need to be sure the group is available first, and that is has the appropriate sudo permissions.

 - name: Ensure Group "support" Exists
  tags: always
  group:
    name: support
    state: present
    gid: 5007

- name: Ensure sudoers File is in Place
  tags: always
  copy:
    src: /opt/ansible/files/sudoers/sudoer_support
    dest: /etc/sudoers.d/support
    owner: root
    group: root

For this to work, we'll need to create the file to copy over with the actual sudo entry. The path and filename are above. The content consists of a single line:

%support ALL=(ALL) NOPASSWD: ALL

Now that we are confident our group will be available, we can create our user account:

 - name: Ensure Admin User Account jdoe is Available
  tags: always
  user:
    name: jdoe
    groups: support
    state: present

- name: Add SSH key for User jdoe
  tags: always
  authorized_key:
    user: jdoe
    key: "ssh-rsa AAAAC3dsfSDF9a79kll0[...]REsd9fdfkjRGLDSFGJ jdoe"

Here, we create the jdoe user, assign and tag the account as a system account. We also assign them to the support group. We also take their public key and add to the authorized_key files in any accounts that are created. We assume jdoe has just become part of the organization, and issued the ssh-keygen command to create their own public/private key pair, and pasted their public key in the play.

 

<PREV - CONTENTS - NEXT>

 

 

An Ansible Journey 11: Managing Services and Editing Files

The primary purpose of this post is to demonstrate how to manage systemctl services. This is useful when deploying a new service and when modifying a service's configuration. Since we need some kind of change that will trigger Ansible to restart the service when done, we also introduce making a change to a text file.

Restarting a Service

The block required for service management is short and not at all complicated:

  - name: Start Nginx Service
    tags: nginx,web,rocky
    service:
      name: nginx
      state: started
      enabled: true
 

We start with name and tags directives, then call the service module. We pass into that module the name of the service as a name directive, and the state (action) we want. Note that just as on the command line, the ".service" is not required at the end of the service name. If you are familiar with systemd, you'll know that services must be both started (so the application runs) and enabled (so it can be started the next time the server boots).

If the service is currently running, and we need to make a change to that service, in many cases, the service must be restarted to pick up the change. When restarting, we don't need to specify the 'enabled' directive. We do, however, need a trigger for the restart. The following can replace the last two lines of the example above:

       state: restarted
    when: nginx_conf.changed

The when directive should be lined up with the 'service:' module line. The when directive in the last line checks the value of a variable, nginx_conf to see if it has been set. If so, the state directive above it will be triggered. We'll discuss how to set that variable below. One final word about the nginx_conf variable before we get there: It is not uncommon to make multiple changes to a file at the same time. When Ansible inspects this variable, it has only one value, and that value reflects the last change that was made. So, if you attempt two different changes to the file at once, the first change is committed, and the second change is not required to be committed (due to impotency), Ansible will not trigger the restart. This is because as mentioned above, the variable only reflects the last change attempt.

Editing a File

We switch now to editing files. If you are familiar with sed and regex expressions, you will find the lineinfile module familiar. It contains many similar constructs, plus many more. The following will modify an existing line in an existing file:

  - name: Enable HTTPS in Nginx - Rocky
    tags: nginx,rocky,web
    lineinfile:
      path: "/etc/httpd/conf.d/httpd.conf"
      regexp: '^ServerAdmin'
      line: "ServerAdmin rlohman@erehwon.net"
    when: ansible_distribution == "Rocky"
    register: httpd_conf

In this  example, we start out by calling the lineinfile module. That will require three parameters: path, regexp, and line. The path parameter tells us in which file the change will be made. The regexp command tells Ansible to search for the string "ServerAdmin," but only when that string starts at the first character in the line (so a line that has one or more spaces in front of Serveradmin will be ignored). Finally, the line parameter instructs Ansible what to change that line to.  The example above was not used in the site.yml file; it is strictly an example to illustrate how to specify changing a line. As such, I'll over the register directive below in the block that I actually used.

Because maintaining configurations is so common, I'll add provide another with a different approach:

  - name: Enable HTTPS in Nginx - Rocky
    tags: nginx,rocky,web
    lineinfile:
      path: "/etc/nginx/nginx.conf"
      line: "        listen       *:443 ssl;"
      insertafter: 'listen       80'
      firstmatch: true
      backup: true
    when: ansible_distribution == "Rocky"
    register: nginx_conf

In this block, we're adding a new line (and new capability) to our configuration file for Nginx. We continue to use the lineinfile module, as well as the path and line directives. These all work identically to those above. In this case, we're modifying our nginx.conf file to allow it to accept HTTPS requests (over port 443) in addition to the standard unencrypted port, 80. I wanted to add this configuration instruction immediately after the one that tells Nginx to open port 80, so I perform a regex search on the line I'm looking for that contains "listen       80" and because I searched using the insertafter directive, my line will be inserted on the next line below the one I searched for. This directive has a complement, insertbefore which works similarly.

The firstmatch directive tells ansible to only add the line after the first occurrence of "listen       80." This is handy as sometimes we may edit a file in which our search string appears more than once (not the case, here, as our search string will only appear once in the nginx.conf file).

The backup directive tells Ansible to make a backup copy of the file if and only if it is going to modify that file. If the desired change is already done, no backup will be created.

This takes us to a new directive, register. Register creates a variable called nginx_conf. It also will modify the value of that variable to indicate that a change was made if the file was modified. This is the variable that we inspected above with this directive:

when: nginx_conf.changed  

This nginx_conf.changed will evaluate to true if a change was made to the file, and in the desired action (restarting the service) will be triggered.

A final note on this example: Your landing page will likely still not be accessible over the HTTPS port, 443, when following this tutorial. This is because Nginx needs to know what certificate to use when processing inbound SSL/TLS connections. We have not specified that so the browser will give an error. That said, running the playbook after adding these plays will still trigger Ansible to restart nginx (which is, after all, the goal of this post).

A Word About Cruft

In the sysadmin world, cruft can be a problem. Cruft is considered anything that remains after some work was performed that isn't necessary for the server or service to operate. In the post on Managing Files (#10), I noted that the .zip file we decompressed was no longer on the server, even though Ansible pulled it down from a website. Had that file been left behind, it would be considered "cruft." In addition, when we made the change above, adding the https configuration line, Ansible left that file sitting in the same directory as the configuration file we changed. Ansible left that file behind as we may need it to revert the change if we inadvertently broke something with our change. That said, it too, is considered cruft.

Cruft can also be a line in a file. For example, we often make small changes to directives in files, by copying a line we want to change, commenting out the original line, and then making the change to the un-commented line. The commented line, just like the backup file, is cruft.

Cruft is generally considered a bad thing in systems administration. It can populate directories with numerous backup files that make looking for a specific file more difficult. Likewise, a configuration file with many commented versions of different configuration parameters can be difficult to read, and may even cause errors when performing a regex search as a prerequisite to committing a change.

There are a number of strategies to keep cruft at a minimum, and they should be employed by the sysadmin. Ansible is a proper tool to use, as is wrapper scripts that may perform some cleanup after the playbook has completed successfully.

My experience teaches me that backup/saved data in files and on disk tend to lose their value after the next change or two (that is, if you need to roll back to a state that has gone through multiple changes, it is more often than not, likely that the rollback would require so many other fixes, that only a good configuration management system will aid in the rollback.

<PREV - CONTENTS - NEXT>

An Ansible Journey 10: Managing Files

This post deals with two methods of managing files with Ansible. We will perform a copy operation, and a pull a compressed file from the internet, and deploy it to rocky-1 (our Ansible host). 

1. Deploy a file to an Ansible Target

We will add the following play inside our hosts: webservers block:

  - name: Deploy Website Landing Page
    tags: nginx,web,rocky
    copy:
      src: /opt/repository/web/landing_page.html
      dest: /usr/share/nginx/html/index.html
      owner: root
      group: root
      mode: 0644
 

The copy module is used to copy files. When configuring this section, you will want to set the src, dest, owner, group, and mode directives to ensure you're pulling the correct file, and when you place it, it has the correct ownership and permissions to be read by Nginx. Not in the example that the source and destination do not need to be the same filename. 

2. Install a Utility from a Compressed Archive

First, our ansible host is not in our inventory file. Even though this is where Ansible runs, it cannot act as an ansible target unless it is in the inventory file. We simply add the following to the end of the inventory file:

[workstations]
192.168.5.249

Next, we'll update our site.yml file to perform the actions we want. Here are the changes:

- hosts: workstations
  become: true
  tasks:

  - name: Install unzip Package
    tags: workstations
    package:
      name: unzip

  - name: Install TerraForm
    tags: workstations
    unarchive:
      src: https://releases.hashicorp.com/terraform/1.11.4/terraform_1.11.4_linux_amd64.zip
      dest: /usr/local/bin
      remote_src: yes
      mode: 0755
      owner: root
      group: root

The first section (in blue) starts a new section that applies only to workstations. (There's nothing special here with respect to a workstation versus a server. The host I'm, rocky-1 on is a Rocky Linux server, but I'm also working on it as if it were my workstation.)

The Install unzip Package section (in orange) ensures we have the unzip package available, as the package we're installing is actually a .zip file. Like the samba package earlier in the tutorial,  the unzip package has the same name, so we can use the package directive instead of dnf. One important note here. If you are referring back to the learnlinux.tv videos, you'll notice that the author does not add the "tags: workstations" line to this block. I expect there was a change in Ansible functionality between the time his videos were recorded and this tutorial was written, as I needed to add this directive in order to get the block to be executed.

The final section (in green) pulls the TerraForm file from Hashicorp's website and decompresses it into the /usr/local/bin directory. It is the unarchive module that manages selecting the utility (unzip) to use. Once complete, you can (as root) issue an updatedb command on the Ansible host followed by:

locate terraform_1.11.4_linux_amd64.zip

to confirm that there is no .zip file leftover after the block completes. The remote_src: directive tells Ansible that the file we are pulling does not exist on the local host. (Note that it was not required in the copy module above, as the source file to be copied was on the same host as Ansible.) One additional note regarding the "hosts: workstations" block. It must be added either at the bottom of the file, or directly above another hosts block. Adding this hosts section in between two unrelated plays from another hosts section will cause errors because Ansible will think that you want the actions to take place in the parent host section. Bottom line: don't try to nest hosts blocks.

 

<PREV - CONTENTS - NEXT>

Ansible Journey 9: Tags

Ansible tags allow you to run specific portions of your playbook, ignoring others. Since each invocation of ansible-playbook against a playbook requires multiple checks to determine whether or not a play should attempt to exectue, running an entire playbook will require more time than just running a portion of that playbook. Tags allow us to target a specific play or task. This can save significant time, particularly during testing, which may result in numerous executions to test individual changes.

We specify tags using the ‘tags’ directive:

tags: tag_1,tag_2...

To see what tags have been applied to a playbook, use ansible-playbook’s ‘--tags’ paraameter:

ansible-playbook --tags web --ask-become-pass playbooks/site.yml

or

ansible-playbook --tags “web,db” --ask-become-pass playbooks/site.yml

Here are two sections of our site.yaml playbook that apply tags:

- hosts: all
  become: true
  pre_tasks:

  - name: Install Updates - Rocky
    tags: always
    dnf:
      update_only: true
      update_cache: true
[...]

 - hosts: web_servers
  become: true
  tasks:

  - name: Install Nginx Web Service
    tags: nginx,rocky,web
    dnf:
      name:
        - nginx
        - pcp-pmda-nginx
      state: latest
    when: ansible_distribution == "Rocky"

The 'always' tag is a special tag that instructs Ansible to always run a play or task, regardless of whether it is specified. It has a complement, "never." These can both be overridden with "--skip-tags."

 

<PREV - CONTENTS - NEXT>