TL:DR

How to quickly configure a fresh system. A quick introduction to Ansible and automation of personal setup for macOS and Arch Linux. Installation of apps, config files, and secrets.

The problem

Setting up a fresh system might be a little cumbersome. You already have your perfect setup. Everything is installed, everything is configured.

Whether you got a new computer, are reinstalling due to an upgrade, or are just setting up a virtual machine, it is time-consuming to set up everything just as you like.

It is not only time-consuming, but it is also easy to forget some things or just forget how to configure something.

How can we solve that problem?

Solutions

There are a few ways to handle this situation.

Dotfiles

The simplest way is to copy config files and store them somewhere, for example, in a git repository. A lot of people store their dotfiles in a repository. We can make some README or other notes to remember how to set it up as we like.

This was also my first approach to this. But wouldn't it be better to automate it?

Bash

I've created a lot of bash scripts to set up everything as I wanted, and it was working just fine. But it was not so good at maintenance.

So, looking for something better, I gave Ansible a try.

Ansible

This is simple to use, easy to maintain, and gives a lot of possibilities to automate virtually anything. So what it is and how to use it?

Introduction to ansible

What is it?

Ansible is software that enables automation and orchestration. It can automate virtually any task.

Ansible is an open source, command-line IT automation software application written in Python. It can configure systems, deploy software, and orchestrate advanced workflows to support application deployment, system updates, and more. ~ ansible.com

It is designed around the following principles:

• agent-less architecture - low maintenance;

• simplicity - simple YAML syntax, using SSH to connect to the machines;

• scalability and flexibility - easily and quickly scale through modular design;

• idempotence and predictability - when the system is in the desired state, it will not change even if the playbook is run multiple times;

How to install

Ansible can be installed using Python and pip. Or, for some systems like Arch Linux and macOS, we may use package managers.

For Arch Linux using yay:

yay -S ansible

For MacOS using brew:

brew install ansible

For more detailed information about installation look here: documentation.

Basics

Inventory

Ansible needs to know which machines it will manage. To achieve that, all we need is a simple inventory file in ini or yaml format.

[webservers]
192.168.1.11
192.168.1.12

[databases]
192.168.1.21
192.168.1.22

or

webservers:
  hosts:
    app: 192.168.1.11
    app2: 192.168.1.12
databases:
  hosts:
    app: 192.168.1.21
    app2: 192.168.1.22

Inventory defines the managed nodes and groups of them. In the above example, we have two groups - webservers and databases. Each group has two IP addresses. By default, Ansible also creates two other groups - all and ungrouped. The first one contains all hosts, and the second one includes all hosts not grouped under anything other than all.

There are some more things that we can set in the inventory, like the user that we are using:

webservers:
  hosts:
    app:
      ansible_host: 192.168.1.11
      ansible_user: app_user

There is more, and we can read about this in the official documentation.

Task

A task is a single automation, like installing a package or creating a directory.

Create task

A task can be created in a separate yaml file or directly in a playbook.

- name: Hello world task
  ansible.builtin.debug:
    msg: "Hello world!"

Modules

Modules or plugins are units that are run from the command line or tasks. Ansible executes each module, usually on a managed node (host), and collects the return value from it. Modules can be collected into collections. We can use built-in modules, modules from the community, or create our own.

Modules can be called in tasks, as in the above task example.

To use the command line:

ansible home -m debug -a "msg=Hello!" -i inventory.yml

home is a group of hosts from the inventory that we specified with -i inventory.yml

This should result in:

pc | SUCCESS => {
    "msg": "Hello!"
}

pc is a host from the inventory.

Collection

Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. They are useful when using something that is not built-in, created by someone else.

To install collections, we can use ansible-galaxy and a requirements file.

Let's say that we need modules to install packages from aur and brew. To do this, we need community.general and kewlfft.aur.

With a requirements.yml file:

collections:
  - community.general
  - kewlfft.aur

We can use galaxy:

ansible-galaxy install -r requirements.yml

It will install all collections from the file, and they will be available for use.

Role

A structured way to organize playbooks into reusable components. It contains tasks, variables, files, templates, and more.

Roles go into the roles directory and have a specified structure. Below we have an example from documentation:

roles/
    common/               # this hierarchy represents a "role"
        tasks/            #
            main.yml      #  <-- tasks file can include smaller files if warranted
        handlers/         #
            main.yml      #  <-- handlers file
        templates/        #  <-- files for use with the template resource
            ntp.conf.j2   #  <------- templates end in .j2
        files/            #
            bar.txt       #  <-- files for use with the copy resource
            foo.sh        #  <-- script files for use with the script resource
        vars/             #
            main.yml      #  <-- variables associated with this role
        defaults/         #
            main.yml      #  <-- default lower priority variables for this role
        meta/             #
            main.yml      #  <-- role dependencies
        library/          # roles can also include custom modules
        module_utils/     # roles can also include custom module_utils
        lookup_plugins/   # or other types of plugins, like lookup in this case

By default, when using a role, Ansible will look for a main.yml (or main.yaml, or main) file.

However, it is still possible to have multiple files there and organize everything according to needs.

Play

This is a section within a playbook that defines what to run and on which host group.

- name: Simple play
  hosts: webservers
  roles:
    - common
  tasks:
    - name: Hello
      ansible.builtin.debug:
        msg: Hello!

This is one play that will run the role common and one task.

Playbook

A playbook is a blueprint for automation. It is a simple configuration for what to run on nodes specified in the inventory.

Each playbook is built from plays that are run from top to bottom.

Facts

Data related to the remote system Ansible manages is available as facts. To access those variables, use ansible_facts or ansible_{name}.

How to use

Let's build a simple example that can be run locally.

Inventory

To achieve this, we need to set up the inventory correctly.

Create inventory.yml:

home:
  hosts:
    pc:
      ansible_host: localhost
      ansible_user: isur
      ansible_connection: local

I have created a group home with the pc host that has some settings:

• ansible_host - host - we want to use it locally, so localhost

• ansible_user- this is the user that will be used on a machine - set it for your user

• ansible_connection: local - we do not want to use an SSH connection to our own system

Running playbook

To run a playbook, use the command ansible-playbook with the path to the playbook as an argument. To select the inventory, use -i:

ansible-playbook play.yml -i inventory.yml

Before running this command, first create a playbook.

Simple playbook

Create playbook.yml:

- name: Simple play
  hosts: home
  tasks:
    - name: Hello user
      ansible.builtin.debug:
        msg: "Hello {{ ansible_user }}!"

Ansible has a built-in debug module that lets you print out messages. We can use variables with templating like above. ansible_user is a fact with information about the username.

Below you can see the result of that task.

TASK [Hello user] *******************************************************************
ok: [pc] => {
    "msg": "Hello isur!"
}

That might be all we need for a simple playbook. But when we have more tasks it might be better to move them to files and import them later.

- name: Simple task imported
  hosts: home
  tasks:
    - ansible.builtin.import_tasks: ./tasks/hello.yml

That way, we can group some tasks or even import tasks into other tasks.

But there is an even better way to organize automation.

With roles

Instead of grouping tasks into random directories, we can use roles.

Let's move the hello task into a role, and now the file structure should look like this:

./
    inventory.yml
    play.yml
    roles/
        hello/
            tasks/
                main.yml

And instead of importing tasks, we can use roles:

- name: Play with roles
  hosts: home
  roles:
    - hello

main.yml in the hello role will look the same as any other task, but it will be easier to maintain and extend. Now we can add something more to it, so we will have some initial information when running the playbook.

- name: Hello user
  ansible.builtin.debug:
    msg: "Hello {{ ansible_user }}!"

- name: Hello system
  ansible.builtin.debug:
    msg: "System: {{ ansible_os_family }}"

Now, after running this, we can see which user and what system is used:

TASK [hello : Hello user] ******************************************************
ok: [pc] => {
    "msg": "Hello isur!"
}

TASK [hello : Hello system] ****************************************************
ok: [pc] => {
    "msg": "System: Archlinux"
}

Those facts ansible_user and ansible_os_family might be useful when we want to do something depending on the operating system or username. We might also use environment variables with ansible_env.

Personal system setup

Goal

The goal is to prepare automation to handle two systems - Arch Linux and macOS. On both of them, I want to install apps, create directories, copy files, create symlinks for configs, and decrypt secrets (ssh).

Plan

To keep everything organized, I will use roles for different groups of tasks. The only requirement is to have Ansible and yay or brew installed, depending on the system.

Collections

Let's start with the required collections. As I am using Arch Linux and yay, I will use kewlfft.aur, and for macOS and brew, community.general. So, requirements.yml will have this:

collections:
  - community.general
  - kewlfft.aur

Inventory

For the localhost machine, the inventory.yml file will be simple. Remember to use the correct user and include ansible_connection: local, so Ansible will not use SSH.

home:
  hosts:
    pc:
      ansible_host: localhost
      ansible_user: isur
      ansible_connection: local

Playbook

The playbook will just load roles. For better organization, I am splitting all tasks into a few roles:

• general - some general stuff that I will always install

• tiling - setting up tiling managers and the system for that usage

• dev - everything I need for software development work

• gaming - some gaming stuff

So the playbook.yml file will look like this:

- name: System Setup
  hosts: home
  roles:
    - general
    - tiling 
    - dev
    - gaming

Roles

Each role will be prepared for both systems, Arch Linux and macOS. The Ansible fact ansible_os_family for Arch is Archlinux and for macOS is Darwin. With when and this fact, we can detect which task to run on which system.

I will not cover my entire setup here, but I will give some examples of how to do some tasks.

Installation role

Let's install discord as an example. First, we need to create a discord role in the roles directory and a main.yml task.

roles/discord/tasks/main.yml

And in this task, we need to remember both systems.

- name: Darwin | Install Discord from brew
  community.general.homebrew_cask:
    name: discord
    state: present
    accept_external_apps: true
  when: ansible_os_family == 'Darwin'

- name: Arch | Install Discord from aur
  kewlfft.aur.aur:
    name: discord
    use: yay
    state: present
  when: ansible_os_family == 'Archlinux'

And that's it. For Arch, it will install discord using yay, and for Mac, it will use brew. If the app is already installed on Mac in a different way, accept_external_apps will not raise an error in that case.

If tasks are more complex, we can split up the file into different files per system.

- name: Darwin | Install Discord
  ansible.builtin.include_tasks: "./darwin.yml"
  when: ansible_os_family == 'Darwin'

- name: Arch | Include Linux specific tasks
  ansible.builtin.include_tasks: "./arch.yml"
  when: ansible_os_family == 'Archlinux'

And put the tasks defined earlier into arch.yml and darwin.yml.

This will load all tasks from the file specified in ansible.builtin.include_tasks.

The file tree right now would look like this:

.
├── collections.yml
├── inventory.yml
├── play.yml
└── roles
    └── discord
        └── tasks
            ├── arch.yml
            ├── darwin.yml
            └── main.yml

This is a simple installation without any additional steps. It might be a little too much work to create a role for each app if we just want to install a few things without anything extra.

We can do this in one role, even in one task, using loops. For example, let's create a role communication which will install apps for communication: discord, slack, and thunderbird.

The system selection task will look exactly the same as above.

- name: Arch | Install communication apps
  kewlfft.aur.aur:
    name: "{{ app }}"
    use: yay
    state: latest
  loop:
    - discord
    - slack-desktop
    - thunderbird
  loop_control:
    loop_var: app

Now name instead of the name of the app is the item from the loop. We can define how the variable is named in loop_control and loop_var. Without defining the variable name, it will be item. This loop will run for each item specified under loop.

We can also move those items into variables.

To do this, we need to create a directory vars in the role and main.yml.

arch_apps:
  - discord
  - slack-desktop
  - thunderbird

darwin_apps:
  - discord
  - slack
  - thunderbird

Now instead of passing a list of items, we pass a variable:

- name: Arch | Install communication apps
  kewlfft.aur.aur:
    name: "{{ app }}"
    use: yay
    state: latest
  loop: "{{ arch_apps }}"
  loop_control:
    loop_var: app

This way, instead of creating multiple roles for a simple installation process, we can just do it like this.

.
├── collections.yml
├── inventory.yml
├── play.yml
└── roles
    ├── communication
    │   ├── tasks
    │   │   ├── arch.yml
    │   │   ├── darwin.yml
    │   │   └── main.yml
    │   └── vars
    │       └── main.yml
    └── discord
        └── tasks
            ├── arch.yml
            ├── darwin.yml
            └── main.yml

There are different ways to organize this stuff; everything is up to you.

Symlink/copy files

What if I want to configure my apps and I have some dotfiles?

Let's configure the tiling manager. I am using i3 on Arch and aerospace on MacOS. We can create different roles for them, or just a role tiling and store everything there.

Inside the role directory, we will now also need a files directory where we store our config files.

Select the system as before and split arch and darwin into arch.yml and darwin.yml.

The file tree will now look like:

.
├── collections.yml
├── inventory.yml
├── play.yml
└── roles
    ├── communication
    │   ├── tasks
    │   │   ├── arch.yml
    │   │   └── main.yml
    │   └── vars
    │       └── main.yml
    ├── discord
    │   └── tasks
    │       ├── arch.yml
    │       ├── darwin.yml
    │       └── main.yml
    └── tiling
        ├── files
        │   ├── aerospace.toml
        │   └── i3
        │       └── config.conf
        └── tasks
            ├── arch.yml
            ├── darwin.yml
            └── main.yml

Tasks for installing tiling stuff look exactly like before with the communication role. But this time, we need some config files:

- name: Arch | Install tiling
  kewlfft.aur.aur:
    name: "{{ item }}"
    use: yay
    state: present
  loop:
    - i3-wm
    - rofi
    - feh

- name: Arch | Config tiling
  ansible.builtin.file:
    src: "{{ role_path }}/files/{{ item.src }}"
    dest: "{{ item.dest }}"
    state: link
  loop:
    - { src: "i3", dest: "{{ ansible_env.HOME }}/.config/i3" }
    - { src: "rofi", dest: "{{ ansible_env.HOME }}/.config/rofi" }

Using the ansible.builtin.file module, we can define where files should be. src points to the source for the file. The role_path variable points to the role path. dest is the destination, and state: link makes it a symbolic link.

This way, both ~/.config/i3 and ~/.config/rofi are symbolic links to the config in our Ansible files.

If we want to copy files instead of making symbolic links, we can use the ansible.builtin.copy module with src and dest. We can also define permissions by using mode with permission numbers.

Decrypt files

What if I tell you that we can store our secrets in a public repository?

For this example, I will set up SSH keys.

Let's create a role ssh and put our SSH key in the files directory.

The role directory will look like this:

    ├── ssh
    │   ├── files
    │   │   ├── id_rsa
    │   │   └── id_rsa.pub
    │   └── tasks
    │       └── main.yml

First, we need to encrypt the files. Go to the directory with the files and:

ansible-vault encrypt id_rsa id_rsa.pub

Set the password and confirm. Done, now the files are encrypted. Or, if you save your password to a file (e.g., in ~/.vault_pass):

ansible-vault encrypt id_rsa id_rsa.pub --vault-password-file=$HOME/.vault_pass

THIS PASSWORD IS SUPER SECRET, NEVER PUT IT INTO THE REPOSITORY.

encrypt will encrypt the file. You can also use edit, view, and decrypt commands.

Now the files are safely encrypted. We can use them in our task to copy to the correct place in our system.

Let's go to roles/ssh/tasks/main.yml.

- name: Ensure that ssh directory exists
  ansible.builtin.file:
    path: "{{ ansible_env.HOME }}/.ssh"
    state: directory
    mode: '700'

- name: Find SSH keys and config files
  ansible.builtin.find:
    paths: "{{ role_path }}/files"
    file_type: file
  register: found_files

- name: SSH Config
  ansible.builtin.copy:
    src: "{{ item.path }}"
    dest: "{{ ansible_env.HOME }}/.ssh/{{ item.path | basename }}"
    decrypt: true
    mode: '600'
  loop: "{{ found_files.files }}"

This will make sure that the ~/.ssh directory exists. Find all the files we want to copy and copy them to the correct place. What is important here:

• mode needs correct permissions so our key will work correctly;

• decrypt - will copy decrypted files;

But how do we pass the password to the task?

It can be done with a password file.

Run

If we have everything ready we can run this playbook.

First, install all required collections:

ansible-galaxy install -r requirements.yml

Now, to run the playbook:

ansible-playbook playbook.yml -i inventory.yml --vault-password-file=$HOME/.vault_pass

So now we have everything we wanted: installing apps, copying and symlinking files, creating directories, and decrypting secrets. Everything is set up in a way that we can use on multiple systems. If we use symlinks for configs and have some common configs between all systems, updating them in the repository will help us keep them on all machines.

If you would like to see my full setup: https://github.com/Isur/dotfiles