Pluggable Authentication Modules (macOS PAM)

Overview

This document describes how to protect PAM-authenticated services on macOS with RCDevs OpenOTP.

Unlike the Linux integration, this macOS integration does not configure NSS, SpanKey, POSIX LDAP extensions, home-directory creation, or user provisioning. The macOS user account must already exist and must already be resolvable by macOS Directory Services / OpenDirectory. The account may be a local account, an iCloud-backed local account, a mobile account, an LDAP account, or an Active Directory account.

OpenOTP is added only at the PAM authentication layer. macOS remains responsible for account lookup, account state, shell, home directory, sudo authorization, Remote Login access lists, and directory-service integration.

Scope and limitations

This guide applies to macOS services that use PAM, such as:

• OpenSSH server: /etc/pam.d/sshd
• sudo: /etc/pam.d/sudo and, on newer macOS versions, /etc/pam.d/sudo_local
• su: /etc/pam.d/su
• other third-party PAM-aware services that provide their own file in /etc/pam.d/

This guide does not cover:

• macOS account creation
• LDAP or Active Directory binding
• home-directory creation
• FileVault pre-boot authentication
• recoveryOS authentication
• Apple Account / iCloud enrollment
• granting sudo privileges

Note that some services configured with PAM authentication may not support OTP challenge mode. In this case, OTP challenge mode must be disabled at the Client Policy level on the WebADM server.

RCDevs does not recommend enabling PAM authentication for the screensaver, login, or any other services unless you fully understand the potential impact.

Prerequisites

Before configuring PAM OpenOTP, confirm that:

1. A working WebADM / OpenOTP infrastructure is available.
2. The macOS host can reach the OpenOTP service URL over HTTPS port 8443.
3. The target macOS users already exist on the Mac or are resolvable through macOS Directory Services.
4. The corresponding users exist in WebADM and are allowed by the relevant client policy.
5. You have an administrator account and a local recovery method before editing PAM files.

Keep one administrator shell open while testing. A broken PAM configuration can prevent sudo, SSH, or console authentication from working.

Verify account resolution on macOS

PAM OpenOTP does not create users. The operating system must already resolve the account.

Check the user with:

id <username>

admin@admins-Mac-mini pam.d % id admin
uid=501(admin) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),702(com.apple.sharepoint.group.2)

For network users, LDAP users, or Active Directory users, the macOS local username must match an account in a directory configured in WebADM.

Install the PAM OpenOTP plugin

First, download the pam_openotp-x.x.x.pkg from RCDevs website and copy it on you macOS machine.

Then, to install the package, the command look like :

sudo installer -pkg /path/to/pam_openotp-1.0.19.pkg -target /

root@admins-Mac-mini Documents # sudo installer -pkg pam_openotp-1.0.19.pkg -target /
installer: Package name is pam_openotp-1.0.19
installer: Installing at base path /
installer: The install was successful.

Configure the PAM OpenOTP client

The OpenOTP PAM configuration file is located in :

/etc/openotp/openotp.conf

The default configuration file looks like below :

#
# OpenOTP Client Configurations
#

# OpenOTP SOAP service URL(s). This is the only mandatory setting.
# Two server URLs can be configured and separated by a comma inside the same quoted string.
# When two servers are configured, you may choose a request routing policy below.
server_url "https://<server>:8443/openotp/"
#server_url1 "https://<server1>:8443/openotp/"
#server_url2 "https://<server2>:8443/openotp/"

# Request routing policy when two server URLs are defined.
# Ordered: First server is preferred (default). When down, second server is used.
# Balanced: Server is chosen randomly. When down, the other is used.
# Consistent: One specific user ID is always routed to the same server (per user routing).
#server_policy "Ordered"

# Domain name
#domain_name "MyDomain"

# Client ID
#client_id "MacOS"

# Challenge suffix
challenge_suffix ": "

# User settings
#user_settings "OpenOTP.KeyExpire=10"

# Client certificate
#cert_file "openotp.pem"
#cert_password "password"

# Trusted CA (WebADM CA certificate)
# Copy the WebADM CA certificate file in /etc/openotp/ca.crt and set the ca_file to enforce SSL server trust.
#ca_file "ca.crt"

# OpenOTP API key
# Wen OpenOTP is configured with 'Required Client Certificate or API Key', you need to create an
# API key in WebADM and set its value here for communicating with the OpenOTP API.
#api_key "MY_API_KEY"

# SOAP timeout
# This is the SOAP request TCP timeout. The default timeout is 30 seconds.
#soap_timeout 30

You must configure at least the OpenOTP service URL but in this example few settings as been configured like client_id and ca_file :

#
# OpenOTP Client Configurations
#

# OpenOTP SOAP service URL(s). This is the only mandatory setting.
# Two server URLs can be configured and separated by a comma inside the same quoted string.
# When two servers are configured, you may choose a request routing policy below.
#server_url "https://<server>:8443/openotp/"

server_url1 "https://webadm1.rcdevsdocs.com:8443/openotp/"
server_url2 "https://webadm2.rcdevsdocs.com:8443/openotp/"

# Request routing policy when two server URLs are defined.
# Ordered: First server is preferred (default). When down, second server is used.
# Balanced: Server is chosen randomly. When down, the other is used.
# Consistent: One specific user ID is always routed to the same server (per user routing).
#server_policy "Ordered"

# Domain name
#domain_name "MyDomain"

# Client ID
client_id "macOS-PAM"

# Challenge suffix
challenge_suffix ": "

# User settings
#user_settings "OpenOTP.KeyExpire=10"

# Client certificate
#cert_file "openotp.pem"
#cert_password "password"

# Trusted CA (WebADM CA certificate)
# Copy the WebADM CA certificate file in /etc/openotp/ca.crt and set the ca_file to enforce SSL server trust.
ca_file "ca.crt"

# OpenOTP API key
# Wen OpenOTP is configured with 'Required Client Certificate or API Key', you need to create an
# API key in WebADM and set its value here for communicating with the OpenOTP API.
#api_key "MY_API_KEY"

# SOAP timeout
# This is the SOAP request TCP timeout. The default timeout is 30 seconds.
#soap_timeout 30

You can optionnaly configure a client authentication with an API key or a client certificate.

The CA file can be downloaded from https://<webadm>/cacert and located /etc/openotp/ folder.

root@admins-Mac-mini openotp # pwd
/etc/openotp

root@admins-Mac-mini openotp # curl -k https://webadm1.rcdevsdocs.com/cacert -o ca.crt

root@admins-Mac-mini openotp # ls -al 
total 24
drwxr-xr-x   5 root  wheel   160 May 21 11:21 .
drwxr-xr-x  80 root  wheel  2560 May 21 10:03 ..
-rw-r--r--   1 root  wheel  1991 May 21 11:21 ca.crt
-rw-r--r--   1 root  wheel  1844 May 21 10:08 openotp.conf
-rw-r--r--   1 root  wheel  1830 May 20 19:02 openotp.conf.default
root@admins-Mac-mini openotp # 

Protect sshd

Configure SSH to use PAM keyboard-interactive authentication

Back up the SSH server configuration:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rcdevs.bak.$(date +%Y%m%d%H%M%S)

Edit /etc/ssh/sshd_config:

sudo vi /etc/ssh/sshd_config

Recommended settings for password + OpenOTP challenge through PAM:

UsePAM yes
KbdInteractiveAuthentication yes
PasswordAuthentication yes

PAM configuration file for sshd

Back up the PAM file:

sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.rcdevs.bak.$(date +%Y%m%d%H%M%S)

Edit the file:

sudo vi /etc/pam.d/sshd

Default file look like this on macOS

# sshd: auth account password session
auth       optional       pam_krb5.so use_kcminit
auth       optional       pam_ntlm.so try_first_pass
auth       optional       pam_mount.so try_first_pass
auth       required       pam_opendirectory.so try_first_pass

account    required       pam_nologin.so
account    required       pam_sacl.so sacl_service=ssh
account    required       pam_opendirectory.so
password   required       pam_opendirectory.so
session    required       pam_launchd.so
session    optional       pam_mount.so

I'm adding the PAM instruction after the last auth line like below :

# sshd: auth account password session
auth       optional       pam_krb5.so use_kcminit
auth       optional       pam_ntlm.so try_first_pass
auth       optional       pam_mount.so try_first_pass
auth       required       pam_opendirectory.so try_first_pass
auth       required       pam_openotp.so

account    required       pam_nologin.so
account    required       pam_sacl.so sacl_service=ssh
account    required       pam_opendirectory.so
password   required       pam_opendirectory.so
session    required       pam_launchd.so
session    optional       pam_mount.so

Keep the existing macOS auth, account and session lines. They are used for macOS account validation, Remote Login access control, and session setup.

Protect sudo

PAM OpenOTP does not grant sudo rights. The user must already be allowed to run sudo through local admin membership, /etc/sudoers, /etc/sudoers.d/, or an enterprise policy.

On the tested version of macOS, the sudo PAM configuration file is /etc/pam.d/sudo.

Back up /etc/pam.d/sudo:

sudo cp /etc/pam.d/su /etc/pam.d/sudo.rcdevs.bak.$(date +%Y%m%d%H%M%S)

Edit the file:

sudo vi /etc/pam.d/sudo

The default file looks like this:

# sudo: auth account password session
auth       include        sudo_local
auth       sufficient     pam_smartcard.so
auth       required       pam_opendirectory.so
account    required       pam_permit.so
password   required       pam_deny.so
session    required       pam_permit.so

The modified file, which includes OpenOTP authentication, looks like this:

# sudo: auth account password session
auth       include        sudo_local
auth       sufficient     pam_smartcard.so
auth       required       pam_opendirectory.so
auth       required       pam_openotp.so client_id=sudo
account    required       pam_permit.so
password   required       pam_deny.so
session    required       pam_permit.so

Note: In this example, a Client ID is configured directly in the PAM configuration file. This overrides the default client_id in /etc/openotp/openotp.conf, allowing a different Client Policy to be applied based on the PAM service being called.

Protect su

Back up /etc/pam.d/su:

sudo cp /etc/pam.d/su /etc/pam.d/su.rcdevs.bak.$(date +%Y%m%d%H%M%S)

Edit the file:

sudo vi /etc/pam.d/su

The default file looks like this:

# su: auth account session
auth       sufficient     pam_rootok.so
auth       required       pam_opendirectory.so
account    required       pam_group.so no_warn group=admin,wheel ruser root_only fail_safe
account    required       pam_opendirectory.so no_check_shell
password   required       pam_opendirectory.so
session    required       pam_launchd.so

The modified file, which includes OpenOTP authentication, looks like this:

# su: auth account session
auth       sufficient     pam_rootok.so
auth       required       pam_opendirectory.so
auth       required       pam_openotp.so client_id=su
account    required       pam_group.so no_warn group=admin,wheel ruser root_only fail_safe
account    required       pam_opendirectory.so no_check_shell
password   required       pam_opendirectory.so
session    required       pam_launchd.so

Note: In this example, a Client ID is configured directly in the PAM configuration file. This overrides the default client_id in /etc/openotp/openotp.conf, allowing a different Client Policy to be applied based on the PAM service being called.

Test:

su - <username>

Client policies

A client policy can be configured globally based on the client_id configuration in /etc/openotp/openotp.conf, or per macOS service. A custom Client ID can be configured in the OpenOTP PAM client configuration, as shown in the previous example.

Client policies can restrict which users or groups may authenticate, which networks are allowed, which MFA methods are accepted, and whether challenge mode is supported.

Refer to Policies & Conditional Access documentation to create and configure your client policies. For PAM services that do not support OTP challenge mode, disable it in the Client Policy configuration.

Uninstall macOS PAM plugin

Before removing the PAM-macOS plugin, ensure that you remove the OpenOTP library from your PAM configuration file or restore the backup file.
You can then execute the following command to completely remove the plugin.

# 1. Verify the installed files
pkgutil --files com.rcdevs.pam-openotp

# 2. Remove the PAM module
sudo rm -f /usr/local/lib/pam/pam_openotp.so

# 3. Remove empty directories if they are now unused
sudo rmdir /usr/local/lib/pam 2>/dev/null
sudo rmdir /usr/local/lib 2>/dev/null

# 4. Remove the package receipt
sudo pkgutil --forget com.rcdevs.pam-openotp

# 5. Remove configuration folder
sudo rm -rf /etc/openotp/

# 5. Verify removal
pkgutil --pkgs | grep -i openotp
ls -l /usr/local/lib/pam/

Troubleshooting

For authentication issues on the OpenOTP server side, consult /opt/webadm/logs/webadm.log.

For issues on macOS, you can use a command similar to the following from the command line:

log stream | grep -E "sshd-session|kernel|openotp|pam"