We were recently faced with the challenge of implementing a centralized directory service for our AWS infrastructure. Having to sync multiple backends for our numerous applications that required authentication was getting cumbersome. Additionally, it was also necessary for us to support managing SSH keys and altering sudo permissions on running instances, as re-deploying infrastructure or performing a configuration management run (in our case, Chef) every time a key changed or a new hire joined had became too tedious. We researched numerous options, but ultimately decided on a combination of Amazon services and new features in Ubuntu 14.04 and CentOS 6.x to achieve our goals. This document will guide you through how and why we made the decisions we did, and how you can duplicate our setup in your own environment. We implemented our setup using Ubuntu 14.04 and CentOS 6.6; however, this cookbook likely supports other Linux distributions.
The first step in implementing a solution was to decide on a backend LDAP-compatible directory. Initially, we considered using OpenLDAP server, but that plan was met with many hesitations. Particularly, we were worried about the amount of work required to expand on the existing OpenLDAP Chef cookbook to meet our needs. Also, we generally prefer to utilize "as a service" tools as often as possible, and this was no exception: maintaining production OpenLDAP infrastructure, including replication, high-availability, and backups, was a responsibility our development team was hopeful to avoid. Eventually, we decided to move forward With Amazon's Directory Service. Although geared as a replacement for Microsoft's Active Directory and therefore targeted for Windows instances, a Simple AD instance is still LDAP (actually, Samba) under the hood. Fortunately, it met all of our requirements for a directory service and was an obvious best choice, albeit with a few limitations which we'll get into in a bit.
From past experience, we were well-aware of the directory integration features of pam and nss with pam_ldap and nss_ldap, respectively, along with nscld. However, these modules have historically been troublesome and often-times not well documented. Furthermore, caching support was non-existent and an outage of LDAP connectivity would leave servers inaccessible and/or unreliable. Enter SSSD, the centralized access point for all authentication and authorization requests for pam, nss, sudo, and more. Additionally, it supports numerous backends, such as LDAP, Active Directory, and FreeIPA, with enhanced caching support to reduce strain on directory servers and provide relief in a service outage. Also, a built-in sss_ssh_authorizedkeys utility allows for the searching of an LDAP backend to retrieve SSH keys. For our setup, SSSD was a no-brainer, especially because Ubuntu 14.04 LTS and CentOS 6.x ship with a recent version that included the enhanced features we need!
While SSSD provides a mechanism for fetching SSH keys from LDAP, OpenSSH still needs to be able to read and trust those keys as if they were in the usual location (.ssh/authorized_keys). A few years ago, a patch to OpenSSH was circulating that added an "AuthorizedKeysCommand" configuration parameter that specified a command string to execute that would return keys for a given user. CentOS 6.x received this patch. Fortunately, a similar patch finally made its way upstream, and as of Ubuntu 14.04 is available as part of the OpenSSH package. No more re-deploying configuration management or running complicated scripts just to replace SSH keys!
The SSSD service allows us to centralize password authentication, public/private key authentication, host access, and sudo roles and access. Unfortunately, the Simple AD service (as of this time) doesn't support SSL connections, which is a requirement for utilizing the LDAP identity provider inside SSSD. We were fortunate enough to be able to work around this problem by utilizing a combination of the AD and LDAP provider backends. As luck would have it, the sss_ssh_authorizedkeys script previously mentioned ignores any backend configuration and will simply use any ldap_* configuration if present, without the requirement of SSL.
To begin, spin up an Amazon Simple AD instance. You'll need to load some custom schema files. First, create sudo.ldif with the following contents, being sure to replace the Base DN references with the Base DN of your directory server (in our examples, we're using example.com):
dn: CN=sudoUser,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.1
schemaIDGUID:: JrGcaKpnoU+0s+HgeFjAbg==
cn: sudoUser
name: sudoUser
lDAPDisplayName: sudoUser
adminDisplayName: sudoUser
adminDescription: User(s) who may run sudo
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
searchFlags: 1
dn: CN=sudoHost,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.2
schemaIDGUID:: d0TTjg+Y6U28g/Y+ns2k4w==
cn: sudoHost
name: sudoHost
lDAPDisplayName: sudoHost
adminDisplayName: sudoHost
adminDescription: Host(s) who may run sudo
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
dn: CN=sudoCommand,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.3
schemaIDGUID:: D6QR4P5UyUen3RGYJCHCPg==
cn: sudoCommand
name: sudoCommand
lDAPDisplayName: sudoCommand
adminDisplayName: sudoCommand
adminDescription: Command(s) to be executed by sudo
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
dn: CN=sudoRunAs,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.4
schemaIDGUID:: CP98mCQTyUKKxGrQeM80hQ==
cn: sudoRunAs
name: sudoRunAs
lDAPDisplayName: sudoRunAs
adminDisplayName: sudoRunAs
adminDescription: User(s) impersonated by sudo (deprecated)
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
dn: CN=sudoOption,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.5
schemaIDGUID:: ojaPzBBlAEmsvrHxQctLnA==
cn: sudoOption
name: sudoOption
lDAPDisplayName: sudoOption
adminDisplayName: sudoOption
adminDescription: Option(s) followed by sudo
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
dn: CN=sudoRunAsUser,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.6
schemaIDGUID:: 9C52yPYd3RG3jMR2VtiVkw==
cn: sudoRunAsUser
name: sudoRunAsUser
lDAPDisplayName: sudoRunAsUser
adminDisplayName: sudoRunAsUser
adminDescription: User(s) impersonated by sudo
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
dn: CN=sudoRunAsGroup,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.7
schemaIDGUID:: xJhSt/Yd3RGJPTB1VtiVkw==
cn: sudoRunAsGroup
name: sudoRunAsGroup
lDAPDisplayName: sudoRunAsGroup
adminDisplayName: sudoRunAsGroup
adminDescription: Groups(s) impersonated by sudo
attributeSyntax: 2.5.5.5
isSingleValued: FALSE
showInAdvancedViewOnly: TRUE
oMSyntax: 22
dn: CN=sudoNotBefore,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.8
schemaIDGUID:: dm1HnRfY4RGf4gopYYhwmw==
cn: sudoNotBefore
name: sudoNotBefore
lDAPDisplayName: sudoNotBefore
adminDisplayName: sudoNotBefore
adminDescription: Start of time interval for which the entry is valid
attributeSyntax: 2.5.5.11
isSingleValued: TRUE
showInAdvancedViewOnly: TRUE
oMSyntax: 24
dn: CN=sudoNotAfter,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.9
schemaIDGUID:: OAr/pBfY4RG9dBIpYYhwmw==
cn: sudoNotAfter
name: sudoNotAfter
lDAPDisplayName: sudoNotAfter
adminDisplayName: sudoNotAfter
adminDescription: End of time interval for which the entry is valid
attributeSyntax: 2.5.5.11
isSingleValued: TRUE
showInAdvancedViewOnly: TRUE
oMSyntax: 24
dn: CN=sudoOrder,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.15953.9.1.10
schemaIDGUID:: 0J8yrRfY4RGIYBUpYYhwmw==
cn: sudoOrder
name: sudoOrder
lDAPDisplayName: sudoOrder
adminDisplayName: sudoOrder
adminDescription: an integer to order the sudoRole entries
attributeSyntax: 2.5.5.9
isSingleValued: TRUE
showInAdvancedViewOnly: TRUE
oMSyntax: 2
dn: CN=sudoRole,CN=Schema,CN=Configuration,DC=example,DC=com
objectClass: top
objectClass: classSchema
governsID: 1.3.6.1.4.1.15953.9.2.1
schemaIDGUID:: SQn432lnZ0+ukbdh3+gN3w==
cn: sudoRole
name: sudoRole
lDAPDisplayName: sudoRole
possSuperiors: container
possSuperiors: top
subClassOf: top
mayContain: sudoCommand
mayContain: sudoHost
mayContain: sudoOption
mayContain: sudoRunAs
mayContain: sudoRunAsUser
mayContain: sudoRunAsGroup
mayContain: sudoUser
mayContain: sudoNotBefore
mayContain: sudoNotAfter
mayContain: sudoOrder
showInAdvancedViewOnly: FALSE
adminDisplayName: sudoRole
adminDescription: Sudoer Entries
objectClassCategory: 1
systemOnly: FALSE
defaultObjectCategory: CN=sudoRole,CN=Schema,CN=Configuration,DC=example,DC=com
Then, create ssh.ldif with the following contents, being sure to replace the Base DN references with the Base DN of your directory server (ie: dc=example,dc=com):
dn: CN=sshPublicKey,CN=Schema,CN=Configuration,DC=example,DC=com
changetype: add
objectClass: top
objectClass: attributeSchema
attributeID: 1.3.6.1.4.1.1466.115.121.1.40
cn: sshPublicKey
name: sshPublicKey
lDAPDisplayName: sshPublicKey
description: Users public SSH key
attributeSyntax: 2.5.5.5
oMSyntax: 22
isSingleValued: FALSE
dn: CN=ldapPublicKey,CN=Schema,CN=Configuration,DC=example,DC=com
changetype: add
objectClass: top
objectClass: classSchema
governsID: 1.3.6.1.4.1.1466.115.121.1.40
cn: ldapPublicKey
name: ldapPublicKey
lDAPDisplayName: ldapPublicKey
description: Used to store SSH keys in LDAP
subClassOf: top
objectClassCategory: 3
mayContain: sshPublicKey
mayContain: uid
Next, create sudoers.ldif with SIMILAR contents to the following, being sure to replace the Base DN references with the Base DN of your directory server (ie: dc=example,dc=com). This example LDIF creates a root sudo role and gives all users access to that role:
dn: CN=Sudoers,DC=example,DC=com
objectClass: top
objectClass: container
cn: Sudoers
name: Sudoers
description: Default container for Sudoers configuration
distinguishedName: CN=Sudoers,DC=example,DC=com
# sudo defaults
dn: cn=defaults,CN=Sudoers,DC=example,DC=com
objectClass: top
objectClass: sudoRole
cn: defaults
description: Default sudoOptions go here
sudoOption: env_keep+=SSH_AUTH_SOCK
dn: CN=root,CN=Sudoers,DC=example,DC=com
objectClass: top
objectClass: sudoRole
cn: root
sudoHost: ALL
sudoCommand: ALL
sudoUser: ALL
To load these LDIFS, execute the following, being sure to replace place-holders with proper values. If you get an error that --user is an invalid option, you'll want to make sure you have the samba package installed alongside ldb-tools:
ldbadd -H "ldap://example.com" sudo.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"
ldbadd -H "ldap://example.com" ssh.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"
ldbadd -H "ldap://example.com" sudoers.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"Now, create a user that will be used to test that the setup is functioning correctly. For our example below (users.ldif), we have "tuser":
dn: CN=Test User,CN=Users,DC=example,DC=com
cn: Test User
sn: User
givenName: Test
displayName: Test User
name: Test User
sAMAccountName: tuser
userPrincipalName: [email protected]
instanceType: 4
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
accountExpires: 9223372036854775807
logonCount: 0
pwdLastSet: 130598522940000000
lockoutTime: 0
userAccountControl: 66048
msDS-SupportedEncryptionTypes: 0
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
And import:
ldbadd -H "ldap://example.com" users.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"Optionally, utlizing a host that has samba-tool installed, set a password for the new account, being sure to replace place-holders with proper values:
samba-tool user setpassword --newpassword "<password>" -H "ldap://example.com" --user "<Admin Account Username>" --password "<Admin Account Password>" tuserYou must also grant authenticated users read access to the CN=Sudoers container that was created via previous LDIFs. Again, using samba-tool against each AD server (as this will not replicate):
samba-tool dsacl set -H "ldap://198.51.100.10" --user "<Admin Account Username>" --password "<Admin Account Password>" --objectdn="CN=Sudoers,DC=example,DC=com" --sddl="(A;CI;GR;;;AU)"
samba-tool dsacl set -H "ldap://203.0.113.20" --user "<Admin Account Username>" --password "<Admin Account Password>" --objectdn="CN=Sudoers,DC=example,DC=com" --sddl="(A;CI;GR;;;AU)"Now add an SSH key to the test user you've created. Be sure to replace the Base DN references with the Base DN of your directory server (again, as an example: dc=example,dc=com) and populating a valid public SSH key:
dn: CN=Test User,CN=Users,DC=example,DC=com
changeType: modify
add: objectClass
objectClass: ldapPublicKey
dn: cn=Test User,CN=Users,DC=example,DC=com
changeType: modify
add: sshPublicKey
sshPublicKey: ssh-rsa .... [email protected]
Apply the LDIF to your directory server:
ldbmodify -H "ldap://example.com" add-key.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"Your Amazon Simple AD instance should now be properly configured. Next, we'll configure SSSD & OpenSSH on a test instance and tie it into the directory service.
First, spin up a vanilla Ubuntu 14.04 LTS instance, switch to a root shell, and edit /etc/resolv.conf to contain something simliar to the following, being sure to replace the search domain with your AD domain and the nameserver IP addresses with the IPs of your Simple AD servers:
search example.com
nameserver 198.51.100.10
nameserver 203.0.113.20
To test connectivity between your instance and Simple AD, execute:
realm listYou should see information about your Simple AD domain. Next, execute the following commands, being sure to replace the username and password references with valid administrator credentials for your directory server and the domain references with your valid AD domain:
apt-get -y update
apt-get -y install sssd sssd-ad sssd-ad-common sssd-tools realmd samba-common samba-libs samba-common-bin
realm join -U <Admin Account Username> example.comNOTE: Encountering a package error during realm join? Check out https://bugs.launchpad.net/ubuntu/+source/realmd/+bug/1333694. We've worked around the problem by temporarily using adcli in our Chef cookbook.
Your server should successfully be joined to the domain. Unfortunately, realm doesn't give us a completely usable SSSD configuration file for our setup, so it's necessary to overwrite /etc/sssd/sssd.conf with the following, being sure to replace any example.com references with your actual configuration:
[sssd]
config_file_version = 2
services = nss, pam, ssh, sudo
domains = example.com
[nss]
filter_users = root,named,avahi,haldaemon,dbus,radiusd,news,nscd,centos,ubuntu
[pam]
[ssh]
[sudo]
[domain/example.com]
id_provider = ad
access_provider = ad
ad_domain = example.com
krb5_realm = EXAMPLE.COM
realmd_tags = manages-system joined-with-samba
cache_credentials = True
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
fallback_homedir = /home/%u
ldap_user_ssh_public_key = sshPublicKey
Finally, add the following lines to /etc/ssh/sshd_config, which will tell OpenSSH to look for SSH keys via SSSD:
AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys
AuthorizedKeysCommand root
Now restart SSSD and OpenSSH, and you should be all set:
service sssd restart
service ssh restartTo test your setup, verify that SSSD can talk to Simple AD and fetch information about your test user (user and group information should be returned):
id tuserNext, verify that SSH keys can be accessed (an SSH key should be returned):
/usr/bin/sss_ssh_authorizedkeys tuserFinally, verify that your test user has root sudo privileges (root sudo privilege information should be returned):
/usr/bin/sudo -U tuser -lTesting the configuration of Simple AD, SSSD, and OpenSSH is simple with our SSSD and OpenSSH chef cookbooks, accessible at:
In order to test that your SSSD & Simple AD setup is working properly, you'll need to run the following on a host that has access to a configured Simple AD instance. At Localytics, we use VPN to gain access from our local machines, but YMMV. While you probably want to setup the following in an organization-specific wrapper cookbook, we've tried to make it as easy as possible to clone the sssd cookbook and simply "get it working".
NOTE: our Chef cookbook temporarily uses adcli instead of realm to join the domain due to https://bugs.launchpad.net/ubuntu/+source/realmd/+bug/1333694. Future versions will revert back to using realm.
To get started, make sure you have the components of chef-dk installed (https://downloads.chef.io/chef-dk/) and a sane ruby setup, then simply:
git clone [email protected]:localytics/chef-sssd
cd chef-sssd
gem install bundler
bundle
kitchen testSome more advanced tests are available via .kitchen.local.yml.EXAMPLE if you've got access to your Simple AD server:
cp .kitchen.local.yml.EXAMPLE .kitchen.local.ymlThen edit your .kitchen.local.yml with the various information necessary, such as instance information, IP addresses of your primary and secondary AD servers, AD domain, and a user that has at least one public key and one sudo role configured. The latter is used by integration tests to verify that all services are functioning correctly. FYI, our default test user, 'Guest', likely does NOT have the configuration necessary to pass the integration tests. You'll want to make this 'tuser' if you used our examples above.
Next, create an encrypted data bag secret key used for locally created data bags:
openssl rand -base64 512 | tr -d '\r\n' > test/integration/with-registration/encrypted_data_bag_secret_keyThen, configure an encrypted data bag with a valid administrative username and password, used for joining the domain with 'realm':
knife solo data bag create sssd_credentials realm -c .chef/solo.rbThe format of the data bag is:
{
"id": "realm",
"username": "administrator",
"password": "password"
}Once you're all set, simply:
bundle exec kitchen test... and watch! If all tests pass, you've successfully configured your Simple AD server!
Comments and feedback on our setup are welcome.