header image

IMAGE: Duke TODAY

Sharing sensitive data in secure manner is important on many critical network environments, and Kerberos security provides much needed security layer for insecure NFSv4 file sharing. This article focuses on setting up, configuring and testing MIT Kerberos V5 + NFSv4 file sharing on Linux environment.

Unlike many blog articles, I take one step further and explain how to set-up such environment with not just two but four individual Linux computers: Combined Kerberos LDAP back-end and user database server, Kerberos KDC & Administration server, NFSv4 server and NFSv4 client.

Table of Contents


Foreword

This article compared to other online sources

Many articles focus setting up just combined Kerberos KDC + NFS server and NFS client setup which is far too simple to be used in every day and complex IT environments. Main issue in these articles is that do not clearly distinguish what is actually required for proper setup on individual computers. Understanding what is needed and where in a practical way is crucial for setting up and scaling up a Kerberos environment in real life organizational IT environments.

Common misconceptions

Official Kerberos and NFS documentation can be a burden itself and is quite fragmented. Not only that but there vendor-specific solutions such as Red Hat's FreeIPA implementation and Oracle's partially own implementation exist, as well.

Once you understand basics, you get quite comfortable with the official documentation within weeks. However, we need lower the barrier at first. We need just a simple and pure but yet complete MIT Kerberos V5 and NFSv4 setup guide with understandable and detailed descriptions. We are humans, after all.

Another major issue is a bit suprising one: NFS Utils, a package provided by kernel.org and available on many Linux distributions, does not very clearly distinguish roles and required services on a NFSv4 server and client, nor if these services are needed just for a pure NFSv4 setup.

NFS Utils include everything for NFSv2, NFSv3 and NFSv4 and yet we need only NFSv4. Older NFSv3 is still another popular choice due to its simplicity, not security. Insecure NFSv4 without Kerberos is also a popular choice among people since Kerberos integration seems too difficult (or not worth the effort) for many who are new to Kerberos security concepts.

Help! Too many daemons and service files! I don't know what to do

Yeah, I know the issue. Kerberos is not the simpliest network/server setup you can do. You may be confused because of

  • mixture of various systemd service files needed, based on a computer role on a network

  • different NFS versions require different configuration and service file mixtures to ‘get it right’

  • custom setups random people use

  • One blogger says something, another says another thing. One uses BSD, another uses RHEL and yet another another OS.

  • “I found documentation that tells me to do something that I don't have on my setup, or simply tells me to use some random configuration without telling me why. Honestly, I don't know what I actually need on a working MIT Kerberos V5 + NFSv4 setup."

And, then there are articles such as this one: Holy shit is Kerberized NFS cancer! - Rudd-O.com

Hint: rely on official documentation, get proper architectural understanding of required Kerberos & NFSv4 stuff for your setup. Get good source materials for this stuff. Really understand how it all works. This is what I attempt to do in this article. You have not hit the wall yet.

Dealing with contradicting recommendations

As far as I've read, NFSv4 security is still in development. Work-in-progress combined with out-dated or conflicting online sources which do not suit to your needs tend to lead to confusion, frustation and finally you give up.

Let me be clear on this one: Some online guides do not directly fit to your setup. Actually, many of them do not perfectly fit to your setup, so you need to practice and integrate Kerberos into your environment.

What do I really need on my servers? Facts - now!

Some guides are also simply too old and talk about configuration which is not required anymore for kerberized NFSv4 setup. For instance, you may still find conflicting information like the following:

  • RPC daemons. NFSv4 Server. It is claimed that RPC SVCGSSD (rpc-svcgssd.service) is required on NFSv4 server. This is completely false. It is replaced by GSS Proxy daemon (see section NFS Server), to be precise: gssproxy.service on Systemd-based Linux systems.

  • NFSv4 Client. On NFSv4 client, rpc-gssd.service is required and gssproxy.service is optional (I recommend it, see Pagure.io website why). This is true. However, rpc-svcgssd.service is never required on NFSv4 client (source: Pagure.io - NFS/GSS Proxy Daemon, see section NFS Client).

    • Conclusion: rpc-svcgssd.service or anything related to it is not required on any computer at all.
  • User and group ID mapping. NFS ID Map service (nfs-idmapd.service) is required on NFSv4 server but not on NFSv4 client. Yet both computers need a proper NFSv4 ID mapping. So how does NFSv4 client do that? Here's the catch: On a client, nfs-idmapd.service is (partially) replaced by Linux kernel internal kernelspace code which calls userspace executable nfsidmap whenever needed. NFSv4 client uses nfs-idmapd.service only as fallback, only and only if nfsidmap executable is not found and not configured in /etc/request-key.conf or in /etc/request-key.d/*.conf file on the NFSv4 client.

  • Unneeded services. These services are not needed on NFSv4 setup at all, no matter what:

    • rpcbind.service, rpc-rquotad.service, nfs-lock.service, rpc-statd.service, rpc-statd-notify.service
  • Commands. kadmin.local command is for local Kerberos administration directly on Kerberos server. kadmin command is for remote computers which have Kerberos installed and access to Kerberos server administration utility. You can configure remote access in /etc/krb5.conf file and by allowing it in your Kerberos KDC & Administration server + firewall policy.

  • Systemd service file contents. I will give you contents of several NFS Utils Systemd files for each computer depending on their role on NFSv4/Kerberos network. The official nfs-utils package seems not to have proper configuration for proper Kerberos V5 + NFSv4 setup by default.

Errors, permission issues! Too much frustation. I give up! Yet I say: Do not give up

Some tricky issues can be, yeah, really, really tricky ones. Even if you enable as much logging as possible (rpcdebug and every possible daemon verbose & debug flags: I am looking at you), you may not clearly see what's wrong in your setup. This is because even Linux kernel does not give enough log data for nasty, yet simple misconfigurations, not talking about userspace NFS daemons. Finally you end up only with your erroneous log entries without clear answers why errors occur in the first place.

My best advice: read official documentation, double check your setup. Take a break. Try different approach. Understand what you read, compare to your own setup. Do not only rely on random Google searches or issue tickets on ServerFault.com or Unix.Stackexchange.com. I say you should use those as well, but not at the expense of the official documentation. See also my Additional guides section.

For instance, I debugged one extremely frustating issue four days straight until I found a solution by just carefully reading Oracle Kerberos and GSS-API documentation, and understood I missed two lines of configuration on my NFSv4 server. Once I fixed that, all my day-long problems were gone and everything worked flawlessly.


Getting started

I go through required configuration steps one by one. It is essential you will have all components in place. Setting up kerberized NFSv4 environment is a complex process for beginners and requires components such as DNS, NTP and Kerberos back-end database which can be full projects only themselves for a sysadmin. However, once you get over these obstacles, and understand key Kerberos concepts, it gets easier to extend your knowledge and troubleshoot issues independently on your own.

This article completely excludes Kerberos KDC server slave (replication) configuration. The article focuses on Linux environment and does not cover BSD.

Prerequisites

I assume you have following hardware & OS resources available:

  • A. Time synchronization: Valid NTP server available (remote or local)

  • B. Centralized domain names: Valid local DNS server

  • 1. NFSv4 user + Kerberos principal back-end server role: Linux computer or VM

  • 2. NFSv4 client role: Linux computer or VM

  • 3. NFSv4 server role: Linux computer or VM

  • 4. Kerberos KDC role: Linux computer or VM

You can always combine roles of, for example, servers 1. and 4., if ever needed or preferred. Do as you like.

DNS and NTP servers. You need a DNS server and reliable time synchronization (NTP) for all computers. Usually NTP servers are located at WAN area unless you want to setup your own local trustworthy NTP server.

As expected, all computers must be networked. All computers should have a firewall software installed for the sake of basic network security (ufw, iptables, firewalld or any other solution you prefer).

Architectural overview

MIT Kerberos V5 + NFSv4 architectural overview is pretty simple. However, it is important to classify and distinguish role of each computer on this setup. One working model example is as follows:

post image
Kerberized NFSv4 architecture (Kerberos logo by MIT)
  • Kerberos KDC stores service, user and computer related principals in a back-end database as shown.

  • Since Kerberos security is partially dependent on exact timestamps, all Kerberos-related computers need a stable and trustworthy NTP server connections.

  • DNS server is needed for centralized management of Kerberos-related computers in a local network, and optionally for special Kerberos server DNS record lookups. See DNS server configuration below for details.

NFSv4 user mapping

Key notions, referring to the image above:

  • NFSv4 server and NFSv4 client form together a common NFSv4 ID Map domain and have mutually equivalent user and group name database. Names, not UIDs or GIDs. The catch is explained in the examples below.
    • Note that NFSv4 ID Map domain doesn't have to be same than your DNS domain or Kerberos REALM name. NFSv4 ID Map domain is and independent entity and it is used purely to detect whether NFSv4 computers are related to each other. The following example explains this better:


NOTE: The NFSv4 user mapping examples above are, of course, simplified ones. In real life, simple lookups by kerberized NFSv4 server are not enough because user authentication and authorization are also based on proper GSSAPI (Generic Security Service Application Program Interface) configuration.

Practically, this means that a user name must be recognizable from Kerberos principal name, otherwise a user is authenticated as nobody, and you will get a lot of permission denied errors and such. To counter that issue, NFSv4 server's /etc/krb5.conf file is configured in a way as shown in NFSv4 server Kerberos configuration section below.


Configuration

All computers

Linux kernel configuration requirements

In this section, you may need to re-compile Linux kernel on/for your target computers with the described configurations enabled.

I assume you know how to compile Linux kernel. If you do not, please refer to online materials and your Linux distribution official manual and compile process steps.

I recommend enabling the following kernel configurations.

You can check these with command zcat /proc/config.gz | grep <config_name>

Please keep in mind I have not actually distinguished which is needed for NFSv4 server and client, so if you want to adapt your kernel specifically for these computers, you need to do check what is needed and what is not by yourself. I prefer flexibility since enabling all flags does not add too much overhead to a Linux kernel image file.


About provided Systemd files

This article provides Systemd file content recommendations. However, always practice critical thinking, and adapt the content to your Kerberos environment. In general, feel free to customize Systemd files as needed so they fit into your specific environment.

As a side note, I always recommend using overrides like this for critical system daemons:

# /etc/systemd/system/krb5-kadmind.service.d/override.conf
------------------------------
[Service]
Restart=always
# Restart=on-failure
RestartSec=10

This override ensures a daemon will restart if it fails, for instance, due to a not-yet-started dependency service. It also ensures the service daemon is started if it fails for some other reason during computer runtime. There is a risk for endless loop and manual intervention requirement if the daemon fails to start again and again. For advanced Systemd service topology and analysis, use commands such as systemctl list-dependencies --reverse <service-file>, and do not forget checking dmesg output, too.





Kerberos back-end and NFSv4 user database server

This article mostly assumes that you have a working LDAP setup and it is running already in your network environment.

In this article, we use OpenLDAP for LDAP Kerberos back-end database. As said, the database is required by Kerberos KDC for storing user, machine and service principals.

For basic OpenLDAP server configuration, see the following links:

For Kerberos, you need to ìnclude kerberos.schema in OpenLDAP's /etc/openldap/slapd.conf file. It is usually provided by Kerberos package and may locate at /usr/share/doc/krb5/examples/kerberos.schema. It is recommended to copy this file into /etc/openldap/schema/. Consider converting this file to LDIF format by using a tool such as schema2ldif.

Refer to your Linux distribution package manager and utilities to find out exact kerberos.schema file location.

For proper Kerberos + OpenLDAP back-end setup, see article Ubuntu.com - Kerberos with OpenLDAP backend.

Host information

hostnameIPv4 addressname on local network
ldapserver-1172.20.0.8/24ldapserver-1.mydomain.com

OpenLDAP Database configuration

NOTE: This section does not describe how to populate LDAP back-end database with Kerberos objects or attributes. See Kerberos configuration - Kerberos database generation steps for LDAP for details. For Kerberos LDAP setup, you need to use Kerberos utility kdb5_ldap_util on Kerberos KDC & Administration server.

Kerberos principal database (container) structure in LDAP database is:



Each Kerberos principal, listed on Required/recommended KDC Principals section, have LDAP entries like:



Depending on your setup, your LDAP users may have additional Kerberos attributes such as:

How do I create NFSv4 LDAP attributes, optionally needed by NFS idmapd.conf files on NFSv4 client/server?

Create new LDAP .schema files with proper attribute definitions. Include your new attributes to a proper LDAP object type (in our case: LDAP user object). Additionally, include .schema files into your OpenLDAP configuration (slapd.conf) file.

If needed, convert .schema files into .ldif format. Use schema2ldif conversion utility for it.

LDAP attributes NFSv4UserName & NFSv4GroupName, might be, for instance:

> nfsv4.schema

attributetype ( 2.16.840.1.113730.3.1.500
    NAME 'NFSv4UserName'
    DESC 'RFC2798: preferred name to be used for NFSv4 user name'
    EQUALITY caseIgnoreMatch
    SUBSTR caseIgnoreSubstringsMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
    SINGLE-VALUE )

attributetype ( 2.16.840.1.113730.3.1.501
    NAME 'NFSv4GroupName'
    DESC 'RFC2798: preferred name to be used for NFSv4 group name'
    EQUALITY caseIgnoreMatch
    SUBSTR caseIgnoreSubstringsMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
    SINGLE-VALUE )

Be aware that I don't use proper SYNTAX field values above for the new LDAP attributes.

Firewall configuration

This section does not cover A) configuration for specific firewall implementation B) advanced firewall policy.

Allowed ports for LDAP database connections are:

Default PortLayer-4 ProtocolApplication ProtocolDirectionEncryptionRequired
389TCPldapInboundNoNo, if port 636 used
636TCPldapsInboundYesNo, if port 389 used

Encrypted connections are always a preferred choise if available. Refer to additional firewall manual or instructions for adding these rules. For ldaps configuration, generate proper certificate files with openssl for OpenLDAP. See man 5 ldap.conf (section TLS OPTIONS) and do proper certificate configurations for nfs-idmapd (man 5 idmapd.conf) + Kerberos configuration files and wherever LDAP connections are used.

Required Systemd process files

These Systemd processes should be run on ldapserver-1:

Systemd fileDescription
iptables.service/ufw.serviceIPv4 Packet Filtering Framework/Uncomplicated firewall …
slapd.serviceOpenLDAP server daemon




DNS server

If you use ISC BIND 9 as your DNS server, see Arch Wiki - BIND and other online documentation for proper configuration. Depending on your setup, other DNS servers can also be used, such as Microsoft DNS Server.

Host information

hostnameIPv4 addressname on local network
dnsserver-1172.20.0.9/24dnsserver-1.mydomain.com

This article assumes that you have a working DNS setup is running already where you can populate additional DNS records to.

Kerberos SRV records are not mandatory. They can be optionally be used if proper Kerberos server entries have not been defined in /etc/krb5.conf files on each Kerberos-related computer.

If you have defined dns_lookup_realm, dns_lookup_kdc or any other DNS settings instead of direct kdc, admin_server, kpasswd_server or such in your /etc/krb5.conf on all hosts, you need to define Kerberos SRV records in you DNS server zone file, as shown below. It is quite obvious these hosts must absolutely use your local DNS server configured in their /etc/resolv.conf resolver configuration file (or by systemd-resolved.service; /etc/systemd/resolved.conf).

A records (and CNAME records if you use them) are strongly recommended for a local Kerberos network environment. Set up your Kerberos computers to use your DNS server as a primary DNS server for name lookups. PTR records are not required.

Firewall configuration

This section does not cover A) configuration for specific firewall implementation B) advanced firewall policy.

Allowed ports are as follows:

Default PortLayer-4 ProtocolApplication ProtocolDirectionEncryptionRequired
53UDPdnsInboundNoYes, by other network hosts
53TCPdnsInboundNoYes, by other network hosts*

* This port is usually used when packet size exceeds UDP single-packet limits.





Kerberos KDC & Administration server

This server uses MIT Kerberos V5 implementation.

Host information

hostnameIPv4 addressname on local network
kerberosserver-1172.20.0.10/24kerberosserver-1.mydomain.com

Kerberos compilation parameters

It is recommended MIT Kerberos packages are compiled with the following configuration parameters on the Kerberos KDC & Administration server.

Kerberos KDC LDAP configuration

This section covers minimal LDAP back-end settings for proper TLS certificate usage on the Kerberos KDC & Administration server.

Kerberos configuration

This section covers crucial configuration files required by the Kerberos KDC & Administration server.

NOTE: You must have properly configured DNS and LDAP servers now. Before proceeding, make sure you have properly configured etc/krb5.conf, /var/lib/krb5kdc/kdc.conf and related files as shown above. After having configured these files, you should restart Kerberos KDC and Administration server: sudo systemctl restart krb5-kdc krb5-kadmind

Once you have properly configured basic LDAP back-end for your Kerberos KDC & Administration server, populate the back-end database with Kerberos principal data (example command given below):

Required/recommended KDC Principals

We use kadmin.local command to access local Kerberos Administration Server. We run it as root since the tool requires read & write access to root-owned system Kerberos keytab file /etc/krb5.keytab.

NOTE: Adding, listing, editing or deleting Kerberos principals require properly configured and fully functional Kerberos principal back-end database connection. In this article, we use a dedicated OpenLDAP server for back-end database purposes.

Additional basic information about key kadmin.local commands are given in Useful commands and hints section in this article.

Keytab configuration

NOTE: Kerberos KDC server uses its back-end principal database to authenticate principals offered by Kerberos clients.

Firewall configuration

This section does not cover A) configuration for specific firewall implementation B) advanced firewall policy.

Allowed ports are as follows:

Default PortLayer-4 ProtocolApplication ProtocolDirectionEncryptionRequired
88TCPkerberosInboundYesYes, by NFSv4 client
464TCPkpasswdInboundYesOptional, for remote kpasswd command
749TCPkerberos-admInboundYesOptional, for remote kadmin command

Encrypted connections are always a preferred choise if available. Refer to additional firewall manual or instructions for adding these rules.

Required Systemd processes

These Systemd processes should be run on kerberosserver-1:

Systemd fileDescription
iptables.service/ufw.serviceIPv4 Packet Filtering Framework/Uncomplicated firewall …
krb5-kdc.serviceMIT Kerberos V5 Key Distribution Center
krb5-kadmind.serviceMIT Kerberos V5 Administration Server

Systemd files configuration





NFSv4 server

Host information

hostnameIPv4 addressname on local network
nfsserver-1172.20.0.11/24nfsserver-1.mydomain.com

NFS Utils compilation parameters

NFS configuration

GSSProxy configuration

Kerberos configuration

Firewall configuration

This section does not cover A) configuration for specific firewall implementation B) advanced firewall policy.

Allowed ports are as follows:

Default PortLayer-4 ProtocolApplication ProtocolDirectionEncryptionRequired
2049TCPnfsInboundNo*Yes, by NFSv4 client

* Unless Kerberos encryption scheme used, as explained in this article.

Encrypted connections are always a preferred choise if available. Refer to additional firewall manual or instructions for adding these rules.

Keytab configuration

Required Systemd processes

These Systemd processes should be run on nfsserver-1:

Systemd fileDescription
iptables.service/ufw.serviceIPv4 Packet Filtering Framework/Uncomplicated firewall …
auth-rpcgss-module.serviceKernel Module supporting RPCSEC_GSS
gssproxy.serviceGSSAPI Proxy Daemon
proc-fs-nfsd.mountNFSD configuration filesystem
nfs-idmapd.serviceNFSv4 ID-name mapping service
nfs-mountd.serviceNFS Mount Daemon
nfsdcld.serviceNFSv4 Client Tracking Daemon
nfs-server.serviceNFS server and services
nslcd.service OR sssd.serviceNaming services LDAP client daemon // System Security Services Daemon

Systemd files configuration

Miscellaneous





NFSv4 client

Host information

hostnameIPv4 addressname on local network
nfsclient-1172.20.0.20/24nfsclient-1.mydomain.com

NFS Utils compilation parameters

NFS configuration

> /etc/hosts (optional)
------------------------------
172.20.0.10 kerberosserver-1.mydomain.com
172.20.0.11 nfsserver-1.mydomain.com

GSSProxy configuration

> /etc/tmpfiles.d/gssproxy.conf (root.root -rw-r--r-- 0644)
------------------------------
d /tmp/gssproxy/clients/ 0755 root root - -

NOTE: systemd-tmpfiles-setup.service must be enabled for tmpfiles.d contents to take effect on a host! Usually it is enabled by default.

Kerberos configuration

Firewall configuration

This section does not cover A) configuration for specific firewall implementation B) advanced firewall policy.

Allowed ports are as follows:

Default PortLayer-4 ProtocolApplication ProtocolDirectionEncryptionRequired
88TCPkerberosOutboundYesYes, by Kerberos KDC
2049TCPnfsOutboundNo*Yes, by NFSv4 server

* Unless Kerberos encryption scheme used, as explained in this article.

Encrypted connections are always a preferred choise if available. Refer to additional firewall manual or instructions for adding these rules.

Keytab configuration

Required Systemd processes

These Systemd processes should be run on nfsclient-1:

Systemd fileDescription
iptables.service/ufw.serviceIPv4 Packet Filtering Framework/Uncomplicated firewall …
auth-rpcgss-module.serviceKernel Module supporting RPCSEC_GSS
gssproxy.serviceGSSAPI Proxy Daemon (optional)
rpc-gssd.serviceRPC security service for NFS client
nslcd.service OR sssd.serviceNaming services LDAP client daemon // System Security Services Daemon
nfs-client.targetNFS client services

Systemd files configuration

Miscellaneous


Useful commands and hints

Foreword

Use secure methologies when you are doing something with keytab files. For instance, transferring keytab files via network must be done either with scp command or with a secure, remote access to Kerberos Administration server (Kerberos clients: command kadmin, when kdc is defined in client's /etc/krb5.conf file).

Kerberos principals

kadmin.local command is used on Kerberos KDC & Administration server.

kadmin command is used on any non-KDC host that has /etc/krb5.conf with proper kdc configuration (or alternative, you have defined KDC in your DNS server's SRV records).

When adding keytab entries on a non-KDC host, it may be recommended to use kadmin command if your firewall and security policy allows it. kadmin is more convinient than using kadmin.local but less secure than transferring generated keytab entries directly from KDC to another host via a secured connection.

In the examples below, kadmin.local command is used. If executed on a non-KDC host, use kadmin instead.

Generating principals

Service/computer principal (no password) example:

> sudo kadmin.local
kadmin.local: addprinc -randkey host/nfsclient-1@MYDOMAIN.COM

User principal (with password) example:

> sudo kadmin.local
kadmin.local: addprinc defaultuser/admin@MYDOMAIN.COM
No policy specified for defaultuser/admin@MYDOMAIN.COM; defaulting to no policy
Enter password for principal "defaultuser/admin@MYDOMAIN.COM":

Keys generated by these commands reflect your settings in Kerberos configuration files.

Adding principals to /etc/krb5.keytab on current host

> sudo kadmin.local
kadmin.local: delprinc defaultuser/admin@MYDOMAIN.COM

Additional Kerberos Administration Server commands

See:

> sudo kadmin.local
kadmin.local: ?

Requesting a new ticket and access to the NFSv4 service

Requesting a TGT is the first step you need as a user when attempting to access Kerberos protected NFSv4 volume on a NFSv4 client.

You request TGT as and for the current user, so do not use sudo with kinit command. The same rule applies when you destroy your current Kerberos session ticket with command kdestroy. You request the TGT for your current Linux LDAP user.

For instance, requesting TGT as logged in LDAP user defaultuser on a properly configured Kerberos client (both NFSv4 server & client are Kerberos clients in our case) could be done as follows:

[defaultuser@nfsclient-1] kinit -V -S nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM defaultuser/admin@MYDOMAIN.COM
Using default cache: /tmp/krb5cc_1000
Using principal: defaultuser/admin@MYDOMAIN.COM
Password for defaultuser/admin@MYDOMAIN.COM:
Authenticated to Kerberos v5

where

  • defaultuser/admin@MYDOMAIN.COM refers to Kerberos principal defaultuser/admin@MYDOMAIN.COM

  • nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM refers to NFSv4 service principal we are requesting with parameter -S for defaultuser/admin@MYDOMAIN.COM

  • Both user & service principals exist and are stored on your Kerberos KDC server.

  • Additionally, your LDAP server database user ldapuser must have previously explained Kerberos LDAP attributes defined in its LDAP user object.

  • Password is a password for your user principal you have defined with Kerberos Administrative Server tool kadmin or kadmin.local.

Issue: Password incorrect while getting initial credentials

Although you have supplied proper credentials, you may still get this message on a Kerberos client.

kinit: Password incorrect while getting initial credentials

Occasionally, I have had this issue and resolved it by resetting the password for this specific principal in Kerberos Administrative Server tool kadmin / kadmin.local. So, reset the password for user principal defaultuser/admin@MYDOMAIN.COM and try again.

> sudo kadmin.local
kadmin.local: change_password defaultuser/admin@MYDOMAIN.COM
Enter password for principal defaultuser/admin@MYDOMAIN.COM:
Re-enter password for principal defaultuser/admin@MYDOMAIN.COM:
Password for defaultuser/admin@MYDOMAIN.COM changed.
kadmin.local:

From Kerberos kadmin documentation: This command requires the changepw privilege, or that the principal running the program is the same as the principal being changed.

Root cause of this issue can be a corrupted file or Kerberos principal data, although I may be wrong.

Ticket cache location on a Kerberos client

Once you have successfully received a session ticket from your Kerberos KDC, it is stored in a cache folder on the Kerberos client computer. It is /tmp by default. This folder acts as your client's Kerberos ticket cache. The default ticket name syntax is krb5cc_<UID> for an user. Machine tickets are stored in the same manner with file name syntax krb5ccmachine_MYDOMAIN.COM.

Custom ticket cache location

To use a custom ticket cache location on a Kerberos client, configure Kerberos client configuration file /etc/krb5.conf:

[libdefaults]
  default_ccache_name = FILE:/tmp/krb5cc_%{uid}

For proper details, see: man 5 krb5.conf

Your software may also implement a variable which defines Kerberos ticket cache location on a Kerberos client for this specific software.

GSSProxy:

Custom ticket cache location for GSSProxy on NFSv4 server & client is defined in a configuration file /etc/gssproxy/*.conf by variable cred_store = ccache:FILE:<folder>/krb5cc_%U

  • You may change %U (user ID) variable to %u (user name) for your needs

    • See also: man 5 gssproxy.conf

How do I view my Kerberos ticket content?

For current user, just use klist command.

On a properly configured NFSv4 computer which uses gssproxy.service and defines Kerberos ticket cache location as shown above, sample ticket cache file content (with NFSv4 service) for LDAP Linux user with UID 1000 could be as follows:

[ldapuser@nfsclient-1] klist -c /tmp/gssproxy/clients/krb5cc_1000

Ticket cache: FILE:/tmp/gssproxy/clients/krb5cc_1000
Default principal: host/nfsclient-1@MYDOMAIN.COM

Valid starting     Expires            Service principal
02/05/21 10:26:39  02/05/21 22:26:39  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
02/05/21 10:26:40  02/05/21 22:26:39  nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM

Ticket renewal

A simple command to renew a Kerberos ticket for current user:

kinit -R

More complex renewal requests can also be made, like:

kinit -V -R -S nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM ldapuser/systemuser@MYDOMAIN.COM

NOTE: Renewal is usually not needed if you use sssd.service or GSSProxy with auto-renewal configuration.

Be aware that renewal has time limit dictated by your Kerberos KDC server after which user must request a new, fresh session (TGT) ticket.

Deleting my current Kerberos ticket cache

Simply issue command

kdestroy

as the current user.

Adding keytab principal entries on Kerberos clients

Basically, we talk about both NFSv4 server and NFSv4 client which both have also a Kerberos client role. This method is useful in situations where

  • Kerberos Administation Server can not be accessed with Kerberos client command kadmin, i.e. the Administation Server has local access only.
  • You can obtain required KVNO number and symmetric key data from Kerberos KDC.

These commands are meant to be run on Kerberos clients. They modify existing keytabs in-place.

Systems & services: with known KVNO number and a symmetric key

This section describes a situation where Kerberos administrator wants to add new entry to a keytab file with known principal name, KVNO number, encryption type and encryption key.

> sudo ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: addent -k <kvno_number> -p <my_principal> -key -e <encryption_type>
> Key for <my_principal> (hex): <hex_number_without_0x_prefix>
ktutil: wkt /tmp/krb5.keytab
ktutil: exit
> mv /tmp/krb5.keytab /etc/krb5.keytab
> ls -l /etc/krb5.keytab
-rw------- 1 root root 646 Feb  4 22:04 /etc/krb5.keytab

where

ParameterDescription
<kvno_number>KVNO number in referring keytab file (for instance, an entry in KDC server /etc/krb5.keytab file)
<my_principal>Principal in referring keytab file (for instance, an entry in KDC server /etc/krb5.keytab file)
<encryption_type>Principal encryption type in referring keytab file (for instance, an entry in KDC server /etc/krb5.keytab file). Use addent ktutil command for each encryption type, if multiple entries defined for a principal.

Example:

...
ktutil: addent -k 3 -p nfs/nfsclient-1@MYDOMAIN.COM -key -e aes256-cts-hmac-sha1-96
> Key for <my_principal> (hex): b064c4ac8562e27587ec15ed583d53ca62697984ff7d0661b71ece12cc478ea7

ktutil: addent -k 3 -p nfs/nfsclient-1@MYDOMAIN.COM -key -e aes128-cts-hmac-sha1-96
> Key for <my_principal> (hex): 68f27b9fd818cd30659fbb2de12aa56f
...

Passwordless non-interactive user authentication

…by using known KVNO number and password for principal.

Adding user principal with a supplied password to a keytab file is very useful if you need to request a ticket granting ticket (TGT) from Kerberos KDC without an interactive password prompt each time you type kinit command.

However, when requesting TGT, shell command should have syntax kinit userwithnopasswordprompt/users@MYDOMAIN.com (with additional parameters you prefer).

Automatic authentication is useful for user scripts and for other automation purposes where access to services is wanted without any user interaction.

> sudo ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: addent -k <kvno_number> -p <my_principal> -password -e <encryption_type>
> Password for <my_principal>: [enter your password]
ktutil: wkt /tmp/krb5.keytab
ktutil: exit
> mv /tmp/krb5.keytab /etc/krb5.keytab
> ls -l /etc/krb5.keytab
-rw------- 1 root root 646 Feb  4 22:04 /etc/krb5.keytab

Example:

ktutil: addent -k 4 -p userwithnopasswordprompt/users@MYDOMAIN.com -password -e aes256-cts-hmac-sha1-96

You don't need this principal entry in KDC's /etc/krb5.keytab file.

Use a proper keytab file for your needs.

Other ktutil commands with descriptions

See

> sudo ktutil
ktutil: ?

Troubleshooting

Foreword

If you encounter issues during configuration, always refer to your server logs, online documentation and as the last straw, sites such as Serverfault.com and Unix.Stackexchange.com. As I said earlier, even logs may not give you enough information so it is important you understand Kerberos basics such as ticket renewal times, symmetric key encryption, how to check current user service ticket and so on. Understanding these and your Kerberos infrastructure in general helps you out when you are debugging any Kerberos/NFSv4 issues.

For common Kerberos issues, see Kerberos Administration Guide - Release 1.19 - MIT, section TROUBLESHOOTING.

The following sections explain most common NFSv4 + Kerberos related issues.

Permission denied when mounting a Kerberos secured volume

mount.nfs: trying text-based options 'sec=krb5p,timeo=900,retrans=5,vers=4.2,addr=172.20.0.11,clientaddr=172.20.0.20'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5p,timeo=900,retrans=5,vers=4,minorversion=1,addr=172.20.0.11,clientaddr=172.20.0.20'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'sec=krb5p,timeo=900,retrans=5,vers=4,addr=172.20.0.11,clientaddr=172.20.0.20'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting nfsserver-1.mydomain.com:/srv/nfs_sec/

Looks familiar? Some hints:

  • Check keytab entry KVNOs and symmetric keys match on Kerberos KDC & Administration server and on NFSv4 computers clients + server keytab file contents.
  • Have you requested a ticket granting ticket (kinit)?
  • Remember, Kerberos has extremely strict policy to time fluctuations. So, on each kerberized computer, check output of date command. To set correct time, enable NTP client service (Systemd: timedatectl (SNTP), ntp, openNTPD, chrony or other ntp client daemon). You can manually set time with command syntax sudo date --set='8 JAN 2021 22:44:12'

There is something odd I have noticed about KVNOs: kvno number listed in the keytab file may not be the actual KVNO Kerberos uses. I have encountered this issue at least once. Reason for this behavior is unknown, likely a bug in the implementation or some underlying feature I am not aware of. To get the actual KVNO number, do the following on a Kerberos client:

kvno -k /etc/krb5.keytab nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM

On the NFSv4 server & client, check output of gssproxy.service process. Do you see any error messages?

sudo journalctl -xe -u gssproxy.service

cannot access XX: Key expired or cannot open directory: Key has expired

Very common and simple issue. You need to either renew your Kerberos service ticket or request it in the first place with command kinit. You can destroy current Kerberos session ticket with kdestroy command.

Permission denied on already mounted NFSv4 volume

You may encounter this issue if you do not have a Kerberos session ticket (TGT) for the current user. This session ticket must have service principal entry for the NFSv4 service (for instance: nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM). Simply request a TGT with kinit command and ensure the ticket does have NFSv4 service, as shown in How do I view my Kerberos ticket content? section.

How to recover from mount failures

At first, attempt sudo mount -a. As an additional and controversial note, after a failed mount operation, it may be necessary to restart rpc-gssd.service on NFSv4 client, as further mount requests may not be passed correctly to the NFSv4 server.

Common debugging

Linux kernel RPC calls

Basically, we enable RPC debugging with rpcdebug command, as shown below. This is useful for debugging NFSv4 server and NFSv4 client.

> for i in nfs nfsd rpc; do sudo rpcdebug -m $i -s all; done
> sudo journalctl -f

Disabling RPC debugging can be done as follows:

> for i in nfs nfsd rpc; do sudo rpcdebug -m $i -c; done

FAQ

How does Kerberos work?

The basic principle is as shown in the following picture.

post image
Kerberos authentication & authorization process (Kerberos logo by MIT)

NOTE: Be aware there are different terms used, depending on the online source material you refer to. Additionally, different articles explain the process in varying detail.

NOTE: If this text gives you headaches, you find alternative articles below.

NOTE: Be aware that the following text reflects my best understanding of the subject. Please let me know if there is any misleading information. Thanks!

By executing kinit on a Kerberos client, there happens a lot of stuff under hood.

In the following example…

  • Kerberos realm is MYDOMAIN.COM

  • user principal defaultuser/admin@MYDOMAIN.COM is used on nfsclient-1.mydomain.com

  • NFSv4 client & Kerberos client computer is nfsclient-1.mydomain.com

    • principal reference: host/nfsclient-1@MYDOMAIN.COM

      • has this principal in a local krb5.keytab file
  • NFSv4 server software is installed on nfsserver-1.mydomain.com which is in the role of a Kerberos client and NFSv4 server

    • service principal reference: nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM

      • has this principal in a local krb5.keytab file
  • Kerberos server software is installed on kerberosserver-1.mydomain.com

    • no principals in a local krb5.keytab file

    • has all mentioned principals in a Kerberos OpenLDAP back-end database on OpenLDAP server ldapserver-1.mydomain.com

  • DNS name resolution & NTP time synchronization must work for these kerberized hosts. DNS & NTP services must be accessible. DNS queries must give valid answers to the kerberized hosts. DNS server must have valid A & PTR record entries for kerberized hosts (forward & reverse lookup zones).

The process

Computer & user authentication:

We do a Kerberos authentication request. The client computer sends the user principal name (defaultuser/admin@MYDOMAIN.COM) to Kerberos server's AS component (Authentication Service) without encryption.

At first, the Kerberos server's AS component checks if the user principal name is found in the Kerberos server's back-end database. If found, the Kerberos server's AS component generates a one-way hash by hashing password for the found user principal. The client computer's principal (host/nfsclient-1@MYDOMAIN.COM) is also validated.

The Kerberos server's AS component then sends a generated server session key (already encrypted with a new secret) and Ticket Granting Ticket (TGT) to the client computer. The Ticket Granting Ticket is encrypted with a secret key of Kerberos server's TGS (Ticket Granting Service) component, unknown to any other parties, including the client computer & NFSv4 server computer.

The client must decrypt the session key with its own secret in order to authenticate against the Kerberos AS component. After successful decryption, the client is authenticated. The server session key is used for further communication with the Kerberos server's TGS component.

The server session key & Ticket Granting Ticket data contents are as follows:

Messageclient computer session keyTicket Granting Ticket
Datatimestampclient computer ID
key dataclient computer network (IP) address
ticket lifetime (validity period)
server session key data

Authorization to the service:

When requesting permission to use a Kerberos protected service, the client computer sends the encrypted Ticket Granting Ticket along with the service's ID (nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM) to the Kerberos server's TGS component.

The client computer also sends a client authenticator message, encrypted with the server session key it received earlier from The Kerberos server's AS component.

client authenticator message data contents are as follows:

Messageclient authenticator message
Datatimestamp
client ID

Kerberos server's TGS component decrypts the received Ticket Granting Ticket data with its own TGS secret key.

As the decrypted Ticket Granting Ticket data includes the client ID and the client computer session key, the Kerberos server's TGS component is able to decrypt the received client authenticator message and furthermore compares if the two client IDs match: one within the client authenticator message and one within the decrypted TGT data. This way the Kerberos server's TGS component can also validate the client in addition to AS.

If the client IDs match, the Kerberos server's TGS component sends a service session key and a service ticket to the client. The service ticket is encrypted with a service's secret key, used by the service principal (nfs/nfsserver-1.mydomain.com@MYDOMAIN.COM).

service ticket data contents are as follows:

Messageservice ticket
Dataclient ID
client computer network (IP) address
ticket lifetime (validity period)
service session key data

Once the NFSv4/Kerberos client has received the encrypted service ticket and the service session key from the Kerberos server's TGS component, it authenticates itself to the service (SS; Service Server), our NFSv4 server. The service is NFSv4 server in our case, and the Service Server is our NFSv4 server.

The client sends a new service authenticator message (which is encrypted with the service session key) and the service ticket (which is encrypted with service's secret key) to the Service Server.

service authenticator message data contents are as follows:

Messageservice authenticator message
Datatimestamp
client ID

The Service Server uses its own secret key to decrypt the received service ticket in order to extract service session key from it. After that, the Service Server decrypts the service authenticator message with the extracted service session key data.

The Service Server checks whether client IDs obtained from the service ticket and the service authenticator message match each other. If they do, the Service Server sends timestamp found in client's authenticator message in an encrypted form back to the client. The timestamp is encrypted with the service session key.

As the client receives the encrypted timestamp from the Service Server, it decrypts it with the service session key it already has obtained earlier. The client computer validates the timestamp. If the validation passes, the client can trust the Service Server and therefore, can finally start further communication with the Service Server.

Sources:

Further reading

There are several good online sources if you are keen to understand more about the internal Kerberos authentication and authorization processes:

Why to read? It may help you understand any issues you may encounter with Kerberos and improves your knowledge to extend Kerberos usage for other services beyond NFSv4, as well.

For common Kerberos configuration, I strongly recommend the following documentation and man pages such as man 5 krb5.conf, man 5 kdc.conf and man 5 gssproxy.conf. It gives you far better understanding of Kerberos and helps you improve your skills further on. Also check out:

Relation between krb5.conf and kdc.conf

On any Kerberos KDC on a domain, krb5.conf and kdc.conf will be merged into a single configuration profile (on Kerberos KDC server).

Although you can use duplicate settings for some settings (such as ticket life times) in both files, I strongly encourage using only a single place where you define something. It keeps your configuration clean, simple and organized.

Obviously, kdc.conf does not have any role on Kerberos clients but you still need a briefly configured krb5.conf on the clients, as well.

For Kerberos clients, only krb5.conf should be configured. For conflicting settings between a Kerberos client and the Kerberos KDC server, it's up to the Kerberos KDC server which policy is accepted on the realm in the end. For instance, Kerberos client can have maximum ticket renewal time of 48 hours in its /etc/krb5.conf but the Kerberos KDC server may enforce 24 maximum renewal time in its combined /etc/krb5.conf and /var/lib/krb5kdc/kdc.conf configuration. Same logic applies to used encryption types and such. Clients do not give orders here, a valid Kerberos server does it.

User principals

Do I need Kerberos principal records for each user in Kerberos back-end database such as in LDAP?

Yes, you need. However, I want to distinguish something here with an example.

Strictly speaking, for NFSv4 to work, you don't need nested Kerberos objects in LDAP records such as in uid=foouser,ou=Users,dc=mydomain,dc=com. However, you must have a LDAP record for your Kerberos principal such as for foouser@MYDOMAIN.COM. This record is located and stored in Kerberos LDAP container, as you have defined in your Kerberos KDC server's /var/lib/krb5kdc/kdc.conf file.

You may also use principal format such as foouser/systemuser@MYDOMAIN.COM as long as you have proper GSSAPI auth_to_local regex configuration defined in NFSv4 server's /etc/krb5.conf file, as described earlier in this article.

Security

Good and compherensive explanations are given in Kerberos Administration Guide - Release 1.19 - MIT, section referrals are shown below. I make short references to key points although there are a lot of good stuff in the linked PDF file, so I strongly recommend reading it.

KDC security policy

Section: 1.1.1 Installing KDC

“It is best to install and run KDCs on secured and dedicated hardware with limited access. If your KDC is also a file server, FTP server, Web server, or even just a client machine, someone who obtained root access through a security hole in any of those areas could potentially gain access to the Kerberos database.

I strongly encourage running Kerberos KDCs in internal and strictly restricted scope in your hardened internal LAN/VLAN network segment. NEVER run KDC or related sensitive services in your DMZ zones. Consider using unprivileged ports (> 1024) for any Kerberos server processes instead of default privileged ports 88, 464 and 749.

Keytab security policy

Section: 1.1.3 UNIX Application Servers - The keytab file

“The keytab file is a potential point of entry for a break-in, and if compromised, would allow unrestricted access to its host. The keytab file should be readable only by root, and should exist only on the machine's local disk. The file should not be part of any backup of the machine, unless access to the backup data is secured as tightly as access to the machine's root password.”

Securing Kerberos computers

Section: 1.1.3 UNIX Application Servers - Some advice about secure hosts

“Kerberos V5 can protect your host from certain types of break-ins, but is is possible to install Kerberos V5 and still leave your host vulnerable to attack. Obviously an installation guide is not the place to try to include an exhaustive list of countermeasures for every possible attack, but it is worth noting some of the larger holes and how to close them.”

“We recommend that backups of secure machines exclude the keytab file. If this is not possible, the backups should at least be done locally, rather than over a network, and the backup tapes should be physically secured.”

Mutually validating Kerberos KDC and clients

I recommend reading the linked official documentation for PKINIT or SPAKE configuration setup.

PKINIT certificates

Section: PKINIT CONFIGURATION

“PKINIT is a preauthentication mechanism for Kerberos 5 which uses X.509 certificates to authenticate the KDC to clients and vice versa. PKINIT can also be used to enable anonymity support, allowing clients to communicate securely with the KDC or with application servers without authenticating as a particular client principal.”

Protection against password dictionary attacks

Password dictionary attacks are concern of principals which use passwords for authentication (principal generated without -randkey parameter is kadmin console). These attacks are not concern of principals generated with -randkey (random key) parameter.

I strongly recommend reading section ADDRESSING DICTIONARY ATTACK RISKS in the referred documentation as well as the following SPAKE section:

Section: SPAKE PREAUTHENTICATION

“SPAKE preauthentication uses public key cryptography techniques to protect against password dictionary attacks. Unlike PKINIT, it does not require any additional infrastructure such as certificates; it simply needs to be turned on. Using SPAKE preauthentication may modestly increase the CPU and network load on the KDC.”

Common security policy

Practice and apply common security policy to your network and host-specific policy. Also consider using SELinux and/or AppArmor or similar solutions for additional security. Practice the best mandatory access control policies in general.

Cyber attacks against Kerberos

Different attack types against Kerberos. You find more about the practical approach of these attacks by checking the URL links on each attack type.

Attack nameDescription
DCShadow attackAn attack where attackers gain enough access inside a network to set up their own fake Domain Controller and to use it in further infiltration
Credential stuffing (Brute force)Automated, continued attempts to guess a password
Encryption downgrade with Skeleton Key MalwareA malware that can bypass Kerberos on a Windows Domain Controller with a master password, but the attacker must have administrative access
Golden ticketA ticket that grants a user domain administration access
Kerberos KDC Spoofing (link 1; link 2)Kerberos Key Distribution Center (KDC) server spoofing. Note: Microsoft Kerberos v5 domain login in Windows 2000 is not vulnerable to this attack
Pass-the-ticketThe process of forging a session key and presenting that forgery to the resource as credentials
Silver ticketA forged ticket that grants access to a service
Kerberoasting Attack (Active Directory) (link 1; link 2)An attacker captures TGT ticket from client computer's memory and cracks the service credential hash offline by using a password cracking tool

Additional sources:

You find more Kerberos exploits on exploit-db.com - Exploit Database by searching Kerberos.

Tools such as MetaSploit-Framework include mimikatz module which has Kerberos and Windows Domain Controller attacks as illustrated in the following picture:

post image
Metasploit Framework: available Kerberos and Windows Domain Controller attack modules (some found modules are irrelevant)

Kerberos critique

Many people find Kerberos difficult to set up or maintain. Others say it's a security framework remnant of yesterday. Usually people tend to be frustated by the lack of good documentation and unnecessary complexity involved. Critique:

Alternatives for secure file sharing

You may ask yourself,

“Do I really have to set up the whole Kerberos infrastructure just for secure file sharing?”

Well, it depends how much security you or your organization wants or requires.

stunnel + NFS

stunnel wrapper is a lot less complex to set up than full Kerberos environment. However…

I once tested stunnel for plain-text NFS (v4 shares. Although my configuration was valid, I had NFS connection stability issues (dropped connections) and weird connectivity issues so I abandoned the approach. You may try stunnel but based on my experience, I don't recommend it for NFS shares. I don't know how it works with Samba or with other NFS versions, though.

Samba shares

Samba shares can only utilize unencrypted LDAP connections for now.

SMB 3.0 protocol will support encryption but as of August 2021, its implementation status is still in progress (see wiki.samba.org - SMB3 kernel status).

So, Samba can't replace Kerberos now.

Basic guide to set up an unencrypted Samba share on Linux: Arch Wiki - Samba.

SSHFS

To-be-written

Additional guides

Conclusions

Setting up a properly working Kerberos environment is a project with many requirements. You need server infrastructure with a working network, good security policy for both hosts themselves but for network topology, as well. Instead of physical computer hosts, virtual machines or containerized solutions may be sufficient for you, or maybe you want to use them even in your production environment.

As Kerberos has many dynamic parts, and the configuration can be done in many ways, there is no a single correct way to set it up. Some basic principles are covered and practiced in this article but you need to understand Kerberos scales up to many, many other services than just NFSv4. In addition, a Kerberos setup can have more encryption schemes and slave/replication mechanisms which are not well covered in this article.

Kerberos is a work horse behind Windows Active Directory, can independently be used in pure Linux environments, or in mixed environment with all Windows/Mac/Linux hosts present. Mixed Kerberos environments is also a topic not covered in this article.

Goal of this article is give you a practical setup that works but actually understanding the logic and extend your knowledge is up to you. I have provided multiple links in this article in different sections to help you out. This is not enough since you need to understand how to plan and configure proper network and security policy, as well. You may have requirements which are not covered in this article. In the end, it is up to your knowledge, technical understanding and goals which is the solution for your kerberized environment.

As my last words I say this: do not even attempt to adapt Kerberos/GSSAPI everywhere. Although some experts consider Kerberos an outdated protocol, it still has its place on many computer networks. Kerberos does not scale to modern cloud solutions where SAML/SAML2, OAuth and such methods have a preference but it still works on securing local network infrastructure when used wisely and practical scope in mind.