Auto-Provisioning User Home Directories in a Multi-Tenant Slurm Cluster with Open OnDemand

Hi all,

I’m working on deploying Open OnDemand on a large, shared Slurm cluster.

Context:

  • It is one central Slurm cluster.
  • Authentication is multi-tenant, using an identity broker (e.g., Keycloak).
  • EduGAIN is supported — so users from different institutions (mostly students) can authenticate using their student credentials.
  • Authentication of local user via local FreeIPA is supported as well

The Challenge:

We need a way to automatically create home directories on the Slurm cluster when a new user logs in for the first time (via Open OnDemand or directly via SSH, if allowed).

So far, I’ve identified a few potential challenges:

  • I don’t want to pre-provision users manually.
  • I need a secure, scalable, and ideally event-driven way to provision a home directory on first login.

I’m looking for guidance or solutions — PAM? OOD hooks? Integration with Keycloak?
Any best practices or tools others use?
Thanks in advance!

I noticed this thread and I hope it is not too late, or if so maybe my approach will benefit someone in the community.

First, a custom mapfile contains a few lines which create a file to serve as the trigger to the inotify daemon:

Check if the user already exists

if id “$INPUT_USER” &>/dev/null; then
echo “$INPUT_USER”
else

Create the file that will trigger inotify to run the user creation script

touch /var/www/html/user_triggers/$INPUT_USER
fi

I deployed an inotify daemon as an ansible task to take action when triggered by the creation of the user file. It could potentially be adapted for your deployment and fit it in to your configuration management tool of choice.


  • name: install inotify-tools
    ansible.builtin.dnf:
    name: inotify-tools
    state: present
    when:
    ansible_distribution == ‘RedHat’

  • name: Create directory for user triggers
    ansible.builtin.file:
    path: /var/www/html/user_triggers
    state: directory
    owner: apache
    group: apache
    mode: ‘0755’

  • name: Create user creation script
    ansible.builtin.copy:
    dest: /usr/local/bin/create_user.sh
    content: |
    #!/bin/bash

    # Directory to watch
    WATCH_DIR="/var/www/html/user_triggers"
    
    # Process each file in the directory
    for FILE in "$WATCH_DIR"/*; do
        if [ -f "$FILE" ]; then
            # Read the username from the filename
            USERNAME=$(basename "$FILE")
    
            # If the username is in an @myinstitution.edu email address, trim it
            if [[ "$USERNAME" =~ @myinstitution.edu$ ]]; then
              USERNAME=${USERNAME%@myinstitution.edu}
            fi
    
            # Create the user
            useradd -m "$USERNAME"
            
            # Wait for the user to be created
            while ! id "$USERNAME" &>/dev/null; do
              sleep 1
            done
    
            # Remove the file after processing
            rm -f "$FILE"
        fi
    done
    

    mode: ‘0755’
    owner: root
    group: root

  • name: Create systemd service for inotify
    ansible.builtin.copy:
    dest: /etc/systemd/system/inotify-user.service
    content: |
    [Unit]
    Description=Inotify User Creation Service

    [Service]
    ExecStart=/bin/bash -c '/usr/bin/inotifywait -m -e create /var/www/html/user_triggers | while read path action file; do /usr/local/bin/create_user.sh; done'
    User=root
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    

    mode: ‘0644’
    owner: root
    group: root

  • name: Reload systemd daemon
    ansible.builtin.systemd:
    daemon_reload: yes

  • name: Enable and start inotify service
    ansible.builtin.systemd:
    name: inotify-user.service
    enabled: yes
    state: started

Also, I should have given a high-level breakdown of why the process lacks simple elegance:

  1. The trigger file is created by the apache user
  2. The inotify daemon runs as root so that it can create the local linux account with the home directory.

If your process only needs to create the home directory without creating a local linux user, you would replace the “useradd” with just a mkdir for the home directory itself, another mkdir for the .ssh subdirectory, a recursive chown, and a restorecon -v -R to fix the selinux labels

Ok, so another update to my last update. I just started out a deployment on a new authentication infrastructure with saml in which I am not using the user_map script and this seems like it could be a sensible solution that would work independently any choices you have had to make regarding authentication mechanisms.

Everything I needed was built in to ood. I was not doing myself any favors by reinventing the wheel (see Fix "no home directory" issue when first time logging in):

Use the pun pre_hook config in your ood_portal.yml file:

pun_pre_hook_root_cmd: ‘/usr/local/bin/create_home_dir.sh’
pun_pre_hook_exports: ‘MELLON_REMOTE_USER,REMOTE_USER’

You’d think the script would be obvious but there is a gotcha here. This is why I am logging the output to /var/log/home_dir.

Using environment variables did not seem like an option because the context of the user changes a couple of times
1. Our researcher logs into Apache
2. The “apache” account executes the pre-pun hook - but with sudo
3. The script is executed as root.

Instead, the script parses out the username from the command line arguments passed to it.


  • name: Create user home directory script
    ansible.builtin.copy:
    dest: /usr/local/bin/create_home_dir.sh
    content: |
    #!/bin/bash

    # Setup logging
    LOG_DIR="/var/log/home_dir"
    LOG_FILE="$LOG_DIR/homedir.log"
    
    # Create log directory if it doesn't exist
    mkdir -p $LOG_DIR
    chmod 755 $LOG_DIR
    
    # Log function
    log() {
        echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> $LOG_FILE
    }
    
    # Start logging
    log "Script started"
    log "Environment variables:"
    env | sort >> $LOG_FILE
    log "Command line arguments: $@"
    
    # Get username from command line arguments
    USERNAME=""
    while [[ $# -gt 0 ]]; do
        case $1 in
            -u|--user)
                USERNAME="$2"
                shift 2
                ;;
            -P|--pre-hook)
                # Skip the pre-hook argument and its value
                shift 2
                ;;
            *)
                shift
                ;;
        esac
    done
    
    if [ -z "$USERNAME" ]; then
        log "ERROR: Username not provided in command line arguments"
        exit 1
    fi
    
    log "Username from command line: $USERNAME"
    
    # Remove @uchicago.edu if present
    USERNAME=${USERNAME%@uchicago.edu}
    log "Final username after processing: $USERNAME"
    
    if [ -z "$USERNAME" ]; then
        log "ERROR: Username is empty after processing"
        exit 1
    fi
    
    # Check if user exists
    if ! id "$USERNAME" &>/dev/null; then
        log "ERROR: User $USERNAME does not exist"
        exit 1
    fi
    
    # Create the home directory with .ssh directory and set permissions and selinux labels
    log "Creating home directory for $USERNAME"
    if [ ! -d "/home/$USERNAME" ]; then
        mkdir -p /home/$USERNAME
        if [ $? -ne 0 ]; then
            log "ERROR: Failed to create home directory"
            exit 1
        fi
        log "Created new home directory"
    else
        log "Home directory already exists, ensuring proper setup"
    fi
    
    # Create .bashrc with proper content
    log "Creating .bashrc"
    cat > /home/$USERNAME/.bashrc << 'EOL'
    # .bashrc created by create_home_dir.sh
    
    # Source global definitions
    if [ -f /etc/bashrc ]; then
            . /etc/bashrc
    fi
    
    # User specific environment
    if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
    then
        PATH="$HOME/.local/bin:$HOME/bin:$PATH"
    fi
    export PATH
    
    # Uncomment the following line if you don't like systemctl's auto-paging feature:
    # export SYSTEMD_PAGER=
    
    # User specific aliases and functions
    if [ -d ~/.bashrc.d ]; then
            for rc in ~/.bashrc.d/*; do
                    if [ -f "$rc" ]; then
                            . "$rc"
                    fi
            done
    fi
    
    unset rc
    EOL
    
    # Create .bash_profile with proper content
    log "Creating .bash_profile"
    cat > /home/$USERNAME/.bash_profile << 'EOL'
    # .bash_profile created by create_home_dir.sh
    
    # Source .bashrc
    . ~/.bashrc
    EOL
    
    # Create and set .ssh directory permissions
    mkdir -p /home/$USERNAME/.ssh
    chown $USERNAME:$USERNAME /home/$USERNAME/.ssh
    chmod 700 /home/$USERNAME/.ssh
    
    # Ensure all files and directories have correct ownership and permissions
    log "Ensuring correct ownership and permissions"
    chown -R $USERNAME:$USERNAME /home/$USERNAME
    if [ $? -ne 0 ]; then
        log "ERROR: Failed to set ownership"
        exit 1
    fi
    
    chmod 700 /home/$USERNAME
    if [ $? -ne 0 ]; then
        log "ERROR: Failed to set home directory permissions"
        exit 1
    fi
    
    # Set file permissions
    chmod 644 /home/$USERNAME/.bashrc
    chmod 644 /home/$USERNAME/.bash_profile
    chmod 644 /home/$USERNAME/.bash_logout
    
    # Log SELinux operations
    log "Setting SELinux contexts"
    # Set context for home directory
    semanage fcontext -a -t user_home_dir_t "/home/$USERNAME(/.*)?" 2>> $LOG_FILE
    restorecon -Rv /home/$USERNAME 2>> $LOG_FILE
    
    # Set specific contexts for files
    chcon -t user_home_t /home/$USERNAME/.bashrc 2>> $LOG_FILE
    chcon -t user_home_t /home/$USERNAME/.bash_profile 2>> $LOG_FILE
    chcon -t user_home_t /home/$USERNAME/.bash_logout 2>> $LOG_FILE
    
    # Set context for .ssh directory
    semanage fcontext -a -t ssh_home_t "/home/$USERNAME/.ssh" 2>> $LOG_FILE
    restorecon -Rv /home/$USERNAME/.ssh 2>> $LOG_FILE
    
    log "Script completed successfully"
    

    mode: ‘0755’
    owner: root
    group: root
    setype: bin_t

  • name: Set SELinux context for log directory
    ansible.builtin.file:
    path: /var/log/home_dir
    state: directory
    mode: ‘0755’
    owner: root
    group: root
    setype: var_log_t

Or if you are not using selinux:


  • name: Create user home directory script
    ansible.builtin.copy:
    dest: /usr/local/bin/create_home_dir.sh
    content: |
    #!/bin/bash

    # Setup logging
    LOG_DIR="/var/log/home_dir"
    LOG_FILE="$LOG_DIR/homedir.log"
    
    # Create log directory if it doesn't exist
    mkdir -p $LOG_DIR
    chmod 755 $LOG_DIR
    
    # Log function
    log() {
        echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> $LOG_FILE
    }
    
    # Start logging
    log "Script started"
    log "Environment variables:"
    env | sort >> $LOG_FILE
    log "Command line arguments: $@"
    
    # Get username from command line arguments
    USERNAME=""
    while [[ $# -gt 0 ]]; do
        case $1 in
            -u|--user)
                USERNAME="$2"
                shift 2
                ;;
            -P|--pre-hook)
                # Skip the pre-hook argument and its value
                shift 2
                ;;
            *)
                shift
                ;;
        esac
    done
    
    if [ -z "$USERNAME" ]; then
        log "ERROR: Username not provided in command line arguments"
        exit 1
    fi
    
    log "Username from command line: $USERNAME"
    
    # Remove @uchicago.edu if present
    USERNAME=${USERNAME%@uchicago.edu}
    log "Final username after processing: $USERNAME"
    
    if [ -z "$USERNAME" ]; then
        log "ERROR: Username is empty after processing"
        exit 1
    fi
    
    # Check if user exists
    if ! id "$USERNAME" &>/dev/null; then
        log "ERROR: User $USERNAME does not exist"
        exit 1
    fi
    
    # Create the home directory with .ssh directory and set permissions
    log "Creating home directory for $USERNAME"
    if [ ! -d "/home/$USERNAME" ]; then
        mkdir -p /home/$USERNAME
        if [ $? -ne 0 ]; then
            log "ERROR: Failed to create home directory"
            exit 1
        fi
        log "Created new home directory"
    else
        log "Home directory already exists, ensuring proper setup"
    fi
    
    # Create .bashrc with proper content
    log "Creating .bashrc"
    cat > /home/$USERNAME/.bashrc << 'EOL'
    # .bashrc created by create_home_dir.sh
    
    # Source global definitions
    if [ -f /etc/bashrc ]; then
            . /etc/bashrc
    fi
    
    # User specific environment
    if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
    then
        PATH="$HOME/.local/bin:$HOME/bin:$PATH"
    fi
    export PATH
    
    # Uncomment the following line if you don't like systemctl's auto-paging feature:
    # export SYSTEMD_PAGER=
    
    # User specific aliases and functions
    if [ -d ~/.bashrc.d ]; then
            for rc in ~/.bashrc.d/*; do
                    if [ -f "$rc" ]; then
                            . "$rc"
                    fi
            done
    fi
    
    unset rc
    EOL
    
    # Create .bash_profile with proper content
    log "Creating .bash_profile"
    cat > /home/$USERNAME/.bash_profile << 'EOL'
    # .bash_profile created by create_home_dir.sh
    
    # Source .bashrc
    . ~/.bashrc
    EOL
    
    # Create and set .ssh directory permissions
    mkdir -p /home/$USERNAME/.ssh
    chown $USERNAME:$USERNAME /home/$USERNAME/.ssh
    chmod 700 /home/$USERNAME/.ssh
    
    # Ensure all files and directories have correct ownership and permissions
    log "Ensuring correct ownership and permissions"
    chown -R $USERNAME:$USERNAME /home/$USERNAME
    if [ $? -ne 0 ]; then
        log "ERROR: Failed to set ownership"
        exit 1
    fi
    
    chmod 700 /home/$USERNAME
    if [ $? -ne 0 ]; then
        log "ERROR: Failed to set home directory permissions"
        exit 1
    fi
    
    # Set file permissions
    chmod 644 /home/$USERNAME/.bashrc
    chmod 644 /home/$USERNAME/.bash_profile
    chmod 644 /home/$USERNAME/.bash_logout
    
    log "Script completed successfully"
    

    mode: ‘0755’
    owner: root
    group: root