Rstudio app works BUT each session begins with a single auth failure that must be ignored

[jasonbuechler]

Hiya everyone,

We’re running centos 6 and slurm 19, and are one OOD release behind.
I just setup the RStudio interactive app but am quite perplexed by this reproducible and ignorable authentication error that follows immediately after clicking “Connect to RStudio Server” for each session:

“Error: Incorrect or invalid username/password”

One can simply go back to the OOD tab and click the button again (with or without closing this goofy error tab) and every subsequent connection to that session works great!

The only logs that seem connected that we’ve so far seen:

___tailing <computenode>:/var/log/messages___
Jan  8 17:14:24 cn68 nslcd[3328]: [456249] nslcd_passwd_byname(): invalid user name
Jan  8 17:14:24 cn68 rserver[67254]: ERROR system error 13 (Permission denied) [user-value=]; OCCURRED AT: rstudio::core::Error rstudio::core::system::user::{anonymous}::userFrom(const rstudio_boost::function<int(T, passwd*, char*, long unsigned int, passwd**)>&, T, rstudio::core::system::user::User*) [with T = const char*] /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixUser.cpp:107; LOGGED FROM: std::string rstudio::server::pam_auth::{anonymous}::userIdentifierToLocalUsername(const string&) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerPAMAuth.cpp:178
Jan  8 17:14:24 cn68 rserver[67254]: WARNING No PAM password provided for user ''; refusing login; LOGGED FROM: bool rstudio::server::pam_auth::pamLogin(const string&, const string&) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerPAMAuth.cpp:456
_____________________________________________

…and just as a reminder, clicking the button from this point on works perfectly fine without touching anything else. Like magic!

This may or may not be related to a much less reproducible issue that might happen on a per-browser basis: After clicking the “Launch” button at the session setup form, and being bounced to https://ondemand.hpc.nau.edu/pun/sys/dashboard/batch_connect/sys/RStudio/session_contexts
…Instead of seeing “Session was successfully created” in a friendly green box, we’ll see “This app does not supply a sub app form file under the directory ‘/var/www/ood/apps/sys/RStudio/local’” in a hangry red box.

Since at this step, it seems it hasn’t yet generated a valid session, our ondemand node itself has some logs:

___tailing ondemand:/var/log/secure___
Jan  8 17:05:43 ondemand sudo:   apache : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/ood/nginx_stage/sbin/nginx_stage pun -u jtb49 -a https%3a%2f%2fondemand.hpc.nau.edu%3a443%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri
Jan  8 17:07:28 ondemand sudo: pam_unix(sudo:auth): authentication failure; logname=jtb49 uid=129387 euid=0 tty=/dev/pts/1 ruser=jtb49 rhost=  user=jtb49
Jan  8 17:07:28 ondemand sudo:    jtb49 : TTY=pts/1 ; PWD=/var/www/ood/apps/sys/RStudio ; USER=root ; COMMAND=/usr/bin/tail -n 500 /var/log/messages
Jan  8 17:08:53 ondemand sudo:    jtb49 : TTY=pts/1 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/tail secure
______________________________________

Is this familiar to anyone?
Best next steps?

Thanks very much!
Jason Buechler
NAU Monsoon

[jeff.ohrstrom]

Do you happen to use CAS authentication? If so, this topic below will be helpful, if not the answer.

Even if you don’t use CAS, it may be worth looking though this anyhow for information about how these apps use the passwords they’re given. It may give you ideas on how to troubleshoot, what files to look for, html elements to inspect, what API calls in your browser to look at and so on.

Jupyter app ask for password first time, works second time

Hi, I am working on adjusting the Jupyter app to our cluster, the app is currently in Sandbox. When I click the button “Connect to Jupyter” it opens a new tab with Jupyter asking for password, if I return and click again it opens Jupyter’s dashboard as it should. I wonder what is the cause of not having Jupyter dashboard displayed from the first click Thank you

Let me know if that’s helpful!

[jasonbuechler]

Ooh thank you for the tip! We do indeed use CAS, and I’ll dig into that thread right this second!

[jasonbuechler]

You nailed it, Jeff! We were missing those CASScope entries in the config, and it now runs like a dream. Thanks so much!
–jason

PS: it seemed this was the root cause of both of the issues. Double win!

[flybirdkh]

Hi everyone ,

I’ve got the same problem, Error: Incorrect or invalid username/password

my authentication is ldapserver,how to set the script.sh.erb.

Thank you for your replay.

[jeff.ohrstrom]

I take it your RStudion install is pretty much just like ours? First, I’d ask if you’re using CAS as you’re single sign on authentication. That what the issue in this topic originally. If not, then let’s debug a bit.

Here’s what a good login to RStudio looks like. You can see in the form data I pass a username (my own) and a password (a temporarily generated one). The response is 302 Moved Temporarily and has a Location header.

Please use Chrome’s dev tools to see what’s being requested and returned. Verify you’re sending a password and it’s the correct one (the same one in connection.yml).

[flybirdkh]

Thank you for your reply. there were no user’s data in auth-do-sign-in.

maybe setup error.
my authentication is signed by ldapserver

and I am a newcomer,I do not know what to do next

[jeff.ohrstrom]

OK so you’re being redirected correctly so that’s good. Are you saying there is no Form Data?

This bit here below request headers? You should be sending at least your username.

You’ll need to look into the working directory of the job. It takes this form.

$HOME/ondemand/data/sys/dashboard/batch_connect/sys/$APP_NAME/output/$UUID

Here’s an actual path I took from my where my $APP_NAME is bc_osc_rstudio_server and UUID is bec8d7f1-45a4-4aea-9247-d716b5def7d8.

/users/PZS0714/johrstrom/ondemand/data/sys/dashboard/batch_connect/sys/bc_osc_rstudio_server/output/bec8d7f1-45a4-4aea-9247-d716b5def7d8

First there’ll be an output.log file. This is the job output log so it’s very useful in debugging.

Secondly, this directory has the connection.yml file. This is the authentication used for applications for this particular job execution, so LDAP doesn’t really play a part here. This file is read and is used for the view to use for when users click ‘connect to the app’ (what you’ve shown there). Make sure this file is being populated with a password. The password is created here in the before.sh.erb. This configuration element is also important.

In that same directory described above (the job’s working directory) you’ll see a job_script_content.sh. This is the script that actually get’s submitted to the scheduler and invokes all the other scripts. You see here, it runs the script.sh (the thing that boots rstudio) but also creates the connection.yml file (in create_yml).

echo "Script starting..."
"/some_directory/script.sh" &
SCRIPT_PID=$!

[[ -e "/some/directory/after.sh" ]] && source "/some/directory/after.sh"

# Create the connection yaml file
create_yml

So, that’s how you verify the password is being created, read and sent.

How you get authorized is through this file. In the script you’ve provided it’s linked to as ${RSTUDIO_AUTH}. Make sure this file is executable and works correctly for you. Here’s where you can also add additional echo statements when trying to debug.

[flybirdkh]

thanks very much
I’ve not got the Form Data

the session show me a error=1

I can find all the file you say,and my work path is hpccf/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/89a0409f-293c-4db1-9ad5-230679da256b/

information about out.log

${RSTUDIO_AUTH} cannot be find in my workdir;

Maybe I should show you my script.sh.erb

[jeff.ohrstrom]

OK, yea I see in your previous image that you’re running an http GET command. Should be POST, this is why you’re not passing any data to the rstudio server (password among it)

Your view should look like ours. In that it is a <form> html element and it has a method=post attribute.

[flybirdkh]

my script.sh.erb

and how to change the http GET o POST

[jeff.ohrstrom]

It’s in your view.html.erb the link I just shared is to ours.

But here’s that file in full

<form action="/rnode/<%= host %>/<%= port %>/auth-do-sign-in" method="post" target="_blank">
  <input type="hidden" name="username" value="<%= ENV["USER"] %>">
  <input type="hidden" name="password" value="<%= password %>">
  <input type="hidden" name="staySignedIn" value="1">
  <input type="hidden" name="appUri" value="">
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-registered"></i> Connect to RStudio Server
  </button>
</form>

[flybirdkh]

view.html.erb

this file has not been edited

[jeff.ohrstrom]

Then I guess I’d say be sure it shows up in your browser like that. Here in dev tools - Inspector you can navigate to that button and see if it is indeed method=post. If it isn’t, then something needs refreshed/bounced because it’s cached in apache or something.

It it is in your browser like that and in the right file, then … I don’t know, there’s something in transit changing it?

[flybirdkh]

It show me method=‘post’

https://studio.hpc.sjtu.edu.cn/rnode/cas037.pi.sjtu.edu.cn/60737/auth-sign-in?error=1
I delete ?error=1 maybe give me the wrong informaion

there is a error when i connect rstudio server

[jeff.ohrstrom]

Sorry that’s a inspection of the open ondemand page that holds the button to ‘connect to rstudio’ at /pun/sys/dashboard/batch_connect/sessions

[flybirdkh]

This is my process:
1.cd $HOME/ondemand/dev/
git clone https://github.com/OSC/bc_osc_rstudio_server.git
2.cd bc_osc_rstudio_server
edit three file:
form.yml(change cluster name)
submit.yml.erb

template/script.sh.erb

above is all my operation.

[jeff.ohrstrom]

Well that should be working just fine. You can see in your post #7 you do a Reqest Method: GET and you can see in my post #6 I do a Request Method: POST . This is your issue. Is there something in between you and OOD (or between OOD and the compute node) that would be changing the request? Do you have something else special in your httpd config that may be doing this? or a web browser plugin that’s changing the method?

Something, somewhere clearly is, though I can’t say who or what. That should work directly, and it seems something outside of OOD is changing the request in flight.

[flybirdkh]

Thank you very much, let me do some checking .

[flybirdkh]

#6 maybe my wrong operation. when I get the address:https://studio.hpc.sjtu.edu.cn/rnode/cas037.pi.sjtu.edu.cn/60737/auth-sign-in?error=1. #6 is showed by deleting the error=1.
Now I reconfig ,the process at #18, and the result is same as before.