Ondemand cert issue

Get Help
1

Here’s an issue i see while running an ansible yum download of ondemand with the rpm

Ansible Error
fatal: [localhost]: FAILED! => {“changed”: false, “msg”: “Failure downloading https://yum.osc.edu/ondemand/2.0/ondemand-release-web-2.0-1.noarch.rpm, Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>”}

My code

  • name: Add Open OnDemand’s repository hosted by the Ohio Supercomputer Center
    yum:
    name: “{{ ood_rpm_repo }}”
    state: present

where ood_rpm_repo : https://yum.osc.edu/ondemand/1.7/ondemand-release-web-1.7-1.noarch.rpm

Looks like ansible does a validate_certs when the yum downloads from a url, and setting validate_certs to false seems to work

2

You know we provide a role for Open OnDemand.

That said - I don’t think I’ve had any issues with our yum certificates. It does look like however the certificate currently being served by yum.osc.edu only started to be valid March 23rd (2 days ago). So it could have just been bad luck to run into it at that time.

3

[root@login001 flask_user_reg]# openssl s_client -connect yum.osc.edu:443
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT

Certificate chain
0 s:/CN=repo.hpc.osc.edu
i:/C=US/O=Let’s Encrypt/CN=R3
1 s:/C=US/O=Let’s Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Server certificate
-----BEGIN CERTIFICATE-----
MIIGcTCCBVmgAwIBAgISA9mj2jE4Z3eSaxaGVkUEt0//MA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjAzMjMxMzUzMzdaFw0yMjA2MjExMzUzMzZaMBsxGTAXBgNVBAMT
EHJlcG8uaHBjLm9zYy5lZHUwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC
AQDQ7FsNjz0WzbhTsJBIEcJFIb7yxcxXEYyo9P84de1i1W0XIqszR5ayUiMZQ2g4
eKsTVPH3yplVUp60noptFpmGJ3lWibZtGaV6zcaAJh8PZ+4sx+RNuUYr+juXclZM
SdrN9ScSq0YtztGBrTepHDD1XeLYl9ig5q4/VE7cnWPsKsbcGdzEVL8KDMsZurIf
w54qDwBPBFfP14pkCSNE1K+XPn78S//EXGKCbqcfJI9aPWLjk99Gu4pb9KIuiSFX
CUCdrLtf5StIcBJmrouZzRXXjGeYF5oumKYUmOll5bSDi/sZxkGnqcLc7fN9x/kd
qCG7fHq1aahWVHTSvbbkl6ucGbaVN9eGIdDt1imVeMmiQPO0GLQEgXjcZPTd9RIJ
NO6gXQ0SpGc00QRA0TcZOYKfpKNCkgcRp1sdblvHcQO8u116nUlf76+7/OtRY+D2
z49PR05iyajwpXzuc5j1r4qgvcf0pm8/XQ15LIrNrMdz6ugrvgbybIR8MixSHHrB
H3DYil79Bqdq0wKjBcFzqaUYBm9jp+m+zMpsLeiJNQAvyg4kVlSfhGPddSdyG9oE
X0h1KJuyy6Bdx5jYhnVv4OMQSBWTpIrKG+7V/9nzF2gC3qB70gA0TpKlsc6FOpgg
gAMsj4FnzLBPex/IO5NlUnsZEOPqt4Lo8DWXxBmpyQDZHQIDAQABo4ICljCCApIw
DgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAM
BgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQJrL7/VTRL3wtvC2hJV+WFV7aqJjAfBgNV
HSMEGDAWgBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYI
KwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0
cDovL3IzLmkubGVuY3Iub3JnLzBlBgNVHREEXjBcggthcHQub3NjLmVkdYIRZG93
bmxvYWRzLm9zYy5lZHWCDm1pcnJvci5vc2MuZWR1ghByZXBvLmhwYy5vc2MuZWR1
ggtyaG4ub3NjLmVkdYILeXVtLm9zYy5lZHUwTAYDVR0gBEUwQzAIBgZngQwBAgEw
NwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5j
cnlwdC5vcmcwggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdgBByMqx3yJGShDGoToJ
QodeTjGLGwPr60vHaPCQYpYG9gAAAX+3RNrAAAAEAwBHMEUCIHC4C383G7giceUv
i4YAOFeJNgZV/mquoMs/VvAjkZ1iAiEAuUhDLN3Yd+L78Vi5pAIs1JSseOmUtjAd
LFrDvgDpIqsAdwApeb7wnjk5IfBWc59jpXflvld9nGAK+PlNXSZcJV3HhAAAAX+3
RNqoAAAEAwBIMEYCIQDDKxP1zEnaCV7BGf48qc1zEJoZ4ujqANuvJV/R4bHYwAIh
AMsr2AeqloW6B9O8ZMRvYCFGMSy2i/OlCZzlEJuX1zOAMA0GCSqGSIb3DQEBCwUA
A4IBAQCcftOmaKMK03QWlMZy48oLganmZnUNJFnJk/pSORjbkdE2hcNVy4zvXdSZ
y9h5gZM1XS0z+TlXlA9KguBExBPAOHFfu2ScZzFu2iuVlKzGoc+J+k2OyowHjWDj
Kd06Coe6APhXQtWc9+nkFmhoelr1D/DdvNC+ezXZNX0/Kts4H5qY3oGV1GU8/RUC
rGFlMQ4XEXw1Tz6xRkmiCymHeVNZcnOG2dqxmXRdk6naBX/v0gju3qPfOOi+4Dwd
pBRIulR5VeQOVNVvI/MSiASUM5xD/TGtWB0IiQwZxlrOTIYX/De//DTj2NDA4oNr
BxBNzzSa7o7TQEzv0O4eebgeLocE
-----END CERTIFICATE-----
subject=/CN=repo.hpc.osc.edu
issuer=/C=US/O=Let’s Encrypt/CN=R3

No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 5292 bytes and written 415 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: CD160806633192F72FC10BB89B050F8867FC34BCC5971F0A75AECBA2F840DABC
Session-ID-ctx:
Master-Key: 0E11B48309BA9020DDC3C56682BBAFF3A29EEA0188E45F14901A68F48EBF63081F524230AE71DD45D3974AE789284A13
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 0c 40 76 1c dd 86 fc da-34 5c b3 40 54 54 94 53 .@v…4.@TT.S
0010 - 78 54 49 19 ce 48 1c 4d-6d a2 df 4e ba ed a4 9f xTI…H.Mm…N…
0020 - f3 43 6e 81 42 6d 7d 37-2a d6 aa 48 32 1f 00 1e .Cn.Bm}7*…H2…
0030 - 4c b9 e0 bd 17 73 2e 65-ad f9 e6 3e 35 08 22 48 L…s.e…>5."H
0040 - b1 40 7f e0 8b 4b 6d 03-a8 ef 68 46 54 c6 cc 9e .@…Km…hFT…
0050 - 28 c9 9e 8a 82 09 7c 82-b9 3c 8e 11 a2 ef e2 bd (…|…<…
0060 - d7 9c 73 6d 3b 2c 37 71-60 79 e5 f8 cf 48 8f 3e …sm;,7q`y…H.>
0070 - 37 3f 88 6d a6 f4 c5 e8-bb 89 5b 8e 94 59 f3 b2 7?.m…[…Y…
0080 - 82 6c f9 88 d7 41 d3 75-d0 3f 1a 0d 06 bf 3f 28 .l…A.u.?..?(
0090 - 5e bd 31 49 90 52 bb b0-b3 13 ce c6 7a 2c b0 31 ^.1I.R…z,.1
00a0 - eb b4 41 56 4c d0 56 72-d5 90 12 6a f2 b9 da ed …AVL.Vr…j…
00b0 - b7 f0 43 50 fe 10 13 72-ce cb bb ed 5a 56 0a 47 …CP…r…ZV.G

Start Time: 1648221427
Timeout   : 300 (sec)
Verify return code: 10 (certificate has expired)

closed
[root@login001 flask_user_reg]# openssl s_client -connect yum.osc.edu:443^C
[root@login001 flask_user_reg]# dig yum.osc.edu

    if 'user' not in session:
        session["user"] = get_authorized_user()

    session['return_url'] = request.args.get('redir', vars.default_referrer)

    if session['user'].get('username') != "u1":
        return render_template('errors/error.html')

    else:
        return render_template('auth/SignUp.html', room_id=session['uid'],
                           username=session['user'].get('username'),
                           fullname=session['user'].get('fullname'), email=session['user'].get('email'),
                           referrer=session['return_url'], cancel_url=vars.default_referrer,
                           welcome_msg=messages.welcome_message,
                           cancel_msg=messages.cancel_message,
                           error_msg=messages.error_message)

@app.route('/error_account')
def error_account_create():
    return render_template('errors/error.html', title='account creation failed')


# misc page error catching
@app.errorhandler(403)
def forbidden(error):
    return render_template('errors/403.html', title='Forbidden'), 403

@app.errorhandler(404)
def page_not_found(error):
    return render_template('errors/404.html', title='Page Not Found'), 404

@app.errorhandler(500)
def internal_server_error(error):
    return render_template('errors/500.html', title='Server Error'), 500

return app

“app/init.py” 75L, 2929C 60,0-1 Bot
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> yum.osc.edu
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38255
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 9

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;yum.osc.edu. IN A

;; ANSWER SECTION:
yum.osc.edu. 361 IN CNAME repo.hpc.osc.edu.
repo.hpc.osc.edu. 41138 IN A 192.148.247.146

;; AUTHORITY SECTION:
hpc.osc.edu. 41138 IN NS ns5.oar.net.
hpc.osc.edu. 41138 IN NS ns6.oar.net.
hpc.osc.edu. 41138 IN NS ns7.oar.net.
hpc.osc.edu. 41138 IN NS ns4.oar.net.

;; ADDITIONAL SECTION:
ns7.oar.net. 39536 IN A 206.244.199.2
ns5.oar.net. 39536 IN A 199.218.199.2
ns4.oar.net. 39536 IN A 199.18.199.2
ns6.oar.net. 39536 IN A 157.134.199.2
ns7.oar.net. 39536 IN AAAA 2610:a8:2007::2
ns5.oar.net. 39536 IN AAAA 2610:a8:2005::2
ns4.oar.net. 39536 IN AAAA 2610:a8:2004::2
ns6.oar.net. 39536 IN AAAA 2610:a8:2006::2

;; Query time: 2 msec
;; SERVER: 10.141.255.254#53(10.141.255.254)
;; WHEN: Fri Mar 25 10:21:06 CDT 2022
;; MSG SIZE rcvd: 334

dig +short yum.osc.edu
repo.hpc.osc.edu.
192.148.247.146

4

Yea it seems like it was just bad timing. openssl s_client -connect yum.osc.edu:443 returns OK now.

    Start Time: 1648223639
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no

I should also note that on Tuesday we had an OSC Downtime, though I didn’t realize certificates are being updated at the same time. @tdockendorf I guess we rotate certs during our downtime? Seems like I should announce the outage either way, whether we bounce the servers or no.

5

This only seems to be a problem when I’m inside a VM. From my mac, it seems fine

6

The InCommon certificate we were using had 13 days left which triggered OSC monitoring alerts. I have been replacing InCommon (not free) with LetsEncrypt (free) and that was done a few days ago. If the certificates are no longer seen as valid on a system then that system likely has out of date root certificates in the global trust store. On RHEL systems this is usually resolved by updating the ca-certificates package to pull in the latest trusted root CAs.

7

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.