That error can happen for one of two reason. One is the CA that signed the certificate is not trusted by the system. See here: Unknown certificate authority when using openondemand-dex. This appears to have happened already with someone using LetsEncrypt.
The other issue that can cause that error is if the URL you referenced is not valid for that certificate. The URL host used for Dex must either match the certificate CN or the Subject Alt Names.