[389-users] Create 389 directory server secure connections

[xinhuan zheng]

Hello All,

I screwed up my 389 directory server console authentication today because I need to set up TLS secure connections. I first started reading this document: http://directory.fedoraproject.org/docs/389ds/howto/howto-ssl.html. The document refers to a nice shell script from github: https://raw.githubusercontent.com/richm/scripts/master/setupssl2.sh. I downloaded the script, read it through. The script allows a couple environment variable setup, one of them is REMOTE variable. I really plan to have another directory server for replication so I thought that would be nice to generate it's certificate, etc beforehand. So I set up that environment variable. Then I ran command below:

REMOTE=labd2.christianbook.com; export REMOTE
./setupssl2.sh /etc/diresrv/slapd-userauth1

The very first time I got error because labd2 remote host doesn't exist yet, the script cannot generate the certificate for it because it cannot connect to it. But I typed in "Directory Manager" password, so it changed dse.ldif file. I tried to restart dirsrv-admin and dirsrv, only dirsrv-admin restarted successfully, the userauth1 instance failed restarting. Then I manually copy back dse.ldif.startOK file to dse.ldif file then restart userauth1 instance. It was restarted successfully. Then I unset REMOTE, re-run the setupssl2 script. Once it's finished, I then restarted both dirsrv-admin and dirsrv. They both restarted successfully. However, when I ran /usr/bin/389-console command, I got below error:

Cannot logon because of an incorrect User ID, Incorrect password or Directory problem.

HttpException:
HTTP/1.1 401 Authorization Required
Status: 401
URL: https://labd1.christianbook.com:9830/admin-serv/authenticate

I also tried to do ldapsearch but wasn't successful either:

# ldapsearch -d 5 -x -L -b 'dc=christianbook,dc=com'
ldap_create
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP localhost:389
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying ::1 389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
ldap_close_socket: 3
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 127.0.0.1:389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
ldap_close_socket: 3
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

It appears that when admin server TLS change takes effect but when the instance TLS wasn't in effect, then admin server cannot reconnect to instance directory server. I don't know how to fix that. Please help. Note this is 389 directory server 1.2.2 and 389 console 1.1.7. They are recent versions running on CentOS 6.7

Thanks,
- xinhuan

[William Brown]

Post by xinhuan zheng
ldap_connect_to_host: Trying 127.0.0.1:389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
ldap_close_socket: 3
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

It appears that when admin server TLS change takes effect but when the instance
TLS wasn't in effect, then admin server cannot reconnect to instance directory
server. I don't know how to fix that. Please help. Note this is 389 directory
server 1.2.2 and 389 console 1.1.7. They are recent versions running on CentOS


First problem to solve is why you can't contact the ldap server here.

Check that the dirsrv processes are starting:

sudo service status dirsrv
sudo service restart dirsrv

If they are starting, you can see them with ps:

ps aux | grep ns-slapd


If there are no processes, you have other issues.


Regardless, if the process is there, check it has listening sockets:

ss -tlnp | grep slapd

If the process is failing to start, check the error log:

tail -n 50 /var/log/dirsrv/slapd-<instance name>/errors

I think that you should check the error log anyway, as it may reveal something
else about the environment.



Once you have the server starting again, then it is time to look at why the admin
system cannot connect correctly to it. 

I hope that this helps guide you to solve the problem,

[xinhuan zheng]

Hello William,

The slapd was down by itself for some reason. Below is from error file this afternoon.

[07/Apr/2016:13:36:33 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[07/Apr/2016:13:36:33 -0400] - Listening on All Interfaces port 636 for LDAPS requests
[07/Apr/2016:14:06:12 -0400] - slapd shutting down - signaling operation threads
[07/Apr/2016:14:06:12 -0400] - slapd shutting down - waiting for 28 threads to terminate
[07/Apr/2016:14:06:12 -0400] - slapd shutting down - closing down internal subsystems and plugins
[07/Apr/2016:14:06:12 -0400] - Waiting for 4 database threads to stop
[07/Apr/2016:14:06:12 -0400] - All database threads now stopped
[07/Apr/2016:14:06:12 -0400] - slapd stopped.

So I started it: service dirsrv restart. From the error file again, it shows the process is started:

[07/Apr/2016:21:25:49 -0400] - 389-Directory/1.2.11.15 B2016.082.1529 starting up
[07/Apr/2016:21:25:49 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[07/Apr/2016:21:25:49 -0400] - Listening on All Interfaces port 636 for LDAPS requests

The 'service dirsrv status' shows it is running. However, I still get the same authorization error like before when using 389-console. I can't do ldapsearch either like before:

ldapsearch -p389 -hlabd1.christianbook.com -x -bdc=christianbook,dc=com
# extended LDIF
#
# LDAPv3
# base <dc=christianbook,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 32 No such object

# numResponses: 1

ldapsearch -p636 -hlabd1.christianbook.com -x -bdc=christianbook,dc=com
(hanging...)

- xinhuan

[xinhuan zheng]

Hello,

I can't get my 389 directory server secure connection to work. The process is started. But I can't do any ldapsearch, nor get 389 console to work. Can I get my non-secure connection work then start all over again from scratch?

- xinhuan

[William Brown]

Post by xinhuan zheng
Hello,
I can't get my 389 directory server secure connection to work. The process is
started. But I can't do any ldapsearch, nor get 389 console to work. Can I get
my non-secure connection work then start all over again from scratch?

Sorry, but I think we need more information to help you.

Can you please show the output of ldapsearch, and the commands you are using? Are
there any errors in the error log? What about the access log? 

By default, both cleartext and TLS secured connections should work if you
followed the setup.

[xinhuan zheng]

Hello Mr. Brown,

I used below ldapsearch command:

ldapsearch -d 5 -H ldaps://labd1.christianbook.com -x -D "cn=Directory Manager" -w****** -s base -b "" objectclass=*

I got below result:

ldap_url_parse_ext(ldaps://labd1.christianbook.com)
ldap_create
ldap_url_parse_ext(ldaps://labd1.christianbook.com:636/??base)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP labd1.christianbook.com:636
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 192.168.13.26:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
TLS: certdb config: configDir='/etc/openldap/certs' tokenDescription='ldap(0)' certPrefix='' keyPrefix='' flags=readOnly
TLS: cannot open certdb '/etc/openldap/certs', error -8018:Unknown PKCS #11 error.
TLS: certificate [CN=CAcert] is not valid - error -8172:Peer's certificate issuer has been marked as not trusted by the user..
TLS: error: connect - force handshake failure: errno 0 - moznss error -8172
TLS: can't connect: TLS error -8172:Peer's certificate issuer has been marked as not trusted by the user..
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Since I ran the setupssl2.sh twice, the first time there were errors in error_log, the second time it didn't appear to be:

[07/Apr/2016:13:21:37 -0400] - Warning: Adding configuration attribute "nsslapd-security"
[07/Apr/2016:13:21:37 -0400] - The change of nsslapd-secureport will not take effect until the server is restarted
[07/Apr/2016:13:23:55 -0400] - slapd shutting down - signaling operation threads
[07/Apr/2016:13:23:55 -0400] - slapd shutting down - closing down internal subsystems and plugins
[07/Apr/2016:13:23:55 -0400] - Waiting for 4 database threads to stop
[07/Apr/2016:13:23:56 -0400] - All database threads now stopped
[07/Apr/2016:13:23:56 -0400] - slapd stopped.
[07/Apr/2016:13:24:17 -0400] - SSL alert: Security Initialization: Can't find certificate (Server-Cert) for family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.)
[07/Apr/2016:13:24:17 -0400] - SSL alert: Security Initialization: Unable to retrieve private key for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.)
[07/Apr/2016:13:24:17 -0400] - SSL failure: None of the cipher are valid
[07/Apr/2016:13:24:17 -0400] - ERROR: SSL Initialization phase 2 Failed.
[07/Apr/2016:13:33:11 -0400] - SSL alert: Security Initialization: Can't find certificate (Server-Cert) for family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.)
[07/Apr/2016:13:33:11 -0400] - SSL alert: Security Initialization: Unable to retrieve private key for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.)
[07/Apr/2016:13:33:11 -0400] - SSL failure: None of the cipher are valid
[07/Apr/2016:13:33:11 -0400] - ERROR: SSL Initialization phase 2 Failed.
[07/Apr/2016:13:35:07 -0400] - 389-Directory/1.2.11.15 B2016.082.1529 starting up
[07/Apr/2016:13:35:07 -0400] - Db home directory is not set. Possibly nsslapd-directory (optinally nsslapd-db-home-directory) is missing in the config file.
[07/Apr/2016:13:35:08 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[07/Apr/2016:13:35:23 -0400] - Warning: Adding configuration attribute "nsslapd-security"
[07/Apr/2016:13:35:23 -0400] - The change of nsslapd-secureport will not take effect until the server is restarted
[07/Apr/2016:13:36:20 -0400] - slapd shutting down - signaling operation threads
[07/Apr/2016:13:36:20 -0400] - slapd shutting down - waiting for 27 threads to terminate
[07/Apr/2016:13:36:20 -0400] - slapd shutting down - closing down internal subsystems and plugins
[07/Apr/2016:13:36:20 -0400] - Waiting for 4 database threads to stop
[07/Apr/2016:13:36:21 -0400] - All database threads now stopped
[07/Apr/2016:13:36:21 -0400] - slapd stopped.
[07/Apr/2016:13:36:33 -0400] - 389-Directory/1.2.11.15 B2016.082.1529 starting up
[07/Apr/2016:13:36:33 -0400] attrcrypt - No symmetric key found for cipher AES in backend userRoot, attempting to create one...
[07/Apr/2016:13:36:33 -0400] attrcrypt - Key for cipher AES successfully generated and stored
[07/Apr/2016:13:36:33 -0400] attrcrypt - No symmetric key found for cipher 3DES in backend userRoot, attempting to create one...
[07/Apr/2016:13:36:33 -0400] attrcrypt - Key for cipher 3DES successfully generated and stored
[07/Apr/2016:13:36:33 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[07/Apr/2016:13:36:33 -0400] - Listening on All Interfaces port 636 for LDAPS requests
[07/Apr/2016:14:06:12 -0400] - slapd shutting down - signaling operation threads
[07/Apr/2016:14:06:12 -0400] - slapd shutting down - waiting for 28 threads to terminate
[07/Apr/2016:14:06:12 -0400] - slapd shutting down - closing down internal subsystems and plugins
[07/Apr/2016:14:06:12 -0400] - Waiting for 4 database threads to stop
[07/Apr/2016:14:06:12 -0400] - All database threads now stopped
[07/Apr/2016:14:06:12 -0400] - slapd stopped.
[07/Apr/2016:21:25:49 -0400] - 389-Directory/1.2.11.15 B2016.082.1529 starting up
[07/Apr/2016:21:25:49 -0400] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[07/Apr/2016:21:25:49 -0400] - Listening on All Interfaces port 636 for LDAPS requests

Below is in my access file:

389-Directory/1.2.11.15 B2016.082.1529
labd1.christianbook.com:636 (/etc/dirsrv/slapd-userauth1)

[11/Apr/2016:20:27:35 -0400] conn=27 fd=64 slot=64 SSL connection from 192.168.13.26 to 192.168.13.26
[11/Apr/2016:20:27:35 -0400] conn=27 op=-1 fd=64 closed - Peer does not recognize and trust the CA that issued your certificate.
[11/Apr/2016:21:14:18 -0400] conn=28 fd=64 slot=64 SSL connection from 192.168.13.26 to 192.168.13.26
[11/Apr/2016:21:14:18 -0400] conn=28 op=-1 fd=64 closed - Peer does not recognize and trust the CA that issued your certificate.

Below is my server certificate output:

# certutil -L -d /etc/dirsrv/slapd-userauth1/

Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI

CA certificate CTu,u,u
Server-Cert u,u,u

Below is my admin server certificate output:

# certutil -L -d /etc/dirsrv/admin-serv/

Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI

server-cert u,u,u
CA certificate CT,,

I also have an orphaned private key:

certutil -K -d /etc/dirsrv/slapd-userauth1/
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
--> < 0> rsa 7ce6fa7d24be45acd0d4e03eb6aea8b2dd62be69 (orphan)
< 1> rsa 51b9007f7669c1aebb9750ed0b24055d22d212fa NSS Certificate DB:CA certificate
< 2> rsa daaf50b878d189db111b5488034d302012538da1 NSS Certificate DB:Server-Cert

I think I'll need to delete all certificates then re-create them all from scratch. One thing I am not sure is the certificate flags, in setupssl2.sh. For "CA certificate" the flag is "CT,,", while for server certificate, it is "u,u,u", is it correct?

Thanks,
- xinhuan

[William Brown]

Post by xinhuan zheng
Hello Mr. Brown,
 
TLS: certdb config: configDir='/etc/openldap/certs' tokenDescription='ldap(0)'
certPrefix='' keyPrefix='' flags=readOnly
TLS: cannot open certdb '/etc/openldap/certs', error -8018:Unknown PKCS #11 error.
TLS: certificate [CN=CAcert] is not valid - error -8172:Peer's certificate
issuer has been marked as not trusted by the user..
TLS: error: connect - force handshake failure: errno 0 - moznss error -8172
TLS: can't connect: TLS error -8172:Peer's certificate issuer has been marked
as not trusted by the user..
 
[11/Apr/2016:21:14:18 -0400] conn=28 fd=64 slot=64 SSL connection from
192.168.13.26 to 192.168.13.26
[11/Apr/2016:21:14:18 -0400] conn=28 op=-1 fd=64 closed - Peer does not
recognize and trust the CA that issued your certificate.
 
I think I'll need to delete all certificates then re-create them all from
scratch. One thing I am not sure is the certificate flags, in setupssl2.sh. For
"CA certificate" the flag is "CT,,", while for server certificate, it is
"u,u,u", is it correct?

The issue is that you do not have your CA cert in /etc/openldap/certs.

I helped with this same issue in the subject titled:

[389-users] Re: admin and Directory Manager accounts cannot log into 389-console


The answer I gave there:


So you should take the current CA cert from the slapd instance:

certutil -L -d /etc/dirsrv/slapd-E2WAN/ -n wsf-LabCA.lab.aero.org -a >
/etc/openldap/cacerts/wsf-LabCA.lab.aero.org.pem

Then you can make these valid for openldap to use:

cd /etc/openldap/cacerts
cacertdir_rehash

This will recreate the hash -> cert symlinks.

From there, re-run your ldap search command:

 ldapsearch -d 5 -x -L -b 'dc=lab,dc=aero,dc=org'



You will need to adjust these commands to match your instances, the CA certs, and
your openldap/certs location. Otherwise, that should fix the issue.

[xinhuan zheng]

Hello Mr. Brown,

I found that the procedure you give me is part of my problem. I ended up with running remove-ds-admin.pl command then re-create my directory server instance and admin server instance. Luckily I kept answers file and ldif data. I also re-run the setupssl2.sh script. Since this time I didn't mess up with environment variables, it did go through successfully. I also did go through your procedure, and successfully run the ldapsearch command. However, how do I know "ldapsearch -x -L -b 'dc=christianbook,dc=com'" uses secure connection or not? I tail my access log, and it shows something like below:

[12/Apr/2016:21:24:47 -0400] conn=46 op=0 EXT oid="1.3.6.1.4.1.1466.20037" name="startTLS"
[12/Apr/2016:21:24:47 -0400] conn=46 op=0 RESULT err=0 tag=120 nentries=0 etime=0
[12/Apr/2016:21:24:47 -0400] conn=46 op=-1 fd=64 closed - Peer does not recognize and trust the CA that issued your certificate.
[12/Apr/2016:21:26:46 -0400] conn=47 fd=64 slot=64 connection from ::1 to ::1
[12/Apr/2016:21:26:46 -0400] conn=47 op=0 BIND dn="" method=128 version=3
[12/Apr/2016:21:26:46 -0400] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn=""
[12/Apr/2016:21:26:46 -0400] conn=47 op=1 SRCH base="dc=christianbook,dc=com" scope=2 filter="(objectClass=*)" attrs=ALL
[12/Apr/2016:21:26:46 -0400] conn=47 op=1 RESULT err=0 tag=101 nentries=103 etime=0 notes=U
[12/Apr/2016:21:26:46 -0400] conn=47 op=2 UNBIND
[12/Apr/2016:21:26:46 -0400] conn=47 op=2 fd=64 closed - U1

It did say "Peer does not recognize and trust the CA that issued your certificate." However, it did return 103 entries for the ldapsearch. I don't understand that. Is the secure connection successfully established or is it actually failed back to non-secure connection?

Thanks,
- xinhuan

[William Brown]

Post by xinhuan zheng
Hello Mr. Brown,
I found that the procedure you give me is part of my problem. I ended up with
running remove-ds-admin.pl command then re-create my directory server instance
and admin server instance. Luckily I kept answers file and ldif data. I also
re-run the setupssl2.sh script. Since this time I didn't mess up with
environment variables, it did go through successfully.

You should not have needed to do that. 

Post by xinhuan zheng
I also did go through your procedure, and successfully run the ldapsearch
command. However, how do I know "ldapsearch -x -L -b 'dc=christianbook,dc=com'"
uses secure connection or not? I tail my access log, and it shows something
[12/Apr/2016:21:24:47 -0400] conn=46 op=0 EXT oid="1.3.6.1.4.1.1466.20037" name="startTLS"
[12/Apr/2016:21:24:47 -0400] conn=46 op=0 RESULT err=0 tag=120 nentries=0 etime=0
[12/Apr/2016:21:24:47 -0400] conn=46 op=-1 fd=64 closed - Peer does not
recognize and trust the CA that issued your certificate.
[12/Apr/2016:21:26:46 -0400] conn=47 fd=64 slot=64 connection from ::1 to ::1
[12/Apr/2016:21:26:46 -0400] conn=47 op=0 BIND dn="" method=128 version=3
[12/Apr/2016:21:26:46 -0400] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn=""
[12/Apr/2016:21:26:46 -0400] conn=47 op=1 SRCH base="dc=christianbook,dc=com"
scope=2 filter="(objectClass=*)" attrs=ALL
[12/Apr/2016:21:26:46 -0400] conn=47 op=1 RESULT err=0 tag=101 nentries=103 etime=0 notes=U
[12/Apr/2016:21:26:46 -0400] conn=47 op=2 UNBIND
[12/Apr/2016:21:26:46 -0400] conn=47 op=2 fd=64 closed - U1
It did say "Peer does not recognize and trust the CA that issued your
certificate." However, it did return 103 entries for the ldapsearch. I don't
understand that. Is the secure connection successfully established or is it
actually failed back to non-secure connection?

If you look in the ldap client config, /etc/openldap/ldap.conf, you likely have
TLS_REQCERT set to NEVER or ALLOW. This means it checks the CA, but will not fail
on an verification failure.

So it is a "secure" connection, but the client hasn't validated the CA.

Post by xinhuan zheng
Thanks,
- xinhuan
--
389 users mailing list