Discussion:
[389-users] dsctl healthcheck bug - or bad at least a bad resolution
Gary Waters
2021-04-15 01:41:18 UTC
Permalink
Hi Guys!

I think I found a bug in dsctl, and wanted to give some background and
see what you guys thought.

I am setting up my ldaphub.. and I am getting an odd issue when running
the dsctl $instance healthcheck on it, but the dsctl $instance
get-nsstate shows that the missing part is right there. I have confirmed
this by looking directly at the dse.ldif file and finding the
"resolution" is already present.

Error and get-nsstate are below. It will be same the error 8 times in a
row.

Hmm.. it seems to be related to maybe how I setup the replication
agreement and consumer, so I added that at the bottom as well.

I found something interesting, if i set the replication ID for the hub,
dsconf wont use the ID number I put in, dsconf puts in a number outside
a valid range 65535. Have you guys seen this ?

Thanks guys for everything!

-Gary


Here is the error (8x):

Severity: MEDIUM
Check: backends:somesuffixroot:mappingtree
Affects:
 -- somesuffixroot

Details:
-----------
This backend may be missing the correct mapping tree references. Mapping
Trees allow
the directory server to determine which backend an operation is routed
to in the
abscence of other information. This is extremely important for correct
functioning
of LDAP ADD for example.

A correct Mapping tree for this backend must contain the suffix name,
the database name
and be a backend type. IE:

cn=o3Dexample,cn=mapping tree,cn=config
cn: o=example
nsslapd-backend: userRoot
nsslapd-state: backend
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree

Resolution:
-----------
Either you need to create the mapping tree, or you need to repair the
related
mapping tree. You will need to do this by hand by editing cn=config, or
stopping
the instance and editing dse.ldif.

dsctl ldaphub get-nsstate

Replica DN:
cn=replica,cn=ou\3dsomesuffix\2co\3dcaltech\2cc\3dus,cn=mapping
tree,cn=config
Replica Suffix:       ou=somesuffix,o=caltech,c=us
Replica ID:           65535
Gen Time:             1618442292
Gen Time String:      Wed Apr 14 16:18:12 2021
Gen as CSN:           607778340002655350000
Local Offset:         0
Local Offset String:  0 seconds
Remote Offset:        7
Remote Offset String: 7 seconds
Time Skew:            7
Time Skew String:     7 seconds
Seq Num:              2
System Time:          Wed Apr 14 17:30:50 2021
Diff in Seconds:      4358
Diff in days/secs:    0:4358
Endian:               Little Endian

Dse.ldif section that already has the resolution present:

dn: cn=ou\3Dsomesuffix\2Co\3Dcaltech\2Cc\3Dus,cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
nsslapd-state: referral on update
nsslapd-backend: somesuffixRoot
cn: ou=somesuffix,o=caltech,c=us
creatorsName: cn=directory manager
modifiersName: cn=server,cn=plugins,cn=config
createTimestamp: 20210415004818Z
modifyTimestamp: 20210415005939Z
numSubordinates: 1
nsslapd-referral:
ldap://supplier2:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
nsslapd-referral:
ldap://supplier1:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
nsslapd-referral:
ldap://supplier0:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
nsslapd-referral:
ldap://supplier4.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
nsslapd-referral:
ldap://supplier5.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
nsslapd-referral:
ldap://supplier3.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus

How I set it set up the hub and the agreement: (note the same commands i
used to setup the suppliers and consumers worked great with only
variance is really the role)

# how i setup the consumer
dsconf -D "cn=Directory Manager" -w XXX ldap://$consumer replication
enable --suffix="ou=somesuffix,o=caltech,c=us" --role="hub"
--replica-id=6001 --bind-dn="cn=replication manager,cn=config"
--bind-passwd=XXX

# how i setup the agreement
dsconf -D "cn=Directory Manager" -w XXXX ldap://supplier repl-agmt
create --suffix="ou=somesuffix,o=caltech,c=us" --host=consumer --port=389 \
     --conn-protocol=StartTLS --bind-dn="cn=replication
manager,cn=config" \
     --bind-passwd=XXXX --bind-method=SIMPLE --init \
     replication-agreement-name-super-awesome
_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io
Mark Reynolds
2021-04-15 14:54:00 UTC
Permalink
Post by Gary Waters
Hi Guys!
I think I found a bug in dsctl, and wanted to give some background and
see what you guys thought.
I am setting up my ldaphub.. and I am getting an odd issue when
running the dsctl $instance healthcheck on it, but the dsctl $instance
get-nsstate shows that the missing part is right there. I have
confirmed this by looking directly at the dse.ldif file and finding
the "resolution" is already present.
I don't see your backend entry in your output, just the mapping tree
entry.  It takes two entgries to define a backend and suffix (annoying I
know but that's how it works).  So how did you create your suffix?  Did
you use ldapmodify or did you use dsconf?


For example, this is how my backend configuration looks:

dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
cn: dc=example,dc=com
cn: dc\=example\,dc\=com
nsslapd-state: backend
nsslapd-backend: userroot
creatorsName: cn=dm
modifiersName: cn=server,cn=plugins,cn=config
createTimestamp: 20210413191244Z
modifyTimestamp: 20210414165444Z
nsslapd-referral: ldap://localhost.localdomain:5555/dc%3Dexample%2Cdc%3Dcom
numSubordinates: 1

...

dn: cn=userroot,cn=ldbm database,cn=plugins,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsBackendInstance
cn: userroot
creatorsName: cn=dm
modifiersName: cn=dm
createTimestamp: 20210413191244Z
modifyTimestamp: 20210413191244Z
numSubordinates: 5
nsslapd-suffix: dc=example,dc=com
nsslapd-cachesize: -1
nsslapd-cachememsize: 939524096
nsslapd-readonly: off
nsslapd-require-index: off
nsslapd-require-internalop-index: off
nsslapd-dncachememsize: 134217728
nsslapd-directory: /var/lib/dirsrv/slapd-localhost/db/userroot



Do you have these two entries in your config?  If so, please show both
of them.
Post by Gary Waters
Error and get-nsstate are below. It will be same the error 8 times in
a row.
Hmm.. it seems to be related to maybe how I setup the replication
agreement and consumer, so I added that at the bottom as well.
I found something interesting, if i set the replication ID for the
hub, dsconf wont use the ID number I put in, dsconf puts in a number
outside a valid range 65535. Have you guys seen this ?
You can not set a replica ID for a hub.  Only supplier replicas get
unique replica ID's.  So when you try and set a replica id on a hub or
consumer it will get replaced by 65535.

Mark
Post by Gary Waters
Thanks guys for everything!
-Gary
Severity: MEDIUM
Check: backends:somesuffixroot:mappingtree
 -- somesuffixroot
-----------
This backend may be missing the correct mapping tree references.
Mapping Trees allow
the directory server to determine which backend an operation is routed
to in the
abscence of other information. This is extremely important for correct
functioning
of LDAP ADD for example.
A correct Mapping tree for this backend must contain the suffix name,
the database name
cn=o3Dexample,cn=mapping tree,cn=config
cn: o=example
nsslapd-backend: userRoot
nsslapd-state: backend
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
-----------
Either you need to create the mapping tree, or you need to repair the
related
mapping tree. You will need to do this by hand by editing cn=config,
or stopping
the instance and editing dse.ldif.
dsctl ldaphub get-nsstate
cn=replica,cn=ou\3dsomesuffix\2co\3dcaltech\2cc\3dus,cn=mapping
tree,cn=config
Replica Suffix:       ou=somesuffix,o=caltech,c=us
Replica ID:           65535
Gen Time:             1618442292
Gen Time String:      Wed Apr 14 16:18:12 2021
Gen as CSN:           607778340002655350000
Local Offset:         0
Local Offset String:  0 seconds
Remote Offset:        7
Remote Offset String: 7 seconds
Time Skew:            7
Time Skew String:     7 seconds
Seq Num:              2
System Time:          Wed Apr 14 17:30:50 2021
Diff in Seconds:      4358
Diff in days/secs:    0:4358
Endian:               Little Endian
dn: cn=ou\3Dsomesuffix\2Co\3Dcaltech\2Cc\3Dus,cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
nsslapd-state: referral on update
nsslapd-backend: somesuffixRoot
cn: ou=somesuffix,o=caltech,c=us
creatorsName: cn=directory manager
modifiersName: cn=server,cn=plugins,cn=config
createTimestamp: 20210415004818Z
modifyTimestamp: 20210415005939Z
numSubordinates: 1
ldap://supplier2:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
ldap://supplier1:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
ldap://supplier0:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
ldap://supplier4.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
ldap://supplier5.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
ldap://supplier3.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
How I set it set up the hub and the agreement: (note the same commands
i used to setup the suppliers and consumers worked great with only
variance is really the role)
# how i setup the consumer
dsconf -D "cn=Directory Manager" -w XXX ldap://$consumer replication
enable --suffix="ou=somesuffix,o=caltech,c=us" --role="hub"
--replica-id=6001 --bind-dn="cn=replication manager,cn=config"
--bind-passwd=XXX
# how i setup the agreement
dsconf -D "cn=Directory Manager" -w XXXX ldap://supplier repl-agmt
create --suffix="ou=somesuffix,o=caltech,c=us" --host=consumer
--port=389 \
     --conn-protocol=StartTLS --bind-dn="cn=replication
manager,cn=config" \
     --bind-passwd=XXXX --bind-method=SIMPLE --init \
     replication-agreement-name-super-awesome
_______________________________________________
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
https://pagure.io/fedora-infrastructure
--

389 Directory Server Development Team
_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the l
Gary Waters
2021-04-15 17:24:05 UTC
Permalink
Hi Mark ,

Thank you so much for your time.
Post by Mark Reynolds
I don't see your backend entry in your output, just the mapping tree
entry.  It takes two entgries to define a backend and suffix (annoying
I know but that's how it works).  So how did you create your suffix? 
Did you use ldapmodify or did you use dsconf?
Do you have these two entries in your config?  If so, please show both
of them.
I used ldapmodify to create my backend and suffix. Here are both from
the dse.ldif:

dn: cn=somesuffixRoot,cn=ldbm database,cn=plugins,cn=config
objectClass: extensibleObject
objectClass: nsBackendInstance
objectClass: top
cn: somesuffixRoot
creatorsName: cn=directory manager
modifiersName: cn=directory manager
createTimestamp: 20210415004818Z
modifyTimestamp: 20210415004818Z
numSubordinates: 4
nsslapd-suffix: ou=somesuffix,o=caltech,c=us
nsslapd-cachesize: -1
nsslapd-cachememsize: 134217728
nsslapd-readonly: off
nsslapd-require-index: off
nsslapd-require-internalop-index: off
nsslapd-dncachememsize: 67108864
nsslapd-directory: /var/lib/dirsrv/slapd-ldaphub/db/somesuffixRoot

dn: cn=ou\3Dsomesuffix\2Co\3Dcaltech\2Cc\3Dus,cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
nsslapd-state: referral on update
nsslapd-backend: somesuffixRoot
cn: ou=somesuffix,o=caltech,c=us
creatorsName: cn=directory manager
modifiersName: cn=server,cn=plugins,cn=config
createTimestamp: 20210415004818Z
modifyTimestamp: 20210415005939Z
numSubordinates: 1
nsslapd-referral:
ldap://supplier2:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
nsslapd-referral:
ldap://supplier1:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
nsslapd-referral:
ldap://supplier0:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
nsslapd-referral:
ldap://supplier4.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
nsslapd-referral:
ldap://supplier5.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
nsslapd-referral:
ldap://supplier3.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2

Cc%3Dus
Post by Mark Reynolds
You can not set a replica ID for a hub.  Only supplier replicas get
unique replica ID's.  So when you try and set a replica id on a hub or
consumer it will get replaced by 65535.
I was thinking that was the case because it was a number larger than
possible for the field.

-Gary
_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the list, report i
Mark Reynolds
2021-04-15 17:59:12 UTC
Permalink
Post by Gary Waters
Hi Mark ,
Thank you so much for your time.
Post by Mark Reynolds
I don't see your backend entry in your output, just the mapping tree
entry.  It takes two entgries to define a backend and suffix
(annoying I know but that's how it works).  So how did you create
your suffix?  Did you use ldapmodify or did you use dsconf?
Do you have these two entries in your config?  If so, please show
both of them.
I used ldapmodify to create my backend and suffix. Here are both from
dn: cn=somesuffixRoot,cn=ldbm database,cn=plugins,cn=config
objectClass: extensibleObject
objectClass: nsBackendInstance
objectClass: top
cn: somesuffixRoot
creatorsName: cn=directory manager
modifiersName: cn=directory manager
createTimestamp: 20210415004818Z
modifyTimestamp: 20210415004818Z
numSubordinates: 4
nsslapd-suffix: ou=somesuffix,o=caltech,c=us
nsslapd-cachesize: -1
nsslapd-cachememsize: 134217728
nsslapd-readonly: off
nsslapd-require-index: off
nsslapd-require-internalop-index: off
nsslapd-dncachememsize: 67108864
nsslapd-directory: /var/lib/dirsrv/slapd-ldaphub/db/somesuffixRoot
dn: cn=ou\3Dsomesuffix\2Co\3Dcaltech\2Cc\3Dus,cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
nsslapd-state: referral on update
nsslapd-backend: somesuffixRoot
cn: ou=somesuffix,o=caltech,c=us
creatorsName: cn=directory manager
modifiersName: cn=server,cn=plugins,cn=config
createTimestamp: 20210415004818Z
modifyTimestamp: 20210415005939Z
numSubordinates: 1
ldap://supplier2:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
ldap://supplier1:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
ldap://supplier0:389/ou%3Dsomesuffix%2Co%3Dcaltech%2Cc%3Dus
ldap://supplier4.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
ldap://supplier5.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
 Cc%3Dus
ldap://supplier3.caltech.edu:389/ou%3Dsomesuffix%2Co%3Dcaltech%2
Cc%3Dus
These entries look fine.  I'm assuming you are running this on a hub or
consumer, is that correct?  Does it work correctly on the supplier
replica?  I think the "nsslapd-state=referral on update" might be
tripping up the healthcheck.

Mark
Post by Gary Waters
Post by Mark Reynolds
You can not set a replica ID for a hub. Only supplier replicas get
unique replica ID's.  So when you try and set a replica id on a hub
or consumer it will get replaced by 65535.
I was thinking that was the case because it was a number larger than
possible for the field.
-Gary
--

389 Directory Server Development Team
_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Gary Waters
2021-04-15 20:23:06 UTC
Permalink
Post by Mark Reynolds
These entries look fine.  I'm assuming you are running this on a hub
or consumer, is that correct?  Does it work correctly on the supplier
replica?  I think the "nsslapd-state=referral on update" might be
tripping up the healthcheck.
Yes I am using this as a hub. The same ldif I use to make the suffix I
use to make the suppliers and consumers, and they work fine (and dsctl
healthcheck says they are ok).  The setting of nsslapd-state was set by
the dsconf command I sent before. I checked a production hub I have
(which this one will eventually replace), and that is the correct setting.

Perhaps this is an issue with dsctl's healthcheck then.

-Gary

_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-i
Mark Reynolds
2021-04-15 21:39:53 UTC
Permalink
Post by Gary Waters
Post by Mark Reynolds
These entries look fine.  I'm assuming you are running this on a hub
or consumer, is that correct?  Does it work correctly on the supplier
replica?  I think the "nsslapd-state=referral on update" might be
tripping up the healthcheck.
Yes I am using this as a hub. The same ldif I use to make the suffix I
use to make the suppliers and consumers, and they work fine (and dsctl
healthcheck says they are ok).  The setting of nsslapd-state was set
by the dsconf command I sent before. I checked a production hub I have
(which this one will eventually replace), and that is the correct setting.
Perhaps this is an issue with dsctl's healthcheck then.
There is definitely a bug, I was just trying to narrow it down. I'll try
and look into this tomorrow...
Post by Gary Waters
-Gary
--

389 Directory Server Development Team
_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-
Gary Waters
2021-04-26 22:51:31 UTC
Permalink
Something to note, as I just triggered this bug on a replica I built
today, this is a new bug.

I have 2 pretty much identical consumer only replicas. The bug triggers
56 times on the new one and not the old one. The replica is a consumer
only, and not a hub. Same ldif file (for db and suffix) to make like 9
of these boxes, this health check happens only on the new ones.

Old box version:

1.4.3.8-6
389-ds-base python3-lib389 389-ds-base-libs

New box version:

1.4.3.22

Trying to downgrade the box.. bah cant start 389 after the downgrade
lol. Rebuilding.

1.4.3.8-7 downgrade (dont see any in between version from 1.4.3.8-7 to
1.4.3.22) .. Ok and downgrading helped! the dsctl healthcheck works!

Hope this helps,

Gary
Post by Mark Reynolds
Post by Gary Waters
Post by Mark Reynolds
These entries look fine.  I'm assuming you are running this on a hub
or consumer, is that correct? Does it work correctly on the supplier
replica?  I think the "nsslapd-state=referral on update" might be
tripping up the healthcheck.
Yes I am using this as a hub. The same ldif I use to make the suffix
I use to make the suppliers and consumers, and they work fine (and
dsctl healthcheck says they are ok).  The setting of nsslapd-state
was set by the dsconf command I sent before. I checked a production
hub I have (which this one will eventually replace), and that is the
correct setting.
Perhaps this is an issue with dsctl's healthcheck then.
There is definitely a bug, I was just trying to narrow it down. I'll
try and look into this tomorrow...
Post by Gary Waters
-Gary
_______________________________________________
389-users mailing list -- 389-***@lists.fedoraproject.org
To unsubscribe send an email to 389-users-***@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-***@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructur
Loading...