Discussion:
x-forwarded-for
(too old to reply)
Robert Viduya
2016-05-17 20:04:49 UTC
Permalink
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves) behind a set of F5 Bigip load balancers. Our Bigip admins recently decided to switch the boxes to "one-armed" mode and that services would have to use X-Forwarded-For headers or equivalent to get the actual client IP address. Obviously, LDAP has no equivalent.

So, I've hacked something together and I'm posting it looking for feedback. If the stuff is actually usable for other folks, that's good too.

On the load balancer side, the code is specifically for the F5 Bigips. But if other load balancers have similar abilities to trigger on events and can insert binary data into the datastream, it should be adaptable.

Essentially what I've done is defined a new LDAP Extended Operation with a payload that's a string containing the source IP address of the incoming connection. The load balancer sends this LDAP operation as soon as it opens a connection to the LDAP server, before any other traffic gets sent. On the directory server side, I've written an Extended Operation plugin that then logs the string. All we need is logging, so that's good enough for us. There's room for improvement there though, like making the address available for IP based ACls (which we don't use).

In order to insert an LDAP operation into the stream, the code on the load balancer needs to choose an LDAP message-id that hopefully the real client isn't going to use. Going on the assumption that the client will start at 0 and increment up, I had the code insert a message-id of 0x70000000. I initially thought I'd have to have the code on the load balancer look for the response message and throw it away so the client doesn't see it, but I've found that the clients just seem to ignore it with no ill effects, so I haven't bothered filtering the response out. The less the code on the load balancer does, the better.

There's a possibility that the client could send it's own xff extended operation, but since the load balancer always sends first, we can just ignore any subsequent log entries.

On the directory server plugin side, I needed to be able to log this to the access log, not to the error log. The slapi_log_access function isn't declared in the plugin header file, so I had to declare it manually.

Here's what our log entries look like now, for both 636 (SSL) and 389 (cleartext before starttls):

[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120 nentries=0 etime=0

[17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156 connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120 nentries=0 etime=0

We haven't switched our Bigips yet, so the "connection from" line still shows the actual client IP address.

F5 Bigip code fragments are called "irules" and are written in TCL. The tar file below contains two different irule files, one for cleartext streams and one for SSL streams. By SSL streams, I mean where SSL connections from the client are terminated at the load balancer and then re-SSLed to the ldap server. Sorry, I'm not writing the other kind.
Rich Megginson
2016-05-18 13:02:44 UTC
Permalink
Post by Robert Viduya
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves)
behind a set of F5 Bigip load balancers. Our Bigip admins recently
decided to switch the boxes to "one-armed" mode and that services
would have to use X-Forwarded-For headers or equivalent to get the
actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for
feedback. If the stuff is actually usable for other folks, that's
good too.
On the load balancer side, the code is specifically for the F5 Bigips.
But if other load balancers have similar abilities to trigger on
events and can insert binary data into the datastream, it should be
adaptable.
Essentially what I've done is defined a new LDAP Extended Operation
with a payload that's a string containing the source IP address of the
incoming connection. The load balancer sends this LDAP operation as
soon as it opens a connection to the LDAP server, before any other
traffic gets sent. On the directory server side, I've written an
Extended Operation plugin that then logs the string. All we need is
logging, so that's good enough for us. There's room for improvement
there though, like making the address available for IP based ACls
(which we don't use).
In order to insert an LDAP operation into the stream, the code on the
load balancer needs to choose an LDAP message-id that hopefully the
real client isn't going to use. Going on the assumption that the
client will start at 0 and increment up, I had the code insert a
message-id of 0x70000000. I initially thought I'd have to have the
code on the load balancer look for the response message and throw it
away so the client doesn't see it, but I've found that the clients
just seem to ignore it with no ill effects, so I haven't bothered
filtering the response out. The less the code on the load balancer
does, the better.
There's a possibility that the client could send it's own xff extended
operation, but since the load balancer always sends first, we can just
ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this
to the access log, not to the error log. The slapi_log_access
function isn't declared in the plugin header file, so I had to declare
it manually.
Here's what our log entries look like now, for both 636 (SSL) and 389
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL
connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT
oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120
nentries=0 etime=0
[17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156
connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT
oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120
nentries=0 etime=0
We haven't switched our Bigips yet, so the "connection from" line
still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL.
The tar file below contains two different irule files, one for
cleartext streams and one for SSL streams. By SSL streams, I mean
where SSL connections from the client are terminated at the load
balancer and then re-SSLed to the ldap server. Sorry, I'm not writing
the other kind.
This is really nice, and a big help for tracing connections through load
balancers and other similar devices.
Are you interested in contributing this to 389?
Post by Robert Viduya
--
389-users mailing list
Robert Viduya
2016-05-18 13:35:00 UTC
Permalink
I have no issues with that. I’m a little concerned about the ldap message id, however.
Post by Robert Viduya
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves) behind a set of F5 Bigip load balancers. Our Bigip admins recently decided to switch the boxes to "one-armed" mode and that services would have to use X-Forwarded-For headers or equivalent to get the actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for feedback. If the stuff is actually usable for other folks, that's good too.
On the load balancer side, the code is specifically for the F5 Bigips. But if other load balancers have similar abilities to trigger on events and can insert binary data into the datastream, it should be adaptable.
Essentially what I've done is defined a new LDAP Extended Operation with a payload that's a string containing the source IP address of the incoming connection. The load balancer sends this LDAP operation as soon as it opens a connection to the LDAP server, before any other traffic gets sent. On the directory server side, I've written an Extended Operation plugin that then logs the string. All we need is logging, so that's good enough for us. There's room for improvement there though, like making the address available for IP based ACls (which we don't use).
In order to insert an LDAP operation into the stream, the code on the load balancer needs to choose an LDAP message-id that hopefully the real client isn't going to use. Going on the assumption that the client will start at 0 and increment up, I had the code insert a message-id of 0x70000000. I initially thought I'd have to have the code on the load balancer look for the response message and throw it away so the client doesn't see it, but I've found that the clients just seem to ignore it with no ill effects, so I haven't bothered filtering the response out. The less the code on the load balancer does, the better.
There's a possibility that the client could send it's own xff extended operation, but since the load balancer always sends first, we can just ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this to the access log, not to the error log. The slapi_log_access function isn't declared in the plugin header file, so I had to declare it manually.
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120 nentries=0 etime=0
[17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156 connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120 nentries=0 etime=0
We haven't switched our Bigips yet, so the "connection from" line still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL. The tar file below contains two different irule files, one for cleartext streams and one for SSL streams. By SSL streams, I mean where SSL connections from the client are terminated at the load balancer and then re-SSLed to the ldap server. Sorry, I'm not writing the other kind.
This is really nice, and a big help for tracing connections through load balancers and other similar devices.
Are you interested in contributing this to 389?
Post by Robert Viduya
--
389-users mailing list
--
389-users mailing list
Rich Megginson
2016-05-18 13:48:58 UTC
Permalink
Post by Robert Viduya
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
Post by Robert Viduya
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
Post by Robert Viduya
Post by Rich Megginson
Post by Robert Viduya
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves)
behind a set of F5 Bigip load balancers. Our Bigip admins recently
decided to switch the boxes to "one-armed" mode and that services
would have to use X-Forwarded-For headers or equivalent to get the
actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for
feedback. If the stuff is actually usable for other folks, that's
good too.
On the load balancer side, the code is specifically for the F5
Bigips. But if other load balancers have similar abilities to
trigger on events and can insert binary data into the datastream, it
should be adaptable.
Essentially what I've done is defined a new LDAP Extended Operation
with a payload that's a string containing the source IP address of
the incoming connection. The load balancer sends this LDAP
operation as soon as it opens a connection to the LDAP server,
before any other traffic gets sent. On the directory server side,
I've written an Extended Operation plugin that then logs the string.
All we need is logging, so that's good enough for us. There's room
for improvement there though, like making the address available for
IP based ACls (which we don't use).
In order to insert an LDAP operation into the stream, the code on
the load balancer needs to choose an LDAP message-id that hopefully
the real client isn't going to use. Going on the assumption that
the client will start at 0 and increment up, I had the code insert a
message-id of 0x70000000. I initially thought I'd have to have the
code on the load balancer look for the response message and throw it
away so the client doesn't see it, but I've found that the clients
just seem to ignore it with no ill effects, so I haven't bothered
filtering the response out. The less the code on the load balancer
does, the better.
There's a possibility that the client could send it's own xff
extended operation, but since the load balancer always sends first,
we can just ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this
to the access log, not to the error log. The slapi_log_access
function isn't declared in the plugin header file, so I had to
declare it manually.
Here's what our log entries look like now, for both 636 (SSL) and
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL
connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT
oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0
tag=120 nentries=0 etime=0
[17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156
connection from 130.207.167.12 to 130.207.183.16
[17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT
oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op"
[17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12
[17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0
tag=120 nentries=0 etime=0
We haven't switched our Bigips yet, so the "connection from" line
still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL.
The tar file below contains two different irule files, one for
cleartext streams and one for SSL streams. By SSL streams, I mean
where SSL connections from the client are terminated at the load
balancer and then re-SSLed to the ldap server. Sorry, I'm not
writing the other kind.
This is really nice, and a big help for tracing connections through
load balancers and other similar devices.
Are you interested in contributing this to 389?
Post by Robert Viduya
--
389-users mailing list
--
389-users mailing list
--
389-users mailing list
Jochen Schneider
2016-05-18 21:04:28 UTC
Permalink
Hi Rich, hi Robert,
Post by Rich Megginson
Post by Robert Viduya
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
Post by Robert Viduya
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
this is quite an interesting approach and it will probably work in the
vast majority of cases.
But instead of adding a separate LDAP _operation_ to the stream (where
we cannot easily ensure uniqueness of the message id) we could implement
the "LDAP Session Tracking Control" from
http://tools.ietf.org/html/draft-wahl-ldap-session and only insert that
_control_ into the first LDAP operation.
There is already a 389 ticket for the server side functionality
(https://fedorahosted.org/389/ticket/47873).
On the F5 there is ASN1::element which makes construction of the control
and adding it to the LDAP operation quite feasible.

Regards,
J.
William Brown
2016-05-18 23:48:28 UTC
Permalink
Post by Jochen Schneider
Hi Rich, hi Robert,
Post by Robert Viduya
I have no issues with that.
Can you open a 389 ticket?  https://fedorahosted.org/389/newticket
Post by Robert Viduya
I’m a little concerned about the ldap message id, however. 
Me too.  Developers, LDAP SMEs, please feel free to chime in.
this is quite an interesting approach and it will probably work in the
vast majority of cases.
I think it sounds a bit fragile, and really relies on the log aggregation techniques of the implementor. I'm not sure I like
this approach. 

As well, how is this locked down to prevent a fradulent client sending a fake control first, then their real operation? It would
certainly cause confusion in the logs ....
Post by Jochen Schneider
But instead of adding a separate LDAP _operation_ to the stream (where
we cannot easily ensure uniqueness of the message id) we could implement
the "LDAP Session Tracking Control" from
http://tools.ietf.org/html/draft-wahl-ldap-session and only insert that
_control_ into the first LDAP operation.
There is already a 389 ticket for the server side functionality
(https://fedorahosted.org/389/ticket/47873).
On the F5 there is ASN1::element which makes construction of the control
and adding it to the LDAP operation quite feasible.
The issue with this is that this control can be forged. We would need to essentially add another layer of "auth" which states
that we allow the control from only certain source ips, ie the load balancer that is allowed to add this control. And even IP
layer checks can be forged unless you turn on things like reverse path checking and such on the host. 


Both solutions really have shortcomings and complexity, but I think I would prefer to implement the draft version of session-03
over the "loosely coupled" approach that Robert developed. We could at least lock it down to a check on the plugin only allowed
some set of IP's to use the control, and put a check in dsktune for rpf. Then the load balancer can inject the control to the
operation as needed.


Saying this, there is no faster way to ruin your network than to allow a load balancer to tamper with your traffic. I've seen
horrible, horrible things happen because of bigip, netscalers, ace, and others, tampered with the internals of DNS, LDAP, HTTP
... At the point you allow a load balancer to actively look at your traffic you are opening yourself to many subtle, hard to
isolate issues.

I really can't recommend that you use one arm mode for LDAP, and highly recomend that you use routed mode, with TLS/SSL
termination on the LDAP servers themself. 

But that's a discussion for another thread I think .....
Post by Jochen Schneider
Regards,
 J.
--
389-users mailing list
--
Sincerely,

William Brown
Software Engineer
Red Hat, Brisbane
Robert Viduya
2016-05-19 02:58:46 UTC
Permalink
My code on the load balancer inserts the XFF message before any client data gets passed. So forging can be handled by ignoring any subsequent XFF messages, only the first one is the one from the load balancer.

However, this doesn’t allow for cascading load balancers. HTML’s X-Forwarded-For header handles this by having each load balancer just append to the string. I’m not sure how important this capability is.
Post by William Brown
Post by Jochen Schneider
Hi Rich, hi Robert,
Post by Rich Megginson
Post by Robert Viduya
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
Post by Robert Viduya
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
this is quite an interesting approach and it will probably work in the
vast majority of cases.
I think it sounds a bit fragile, and really relies on the log aggregation techniques of the implementor. I'm not sure I like
this approach.
As well, how is this locked down to prevent a fradulent client sending a fake control first, then their real operation? It would
certainly cause confusion in the logs ....
Post by Jochen Schneider
But instead of adding a separate LDAP _operation_ to the stream (where
we cannot easily ensure uniqueness of the message id) we could implement
the "LDAP Session Tracking Control" from
http://tools.ietf.org/html/draft-wahl-ldap-session and only insert that
_control_ into the first LDAP operation.
There is already a 389 ticket for the server side functionality
(https://fedorahosted.org/389/ticket/47873).
On the F5 there is ASN1::element which makes construction of the control
and adding it to the LDAP operation quite feasible.
The issue with this is that this control can be forged. We would need to essentially add another layer of "auth" which states
that we allow the control from only certain source ips, ie the load balancer that is allowed to add this control. And even IP
layer checks can be forged unless you turn on things like reverse path checking and such on the host.
Both solutions really have shortcomings and complexity, but I think I would prefer to implement the draft version of session-03
over the "loosely coupled" approach that Robert developed. We could at least lock it down to a check on the plugin only allowed
some set of IP's to use the control, and put a check in dsktune for rpf. Then the load balancer can inject the control to the
operation as needed.
Saying this, there is no faster way to ruin your network than to allow a load balancer to tamper with your traffic. I've seen
horrible, horrible things happen because of bigip, netscalers, ace, and others, tampered with the internals of DNS, LDAP, HTTP
... At the point you allow a load balancer to actively look at your traffic you are opening yourself to many subtle, hard to
isolate issues.
I really can't recommend that you use one arm mode for LDAP, and highly recomend that you use routed mode, with TLS/SSL
termination on the LDAP servers themself.
But that's a discussion for another thread I think .....
Post by Jochen Schneider
Regards,
J.
--
389-users mailing list
--
Sincerely,
William Brown
Software Engineer
Red Hat, Brisbane
--
389-users mailing list
Continue reading on narkive:
Loading...