The Netscaler is hiding stuff from you...

I have been thinking recently about how to hid my infrastructure info from the public, and one easy way is to stop telling the world what type of webserver you are running.  Now I am not going to get into the discussion of whether or not "security through obscurity" works... but this is so easy, even if it hinders some script kiddies, I will be happy.

There are lots of ways to see the response headers from your webserver, and I found a scanner that will tell you that and a bit more:  https://securityheaders.io I ran some of my URLs though the device, and sure enough, they are blabbing to the world what versions of whatever it has.... 

So instead of trying to figure out how to get all of my webservers to shut up, I decided to use the Netscaler to just remove the headers before they are presented to the client.  I must admit, it was pretty easy... from the CLI.  When I tried it from the GUI, I had a strange message and didn't want to fuss around with it much more.

 -----------------------------

Remove "Server" header:

add rewrite action Delete_server_header_action delete_http_header Server -bypassSafetyCheck YES -comment "This will delete the Server Header field from Server's response before sending to client"

add rewrite policy Delete_server_header_policy "HTTP.RES.HEADER(\"Server\").EXISTS" Delete_server_header_action -comment "This will delete the Server header field from server\'s response before sending to client"

Now to remove "x-powered-by" header:

add rewrite action Delete_x-powered-by_header_action delete_http_header X-Powered-By -comment "This will delete the X-Powered-By Header field from Server's response before sending to client"

add rewrite policy Delete_x-powered-by_header_policy "HTTP.RES.HEADER(\"X-Powered-By\").EXISTS" Delete_x-powered-by_header_action -comment "This will delete the X-Powered-By header field from server\'s response before sending to client"

-------------------------

then bind them both to your Content Switching Virtual Server, give it priority of 85 (in my case I had a few others I want to run afterwards), and change "goto expression" to "NEXT"

Easy, right?  Now run your test again, and those headers are now missing... of course, you could replace the headers with something fun, like "X-Powered-By: The Dark Side" or whatever.... but I am not sure my employer would appreciate the humor as much as I would.

Netscaler cert... damn thing hung up on me again

Netscaler has a strange GUI that I think was designed as an "afterthought" by the developers.  The more you use it, the more you try to figure out why stuff is in the order it is, or the grouping it is in.  Sometimes the Netscaler will perform an operation, and drop your connection without warning.  So here is how to install Certificates, which might end in a dropped connection when associating a cert to the device management interface.

Request a certificate as you would normally do, using IIS.  This has been documented plenty of other places, so skipped here. 

Because of the HA pair, you will need one cert, but make it good for 2 DNS names, including NS.blah.com and NS-Otherlocation.blah.com

This is also good for moving a site's SSL certificate to the Netscaler for load balancing from an IIS host.

Visit http://www.derekseaman.com/2013/05/import-iis-ssl-certificate-to-citrix-netscaler.html on how to export this new certificate into the Netscaler UNTIL the section where you have to upload it to the NETSCALER

(Mr Derek Seaman's instructions are good, but not for our NS version.  You can probably figure it out with clicking around, but just in case:)

 At this point, on the Netscaler, you select Traffic management --> ssl and "import PKCS#12"
Most of Mr Seaman's instructions will still work, but things may be very slightly out of order, like the order to click "browse" or whatever... but it is much easier with his diagrams than I can explain here.  Remember to use a good password manager to generate and store any passwords you use in this process.
When you are finished, the cert is ready to be used with your VIP.

IF YOU ARE INSTALLING THE CERT FOR THE NETSCALER DEVICE ITSELF:

Skip the step above where you upload the certificate, or remove it from the Netscaler if you have already uploaded it.

Download the "X509 Certificate only, Base64 encoded" file and open it in a text editor.
blah_com_cert.cer

Download the "X509 Intermediates/root only Reverse, Base64 encoded: " file and open it as well.
blah_com_interm.cer

Create a new text file.  Copy the X509 Certificate only, Base64 encoded  cert to it first and then copy the NEXT two X509 Intermediates/root only Reverse, Base64 encoded certs from the file below the first. (They will be the first two in the blah_com_interm.cer file. The root is the last one in that file and you don't want it.) Now save the file with a meaningful name, like "blah_com-bun-noroot.crt".  (bun=bundle)

Login into the NS and upload your cert bundle and private key. In this example they would be blah_com.key and blah_com-bun-noroot.crt.
Then under SSL/Certificates select the ns-server-certificate and update it. There's a check box on the Update Certificate window that says, Click to update Certificate/Key. Select that and then browse for the two files you just uploaded.  Also check the box "no domain check" if you are switching domain suffixes . 

 After clicking "ok", wait a min or two... then you will have to reconnect your browser.

Check the certificate in the browser, it should list new certificate.

 

You do not have to modify the other Node in the HA pair, the HA standby member gets updated automatically.

You can diagnose/view the certs you uploaded using "shell" and "openssl x509 -in NAMEOFFILE.cer -text -noout"

Happy balancing,
-_Bryan

Netscaler N00b no more?

My employer purchased a few Netscalers (NS) and put me and another dozen or so folks through training on how to configure and use it. But, nothing prepared me for the strange way Citrix and Citrix-fans write their documentation.

So here are a few things I learned in the last few weeks working with it:

 1) AD and AAA - there are many articles on how to use AAA with Active Directory. (http://support.citrix.com/article/CTX111079 for example)  I searched for quite some time trying to figure out how to "link" an AD group to a NS group.  I assumed I would have to create a group on the NS, then tell the NS to associate that group with the AD group.... But no.  I wanted to customize what the NS group was called, but you cannot.  So here was my trick:  Create a group on the NS EXACTLY the way it is spelled in AD and associate it with a policy ( I used a built-in NS policy).  Then under "System, authentication, LDAP, Servers tab, the value I used was:
 "memberOf=CN=ad_group_name_here,OU=ou_where_group_is_located,DC=domain_name,DC=dopmain_name,DC=domain_name,DC=com" .  In short, a NS local group of the same name is associated with a local LDAP policy which is searched for using location in AD.
Also, here is another thing that I hope will save you some time.  LDAP using FQDNs did not work for us.  Instead, I had to create one server LDAP policy for each of our domain controllers using their individual IPs.  DNS is setup, and seems to work great with everything else tested (ping, traceroute), but it appears the Nestcaler does not handle multiple A record responses from a DNS server in this circumstance.

2) AppExpert Templates - I was experimenting with SharePoint behind the NS, and I came across the AppExpert Template for it.     It looked pretty neat, with lots of optimizations and promises of improved performance. 

I tried using many walk-troughs to implement it, but none of them seemed to work:
https://www.citrix.com/content/dam/citrix/en_us/documents/products-solutions/microsoft-sharepoint-2013-with-citrix-netscaler.pdf
https://www.paloaltonetworks.com/content/dam/paloaltonetworks-com/en_US/assets/pdf/technology-solutions-briefs/citrix/panw-netscaler-sharepoint.pdf
https://www.citrix.com/content/dam/citrix/en_us/documents/products-solutions/citrix-netscaler-datasheet-microsoft-sharepoint-2013.pdf

So, here is what I had to know to get the template properly installed:
-- The NAME you first enter MUST NOT include any special characters.  It will error towards the end with a strange message if you do.
-- You must not already have the resource of the "public endpoint" as a current Load balanced server or anywhere else if you can help it. The only thing you should have configured is the VIP configured on the NS itself.   The template will create the server for you.
--  Only a N00b like me probably thought this, but the template does not create a "load balanced" server, it created a "content switching server."  
-- If you need to delete the template, you will have to remove all of the Response, Rewrite, and all the other Policies and Actions it creates.  Luckily they will start with the name you provided above.

3) Mac address filtering -  We have some "real servers" and some VIPs behind and in front of the NS.  We could not figure out why the traffic would disappear.  It turned out that you must enable "
MAC based forwarding" under System, Settings, Configure Modes.  The networking team here hated me for a few days, and they thought I was an idiot, as their NS instance worked fine... but they didn't have anything that had to traverse a firewall.  This little checkbox was the reason it only half-worked for me.

4) Routes -  For me, the GUI is really confusing, as they have a column that says "Gateway/Owned IP/Name."  Long story short, add Routes using the CLI as it makes a hell of a lot more sense.

May your life and servers be forever balanced,
-_Bryan