Creating a Citrix NetScaler Test environment

Being a Citrix Certified Instructor I am very much aware of the Red/Green/Blue website used during official Citrix NetScaler training (CNS-220, CNS-222). I created my own test website. I usually use it during product demonstrations to present anything from basic load balancing to web application firewall.

I am also aware about problems with the original Citrix labs: They sometimes seem to not load balance. Actually they do, but, because this page is compromised of several files, it may appear to show the same colour all the time. I wanted to avoid this, so my pages don’t use external style-sheets, scripts and images, instead I added everything into the HTML file (you may include images using base 64 encoding).

You may download my test website from here. I will update my page every now and then. You can download it as often as you like. The download will ask you for your E-Mail address. I promise not to send any SPAM to you, instead I’ll just count the numbers of downloads.

Requirements and prerequisites

My environment is made of a single Windows server (I tested using 2012R2 Server) and a NetScaler VPX. You may very well use some entry level virtualization solution like VMWare workstation or Hyper-V on your laptop computer, but professional environment like Xen-Server, KVM and simmilar may also be used of course.

My download does not include the machines, but the website only. There is no license included, however you may request a demo license using your Citrix account)

Installation procedure

Import a Citrix NetScaler VPX into your virtualization solution. (www.citrix.com -> downloads -> NetScaler ADC -> Reliese xxx -> Virtual Appliances).

Install a Windows Server (I tested using 2012R2, but I guess it will work with any version from 2008). This server should have 4 GB RAM as a minimum

IP addressing

I used 192.168.0.100 as a NSIP, 192.168.0.110 as a SNIP, 192.168.200 ff for virtual servers

Windows machine used 192.168.0.20 to 24

Windows set up

Roles and features

After setting up this windows machine you have to set up IIS. Start Server Manager (if it’s not already started) and click “add roles and features”. Click Next 3 times.

Select Active Directory Certificate authority, Web Server IIS and DNS. If asked select following roll- services:

.NET Extensibility 4.5
ISAPI Extensions
ISAPI Filters
.NET Extensibility 3.5
Certificate Authority
Certificate enrolment web service

Setingt up the Certificate Authority:

stand alone CA
root CA
create a new key
SHA 256 (or highter)
confirm all the rest of the questions

IP configuration

select your network adapter. Change IP address. Set 192.168.0.20 255.255.255.0 as an IP address (you may use any other address range you like, but I use 192.168.0.x in my example). DNS should be 127.0.0.1, gateway depending on your settings.

Click advanced. add 4 more IP addresses (192.168.0.21 to 192.168.0.24).

IIS settings

Copy my files into c:\inetpub directory.

Open Internet Information Server Management.

Open your server and select sites. Right click your server and select add website. Create 4 virtual websites:

Sitename: Sitie1 (2,3,4)
Site path: C:\inetpub\wwwroot1 (2,3,4)
type: http
IP address: 192.168.0.21 (22,23,24)
hostname: (empty)

ASPx is just needed for the Citrix NetScaler Web Application Firewall test page. Check, if ASPX works correctly surfing to http://192.168.0.24/Allow.aspx. If it does not: follow this Microsoft instructions.

additional software

If you want to use this machina as a workstation as well install Google’s chrome Browser and Mozilla FireFox. Alternatively you may create a dedicated work station or use your desktop work station.

You’ll very likely need the SSH terminal putty, the secure copy tool WinSCP and the network monitor WireShark. They can be considered to be the tools used by a NetScaler admin during his daily work.

Labs:

Prerequisites

in DNS manager create a new Forward lookup zone called test.lab.

Create hosts:

colours.training.lab 192.168.0.200
cs-test.training.lab 192.168.0.201
aaa.training.lab 192.168.0.202

1st lab: create a load balancing vServer

Server:

srv_red -> 192.168.0.21
srv_green -> 192.168.0.22
srv_blue -> 192.168.0.23

Services:

svc_red (HTTP/80)
svc_green (HTTP/89)
svc_blue (HTTP/80)

Loadbalancing vServer

lb_vsrv_colors (192.168.0.200/HTTP/80)

additional labs:

add persistence (source IP, cookie based, …)
disable services and see what hapens (re-enable these)
unbind red service, create an additional loadbalancing vServer (non addressable), called lb_vsrv_red. Set this one in protection as a backup virtual server. Disable service blue and green. Which status does lb_vsrv_colors have now? Does it work? Why? rebind red service.

2nd lab: certificates

use the wizard to create a key and a CSR (hostname *.training.lab). Surf to 192.168.0.20/certsrv. Request a certificate. download this certiticate as BASE 64. Install it into NetScaler
create a lb vServer lb_vsrv_colors_secure (192.168.0.200/SSL/443). Bind the 3 services and your newly created certificate. Surf to https://colours.training.lab

3rd lab: content switching

create a new content switching vServer cs_vsrv_browser 192.168.0.201/HTTP/80
create two new cs-policies
- HTTP.REQ.HEADER(“User-Agent”).CONTAINS(“Trident”)
- HTTP.REQ.HEADER(“User-Agent”).CONTAINS(“Chrome”)
bind these policies to cs_vsrv_browser. The Trident policy should invoke the red, the Chrome policy the blue server. Surf to cs-test.training.lab using an MS- Internetexplorer, a Google Chrome and a FireFox.

4th lab: responding

create a responder policy to forward users from http://colors.training.lab/ to https://colors.training.lab/ and bind it to lb_vsrv_colours
create a responder policy forwarding users from https://colors.training.lab/ to https://colors.training.lab/home.htm
unbind the responder policy from lb_vsrv_colours

5th lab: rewriting

create a rewriting policy rewriting requests for http://colors.training.lab into http://colors.training.lab/home.htm and bind it to lb_vsrv_colours
remove server header from HTTP-response and bind it to lb_vsrv_colours
add a server header into http response stating your server to be an Apache and bind it to lb_vsrv_colours

Why would you like to customize a 404 page?

Well It’s all about misleading information. A hacker has very limited chance to get friend with your web server. On the other way, he needs to find out as much as any possible. The more he knows, the more likely his attack will be successful. On the other hand he has to let sleeping dogs lie. With other words: He must not alarm you.

One of the most important things to know is: What kind of web server do I have to deal with?

The first source to look into is a HTTP response header called Server. Information here may be very verbose. I don’t know why this header is part of HTML standard, but actually it is.

The Server response-header field contains information about the software used by the origin server to handle the request. The field can contain multiple product tokens (section 3.8) and comments identifying the server and any significant subproducts. The product tokens are listed in order of their significance for identifying the application. (RFC 2616)

This is an example server header:

Apache/1.3.28 (Unix) mod_ssl/2.8.15 OpenSSL/0.9.7c mod_perl/1.27 PHP/4.3

In this case, it’s a very outdated Apache, using an outdated SSL module, outdated Perl and outdated PHP. It’s easy to change this information using Citrix NetScaler rewrite policies (DELETE_HTTP_HEADER and INSERT_HTTP_HEADER).

But hackes are not that stupid. They will probably verify this information. My personal next try would be: check for a non existing site. We will see a 404, page not found. Being careful I would use an existing URL, however do a minor typo, just like http://norz.at/default.html instead of http://norz.at/default.htm. You would probably not be scared if you would see a request like that watching your logs.

The next thing he would see is a 404, Not Found. It will be specific to your server, if you don’t change it. And a 404 page originating from an IIS6 would, for sure, come from an IIS 6, no matter what the server header tells you.

More reasons to change the 404 page

of course there are even more reasons to change the 404 page: customized 404 pages seem to be funny, they may help people to find the content needed, and so on.

Why not change your web server?

This would be possible. However, you would need to change all your load balanced web servers. There is an other reason: Responder policies. I will never return a “401 Unauthorized” or “403 forbidden“. I would rather return a “404 not found”. Being a hacker I would be very excited to see a 401 or 403!

I would think: here it is, but someone protects it from being accessed. But how could I find out what’s going on, if a Citrix NetScaler uses exactly the same 404 page as the original web server? I would probably think the file is not there.

My solution

My first attempt was creating a simple rewriting policy changing the body with something like “HTTP/1.1 404 OK\n\r\n\r<html><head><title>404 File not found</title></tead><body><h1><font color=\”#802020\”>404 File not found!</font></h1><p><font color=\”#802020\”>The file you requested is not on this server.</font></p></body></html>” in it.

The length of the text is limited, so this is not a good solution. And I would rather like to place the file “somewhere” on my web server, so it’s pretty easy to change.

I spent some time thinking what to do and made up my mind to use the HTTP callout feature. It was my first ever attempt to use HTTP callout, and I’ll describe how it works.

NetScaler’s HTTP callout feature

HTTP callout is intended to be used in policies to check something, i.e. an IP address, against a web based service. So I could send an IP address (CLIENT.IP.SRC) to a web server containing an IP black list. This web server then would respond with something indicating good or bad.

I do something completely different: I will retrieve the content of the 404 page from a web server. To do so I have to navigate to App Expert -> HTTP Callouts.

Like any policy it has to get a name. I do my callout to a vServer, so I have to specify the server here. My request will be attribut based, that means, I will be able to send regular HTTP requests, mine is a HTTP GET. My web server uses several host names for various virtual pages, so I have to specify a proper host expression. This makes sure, we retrieve the file from the right source. The URL Stem Expression is the URL we want to retrieve.

We scroll down to the bottom and select the return type TEXT and the expression should be HTTP.RES.BODY(65538). The number is the number of bytes to retrieve.

So, my policy will connect to a NetScaler vServer called cs_vsrv_norz.at to retrieve a file called /notfound.htm, setting the header Host to norz.at (i.e: http://norz.at/notfound.htm). It will then return all the body of this file, containing links to style definitions, pictures and so on.

command line version:

add policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"
set policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"

The rewrite policy

The rewrite policy should be a very simple thing:

The NetScaler rewrite action using a HTTP callout

add rewrite action callout404 replace_http_res "SYS.HTTP_CALLOUT(callout_retrieve_404 )"

It’s a replace policy. Expression to choose target location is all of the HTML body, so HTTP.RES.BODY (65536). To be more precise, it’s the first 65536 byte of the body (a 404 page typically is by far smaller). The Expression is the text we will use to replace the former body with. It is the HTTP callout request, in my case SYS.HTTP_CALLOUT(callout_retrieve_404).

The NetScaler rewrite policy

add rewrite policy rw_pol_404 "HTTP.RES.STATUS.EQ(404)" rw_act_404

This policy will get applied if the HTTP response status is a 404 (HTTP.RES.STATUS.EQ(404)). I then bound this policy to my web server. That’s it. It was pretty easy.

I was so enthusiastic, when I found out about NetScaler admin partitions! What a great extension to existing NetScalers! However I got disillusioned finding out about limitations. It took me some time to find out how to overcome this issues, but there are still some features missing.

The feature I missed most is doing traces. It’s not listed in the compatibility list, so it’s intended to be there. But it is not! If you click into System and Diagnostics you’ll see just very little content, and definitely no nstrace (this is about NetScaler versions up to 11.1 48.10).

Citrix documentation is always right, and if it’s not, it’s right even though. So I tried to do a nstrace from commandline. It started and stopped without any problem. Unfortunately I could not find the output of my nstrace in /var/nstrace subdirectory.

So I searched for it, and found it in /var/partitions/<partitionname>/nstrace.

So that’s how I do an nstrace inside a NetScaler admin partition:

I use putty to connect to my NetScaler. Masochists might prefer to use the built in terminal from GUI, however I don’t tend to masochism.

switch partition <partitionname>
start nstrace -filter “CONNECTION.SRCIP.EQ(<source IP>)” -size 0 -time 3600 -link
stop nstrace

So I log into my Citrix NetScaler. I change into my partition (currently partition names can’t auto complete, so make sure you know the name; show ns partition will list all partitions.

Next I start the trace. To do so I follow CTX120941.

start ns trace will simply start the trace. -filter will filter a connection. Usually you would use connection objects like CONNECTION.SRCIP.EQ(<source IP>) or CONNECTION.DSTIP.EQ(<destination IP>) to limit the amount of data captured. -size=<size> will limit the amount of data captured per packet. If you want to debug HTTP problems you would very likely set the size to 0, as this would capture all of the packet (0 sometimes means unlimited). -time=<time> will automatically stop the trace after <time> seconds.

After doing your trace you may execute a stop nstrace command to stop your trace. This is not needed if you set the time parameter, but I prefer to stop traces instead of setting a time parameter.

In the end you need to download the trace file. I usually use winscp as a secure FTP client, download it from /var/partitions/<partitionname>/nstrace and view content in your favourite network monitor. I prefer to use WireShark, as it fully supports NetScaler. Citrix support also uses WireShark.

Additional parameters for tracing

-tcpdump ENABLED switches to TCPdump format. TCPdump is a standard UNIX® format for network tracing. Different to NStrace it does not contain L1 information (ports), but it is understood with most network tracing utilities. You may want to use it together with -perNIC ENABLED if you want to debug routing problems. This will create a separate trace file pre NIC. You then have to scroll down both instances of your network monitor in parallel (and synchronize these 2 windows if you scroll down). However you may prefer to download free WireShark and use it instead as it understands NStrace: one window, all L1 information is contained in your trace.

–link Also trace filtered connection’s peer traffic. Only makes sense in combination with -filter. It will trace all traffic filtered plus all traffic resulting from your filtered traffic, so traffic from client to VIP and traffic from SNIP to your back end server. This is a very good one!

-mode SSLplain will decrypt all SSL traffic. Because of this you won’t see any SSL hand shake, instead, all SSL traffic will appear to be plain text. This may be beneficial if you want to debug encrypted traffic. Caution: this may expose sensitive data to you (the admin)

There are several more parameters. You may find them in Citrix NetScaler product documentation.

I recently had to protect a website using IP reputation feature. There is some good information about this feature, however I decided to glean information here.

Facts about this feature

IP reputation is a platinum feature. It is included in web application firewall (there are extra licenses for the WAF available, they also contain IP-reputation).

IP-reputation feature provides us with a constantly updated feed of “known” malicious IP addresses. This database maintained by webroot. This database is dynamically generated and updated every 5 minutes, so it will never be outdated. Webroot uses sensor networks for this fully automated process. You may use this database manually from here.

It is designed to check for the reputation of an IP address, so to find out if this address is a well known malicious one, or not. All IPs not found in this database are considered to be non-malicious.

Requirements

IP-reputation does HTTP call-outs to api.bcss.brightcloud.com on port 443. You therefore need to be able to:

have a valid license (i.e. a WAF license)
resolve api.bcss.brightcloud.com from your NetScaler
connect from NSIP to this IP via port 443
advanced feature Reputation has to be enabled (enable feature reputation)

Yes. that’s right: NSIP. NetScaler BSD system always uses NSIP, and IP-reputation is done in BSD, not inside the NetScaler subsystem.

How does it work

NetScaler stores a copy of WebRoot’s database for off-line use (and to avoid undesired latency). It automatically checks for updates every 5 minutes.

During first start of the IP-reputation service Citrix NetScaler does an initial call-out to api.bcss.brightcloud.com from it’s NSIP via port 443 to fetch the database. This process is logged into /var/log/iprep.log

Oct 4 03:50:00 82e6de130138 iprep: iprep process started... Oct 4 03:50:00 82e6de130138 iprep: iprep_get_schema_version:134 current schema version:1.0 Oct 4 03:50:00 82e6de130138 iprep: iprep_check_db_upgrade:296 DB schema is not up-to-date. Oct 4 03:50:00 82e6de130138 iprep: iprep_upgrade_db:242 upgrading schema version from 1.0 to 1.1. Oct 4 03:50:01 82e6de130138 iprep: IPREP update versions: major version:1 minor version:1068 update version:231 total ips:1640158 last update time:1485009026 Oct 4 03:50:01 82e6de130138 iprep: Webroot credentials from PE. oem_id:Citrix device_id:450000 user_id:HE2H91SCZ6. Oct 4 03:50:01 82e6de130138 iprep: PE update versions: major version:0 minor version:0 update version:0 total ips:0 last update time:0 Oct 4 03:50:01 82e6de130138 iprep: outfile:/var/nslog/iprep/webroot_http_resp_1507089001.xml

This database is updated every fife minutes. Database updates also get logged:

Oct 4 04:15:21 <local2.info> 82e6de130138 iprep: File:update_1.1332_107.txt no of ips:0. Oct 4 04:15:21 <local2.info> 82e6de130138 iprep: WebRoot update versions: major version:1 minor version:1332 update version:107 total ips:2110941 last update time:1507090521 Oct 4 04:15:22 <local2.info> 82e6de130138 iprep: This update version doesn't have any new ip data. Oct 4 04:15:22 <local2.info> 82e6de130138 iprep: iprep_update_pe_cur_versions:430 updating PE with DB versions..
The database is stored in /var/nslog/iprep/iprep.db (the file mentioned in the logs is a temporary file and gets deleted immediately).

root@myNetScaler# ls -l total 88080 -rw-r--r-- 1 root wheel 90113024 Oct 4 04:20 iprep.db
So my reputation file is round about 90 MB in size. It’s a binary file, so there is no point in looking into it.

Thread categories

NetScaler has two built in functions:

IPREP_THREAT_CATEGORY(category)
IPREP_IS_MALICIOUS

While the later is a general one, the first one is very specific. There is a set of threat categories, and you have to specify the ones you’re interested in.

In a reverse proxy deployment you would filter malicious clients: CLIENT.IP.SRC. If you want to protect your clients from connecting to a malicious server you would rather filter potentially malicious server IPs: CLIENT.IP.DST

There are several thread categories (sources: Product documentation by BrightCloud, Citrix)

SPAM_SOURCES: The Spam Sources category includes tunneling spam messages through a proxy, anomalous SMTP activities, and forum spam activities.

WINDOWS_EXPLOITS: The Windows Exploits category includes active IP addresses offering or distributing malware, shell code, rootkits, worms or viruses.

WEB_ATTACKS: The Web Attacks category includes cross site scripting, iFrame injection, SQL injection, cross domain injection, or domain password brute force attack.

BOTNETS: The Botnets category includes Botnet C&C channels, and infected zombie machines controlled by Bot master.

SCANNERS: The Scanners category includes all reconnaissance such as probes, host scan, domain scan, and password brute force attack.

DOS: The Denial of Service category includes DOS, DDOS, anomalous sync flood, and anomalous traffic detection.

REPUTATION: The Reputation category denies access from IP addresses currently known to be infected with malware. This category also includes IPs with average low Webroot Reputation Index score. Enabling this category will prevent access from sources identified to contact malware distribution points.

PHISHING: The Phishing category includes IP addresses hosting phishing sites and other kinds of fraud activities such as ad click fraud or gaming fraud.

PROXY: The Proxy category includes IP addresses providing proxy services.

NETWORK: IPs providing proxy and anonymization services including The Onion Router aka TOR or darknet.

CLOUD_PROVIDERS: I didn’t find any information about this category. As far as I understood, this means, the IP belongs to a cloud provider like AWS, Azure, … So it does not indicate a malicious IP at all.

MOBILE_THREATS: I didn’t find any information about this category. It seems to be a collection of IPs harmful for mobile devices

How to use IP-Reputation service?

Usually I create responder policies with IP reputation feature.

Proxying outside

During proxying to outside I usually use responder policies redirecting to an error page, or respond with a predefined error message telling the user about the reason for blocking.

Action

add responder action res_act_block_malicious respondwith q{"HTTP/1.1 401 Requested URL not allowed\r\n\r\nYour URL had been blocked due to security concerns about target IP "+ CLIENT.IP.DST}

This action responds with a HTTP 401 (unauthorized) text telling the user about the problem. This will help both, user ans help desk, understanding what’s going on. I would not reset the connection (from technical point of view: send a TCP reset), as a user would not understand, what’s going on, nor would I drop the connection, which would be more or less the same, from perspective of a user, but by far slower (as there is no reply from the server).

Policy

add responder policy res_pol_block_malicious "CLIENT.IP.DST.IPREP_THREAT_CATEGORY(WINDOWS_EXPLOITS) || CLIENT.IP.DST.IPREP_THREAT_CATEGORY(PHISHING)" res_act_block_malicious

This policy checks if server’s IP is either known for spreading windows exploits or hosting sites used for phishing.

Proxying inside: a Reverse Proxy

if we proxy to inside (which is more common than load balancing outbound) we have several options: Responding with a HTML page (i.e. an error message), blocking or dropping.

Respond with an HTML page seems to be a good idea. My action would look simmilar to the action above. But it would inform the attacker about the reason why he is unable to connect and thereby what to do to connect even though (get an other IP).
Blocking in my opinion is stupid. What would you think if I would block you? My first guess would be: What can I do to be blocked no more? Maybe use a different IP? Use TOR network? So blocking does not seem to be the right thing!
Dropping is my choice. Using a well known bad IP my server would look like not up and running. Being a Black-Hat I’d give up (or invest a serious amount of time on examining the issue).

The policy

add responder policy res_pol_dropmalicious "CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(DOS) || CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(BOTNETS)" DROP -logAction log_drop_malicious

so this policy will drop a client request originating from either a well known source of DOS attacks, or from a well known bot network.

some concerns?

Services like IP reputation may tend to false positives. I’d strongly recommend to log, so you’ll be able to investigate issues. My policies therefore usually contain a logging policy.

But think of one of your main customers using an IP with bad reputation? You could ask WebRoot to change the reputation of this very IP address. However this does not work well and takes time. Webroot rates IPs for reason, so it’s very likely to reappear within some days. This may be an IP of a proxy also used by bad guys.

We could white list IPs by just combining the policy with
&& CLIENT.IP.SRC.IN_SUBNET(98.12.43.5/26).NOT

However this would lead to an endles list of exceptions, unreadable to humans, inefficient in NetScaler and unmanageable. So I would rather use a data set.

Data sets are lists of numbers, in this case: IP addresses. I use these lists to white list IP addresses.

add policy dataset Alowed_IP_List ipv4 bind policy dataset Alowed_IP_List 82.218.161.177 -index 1 ...
The policy would now look like that:
add responder policy res_pol_block_malicious "res_act_block_malicious((CLIENT.IP.DST.IPREP_THREAT_CATEGORY(WINDOWS_EXPLOITS) || CLIENT.IP.DST.IPREP_THREAT_CATEGORY(PHISHING)) && CLIENT.IP.SRC.TYPECAST_TEXT_T.CONTAINS_ANY(\"Alowed_IP_List\")).NOT"

So we now first check if the IP gets blocked by any of my IP reputation checks, and then if it’s not in my allow list.

I hope, this short summary helps.

Cheers

Johannes

Sometimes we have to schedule commands in a Citrix NetScaler. A good example would be:
force HA failover
It’s obvious, we don’t want to fail over during day time to not disconnect TCP connections, to not interrupt users. The best time would be something like 3:30 AM. It’s obvious, we don’t want to set an alarm for 3:00 to get up, take a shower, brush teeth, just to force an HA fail over. At least I don’t want!

Scheduling an HA fail over for off peak hours is important for both, Citrix NetScalers proxying big files for download and for NetScaler Gateways: During HA fail-over we will loose TCP-sessions, so downloads will break and HDX (ICA) sessions will get disconnected.

Starting to dig into Citrix NetScaler

Inside a NetScaler there are two operating systems working at the same time and therefore two different shells:

the Citrix NetScaler shell, the first one you connect too using putty (or even better: smarTTY)
the BSD shell. It can be reached typing
shell
into NetScaler’s command line

There is no chance to schedule commands in NetScaler OS. But BSD is just an ordinary UNIX (please don’t call BSD a Linux, it is not). My first guess would be to use at, however at is not there. So we need to use crontab.

Crontab in UNIX is used to schedule commands on a regular base. So crontab would be great to schedule a backup of Citrix NetScaler configuration, it’s not perfect for one time commands.

We could install at into BSD, but I never install software into a NetScaler and I would strongly advise you to keep away from doing this. So we need to use crontab.

How to execute a NetScaler shell command from BSD?

That’s a big question. BSD shell just allows to execute BSD commands. So what now?

nscli

nscli is a UNIX command on a NetScaler, allowing users to execute NetScaler commands from BSD

root@82e3d3135738# man mscli No manual entry for mscli

shit.

root@82e3d3135738# nscli --help Usage: nscli [-norc] [-U []:] [-D ] [-s] [[-k] ]>

where: -norc causes the personal initialization file, ~/.nsclirc, to be skipped is the IP of the target NetScaler is used to log in to the target NetScaler is an integer between 0 and 9 -s stifles "exec:" and "Done" messages is any nscli command and -k causes the program to keep-a-going after command root@82e3d3135738#

much better! so we have to execute a command like that:

nscli -U 127.0.0.1:nsroot show ns runningconfig

so we specify a NetScaler IP (no SNIP, sorry guys, we’re dealing with BSD!), an user name and NetScaler commands after this.

It works fine, unfortunately we get prompted for a password. So we can’t easily use this command in a batch file? Yes we can. There is some information missing: we may specify a password as well. No too beautiful, as this batch file will also contain the password in plain text, but possible. The command would look like that:

nscli -U 127.0.0.1:nsroot:your_Password_goes_here show ns runningconfig

easy? Yes, it is! You may even skip the IP using this command locally:

nscli -U :nsroot:your_Password_goes_here show ns runningconfig

This leading : assumes an IP of 127.0.0.1.

Using crontab on a NetScaler

Using crontab on a NetSaler would be more than just easy. Just add a standard crontab entry into /etc/crontab.

30 3 * * * root nscli -U 127.0.0.1:nsroot:your_Password_goes_here force ha failover -force

That’s simple.

Next we’ll have to kill cron and start cron (cron start) again, so it will reread crontab.

root@82e3d3135738# cron start cron: cron already running, pid: 965 root@82e3d3135738# kill 965 root@82e3d3135738# cron start

Unfortunately this entry won’t disappear after executing, so it will get executed tomorrow and the day after tomorrow as well. So you have to remove this entry tomorrow morning. Still by far better than getting up in the middle of the night, isn’t it?

What else could we do?

We could also use this for daily tasks, such as backing up ns.conf, purging log files and many more!

BUT

never reboot your NetScaler! Why? All content in /etc gets discarded. /etc is just RAM, no disk based file system.

What to do?

Well we need to rewrite /etc/crontab with every reboot! I’m pretty sure you won’t like to do this. There has to be an other way, a more automatic way, to write data into crontab!

We could use /etc/rc.conf to fill crontab after reboot. Unfortunately we face the same problems here: It will get discarded during boot. However there is a file called /flash/nsconfig/rc.netscaler (see CTX122271). This is the template for the /etc/rc.conf.

There is a good description in Citrix forums by Rob Harp about how to use it. Rob’s example is about doing daily backups. I’d suggest reading his article.

An important note in the end

Keep in mind: Changes to BSD shell is executed on this very Citrix NetScaler only. It will never get executed on the other node of a HA or cluster! You’ll probably have to do these changes with all nodes!

What’s an IP address calculator?

I’m pretty sure it’s something you won’t need. It will help understanding IP addresses. It does calculations on IP addresses and will tell you, if the address is valid (or a network / broadcast address), and if two addresses are on the same subnet.

Why did I create an IP address calculator like that?

My daughter started studying informatics and she most of their fellow students had problems understanding IP addresses. It took me some time to see why. And yes, I started remembering the time, back in the late 1980s, when I started creating networks of UNIX hosts, and how helpless I had been about IP addressing.

So I created an IP address calculator for my daughter.

What does this IP address calculator do?

Translating IP addresses into binary / hexadecimal

It’s essential to go away from decimal notation if you want to understand an IP address. So if you would enter 192.168.15.2/24 into my IP address calculator it will return:

An IP address explained using binary

It will also return: IP address 192.168.15.2 is a class C (255.255.255.0 /24) following RFC 791. You may also ping this IP using any of these notations: 0xc0.0xa8.0xf.0x2 oder 3232239362. The IP address is valid.

Why did I mention this fancy decimal number? Well I want students to understand IP addresses. ((192*256+168)*256+15)*256+2 = 3232239362. It’s a 4 byte number, so it’s just an other notation. IP addresses are 4 byte numbers.

Validations

My IP address calculator will check, if this IP is valid, or not. It will not allow IPs other than class A, B and C (actually it will allow Class D, multicast as well, but it will mention)

If you do calculations on any forbidden address (192.168.15.0/24, 192.168.15.255/24, 127.x.y.z, …) it will tell you it’s an invalid IP, and why. In most cases it will refer to an RFC.

If the subnet mask is not following current RFCs (so not all 1 to the left, all 0 to the right) it will tell you about this problem (i.e. 232.255.255.0). It will do all calculations based on this subnet mask.

Routing

One of the most challenging things for beginners is routing. Why is 192.168.15.2/24 and 192.168.15.254 the same subnet, while 192.168.51.2 is a different? Well, once more I’ll show IP1, IP2 and subnet mask in binary. I will use different colours for network- and host number. I try to make clear where differences are.

IP routing explained using binary

In addition I also add a small sketch showing either two hosts directly connected to each other, or connected via a router.

Network and broadcast addresses

These students always need to specify both, the network address and the broadcast address. So I also do this.

Feedback wanted

Please tell me if you don’t like my explainations, if you think they are wrong, miss-leading or you can hardly understand!

Using my calculator / terms and conditions

like always I will be happy to see usage. You are allowed to link to my IP calculator, use it in and for your classes, preparation for exams and your daily work. Like any software my IP calculator is buggy, so never use if for a production environment, never use it during exams. You can’t make me responsible for miscalculations my did!

I needed to use a Citrix NetScaler both, as a SAML identity provider (IDP) and service provider (SP). So I set up my test environment accordingly.

What my test environment looked like:

You see, I created two admin partitions on my Citrix NetScaler, one for the service provider (SP partition), cotaining both, the SAML SP and a web server, and one for my identity provider (IDP partition), containing the IDP.

I used this partitions to emulate “2 different NetScalers” as it does not make sense to have both, SAML-SP and SAML-IDP in the same data center (you could do conventional LDAP/RADIUS/TACACS authentication instead).

How SAML works:

SAML authentication uses an external server for authentication, the so called SAML Identity Provider (SAML-IDP).

The SAML Service Provider (SAML-SP) is local, close to the resource, and calls for Authentication to the SAML-IDP.

So a user connects to a resource. If the user had not been authenticated before, he gets gets forwarded to the logon server, the so called SAML-SP.

The SAML-SP forwards the user to the SAML-IDP for actual authentication. The SAML-IDP does the authentication.

After successful authentication, the SAML-IDP forwards the user to the SAML-SP. It also passes the so called assertion, the prove this user was authenticated successfully. You could think of an assertion like a man’s ID card. As soon as the SAML-SP has validated the assertion, it forwards the user to the resource.

SAML-SP and resource are always located on the same Citrix NetScaler, the SAML-IDP is usually located “somewhere else on the internet”.

Certificates

SAML uses certificates to establish trust between SAML-SP and SAML-IDP.

The SAML-SP uses a server certificate to authenticate to the SAML-IDP. This certificate (not the private key, of course) has to be on the SAML-IPD as well, so it can get checked.

The SAML-IDP uses a certificate to digitally sign (and encrypt) the assertion. This certificate (again: not the private key) has to be present on the SAML-SP, so the SAML-SP is able to decrypt and validate the assertion.

It’s possible to use the same certificates for both, SSL between client and SAML-IDP / SAML-SP, and to prove identity, however I would rather use private (and therefore more trustworthy) certificates to prove identity.

NetScaler as a SAML Service Provider (SAML-SP)

A Citrix NetScaler may be a SAML identity provider for any SAML service provider. An other NetScaler may be the service provider, but also services like Microsoft Azure, Microsoft Office 365, Citrix Sharefile and many more may use a NetScaler as an authentication source.

In my example I just created a simple load-balancing vServer and added authentication to it. There is nothing special about it, in fact I used my test server (a description might be found there).

add server www 10.127.255.250 add service lb_svc_www www HTTP 80 add lb lb_vserver lb_vsrv_www HTTP 192.168.0.4 80 bind lb lb_vserver lb_vsrv_www lb_svc_www

The NetScaler SAML Authentication policy

The NetScaler SAML Service provider action

GUI: Navigate to:

Security → AAA-Application Traffic → Policies → Authentication → Basic Policies → SAML

With SAML Actions click Add.

add authentication samlAction saml_sp_server -samlIdPCertName lets_encrypt -samlSigningCertName lets_encrypt -samlRedirectUrl "https://idp.norz.at/saml/login" -samlUserField "Name ID" -samlIssuerName "https://sp.norz.at"

IDP Certificate Name*	SAML IDP’s certificate
Redirect URL*	The URL of the SAML IDP in use, if IDP is a NetScaler: /saml/login
User Field	User Name in assertion, if IDP is a NetScaler this is Name ID
Signing Certificate Name	a certificate used to sign the SAML assertion (a normal server certificate)
Issuer Name	The FQDN of the SAML Service Provider (this AAA server)

The NetScaler SAML Service provider policy

GUI: Navigate to:

Security → AAA-Application Traffic → Policies → Authentication → Advanced Policies → SAML

add authentication Policy SAML_SP_pol -rule true -action saml_sp_server

The SAML Service Provider (SAML-SP) Authentication vServer.

Click add

Provide name and IP (port s usually 443, protocol can’t be changed)

bind a server certificate (this one gets exposed to users, so it has to be trusted!)

bind the authentication policy you previously created

add authentication vserver SAML_SP SSL 192.168.0.4 443 set ssl vserver SAML_SP -ssl3 DISABLED bind authentication vserver SAML_SP -policy SAML_SP_pol -priority 100 -gotoPriorityExpression NEXT

NetScaler as a SAML Identity Provider (SAML IDP)

A Citrix NetScaler may also get used as a SAML Identity Provider (SAML-IDP). This allows to authenticate to any authentication source like LDAP, RADIUS, Certificates, TACACS, local (to IDP), Negotiate, O-Auth, SAML, WebAuth, EPA or Citrix StoreFront. In my example I authenticate to TACACS (TACACS policy is not included)

Creating the a SAML Identity Provider Policy

Creating the a SAML Identity Provider Action (Saml IDP Action) on a Citrix NetScaler

Navigate to:

Security → AAA-Application Traffic → Policies → Authentication → Advanced Policies → SAML IDP. Go to Profiles.

Click Add.

add authentication samlIdPProfile SAML_IDP_profile2 -samlSPCertName SP-assertions-signing-cert -samlIdPCertName IDP-Signing-Cert -assertionConsumerServiceURL "https://sp.josel.net/cgi/samlauth" -samlIssuerName "https://sp.josel.net" -signatureAlg RSA-SHA256 -digestMethod SHA256

Assertion Consumer Service URL	The URL of the Service provider (if NetScaler: *https://FQFN/cgi/samlauth*)
IDP Certificate Name	Certificate used to digitally sign the assertion (a normal server certificate)
SP Certificate Name	Certificate used by the service provider, so it can be trusted (see above)
Encrypt assertion	keep SAML traffic a secret (best practice)
Issuer Name	The FQDN of the SAML Identity Provider (this SAML IDP’s name)

The SAML-IDP policy

add authentication samlIdPPolicy SAML_IDP_Policy -rule true -action saml_idp

The authentication policy

I don’t go into authentication policies here. Just follow Citrix bast practices, there are many guides out there. I created a policy similar to CTX113820.

The SAML Identity Provider (SAML-IDP) Authentication vServer.

click add

provide name, IP address and port, usually 443 (the protocol can’t get changed)

bind a server certificate. This one gets exposed to the user, the user has to trust this certificate!

bind an authentication method and a SAML IDP policy

select both, the IDP and the authentication policy

Trouble shooting

I used following tools:

Citrix NetScaler’s log (Yes, there is a log on a NetSaler and SAML issues get logged there! You look at /var/log/ns.log)

FireFox add-on SAML-Message Decoder (also available for Chrome)

Citrix NetScaler Network traces

Issues:

I have seen several issues recently:

SAML-SP fails to forward to SAML-IDP

detected: error in browser

check settings on in SAML-SP’s SAML Authentication action: Redirect URL

SAML-IDP fails to forward to SAML-SP

detected: error in browser

check settings on in SAML-IDP’s SAML-IDP Authentication action: Assertion Consumer Service URL

Certificate not trusted on SAML-IDP

detected: confusing message in browser, log in IDP’s /var/log/ns.log

add SAML-SP’s signing certificate to SAML-IDP’s SAML-IDP profile: SP-Certificate Name

I hope. that helps. Just drop me a message if you need more information. You’re very much welcome to link to my blog / my website. Thanks!

last update: November 14 /2017

Or: The power of the ANY service type

This is a work around for a well-known problem in NetScaler: Binding NetScaler Gateways to content switching vServers.

This solution does not follow Citrix best practices. Avoid using it, if you can!

My solution will work with NetScaler 10 upward. I didn’t test with 9.x as they are not considered to be secure any more.

The Problem

Up to 11.0 it was impossible to bind a NetSaler Gateway to a Content Switching vServer. By now (firmware versions 12) this is limited to a single NetScaler Gateway. This limitation may be an obstacle to overcome in certain environments. Most companies nowadays suffer under a lack of public IPs. But mos of all: Users don’t like complex environments with tons of different URLs to handle, one for mobile devices, one for PCs, one for trusted, one for untrusted devices and so on. Instead they want to use a single URL for all use cases.

Content switching may mitigate this issue by hiding very different configurations behind a single URL. But this is not true for NetScaler Gateways. In days of old we could not bind any gateway to a content switching vServer at all, now (starting from version 11) we can bind a maximum of one gateway to it.

Why may one gateway not be enough? First of all, it is complexity. It may confuse you if you have to bind tons of different scenarios to one gateway. In my real world experience I see often buggy environments being buggy, as complexity may over work the admins. But there may also be technical reasons. One of my costumer would have to bind round about 50 LDAP sources of costumers and partners. All of them are geographical dispersed and some of them may even be misconfigured and therefore slow. Logon to the last ADs in the list would be painful. Splitting the gateway up into some gateways would speed up things very much.

The solution

This question came up in one of my NetScaler classes. We set up all needed NetScaler Gateways. They are addressable and use private addresses of a separate address space (this address space does not exist outside of NetScaler).

We set up a content switching vServer. I would prefer a SSL-bridge to avoid SSL offloading, however we needed something to base content switching on, so we used a SSL vServer. This is far from being a perfect solution, but it works.

How to bind them together?

My first thought was: pointing the services of the load balancing vServer to the NetScaler gateways. But this does not work, we faced an error stating this IP address is already in use.

That’s my trick: I create load balancing vServers of type ANY and point its services to the corresponding gateways. That’s why these gateway servers use private addresses that don’t exist in your environment. This traffic will never leave this NetScaler.

^NetScaler CS-Vserver loadbalancing many NetScaler Gateways

(graphic by courtesy of Andre Buck)

What’s wrong about this setup?

It does not follow Citrix best practices. So you should avoid using it. On the other hand: everything we do is fully supported: The content switching vServer, the load balancing vServers bound to it, load balancing vServers of type any, and last, not least, the gateways.

We won’t be able to log on to the NetScaler Gateways using smart cards (certificate based logon), if we use SSL-Offloading lb vServers, as these certificates won’t be visible to the NetScaler Gateway.

Why would you use it even though?

It’s currently the only chance to bind more than one NetScaler Gateway to a content switching vServer on a NetScaler.

I recently was asked to teach Citrix SD-WAN. My first thought was: wtf? I asked Google, and Google, knowing everything, spoke to me in infinite wisdom: Citrix SD-WAN’s previous name is Branch Repeater. And Branch Repeater, I did already know this, once was the new name for WanScaler (a product I have been certified on, but never used in real life). Meanwhile the product got rebranded again and is now called Citrix NetScaler SD-WAN. So it is just a rebranded product?

WanScaler once was a great product, caching WAN traffic, and thereby preventing content from traversing a WAN multiple times. “Compression rates” of 1:100,000 had been possible, would probably still be possible, if … Yes, if we nowadays would not encrypt everything. Caching and encryption don’t go together well, never did and will never ever do. That’s why I didn’t recently hear much about WanScaler or CloudBridge. It has its reason to exist, mainly in ICA environment, but never got a big success.

But I do what people want me to do, so I started reading into it. I built my own test environment consistng of 2 SD-WANs, 2 WanEms, a server and a client. And found absolutely thrilling information! It is not just a 3^rd re-branding of a product of very limited area of application, instead it is a brand new approach to WAN: Software Defined WAN, SD-WAN.

There are two different types of appliances: These classical WanScalers, Repeaters, which ever name you want to use (SD-WAN WO [WAN-optimization]), and this brand new type of software defined WAN (SD-WAN SE [Standard edition]). And, in addition, and a blended version, called NetScaler SD-WAN enterprise of course, worth thinking about it!

What’s so totally new about it?

I will just focus on software defined WAN (SD-WAN SE). SD-Wan nowadays is a hype. Gartner says:

By end of 2019, 30% of enterprises will use SD-WAN products in all their branches, up from less than 1% today.

That’s an ambitious prognosis! And Citrix is right here, one of just 5 solutions currently on the market. Gartner:

Organisations looking for WAN optimization or dynamic selection capabilities should consider this vendor, especially when Citrix applications are also present

What means: dynamic selection capabilities?

It’s all about finding out, where to send packets too. Still not clear?

Current deployments:

Usually we have a MPLS connection to branches. MPLS is fast, has low jitter and is reliable, guaranteed SLAs of 99,9% are usual (this means: less than 1 hour down per month), in real life European SLAs will be even much higher. It’s very common to bundle MPLS with a GSM LTE or GSM G4 connection in active passive, as a last mile outage due to construction works is a very likely thing to happen. This will result in a SLA of 99,999%, meaning: 5 minutes down time per year. In addition we usually also have Internet connections (with much lover SLAs of about 98-99%, 7 to 15 hours down per month) in place.

Citrix NetScaler SD-WAN

What’s wrong about it?

Simple like that: we have 3 connections, one costly, one moderate and one cheap. And we only use the costly one for WAN transfer. If we need to upgrade (Gartner speaks about 15% increase of WAN traffic per year, so there are always upgrades coming up), we need to upgrade the most costly one. It’s a damn expensive solution.

Can we make things better?

Yes we can! SD-WAN would be a solution allowing all this 3 connections to be used at the same time. You think about link load balancing? You’re totally wrong! Keep on reading.

A Citrix NetScaler SD-WAN is a virtual WAN solution. Tracerouting your WAN from inside, you would just see a single hop, not two of them, so it’s technicaly to be considered as a tunnel. It’s a tunnel, aggregating of all this three connections. But the really important thing is: it’s a UDP based tunnel using UDP port 4980.

What’s great about an UDP based tunnel? It’s just a less reliable tunnel, isn’t it?

Yes and no. UDP is not reliable at all. That’s true. UDP does not have connections like TCP has. No sequence numbers, no acknowledgement numbers. If we need reliability, we have to add it on a higher layer. However, using a stateless protocol, we can send one packet of a single TCP connection on MPLS, the next one on GSM. Even more: the packet and its acknowledge don’t even need to use the same connection. And it’s easy to avoid a congestion, because we can dynamically swerve a line if quality of this connection is decreasing.

And that’s what it is: We have a tunnel between data centre and branch office. And the tunnel intelligently and dynamically selects the best matching connection for a certain kind of traffic. We can hardly predict where a packet would flow. Its policy based, so we can assign certain requirements about quality for each kind of traffic.

Asymmetric connections? What about our firewall?

It’s all based on UDP. There is no connection on layer 4. Our Sessions are layer 7 only, and a firewall is a L3/4 device. The only thing you have to do: allow UDP 4980 originating from all branches to traverse your firewall!

Which connection is the best one?

It depends (this is the universal answer for every architect to all kind of questions).

Think about ICA? It’s the least latency one. User experience is very sensitive about latency. Jitter? We don’t care much about it, as long as we can keep latency below a certain value.

SIP (VOIP telephony)? Latency is not a big issue. Latency above a certain amount will make our phone calls less interactive, but everything below 200ms would be fine. Instead its jitter we have to care about. Jitter would distort spoken words, so they are harder to understand. And we are concerned about packet loss of course, as packet loss has an even stronger impact on understandability than jitter.

TCP based Videos? Well, neither of them is a problem, as we usually buffer videos. Even packet loss is not a big problem, as long as it doesn’t exceed a certain value. Instead its mere bandwidth. Videos nowadays fill up our costly WAN links causing congestion and packet loss.

File transfer? During browsing of directories, latency clearly is an issue, but this is not true about up- or download of files. WanScalers always had their method to deal with this latency issues, and it didn’t disappear in the enterprise or WO version. Again, it’s just bandwidth. However we would prioritize it a bit higher than video as less speed immediately impacts user experience.

So every type of connection has its own, very special, requirements. Pure prioritization won’t be sufficient.

All our Connections are pooled into the so called SD-WAN network connection. Our NetScaler SD-WAN Box will continuously monitor all possible connections and select the one best matching for every TCP stream, even for every single TCP packet. And even more: We have no need for symmetry. We could send an ICA packet on MPLS while we get the acknowledge packet over the internet! So asymmetric connections (like ADSL or cable TV) are an issue no more.

I did a short survey with my costumers: SIP traffic is not increasing very much. ICA traffic is increasing slowly while bulk traffic, mainly HTTP and videos, is exploding. Gartner calculates with annual growth of 15% continuing until 2019. So why upgrade our expensive MPLS connections and not use cheap internet instead?

How often do we measure latency?

At least every 50 ms on an idle connection. SD-WAN protocol sends measurement information with each and every packet it transmits. So the more busy your connection is, the more measurement data we transmit.

Security

Sending data over the internet is a risky thing. Citrix NetScaler SD-WAN uses 256 BIT AES IPSec. No data will traverse the internet in an unencrypted way. The crypto library supports NSA Suite B, I think, this is sufficient.

There is another aspect. If I want to hack into a connection, I need to pick up every single packet of this very data stream. I face dramatically more overhead if a single packet is missing. How can I collect all packets, if packets are distributed randomly over several very different connections, provided by different providers? This would even be challenging for these intelligence services all over the world! How could they ever reassemble a TCP stream? In my opinion this is a strong plus on security, even a plus over MPLS.

Resilience

We continuously measure the quality of a connection. If we see any parameter changing (latency, jitter, congestion, bandwidth) we will immediately change our assessment. Internet can have both, less latency and less jitter than MPLS. But it might change within milliseconds. It’ important to react. SD-WAN does immediately!

SIP (VOIP) is very sensitive to both, packet loss and jitter. At the same time it’s not a top bandwidth waster. We could therefore easily duplicate data over two or more lines and send it simultaneously. The data arriving first will be forwarded to the user, the other packets will be discarded. This would reduce the risk of packet loss and, at the same time, reduce both jitter and latency. This is an other great feature of Citrix NetScaler SD-WAN, and turned on by default for SIP. Isn’t it a great chance for our VOIP calls?

There is a great Video on YouTube about resilence. It’s a marketing video, taken at Synergy 2016, but I like it, as it truly shows how it works.

More chances to safe costs

We currently use MPLS because it is secure, available and offers guaranteed SLAs. Using Citrix NetScaler SD-WAN we don’t need to care about security as it’s built into SD-WAN. However SLAs of internet connections are not on top. What about using three internet connections at the same time. Three connections using very different technology like a GSM based, a cable based and a DSL based connection, instead of MPLS? This would offer SLAs above MPLS and, at the same time, be by far less costly? I think, evaluating this would be worthwhile!

Links

I have spoken to guys from Danish government evaluating Citrix NetScaler SD-WAN, some weeks ago. They are very interested in SD-WAN, mainly as they already have a department using Citrix NetScaler SD-WAN: Danish AgriFish. It’s all about costs and random disruption of ICA connections, and it works perfectly well. AgriFish is enthusiastic about it, other governmental authorities will follow. I link this (Citrix) success story here. I’m sceptic about success stories (never trust statistics you didn’t fake yourselves, but these guys I met face to face could prove this AgriFish one, so I can trust in it!

Why would you like to customize a 404 page?

One of the most important things to know is: What kind of web server do I have to deal with?

The first source to look into is a HTTP response header called Server. Information here may be very verbose. I don’t know why this header is part of HTML standard, but actually it is.

The Server response-header field contains information about the software used by the origin server to handle the request. The field can contain multiple product tokens (section 3.8) and comments identifying the server and any significant subproducts. The product tokens are listed in order of their significance for identifying the application. (RFC 2616)

This is an example server header:

Apache/1.3.28 (Unix) mod_ssl/2.8.15 OpenSSL/0.9.7c mod_perl/1.27 PHP/4.3

More reasons to change the 404 page

of course there are even more reasons to change the 404 page: customized 404 pages seem to be funny, they may help people to find the content needed, and so on.

Why not change your web server?

My solution

The length of the text is limited, so this is not a good solution. And I would rather like to place the file “somewhere” on my web server, so it’s pretty easy to change.

I spent some time thinking what to do and made up my mind to use the HTTP callout feature. It was my first ever attempt to use HTTP callout, and I’ll describe how it works.

NetScaler’s HTTP callout feature

I do something completely different: I will retrieve the content of the 404 page from a web server. To do so I have to navigate to App Expert -> HTTP Callouts.

We scroll down to the bottom and select the return type TEXT and the expression should be HTTP.RES.BODY(65538). The number is the number of bytes to retrieve.

command line version:

add policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"
set policy httpCallout callout_retrieve_404 -vServer cs_vsrv_norz.at -returnType TEXT -hostExpr "\"norz.at\"" -urlStemExpr "\"/notfound.htm\"" -scheme http -resultExpr "HTTP.RES.BODY(65538)"

The rewrite policy

The rewrite policy should be a very simple thing:

The NetScaler rewrite action using a HTTP callout

add rewrite action callout404 replace_http_res "SYS.HTTP_CALLOUT(callout_retrieve_404 )"

The NetScaler rewrite policy

add rewrite policy rw_pol_404 "HTTP.RES.STATUS.EQ(404)" rw_act_404

This policy will get applied if the HTTP response status is a 404 (HTTP.RES.STATUS.EQ(404)). I then bound this policy to my web server. That’s it. It was pretty easy.

So I searched for it, and found it in /var/partitions/<partitionname>/nstrace.

So that’s how I do an nstrace inside a NetScaler admin partition:

I use putty to connect to my NetScaler. Masochists might prefer to use the built in terminal from GUI, however I don’t tend to masochism.

switch partition <partitionname>
start nstrace -filter “CONNECTION.SRCIP.EQ(<source IP>)” -size 0 -time 3600 -link
stop nstrace

So I log into my Citrix NetScaler. I change into my partition (currently partition names can’t auto complete, so make sure you know the name; show ns partition will list all partitions.

Next I start the trace. To do so I follow CTX120941.

After doing your trace you may execute a stop nstrace command to stop your trace. This is not needed if you set the time parameter, but I prefer to stop traces instead of setting a time parameter.

Additional parameters for tracing

There are several more parameters. You may find them in Citrix NetScaler product documentation.

last update: April 16^th 2018

I had been asked recently: Johannes, how can we log data about NetScaler Application Firewall policy hits in detail?

The standard NetScaler Web Application Firewall log-files

NetScaler’s Web Application Firewall logs to /var/log/ns.log. These logs are fine for trouble shooting. There is a good description about these logs here. This is a sample log, stolen from a Citrix blog about NetScaler Web Application Firewall (WAF) logging:

Jun 22 19:14:37  10.217.31.98 06/22/2015:19:14:37 GMT ns 0-PPE-1 :d
efault APPFW APPFW_XSS 60 0 :  10.217.253.62 616-P
PE1 y/3upt2K8ySWWId3Kavbxyni7Rw0000  pr_ffc http://aaron.stratum8.n
et/FFC/login.php?login_name=abc&passwd=12345&drinking_pref=
on&text_area=%3Cscript%3E%0D%0A&loginButton=ClickToLogin&am
p;as_sfid=AAAAAAWEXcNQLlSokNmqaYF6dvfqlChNzSMsdyO9JXOJomm2vBwAMOqZI
Chv21EcgBc3rexIUcfm0vckKlsgoOeC_BArx1Ic4NLxxkWMtrJe4H7SOfkiv9NL7AG4
juPIanTvVo%3D&as_fid=feeec8758b41740eedeeb6b35b85dfd3d5def30c C
ross-site script check failed for field text_area="Bad tag: script"
 <blocked>

This log is showing a blocked cross-site script (XSS) attack. It’s a reply to a form using http GET method.

A custom error page?

Yes, this is something we could definitely do: Create a custom error page, displaying details about the reason for the attack to be blocked. It’s obvious: We don’t want to display informations like this to an attacker, so we will never use a page like this in production. However it would be a nice-to-have in a test environment. So I created a file, following CTX140293. This file can be imported easily into a WAF-Profile and will display information about transaction ID, session ID, Violation category, the log entry displayed in the event log and the corresponding session cookie. Feel free to download this file following this link.

I could think of using a page displaying the transaction ID only. So a person blocked by mistake could specify this ID to the help-desk and it would be easy for the help-desk to find this log entry in your Syslog server.

Unfortunately all this is not detailed enough during test and pre-prod / implementation phase. There is so much data missing: Cookies beyond session cookies, http PUT or POST data, headers and many more. So we have to do network traces. But traces are time consuming to read. So what to do?

I have written an other blog about logging responder and rewrite policies. Similar to responder and rewriting policies we may log app-fw policy hits. So turn on “User Configurable Log Messages” in “Change Auditing Syslog Settings”

Useful logging policies for NetScaler Web Application Firewall:

Well, don’t do this during production phase, it might lead to duplicated log entries and might print sensitive data like usernames and passwords into your logs (which is not desired). I suggest using this policies in test and pre-prod/staging environments only!

I use warning as a servity as NetScalers Web Application Firewall’s messages (different to other messages) usually use warning as a servity.

Logging HTTP headers

add audit messageaction log_URL WARNING "\"Client with IP \" + CLIENT.IP.SRC + \" has accessed \" + HTTP.REQ.URL.PATH_AND_QUERY.MARK_SAFE + \" Headers \" + HTTP.REQ.FULL_HEADER"

This will result in a log entry like this:

Apr  5 09:22:50  192.168.10.101 04/05/2018:09:22:50 GMT ns_vpx_01 0
-PPE-0 : default APPFW Message 1263 0 :  "Client with IP 192.168.10
.10 has accessed /images/FormField.png Headers. GET /images/FormFie
ld.png HTTP/1.1^M Host: afweb.training.lab^M User-Agent: Mozilla/5.
0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0^M A
ccept: */*^M Accept-Language: en-US,en;q=0.5^M Accept-Encoding: gzi
p, deflate^M Referer: http://afweb.training.lab/defaultstyles.css^M 
Cookie: ASPSESSIONIDQQDTACAQ=OKDFMLOCFLFKBKOINCMPPMPN; ASPSESSIONIDS
SATBCBQ=KPCFHFPCOOMBBJOINHDNNLHC^M Connection: keep-alive^M Cache-Co
ntrol: max-age=0^M ^M "

So we can see not just the usual data, but also all HTTP headers in the log.

Logging HTTP POST data

add audit messageaction log_URL WARNING "\"Client with IP \" + CLIENT.IP.SRC + \" has accessed \" + HTTP.REQ.URL.PATH_AND_QUERY.MARK_SAFE + \" HTTP-Body: \" + HTTP.REQ.BODY(HTTP.REQ.CONTENT_LENGTH)"

Apr  5 09:56:34  192.168.10.101 04/05/2018:09:56:34 GM
T ns_vpx_01 0-PPE-0 : default APPFW Message 1593 0 :  "Client with
 IP 192.168.10.10 has accessed /cookie.asp HTTP-Body: name=Johanne
s;password=You'll_never_guess"

So we can nor clearly see all HTTP post data.

Logging both, full HTML-Headers and Post/Put data

add audit messageaction log_URL WARNING "\"Client with IP \" + CLIENT.IP.SRC + \" has accessed \" + HTTP.REQ.URL.PATH_AND_QUERY.MARK_SAFE + \" || Headers \" + HTTP.REQ.FULL_HEADER + \" || HTTP-Body: \" + HTTP.REQ.BODY(HTTP.REQ.CONTENT_LENGTH)"

I’d be happy to see some notes, remarks, feel free to tell me if you like my blog article, or if you don’t agree. Comments are a thank you to the blogger! Feel free to share this blog!

Let’s talk about a WAF, a Web Application Firewall on a Citrix NetScaler. What’s to be concerned off? Is it worth while considering a NetScaler to be your WAF?

I do work for several companies, including Citrix Consulting Services. Recently I worked on some Web Application Firewall projects, so I have some experience on it.

Usual concerns

will a Citrix NetScaler be really safe WAF?
How well does it scale?
Is it easy to implement?

1: How safe is a Citrix NetScaler Web Application Firewall (WAF)

As far as I know one of the biggest websites world wide is using NetScaler WAF. They are storing hundreds of millions of customer records (including billing and credit card information). As far as we know, they never got hacked so far. Their website seems to be safe. Same about a huge NGO with is political exposed very much. They are attractive to hackers from all over the world. They also still are not known to be hacked during the last some years.

I also know of banks trusting in Citrix NetScaler Web Application Firewall (WAF), they are successful.

Nicht zuletzt gibt es mehrere Zertifizierungen, die unsere NetScaler WAF derzeit durchführt: NSS-Labs empfiehlt NetScaler WAF genauso wie ICSA-Labs.

So I consider NetScaler WAF to be secure, if it’s set up correctly.

2: How well will it scale?

Well, that’s a problem indeed. And it depends (every architect’s standard answer to each and every question). To be honest, a WAF is overhead. Huge overhead. Every single packet, flowing in (and flowing out in many cases), has to get inspected. So WAF has to be considdered a burden for the CPU of a NetScaler.

Like every feature on NetScaler, WAF is not multithreaded, meaning: Every Packet Processing Engine (PPE) is processing a TCP packet flow, independently from all other PPEs. And does everything on it’s own, not calling a singe operating system funchtion. There is just one thread, picking up the packet, doing all policies (responder, rewriting, WAF, …) and forwarding it to it’s destination. This feature is great, as it makes a NetScaler a very stable box, but it may cause some CPU-cores to be overloaded for a relatively long time. Overload on a CPU-core means latency for a user. To avoid overload, average CPU has to be under 75-80 %.

So, if you go NetScaler WAF, you’ll have to be able to scale out. Scaling out may mean, to upgrade your box with bigger licenses. Bigger licenses may mean: unlocking CPU cores and RAM. But it may also mean: Add more NetScaler boxes. Adding more NetScaler boxes seems to mean: Cluster. But as I personally would avoid a Cluster, I’d rather load-balance NetScalers. So a typical WAF-deployment would look like this:

A pair of NetScaler VPX (or SDX, MPX) boxes (in HA, tier 1) load balancing NetScaler MPX boxes (tier 2). These MPX boxes do both SSL-on- / off-loading and WAF. HA is not needed.

The tier 1 HA pair is just a load balancing vServer of type SSL bridge, using SSL session ID for persistence. The vServer is in source IP mode (SIP-mode), to preserve IP addresses for tier 2

This setup scales up easily and – at the same time – avoids cluster typical problems like features not being available or being hardly tested. We don’t use HA as load balancing in tier 1 takes care of high availability. We may scale out easily, if performance is insufficuient. We may even upgrade these boxes independent from each other. Of course we need one additional box, in case of one of these boxes is going down (n+1 principle)

3: Is it easy to implement?

Citrix sales (and some consultants) tend to answer this questions with a clear and simple yes, as NetScaler comes with an integrated learning feature doing all stuff for you. That’s really great!

Me, being rather a consultant than a sales guy – however – would rather say no. Being a customer I’d absolutely like to have a consultant with long-term experience working on this project.

One of the biggest problems in security is a false feeling of safety. A WAF will always give you a sound feeling of security. Feeling secure, makes people careless. But bever forget: What if there is something wrong about your WAF?

You got it: Your feeling of security may be as same as wrong, as your WAF setup.

All of us are always a bit shy looking at Citrix Synergy: What will it bring? Well, this time, Citrix comes up with brand new names for all products. It’s the first time Citrix is renaming the product. Until now the mane resisted all renaming by marketing departement.

Citrix aquired NetScaler back in 2005. The original company “NetScaler” was founded by Michel K Susai in 1997.

It’s nothing less but the biggest name change of all time in Citrix history.

I currently don’t know the upcoming name for Citrix Receiver (was Receiver, Plugin for Hosted Applications, Plugin, Citrix Client and many, many more)

So which names will we have to deal with in future?

There is a white paper about name changes. All products will get renamed. There are three fields of products:

Citrix Workspace
- Citrix Content Collaboration (was ShareFile)
- Citrix Entpoint Management (was XenMobile)
- Citrix Secure Browsing (was XenApp secure Browser)
- Citrix Hypervisor (was XenServer)
- Citrix App Layering
- Citrix Virtual Apps (was XenApp)
- Citrix Virtual Desktops (was XenDesktop)
- Citrix Endpoint Management
  - Citrix Secure Mail
  - Citrix Secure Web
Citrix Networking
- Citrix ADC (was NetScaler ADC – “NetScaler“)
- Citrix SD-WAN (was WanScaeler, Cloudbridge, SD-WAN, NetScaler SD-WAN)
- Citrix Web App Firewall (was NetScaler App Firewall, NetScaler App Security)
- Citrix Gateway (was NetScaler Unified Gateway)
- Citrix Application Delivery Management (was NetScaler MAS, MAS, NMAS)
- Citrix Secure Web Gateway (was NetScaler Secure Web Gateway)
- Citrix Intelligent Traffic Management (Cedexis Plattform)
Citrix Analytics
- Citrix Analytics for Networking
- Citrix Analytics for Workspaces

So we see: Everything is less difficult than it was before. We will clearly understand each other during our conversations through out the next some years. We have to be happy!

Last update: May 31^th, 2018

tested using firmware 11.1

If you read about slowloris, you always read about NetScaler doing a great job. Tests in our lab environment show: NetScaler will successfully block these attacks. And there is hardly anything we have to do about it: it’s built into the system. Great news indeed!

The only thing we have to do is reduce client idle timeout to a lower value (default 180 seconds). I’d propose something below 20 seconds.

Unfortunately NetScaler will not log these attacks. WTF? Yes, that’s true. NetScaler won’t log a blocked slowloris attack. I recently set up a Citrix NetScaler WAF in the lab environment of a big bank, and they wanted me to log these Slowloris. I understand very well, why they want to log these. However we can’t.

This blog article could be over right now, but I did some research. Of course we have counters for this kind of attack. And we expose many of them to NITRO. So it could be possible, to do logging based on NITRO calls. Lets dig a little bit into NITRO. To do so I open my browser and surf to https://SNIP/nitro/v1/stat/protocolhttp (SNIP is the subnet address of my NetScaler). After logging on it returns a JSON list of counters. Most of them are of no importance for us, but I’m interested in httperrincompleteheaders.

That’s a good point to start from!

How to log httperrincompleteheaders on Citrix NetScaler ADC

So my approach would be like this:

do a query to NITRO API and store the results in a NetScaler variable.
log, if this counter increases.

There are some obstacles to overcome.

First of all, we need to find a way to query NetScaler NITRO from within a NetScaler. And here it is:

define a HTTP Callout.

We may use an HTTP callout to query Nitro. That’s quite simple to do, it’s just an http get. Problem here is authentication. I found a solution, unfortunately it’s not a very elegant one.

add policy httpCallout query_incomplete_header_neu_num -IPAddress 192.168.30.110 -port 80 -returnType NUM -urlStemExpr "\"/nitro/v1/stat/protocolhttp\"" -headers X-NITRO-USER("nsroot") X-NITRO-PASS("nsroot") Accept("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") host("192.168.30.110") -scheme http -resultExpr "HTTP.RES.BODY(2048).XPATH_JSON(xp%/protocolhttp/httptotrequests%).TYPECAST_NUM_AT"
In GUI, callouts are located in AppExpert

What’s in this policy?

-IPAddress: This is the IP address we’re actually calling. It has to be a SNIP, HTTP access enabled.
-Port: The port, usually 80 (SSL doesn’t make any sense for NetScaler internal communication; it’s waste of ressources)
-returnType: The type of data this callout has to return. Possible values: TEXT, NUM or BOOL. We’re interested in numbers.
-urlStemExpr: The URL we call (/nitro/v1/stat/protocolhttp)
-headers: headers we have to set. They are nescessary, see below
- X-NITRO-USER(“nsroot”) username
- X-NITRO-PASS(“nsroot”) password for this user
- Accept(“text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8”) the encoding, our policy can understand
- host(“192.168.30.110”) server’s hostname. Don’t skip this one!!!
-scheme: htt/https
-resultExprthe: data we’re interested in

This query should return a number: The number of requests containing incomplete headers. So we have to take a closer look at the http response. In RAW format it would look like this:

{ "errorcode": 0, "message": "Done", "severity": "NONE", "protocolhttp": { "spdytotst
reams": "0", "spdystreamsrate": 0, "httptotrequests": "13211", "httprequestsrate": 3,
 "httptotresponses": "13209", "httpresponsesrate": 3, "httptotrxrequestbytes": "52418
9", "httprxrequestbytesrate": 84, "httptotrxresponsebytes": "2498358", "httprxrespons
ebytesrate": 1004, "httptotgets": "1244", "httpgetsrate": 0, "httptotposts": "12", "h
ttppostsrate": 0, "httptotothers": "11955", "httpothersrate": 2, "httptot10requests":
 "0", "http10requestsrate": 0, "httptot11requests": "2102", "http11requestsrate": 0, 
"httptotclenrequests": "20", "httpclenrequestsrate": 0, "httptotchunkedrequests": "0"
, "httpchunkedrequestsrate": 0, "httptottxrequestbytes": "0", "httptxrequestbytesrate
": 0, "httptot10responses": "4166", "http10responsesrate": 1, "httptot11responses": "
9043", "http11responsesrate": 2, "httptotclenresponses": "8185", "httpclenresponsesra
te": 2, "httptotchunkedresponses": "1", "httpchunkedresponsesrate": 0, "httperrnoreus
emultipart": "0", "httperrnoreusemultipartrate": 0, "httptotnoclenchunkresponses": "4
166", "httpnoclenchunkresponsesrate": 1, "httptottxresponsebytes": "0", "httptxrespon
sebytesrate": 0, "httperrincompleteheaders": "0", "httperrincompleterequests": "0", "
httperrincompleterequestsrate": 0, "httperrincompleteresponses": "0", "httperrincompl
eteresponsesrate": 0, "httperrserverbusy": "4", "httperrserverbusyrate": 0, "httperrl
argecontent": "0", "httperrlargechunk": "0", "httperrlargectlen": "0", "spdyv2totstre
ams": "0", "spdyv2streamsrate": 0, "spdyv3totstreams": "0", "spdyv3streamsrate": 0 } 
}

We have to extract data behind “httperrincompleteheaders”: and before the next “, and convert this to a number. A possible string would be:

HTTP.RES.BODY(20480).AFTER_STR("httperrincompleteheaders\": \"").BEFORE_STR("\"").TYPECAST_NUM_AT

But our experience shows: This is inefficient! So we need to find a better way: NetScaler understands JSON. We will rather use this one:

HTTP.RES.BODY(2048).XPATH_JSON(xp%/protocolhttp/httptotrequests%).TYPECAST_NUM_AT

HTTP Callout is finnished!

Next we need a NetScaler variable.

Creating a Variable to store our data in

Citrix NetScalers have built in functionality called variables. They can be created either from GUI or command line.

add ns variable HTTP_INCOMPLETE_HEADERS -type ulong

In GUI, variables are located in AppExpert
Assigning data to this variable

It’s surprisingly difficult to assign data to a variable! A simple $variable=7 won’t do the job. Instead we have to create an assignment. Luckily assignments are – at the same time – policy expressions for responder and rewriting policies. So we’ll have to create an assignment for this variable:

add ns assignment set_incomplete_header -variable "$HTTP_INCOMPLETE_HEADERS" -set "SYS.HTTP_CALLOUT(query_number_req)"

In GUI assignments are located in Appexpert

This assignment will replace the value in my variable defined above with the return value of the callout created first.

The trigger

We could check for slowloris when ever a request comes in, however this may be way too often, as NITRO calls are of some overhead for our system. So I created a trigger. I won’t do a step by step instruction for this here, I will just give you an idea how it works:

I create a Citrix NetScaler service of type ANY, pointing “somewhere”. (it does not matter as we won’t use this service at all). I assign a health monitor of type HTTP to it, specifying the IP the HTTP vServer (CS or LB does not matter). This health monitor periodically sends HTTP requests there, they are easy to identify and we are free to define the frequency we like. This health monitor is our trigger.

The policies we will use

Next to do is creating policies. We may use responder or rewriting policies (I used responder policies, but it does not matter). This policy will do the callout and compare it to the stored value in our Citrix NetScaler variable. We would need to bind a logging policy to it (see here). Unfortunately this is not possible. It’s a restriction in NetScaler: “Log action is not suported with assignment action“. So we have to do two identic policies, one doing nothing but logging, the other incrementing the counter.

Logging policy

add responder policy log_IncompleteRequests "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT " NOOP -logAction Log_new_number rule "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT "
This Citrix NetScaler responder policy checks if client’s IP is the SNIP and the incomplete header counter increased. If so the responder policy actually does nothing (NOOP), but it logs.

incrementing policy

add responder policy count_IncompleteRequests "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT" set_incomplete_header rule "CLIENT.IP.SRC.EQ(192.168.30.110) && $HTTP_INCOMPLETE_HEADERS.EQ(SYS.HTTP_CALLOUT(query_incomplete_header)).NOT"

This policy can’t log (see above), however set the counter. Policy expression is the same as above.

Binding the policies

The last task is binding these policies. Simple like that. Bind the logging policy first with “goto next”, bind the incrementing policy (with “goto next”, if you still have more policies to check for).

I hope you liked my tricks. I’d be happy to hear your thoughts on this, just drop some words in the comment box to let me know about your thoughts. Feel free to link to my page when ever you like.

Johannes

How to protect your cookies using Citrix NetScaler

I recently did a web application firewall (WAF) project for a big company owning and hosting hundreds of websites. They did several penetration tests. One of them focussed on cookies. Citrix NetScaler did a great job protecting cookies, cookie tampering was impossible, but they had been able to steal cookies.

Stealing cookies is not that easy, especially if a website is well protected and XSS (cross site scripting) is blocked it is near to impossible. It would be easy stealing cookies using XSS: post document.cookies to a website of attacker’s choice, that’s it. But it’s also easy in a lab environment: Just pick them up, copy them to an other box and feed them back in. This is, what they did. And NetScaler failed (actually no surprise to me).

How does Cookie protection work?

We have several methods. In most cases, we need sessionization. Citrix NetScaler will store information about cookies (hashes) and will drop tampered cookies. We also could encrypt cookies. This would make cookie tampering hard for an attacker, as he has to guess (brut force) the key. We could cache (session-) cookies on Citrix NetScaler. In addition we could mark cookies HttpOnly or Secure. All these methods target cookie tampering, not cookie stealing.

A Cookie-Monster stealing cookies? What can we do?

Sure, NetScaler can’t do anything. NetScaler will add a session cookie to the existing cookies (change the default name from citrix_ns_id to something else, an attacker does not nescessarily need to know about Citrix NetScaler ADC protecting our website). A NetScaler will make 100% sure none can tamper cookies. But it will definitely allow stealing of cookies.

The solution

We need to put additional cookies into the data stream identifying the client. We need to find something specific.

The IP address

Unfortunately the IP address is not half as specific as people think. Mobile phone networks are typically NATing their users to the internet. It’s rather likely for us to share the same IP if we share the same mobile phone provider. An other drawback of the IP is: it may change while users move from one network to an other, let’s say move from home (there is Wi-Fi connected to DSL) to the street (using LTE coming from a totally different provider).

Even though there are some draw-backs: The IP address is a good thing to use. It would be easy for an attacker to fake it, if he only knew. So we have to keep it a secret and encrypt it.

User-Agent

There is a wide variation in User-Agent strings being sent from client to server. Clients differ in language, browser type and version, operating system and many more. It would be easy for an attacker to fake this string, if he only knew. So we have to keep it a secret and encrypt it.

More things?

sure, what ever you have, use it. My customer uses several things in parallel.

Implementing the solution

General thoughts?

We will add cookies, so we need names for these. I’m a great fan of cheating. The more you cheat the less likely an attacker would understand your setup. So I’ll call my example cookies Tmp-Data and Default-Printer.

Creating these cookies

I create two rewriting policies in response direction:

Citrix NetScaler Policy Actions:

add rewrite action rw_act_setCookie_IP insert_http_header Set-Cookie "\"Default-Printer=\" + CLIENT.IP.SRC.TYPECAST_TEXT_T.ENCRYPT"

This policy action extracts client’s IP from HTTP request, converts it into text, and encrypts it.

add rewrite action rw_act_setCookie_User-Agent insert_http_header Set-Cookie "\"Tmp-Data=\" + HTTP.REQ.HEADER(\"User-Agent\").ENCRYPT"

This policy action extracts the User-Agent string from the original HTTP request and encrypts it.

Citrix NetScaler Policies

add rewrite policy rw_pol_setCookie_IP true rw_act_setCookie_IP
add rewrite policy rw_pol_setCookie_User-Agent true rw_act_setCookie_User-Agent

We are using true as a policy condition because we want this to be in done every request.

Binding these Policies

Just bind these policies to a vServer of choice. No matter if it’s a cs vServer, or a lb vServer.

Checking incoming traffic

General thoughts?

Citrix NetScaler WAF will protect all cookies, including the ones we created, from being tampered. So we don’t have to worry about these cookies being tampered. But what if a request comes in, not containing these cookies? That’s more than possible: Every user session starts with a request not containing cookies. So we must allow requests without pre-existing cookies, or we could strip all cookies from an initial request. We must not allow requests containing all application cookies, but not our ones. If you are dealing with an existing website: There may already be persistent cookies stored on a client device. Persistent cookies usually don’t contain sensitive information. They store things like settings rather than the user’s identity.

What kind of policy will we use to check for cookies? There are two possible answer. My first one would be: Responder. Just drop requests, or redirect them to a safe location. Drawback of this is, it’s not all done at the same place. And responder policies usually don’t log (we may force them to log). So why not use Application Firewall policies instead? There are three built in profiles: APPFW_BYPASS, APPFW_RESET, APPFW_DROP, APPFW_BLOCK (for details see here). I will use APPFW_DROP to drop silently.

The Citrix NetScaler Application Firewall policy:

the policy dropping requests containing stolen cookies

add appfw policy appfw_pol_drop_wrongcookie "HTTP.REQ.COOKIE.VALUE(\"Default-Printer\").EQ(\"\").NOT && (HTTP.REQ.COOKIE.VALUE(\"Default-Printer\").DECRYPT.EQ(CLIENT.IP.SRC.TYPECAST_TEXT_T).NOT || HTTP.REQ.COOKIE.VALUE(\"Tmp-Data\").DECRYPT.EQ(HTTP.REQ.HEADER(\"User-Agent\")).NOT)" APPFW_DROP

This policy will be triggered if cookie “Default-Printer” is not empty and either “Default-printer” does not match the IP or “Temp-Data” does not match User-Agent.

the policy dropping requests with missing cookies

add appfw policy appfw_pol_drop_missingcookie "(HTTP.REQ.COOKIE.VALUE(\"Default-Printe\").EQ(\"\") || HTTP.REQ.COOKIE.VALUE(\"Default-Printe\").EQ(\"\")) && (HTTP.REQ.COOKIE.VALUE(\"<the session cookie of your application goes here>\").EQ(\"\").NOT" APPFW_DROP

This policy drops requests not containing “Default-Printer” cookie or “Temp-Data” cooke and at the same time contains your application’s session cookie.

I hope you liked my tricks. I’d be happy to hear your thoughts on this, just drop some words in the comment box to let me know about your thoughts. Feel free to link to my page when ever you like.

Johannes

Recently I had several requests related to NITRO. NITRO is Citrix NetScaler’s API. Any device may communicate to a NetScaler using NITRO. Even a browser! Citrix exposes several settings and counters and even allows changes. NITRO is the central source for scripting NetScalers.

I, being rather an administrator than a programmer, am not that much interested in using NITRO with C++/C#, Java, …, instead I have an administrator’s view on it. My first steps with NITRO had been around retrieving and logging counters. I wrote a blog about logging slowloris attacks. This was NITRO calls from within a NetScaler.

This one is about NITRO in general.

NITRO in a nutshell

Citrix NetScaler’s NITRO is an API. Even though it’s partly possible to be useed it with standard HTTP, it’s based on REST and JSON. That means: Requests and responses follow a structured, XML like, format. That’s fine from one side, but turned out to be a bit challenging for me.

What does it expose?

There is a good SDK available. NITRO exposes:

Configurations. http(s)://<netscaler-ip-address>/nitro/v1/config/<resource-type>
Statistics. http(s)://<netscaler-ip-address>/nitro/v1/stat/<resource-type>

So it’s possible to get read/write access to Citrix NetScaler’s configuration as well as read access to NetScaler statistics!

The Nike^® way: Let’s do IT (or: a first try)

My Citrix NetScaler ADC Testsystem’s NSIP is 192.168.30.100. So I surf to http://192.168.30.100/nitro/v1/stat/.

I get promptet for username and password. An easy one for all of you as my testsystem uses nsroot/nsroot . If you want to logon using a script you would follow these guide lines in Citrix NetScaler NITRO SDK.

Immediately after sending my credentials our first success:

Or, if I select RAW data in Firefox:

This is a complete list of objects Citrix NetScaler NITRO exposes counters for.

Next step: Retrieve counters for a specific class:

Let’s say: I’m currently mainly interested in WAF, so the counters I’ll try to extract is appfw. My URL would be http://192.168.30.100/nitro/v1/stat/appfw. And here they are! Again, this is a great JSON file and can be processed easily. This can even be done from inside Citrix NetScaler using http callout.

Let’s go into the config side!

So my first guess would be: surf to http://192.168.30.100/nitro/v1/config/. And it works!

View Citrix NetScaler Firmware version:

http://192.168.30.100/nitro/v1/config/nsversion

Get Citrix NetScaler basic configuration

http://192.168.30.100/nitro/v1/config/nsconfig

This is some information like NS-IP, cookie version, HA status, time zone, last config update / save, system time and more.

I see, I would have to save my “valuable” configuration. This would be possible using http://192.168.30.100/nitro/v1/config/nsconfig?action=save, however it does not work. Why? Because I send a http get instead of a put. See here for details. I could use fiddler to change my get into a put (putting the right content into the body), but that’s way to complex for me to do, so I left my configuration unsaved.

(just kidding, of course I did! It’s not that easy, but in the end I made it, that’s why I’m still here, some minutes before midnight)

Nitro: Which lb-vServers are on my Citrix NetScaler?

http://192.168.30.100/nitro/v1/config/lbvserver (http://192.168.30.100/nitro/v1/config/csvserver)

(I have collapsed several servers, so you can see there are several of them)

This will retrieve a list of all vServers out of Citrix NetScaler using a Nitro call (I have collapsed several servers, so you can see there are several of them)

http://192.168.30.100/nitro/v1/config/lbvserver/lb_vsrv_colors

Nitro: give me a list of SSL certificates?

http://192.168.30.100/nitro/v1/config/sslcertkey

A complete list. Simmilar to lb-vServer a specific certificate would be: http://192.168.30.100/nitro/v1/config/sslcertkey/ns-server-certificate.

There is much more to query for. NetScaler GUI constantly does NITRO queries to Citrix NetScaler. Using a proxy like fiddler could help you seeing these calls.

I hope, you liked my blog. It’s fun to play around, dig into NetScaler and I really had fun to see my NetScaler from a programmer’s perspective. I’d be more than just happy to see comments on this article. Your comments keep me writing more blogs …

This will be my shortest blog about the subject ever. Citrix finally did it! They created a “Built-in secure front-end SSL profile” called ns_default_ssl_profile_secure_frontend.

What do you need to do? Just bind this profile to your vServer. That’s it. Isn’t it great? Compare this to my last blog about the subject!

That’s the way it should be done. Hey, Citrix ADC guys, you’re doing great!