Using NetScaler and MaxMind API to insert GeoIP headers to backend

Posted in : NetScaler Av Simon Gottschlag Översätt med Google ⟶

8 years ago

There are some really great blog posts out there regarding GeoIP and NetScaler (Neil Spellings for example). In this case, I need to insert City, Country and Continent into headers to the backend.
My first thought was to use the internal GeoIP (location) database in NetScaler, but it isn’t supported yet to print information from it (send me an email if you want the RFE ID). The next step was to start working with the latest MaxMind API which supports all three data sets. Unfortunately, there’s a bug in the XPATH_JSON parser in NetScaler that can’t handle 3-byte UTF-8 characters (Japanese characters for example). This is something being fixed and will be released later (same here, send me an email if you want the BUG ID). Since NetScaler is the best product out there, and the NetScaler team is the greatest ever, they made a policy exentsion to fix the issue until it’s fixed in the code.
Until the bug is fixed, the following policy extension is needed:

-- This NetScaler Policy Extension function removes all 3-byte UTF-8 characters
-- from its input. It is intended to be used with XPATH_JSON(), to compensate
-- for a bug in the JSON parser that does not correctly recognize 3-byte UTF-8
-- characters, which are used for Asian character sets like Katakana and CJK
-- Unified. This function can be used if the 3-byte characters can be ignored
-- in a policy expressions. For example:
--
-- HTTP.RES.BODY(1000).FILTER_UTF8.XPAHT_JSON(xp%/city/names/en%)
--
-- where the selected values will not contain 3-byte UTF-8 characters.
function NSTEXT:FILTER_UTF8(): NSTEXT
    -- Use an array to accumulate the output characters, then concatenate
    -- the characters at the end to make the output string. This is more
    -- efficient that concatenating each character to the output string.
    local output = {}
    local len = string.len(self)
    local i = 1  -- index for the input string (self)
    local j = 1  -- index for the output array
    local b      -- first byte of a character
    while i <= len do
       b = string.byte(self, i)
       if b <= 0x7F then
           -- 1-byte UTF-8 character 0xxxxxxx
           -- Copy the byte.
           output[j] = string.char(b)
           i = i + 1
           j = j + 1
       elseif bit32.rshift(b, 5) == 0x06 then
           -- 2-byte UTF-8 character 110xxxxx 10xxxxxx
           if i + 1 > len then
               break -- missing a byte
           end
           -- Copy the two butes.
           output[j] = string.char(b)
           i = i + 1
           j = j + 1
           output[j] = string.char(string.byte(self, i))
           i = i + 1
           j = j + 1
       elseif bit32.rshift(b, 4) == 0x0E then
           -- 3-byte UTF-8 character 1110xxxx 10xxxxxxi 10xxxxxx
           if i + 2 > len then
               break -- missing a byte or two
           end
           -- Don't copy the bytes.
           i = i + 3
       elseif bit32.rshift(b, 3) == 0x1D then
           -- 4-byte UTF-8 character 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
           if i + 3 > len then
               break -- missing a byte or three
           end
           -- Copy the four bytes.
           output[j] = string.char(b)
           i = i + 1
           j = j + 1
           output[j] = string.char(string.byte(self, i))
           i = i + 1
           j = j + 1
           output[j] = string.char(string.byte(self, i))
           i = i + 1
           j = j + 1
           output[j] = string.char(string.byte(self, i))
           i = i + 1
           j = j + 1
       else
           break -- invalid initial UTF-8 byte
       end
    end
    return table.concat(output)
end

Now, let’s take a look at how we can utilize MaxMinds API to create these headers.
First off, create a load balancing vserver for MaxMind:

add server SRV-MAXMIND_GEOIP geoip.maxmind.com -domainResolveRetry 10
add lb monitor MON-MAXMIND_GEOIP HTTP -respCode 301 -httpRequest "HEAD /" -customHeaders "Host:geoip.maxmind.com\r\n" -LRTM DISABLED -interval 5 MIN -secure YES
add service SVC-MAXMIND_GEOIP SRV-MAXMIND_GEOIP SSL 443 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -sp OFF -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES
bind service SVC-MAXMIND_GEOIP -monitorName MON-MAXMIND_GEOIP
add lb vserver LB-MAXMIND_GEOIP HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180
bind lb vserver LB-MAXMIND_GEOIP SVC-MAXMIND_GEOIP

Next step, create the callout (and use the policy extension until the bug is fixed):

add policy httpCallout httpcallout_maxmind_geoip -vServer LB-MAXMIND_GEOIP -returnType TEXT -hostExpr "\"geoip.maxmind.com\"" -urlStemExpr "\"/geoip/v2.1/insights/\"+CLIENT.IP.SRC" -headers Authorization("Basic ****") Accept("application/json") -scheme http -resultExpr "HTTP.RES.BODY(2048).FILTER_UTF8.XPATH_JSON(xp%/city/names/en%) + \";\" + HTTP.RES.BODY(2048).FILTER_UTF8.XPATH_JSON(xp%/country/names/en%) + \";\" + HTTP.RES.BODY(2048).FILTER_UTF8.XPATH_JSON(xp%/continent/names/en%)" -cacheForSecs 3600

Please note: Change the authorization header to your own.
This will now cache the following (if IC is licensed and enabled): ”City;Country;Continent”
Now, let’s create the rewrites and add them to a vserver:

add rewrite action RWA-REQ-ADD_GEOIP_CITY insert_http_header X-GeoIP-City "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).TYPECAST_LIST_T(\';\').GET(0)"
add rewrite action RWA-REQ-ADD_GEOIP_COUNTRY insert_http_header X-GeoIP-Country "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).TYPECAST_LIST_T(\';\').GET(1)"
add rewrite action RWA-REQ-ADD_GEOIP_CONTINENT insert_http_header X-GeoIP-Continent "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).TYPECAST_LIST_T(\';\').GET(2)"
add rewrite policy RWP-REQ-ADD_GEOIP_CITY true RWA-REQ-ADD_GEOIP_CITY
add rewrite policy RWP-REQ-ADD_GEOIP_COUNTRY true RWA-REQ-ADD_GEOIP_COUNTRY
add rewrite policy RWP-REQ-ADD_GEOIP_CONTINENT true RWA-REQ-ADD_GEOIP_CONTINENT
add rewrite policy RWP-REQ-ADD_GEOIP-NOOP "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).STRIP_CHARS(\";\").LENGTH.GE(1)" NOREWRITE
add rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS http_req
bind rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS RWP-REQ-ADD_GEOIP_CITY 100 NEXT
bind rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS RWP-REQ-ADD_GEOIP_COUNTRY 110 NEXT
bind rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS RWP-REQ-ADD_GEOIP_CONTINENT 120 END
bind cs vserver <CS vServer Name> -policyName RWP-REQ-ADD_GEOIP-NOOP -priority 100 -gotoPriorityExpression NEXT -type REQUEST -invoke policylabel RWPL-REQ-ADD_GEOIP_HEADERS

See below for how it’s cached:

> sh cache object -locator 0x00000007b16e00000020
        Integrated cache object statistics:
        Locator: 7b16e00000020
        Response size: 32 bytes
        Response header size: 0 bytes
        Response status code: 200
        ETag: NONE
        Last-Modified: NONE
        Cache-control: NONE
        Date: Sat, 04 Mar 2017 08:36:07 GMT
        Contentgroup: calloutContentGroup
        Complex match: NO
        Host: geoip.maxmind.com
        Host port: 443
        URL: /geoip/v2.1/insights/<Source IP>
        Destination IP: 0.0.0.0
        Destination port: 57810
        Request time: 2764 secs ago
        Response time: started arriving 2764 secs ago
        Age: 2767 secs
        Expiry: 836 secs left to expiry
        Flushed: NO
        Prefetch: 611 secs left to prefetch
        Current readers: 0
        Current misses: 0
        Hits: 219
        Misses: 1
        Compression Format: NONE
        HTTP version in response: 1.1
        Weak ETag present in response: NO
        Negative marker cell: NO
        Auto poll every time: NO
        NetScaler ETag inserted in response: NO
        Full response present in cache: YES
        Response data present in Secondary: NO
        Destination IP verified by DNS: NO
        Stored through a cache forward proxy: NO
        Delta basefile: NO
        Waiting for minhits: NO
        Minhit count: 0
        App Firewall MetaData Exists: NO
        HTTP request method: GET
        Stored by policy: NONE
        HTTP callout cell: YES
        HTTP callout name: httpcallout_maxmind_geoip
        HTTP callout type: TEXT
        HTTP callout response: Gothenburg;Sweden;Europe
         Done

I just tried it out with NGINX as backend with the following configuration:

    location /raw/123 {
     default_type application/json;
     return 200 '[{"City":"$http_x_geoip_city","Country":"$http_x_geoip_country","Continent":"$http_x_geoip_continent"}]';
    }

And we get a nice and clean response:

[{
	"City": "Gothenburg",
	"Country": "Sweden",
	"Continent": "Europe"
}]

Feel free to leave a comment if you have issues with anything or if you know better ways of doing it!

Tags : GeoIP, Lua, NetScaler, Policy Extension