Posted in : NetScaler Av Simon Gottschlag Översätt med Google ⟶
7 years ago
There are some really great blog posts out there regarding GeoIP and NetScaler (Neil Spellings for example). In this case, I need to insert City, Country and Continent into headers to the backend.
My first thought was to use the internal GeoIP (location) database in NetScaler, but it isn’t supported yet to print information from it (send me an email if you want the RFE ID). The next step was to start working with the latest MaxMind API which supports all three data sets. Unfortunately, there’s a bug in the XPATH_JSON parser in NetScaler that can’t handle 3-byte UTF-8 characters (Japanese characters for example). This is something being fixed and will be released later (same here, send me an email if you want the BUG ID). Since NetScaler is the best product out there, and the NetScaler team is the greatest ever, they made a policy exentsion to fix the issue until it’s fixed in the code.
Until the bug is fixed, the following policy extension is needed:
-- This NetScaler Policy Extension function removes all 3-byte UTF-8 characters
-- from its input. It is intended to be used with XPATH_JSON(), to compensate
-- for a bug in the JSON parser that does not correctly recognize 3-byte UTF-8
-- characters, which are used for Asian character sets like Katakana and CJK
-- Unified. This function can be used if the 3-byte characters can be ignored
-- in a policy expressions. For example:
--
-- HTTP.RES.BODY(1000).FILTER_UTF8.XPAHT_JSON(xp%/city/names/en%)
--
-- where the selected values will not contain 3-byte UTF-8 characters.
function NSTEXT:FILTER_UTF8(): NSTEXT
-- Use an array to accumulate the output characters, then concatenate
-- the characters at the end to make the output string. This is more
-- efficient that concatenating each character to the output string.
local output = {}
local len = string.len(self)
local i = 1 -- index for the input string (self)
local j = 1 -- index for the output array
local b -- first byte of a character
while i <= len do
b = string.byte(self, i)
if b <= 0x7F then
-- 1-byte UTF-8 character 0xxxxxxx
-- Copy the byte.
output[j] = string.char(b)
i = i + 1
j = j + 1
elseif bit32.rshift(b, 5) == 0x06 then
-- 2-byte UTF-8 character 110xxxxx 10xxxxxx
if i + 1 > len then
break -- missing a byte
end
-- Copy the two butes.
output[j] = string.char(b)
i = i + 1
j = j + 1
output[j] = string.char(string.byte(self, i))
i = i + 1
j = j + 1
elseif bit32.rshift(b, 4) == 0x0E then
-- 3-byte UTF-8 character 1110xxxx 10xxxxxxi 10xxxxxx
if i + 2 > len then
break -- missing a byte or two
end
-- Don't copy the bytes.
i = i + 3
elseif bit32.rshift(b, 3) == 0x1D then
-- 4-byte UTF-8 character 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
if i + 3 > len then
break -- missing a byte or three
end
-- Copy the four bytes.
output[j] = string.char(b)
i = i + 1
j = j + 1
output[j] = string.char(string.byte(self, i))
i = i + 1
j = j + 1
output[j] = string.char(string.byte(self, i))
i = i + 1
j = j + 1
output[j] = string.char(string.byte(self, i))
i = i + 1
j = j + 1
else
break -- invalid initial UTF-8 byte
end
end
return table.concat(output)
end
Now, let’s take a look at how we can utilize MaxMinds API to create these headers.
First off, create a load balancing vserver for MaxMind:
add server SRV-MAXMIND_GEOIP geoip.maxmind.com -domainResolveRetry 10 add lb monitor MON-MAXMIND_GEOIP HTTP -respCode 301 -httpRequest "HEAD /" -customHeaders "Host:geoip.maxmind.com\r\n" -LRTM DISABLED -interval 5 MIN -secure YES add service SVC-MAXMIND_GEOIP SRV-MAXMIND_GEOIP SSL 443 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -sp OFF -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES bind service SVC-MAXMIND_GEOIP -monitorName MON-MAXMIND_GEOIP add lb vserver LB-MAXMIND_GEOIP HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 bind lb vserver LB-MAXMIND_GEOIP SVC-MAXMIND_GEOIP
Next step, create the callout (and use the policy extension until the bug is fixed):
add policy httpCallout httpcallout_maxmind_geoip -vServer LB-MAXMIND_GEOIP -returnType TEXT -hostExpr "\"geoip.maxmind.com\"" -urlStemExpr "\"/geoip/v2.1/insights/\"+CLIENT.IP.SRC" -headers Authorization("Basic ****") Accept("application/json") -scheme http -resultExpr "HTTP.RES.BODY(2048).FILTER_UTF8.XPATH_JSON(xp%/city/names/en%) + \";\" + HTTP.RES.BODY(2048).FILTER_UTF8.XPATH_JSON(xp%/country/names/en%) + \";\" + HTTP.RES.BODY(2048).FILTER_UTF8.XPATH_JSON(xp%/continent/names/en%)" -cacheForSecs 3600
Please note: Change the authorization header to your own.
This will now cache the following (if IC is licensed and enabled): ”City;Country;Continent”
Now, let’s create the rewrites and add them to a vserver:
add rewrite action RWA-REQ-ADD_GEOIP_CITY insert_http_header X-GeoIP-City "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).TYPECAST_LIST_T(\';\').GET(0)" add rewrite action RWA-REQ-ADD_GEOIP_COUNTRY insert_http_header X-GeoIP-Country "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).TYPECAST_LIST_T(\';\').GET(1)" add rewrite action RWA-REQ-ADD_GEOIP_CONTINENT insert_http_header X-GeoIP-Continent "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).TYPECAST_LIST_T(\';\').GET(2)" add rewrite policy RWP-REQ-ADD_GEOIP_CITY true RWA-REQ-ADD_GEOIP_CITY add rewrite policy RWP-REQ-ADD_GEOIP_COUNTRY true RWA-REQ-ADD_GEOIP_COUNTRY add rewrite policy RWP-REQ-ADD_GEOIP_CONTINENT true RWA-REQ-ADD_GEOIP_CONTINENT add rewrite policy RWP-REQ-ADD_GEOIP-NOOP "SYS.HTTP_CALLOUT(httpcallout_maxmind_geoip).STRIP_CHARS(\";\").LENGTH.GE(1)" NOREWRITE add rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS http_req bind rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS RWP-REQ-ADD_GEOIP_CITY 100 NEXT bind rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS RWP-REQ-ADD_GEOIP_COUNTRY 110 NEXT bind rewrite policylabel RWPL-REQ-ADD_GEOIP_HEADERS RWP-REQ-ADD_GEOIP_CONTINENT 120 END bind cs vserver <CS vServer Name> -policyName RWP-REQ-ADD_GEOIP-NOOP -priority 100 -gotoPriorityExpression NEXT -type REQUEST -invoke policylabel RWPL-REQ-ADD_GEOIP_HEADERS
See below for how it’s cached:
> sh cache object -locator 0x00000007b16e00000020
Integrated cache object statistics:
Locator: 7b16e00000020
Response size: 32 bytes
Response header size: 0 bytes
Response status code: 200
ETag: NONE
Last-Modified: NONE
Cache-control: NONE
Date: Sat, 04 Mar 2017 08:36:07 GMT
Contentgroup: calloutContentGroup
Complex match: NO
Host: geoip.maxmind.com
Host port: 443
URL: /geoip/v2.1/insights/<Source IP>
Destination IP: 0.0.0.0
Destination port: 57810
Request time: 2764 secs ago
Response time: started arriving 2764 secs ago
Age: 2767 secs
Expiry: 836 secs left to expiry
Flushed: NO
Prefetch: 611 secs left to prefetch
Current readers: 0
Current misses: 0
Hits: 219
Misses: 1
Compression Format: NONE
HTTP version in response: 1.1
Weak ETag present in response: NO
Negative marker cell: NO
Auto poll every time: NO
NetScaler ETag inserted in response: NO
Full response present in cache: YES
Response data present in Secondary: NO
Destination IP verified by DNS: NO
Stored through a cache forward proxy: NO
Delta basefile: NO
Waiting for minhits: NO
Minhit count: 0
App Firewall MetaData Exists: NO
HTTP request method: GET
Stored by policy: NONE
HTTP callout cell: YES
HTTP callout name: httpcallout_maxmind_geoip
HTTP callout type: TEXT
HTTP callout response: Gothenburg;Sweden;Europe
Done
I just tried it out with NGINX as backend with the following configuration:
location /raw/123 {
default_type application/json;
return 200 '[{"City":"$http_x_geoip_city","Country":"$http_x_geoip_country","Continent":"$http_x_geoip_continent"}]';
}
And we get a nice and clean response:
[{
"City": "Gothenburg",
"Country": "Sweden",
"Continent": "Europe"
}]
Feel free to leave a comment if you have issues with anything or if you know better ways of doing it!
Tags : GeoIP, Lua, NetScaler, Policy Extension
Personlig rådgivning
Vi erbjuder personlig rådgivning med författaren för 1400 SEK per timme. Anmäl ditt intresse i här så återkommer vi så snart vi kan.
Add comment