This article shows how to allow blank referrers for social network crawlers when HTTP referrer protection is enabled. Social networks use different user agents to access content, so we will allow blank referrer for these user agents only. Any other user agents attempting to access CDN assets will receive a 403 Forbidden status code, as well as requests with non-listed referrers.
The HTTP referer (originally a misspelling of referrer) is an HTTP header field that identifies the address of the webpage (i.e. the URI or IRI) that linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated. -Wikipedia
Skill Level: Intermediate
Time Required: 10 minutes
Tools Required: EdgeRules
Edge Rule Setup
- Log into the StackPath Control Panel
- Select Sites and then select Manage next to the site you wish to create the edge rule with
- Select CDN and choose EdgeRules
- Select Add New Rule and then + New Rule
- Enter in the following directives:
- Add directive WHITELISTED REFERRERS and set it to www.yourdomain.com
- Add directive SET VARIABLE and set it to $flag 1
- Add directive SET VARIABLE and set it to $flag 0 with the following condition:
- $http_user_agent as RegEx(Case Insensitive) to value of relevant social network user agent
- Add directive SET VARIABLE and set it to $invalid_referer $flag with the following condition:
- $http_referer as RegEx(Case Insensitive) yo ^$
- Add directive STATUS CODE and set it to 403 with the following condition:
- $invalid_referer as (Case Insensitive) to 1
Testing
To test this Edge Rule please ues the following two Curl examples. The results should match the examples below.
Curl example without referrer:
curl -I http://site.company.stackpathdns.com/image.png
HTTP/1.1 403 Forbidden
Date: Tue, 24 Jan 2017 15:55:08 GMT
Content-Type: text/html
Content-Length: 162
Server: NetDNA-cache/2.2
X-Cache: MISS
Connection: keep-alive
Curl example with valid Google user agent:
curl -I http://site.company.stackpathdns.com/image.png --referer http://www.b92.net -A "User-Agent: Google"
HTTP/1.1 200 OK
Date: Tue, 24 Jan 2017 15:55:45 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 27556
Set-Cookie: SPSI=6d9b4bed11b5ba69b8f7f8510c23612b; Path=/
Set-Cookie: adOtr=obsvl; Expires=Sat, 21-Jan-17 15:55:45 GMT; Path=/
Set-Cookie: UTGv2=D-h4e259c9224bd7c18e3f1735ce58aef01378; Expires=Wed, 24-Jan-18 15:55:45 GMT; Path=/
cache-control: no-store, no-cache, max-age=0, must-revalidate, private, max-stale=0, post-check=0, pre-check=0
Server: NetDNA-cache/2.2
X-Cache: MISS
Connection: keep-alive