My last post was about my plans to make a system that detects and blocks leeching attempts. My plan was to create a system where I can filter my traffic by determining who was trying to access the requested files based on the referrer string sent by the browser. So I developed a system that tracked and organized all the requests for any file in my files folder. It only took a couple days to get the tracking system sorted out but I wanted a clear idea of where most of the traffic was coming from so for the last month I left it running. Today, I finally have my results.

As well as tracking the referring website I also tracked the filename, the file type, the user agent, the base website, and how many times that file was requested by that referrer. On top of that, if there was no referrer or if the referrer came from my website, I tracked their IP address and the country that IP address belongs to.

To my dismay the majority of my traffic had blank referrers, meaning that I had absolutely no way of determining how they found the file that they were requesting. I attribute most of these blank referrer strings to crappy flash players that choose to not send any referrer information with their request. Most notably of which is "NSPlayer". What is really interesting about the requests from the NSPlayer user agent is that they are all from China. Which, at risk of generalizing internet traffic, probably means that it is up to something sinister.

Another interesting factor is referrer spoofing. In my first version of my tracker I naively thought that I could trust traffic coming from my own domain (saurdo.com) but upon removing it I noticed that there were a lot of bogus referrer strings accessing files from places on my website where the file was never referenced. With my ability to track the user agent and country of origin I have determined that these bogus referrers were bots. I haven't determined what they are trying to achieve but I do know that they scan my files folder at least once a day. I determined they were bots because they all used internet explorer 6 and they were all from China (another generalization).

The unreliability of the referrer string pretty much ruins my idea because I cannot move any of my filtering to a lower level, such as an htaccess file. In order to properly determine the legitimacy of each referrer I need to test their reliability. To do this on all requests, including ones that your website sends, just sounds horribly inefficient.

I haven't come up with a better idea yet.