Covering the basic concepts behind a modern spider trap, this guide discusses the core components you require to create and operate a spider trap through to the various techniques you can use to identify crawler or spider traffic - such as User-Agent checks, IP address checks and more advanced methods such as weighted results plus behavioural quirks and analysis.
