Offline Website grabbers / broswers

News related to the site
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Offline Website grabbers / broswers

Post by Iain »

I have noticed a couple of people using website grabbers (ie programs which download the WHOLE of a site) over the past few months. I was letting it go because while they were using a good bit of bandwidth, I still had lots to spare but now it's getting silly.

It's only 10 days into February and one guy from 217.43.159.22 who's using a website grabber has used 12.25% of the website bandwidth so far. He's downloaded over 200Mb of files in only 4 visits!!!

It pisses me off that there's always a couple of people who try to abuse things and make it worse for other people (ie, probably slowing down the site).

Now, because of this guy and probably a few others, I have to waste time and figure out a way to stop these website grabbers eating all my bandwidth.

Die leechers die! :-E

[update]

Just noticed that the same guy has used 5.77% of the bandwidth of my other site, which is used to host some of the issue scans

[/update]
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

Damn!! It took until now (2 hours) to get it working correctly. I should be a bit of an expert of robot.txt and .htacces files now!!

This should stop this guy.. at least for now, I'll have to add more barriers later.

The Internet is really like the wild west these days :(

Let me know if I've broken anything on the site with these access changes!!
User avatar
Lloyd Mangram
King of Ludlow
Posts: 1151
Joined: Thu Jun 19, 2003 10:22 pm
Location: Ludlow
Contact:

Post by Lloyd Mangram »

Oh dear.... :shock: :oops:

I wanted to ask your permission first, and later I wanted to inform you about it but forgot, but errr... I have to admit I 'mirrored' your whole site
AFTER the forum hijack, unfortunately! :evil:

I thought if anything would happen in the future I would have a backup at least, preserving all your hard work if we'd ever run into trouble (again).
Of course, I had no intentions to steal your bandwidth or anything, it was just a one timer for me, only to occasionally upgrade the backup if needed.
(with all the evil scripts you have running now, I can forget that now 8) )

So, I'm really sorry about it all! Don't ban me! :twisted:

Ta!
Rob
:D
Once again I emerge from beneath a massive pile of paper which makes my desk groan to bring you the world’s most amazing posts.
User avatar
LeeT
Director
Posts: 460
Joined: Wed Jun 18, 2003 8:19 am
Location: United Kingdom, Hartlepool

Post by LeeT »

On a separate issue, I noticed that there was '32 people on-line' yesterday when I was logged in. :?
Contributor to Def Tribute to ZZAP!
Compilation64 - http://compilation64.zzap64.co.uk/
Crazy about the C64? - www.lemon64.com
Amiga Anguish? - www.lemonamiga.com
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

Rob!!!!!

You should have let me know before!! But... you only did it once? And how long ago was that? Were you using Httracter?

The above program has been taking about 10% of my bandwidth for the past few months though...
User avatar
Lloyd Mangram
King of Ludlow
Posts: 1151
Joined: Thu Jun 19, 2003 10:22 pm
Location: Ludlow
Contact:

Post by Lloyd Mangram »

iain wrote:Rob!!!!!

You should have let me know before!! But... you only did it once? And how long ago was that? Were you using Httracter?

The above program has been taking about 10% of my bandwidth for the past few months though...
Hi Iain, don't hate me. :wink:
I did it with Httracker, (excellent proggie, btw) in a couple of sessions.
I back-upped the whole site, including the forum, but NOT THE SCANS! (of course, and why should I, I have them all on cd/hd)
You can choose which folders you want to 'mirror' and I excluded, or killed, all folders with scans. Occasionally I seemed to have mirrored some other stuff as well on the same server. (tfyatdt-stuff)
It was in the week (or two weeks ,max) after the Rrap got hacked, and yesss..I only did it ONCE.
I'm really, really sorry. :cry:
I never meant to get you into any sort of trouble as I really appreciate what you are doing. I'm sure you're aware of that fact. :)

Hugs? :D 8)
Rob
Once again I emerge from beneath a massive pile of paper which makes my desk groan to bring you the world’s most amazing posts.
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

Well, I didn't get into any trouble, I'm not over my bandwidth limit but I don't want lots of people grabbing the whole site lots though.

I forgive you this time Rob! But don't worry, there's other people doing it as well.

By modifying the .htacess file I should be able to stop a lot of email harvesters etc. so the extra knowledge I have gained nwill be usefull.
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

Gawd damn it, another person used a web grabber yesterday and used up 20% of my bandwidth so far this month. Have to go changing the server files again to block it :((
User avatar
Lloyd Mangram
King of Ludlow
Posts: 1151
Joined: Thu Jun 19, 2003 10:22 pm
Location: Ludlow
Contact:

Post by Lloyd Mangram »

iain wrote:...another person used a web grabber yesterday and used up 20% of my bandwidth so far this month... :(
Woo! :evil:
Well, it wasn't me. :wink:
Once again I emerge from beneath a massive pile of paper which makes my desk groan to bring you the world’s most amazing posts.
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

I don't fucking believe it! The website is getting hammered again, prob by offoline website grabbers, 17,000 pages have been served already today (the avg for a full day is 3000-4000).

My months bandwidth is almost up, if this continues I'll have to shut down the site for a day or two!!!

So much hassle!!!! :-E
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

The gobshite is still at it, his IP is 212.69.226.40 from legend.co.uk. Feel free to bomb him, he has used 1/2 gig of bandwidth so far today!!!
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

Look at all the bandwidth these fuckers have used. You can see the crazy spikes of bandwidth usage at the beginning of this month and again today.

Anyway, I have banned the current ones IP now (couldn't do it from college earlier!)

Image

I'll have to do some serious anti crawler etc. coding for the site after my exams!
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

Unbelievable, the website is under attack again from the 212.69.226.40 gobshite at legend.co.uk. Guess I didn't implement my ban correctly!

The worm that's trying to hack phpBB forums must also be all over the place, I've had 11609 forbidden access today as well!

The Internet is just like a war zone at the moment, it's sad how it's gotten really, compared to how it used to be back when I started using it in 1994 :(

The joys of running a website! :(
CraigGrannell
King of Zzap Towers
Posts: 610
Joined: Thu Jun 19, 2003 6:45 pm
Location: UK
Contact:

Post by CraigGrannell »

I take it you've contacted the ISP?
User avatar
Iain
Admin
Posts: 2222
Joined: Tue Jun 17, 2003 6:42 pm
Location: Cavan, Ireland
Contact:

Post by Iain »

his ISP? not as of yet, but if it continues, I will when I get home and have a good look at my logs to see the details, need to add more security to issue downloads I think :( Maybe a fax of birthcert , photo ID and signed letter before you can download them? :-D
Post Reply