# Site Running Slow



## Shaun (29 Nov 2011)

You've probably noticed, but there is an on-going issue with the CC server at the moment where it is randomly using up all of its available memory and grinding to a halt.

I'm making small adjustments to the server/database settings each time it happens to try and find the best compromise between stability and speed.

Apologies for the treacle-like access and sporadic down-time, but please bear with me and I should be able to resolve it.

Thanks,
Shaun


----------



## summerdays (29 Nov 2011)

Years ago we use to describe the speed our work computer ran at, in terms of dead dog with x number of legs - (can't quite remember the expression), is that still used in computing terms?


----------



## Shaun (29 Nov 2011)

summerdays said:


> Years ago we use to describe the speed our work computer ran at, in terms of dead dog with x number of legs - (can't quite remember the expression), is that still used in computing terms?


 
Sounds about right for the way the CC server is running this morning ... lol


----------



## TheDoctor (29 Nov 2011)

These things happen.


----------



## Shaun (29 Nov 2011)

The hosting company are going to install some analysis tools on the server this evening for me. Hopefully this will help to discover the source of the slowness.


----------



## Mozzy (29 Nov 2011)

Ahhh that explains why it has been hanging. It was soooo slow last night for a period and about twenty minutes ago. I gave up. Fine now though Thanks for all you do Shaun.


----------



## accountantpete (18 Dec 2011)

It's been slow all weekend for me - and others presumably judging by the number of repeated identical posts.


----------



## Zoiders (18 Dec 2011)

Slow?

It's like an asthmatic ant with heavy shopping.


----------



## Shaun (18 Dec 2011)

Sorry, I haven't been on CC for a few days because I've been attending to some personal/family matters and hadn't realised this was going on.

I've had a look at the server logs and it seems we're being overrun by a very aggressive spider-bot from a Chinese search engine called Baidu that has been flooding the CC server with requests and using up all available resources.

I need to install some new software on the server to combat this (_by blocking their access completely_) and will try to get it on and working tonight if I can.

Apologies for the problems.

Cheers,
Shaun


----------



## Shaun (18 Dec 2011)

As a temporary measure I've reconfigured Apache to limit the number of simultaneous requests it serves - which will mean there will still be pauses and queuing of requests, but because it will be doing less at once it should make it work more efficiently until I can stop the mass of requests from China.


----------



## StuAff (18 Dec 2011)

So far, so good. Definitely much faster.


----------



## Shaun (18 Dec 2011)

Good ... I need to go an iron Little Ms. Admin's school uniform (_if I don't Mrs. Admin will make the Baidu spider-bot look like Mr. Nice Guy!!!_) but I will have a look at a proper fix after I've got the kit done ... 

Cheers,
Shaun


----------



## jonathanw (18 Dec 2011)

Thanks Shaun

Major improvement


----------



## ColinJ (18 Dec 2011)

I found this article which suggests solutions to the problems caused by Baidu and other spiders.


----------



## dave r (18 Dec 2011)

Admin said:


> As a temporary measure I've reconfigured Apache to limit the number of simultaneous requests it serves - which will mean there will still be pauses and queuing of requests, but because it will be doing less at once it should make it work more efficiently until I can stop the mass of requests from China.


 
Cheers Shaun


----------



## Speicher (18 Dec 2011)

I am very pleased Admin, that you have been able to iron out some of the problems.


----------



## Speicher (18 Dec 2011)

Or should I say, I am very impressed with your success with this board this evening. 
Seams it was nothing to get steamed up about! A hem!


----------



## MacB (18 Dec 2011)

Speicher said:


> Or should I say, I am very impressed with your success with this board this evening.
> Seams it was nothing to get steamed up about! A hem!


 
that'll be enough of that sort of stuff, I'm pretty sure excessive punning can have an even worse effect than spiderbots


----------



## potsy (18 Dec 2011)

MacB said:


> that'll be enough of that sort of stuff, I'm pretty sure excessive punning can have an even worse effect than spiderbots


Why break the habit of a lifetime though?


----------



## Speicher (18 Dec 2011)

An iron ic post, Potsy, which just in creases the chances of it happening even more


----------



## summerdays (19 Dec 2011)

Admin said:


> we're being overrun by a very aggressive spider-bot from a Chinese search engine called Baidu that has been flooding the CC server with requests and using up all available resources.


 
OK ... in simple terms ... someone what to tell me what a spider-bot was doing ... I've a vague impression of Aragog fiddling with the insides of a computer... (I don't think it helps that he has a name)

Is Baidu (your friendly Chinese spider), trying to use your computer resources to do something else with it? What does he gain?



Admin said:


> Good ... I need to go an iron Little Ms. Admin's school uniform


Has she not broken up from school yet?


----------



## Shaun (19 Dec 2011)

summerdays said:


> OK ... in simple terms ... someone what to tell me what a spider-bot was doing ... I've a vague impression of Aragog fiddling with the insides of a computer... (I don't think it helps that he has a name)


 
Spiders are little software robots that Google and other search engines use. They come along to your web site and download the page content to add to their search results.

Some are very good and will respect your site and not harvest too much information all at once. Some are less considerate and will send lots and lots of these little bots to grab your site data all at once - so much so that the server can't cope with volume and grinds to a halt.



summerdays said:


> Is Baidu (your friendly Chinese spider), trying to use your computer resources to do something else with it? What does he gain?


 
I presume it uses the data in the same way as Google, but I have yet to see any traffic generated from it and after reading other web site owner resources it seems many people are more than happy to block it completely to save on bandwidth/resources.



summerdays said:


> Has she not broken up from school yet?


 
Nope - not until Wednesday!!


----------



## Shaun (19 Dec 2011)

Just an update - I've spoken to the excellent technical bods who host the CC server and they're going to help me to block a few of the more aggressive spiders, which will hopefully reduce the load on the server and get us back up to speed.

Cheers,
Shaun


----------



## ianrauk (19 Dec 2011)

nice one...


----------



## summerdays (19 Dec 2011)

Thank you ... I understood that ... _(I think )_


----------



## postman (19 Dec 2011)

Well done Admin.Now seeing how you like ironing.One of the retired tasks i hate.Iv'e just popped a load into a Registered envelope and it's on its way to you.


----------



## Shaun (19 Dec 2011)

postman said:


> Well done Admin.Now seeing how you like ironing.One of the retired tasks i hate.Iv'e just popped a load into a Registered envelope and it's on its way to you.


 
Hey, hang on - I never said I liked it ...


----------



## Scilly Suffolk (19 Dec 2011)

Noticeable improvement!


----------



## Shaun (20 Dec 2011)

Sorted. 

*For the techies:* The Shorewall firewall has been installed on the server and the Baidu IP ranges have been blacklisted for the TCP protocol (on port 80).

*For the non-technical CC'ers:* The do-dahs have now been added to the thing-a-me-jig and the baddies can't hog the server so much now! 

Cheers,
Shaun


----------



## accountantpete (20 Dec 2011)

Thanks Shaun - much faster now.


----------



## ianrauk (20 Dec 2011)

glad to see the watchyamacallit thingumybob in the dilithium crystals have vectored and transmogrified the motherboard.


----------



## Shaun (20 Dec 2011)

ianrauk said:


> glad to see the watchyamacallit thingumybob in the dilithium crystals have vectored and transmogrified the motherboard.


 
Don't tell anyone, but I just banged the top of it and it started working again ...


----------



## potsy (20 Dec 2011)

Admin said:


> Don't tell anyone, but I just banged the top of it and it started working again ...


I think we all guessed that 
Well done on hitting it in just the right place though


----------



## ianrauk (20 Dec 2011)

and no accidentally tipping the water from the vase down the back again...


----------



## Shaun (20 Dec 2011)

ianrauk said:


> and no accidentally tipping the water from the vase down the back again...


 
Ah ha ... so you had one of those green-screen Sony TVs as well? Mum stopped putting plants on the top of TVs after that.


----------



## ianrauk (20 Dec 2011)

Admin said:


> Ah ha ... so you had one of those green-screen Sony TVs as well? Mum stopped putting plants on the top of TVs after that.


 

Oh yes indeed...


----------



## coffeejo (20 Dec 2011)

'specially for Admin...


----------



## Shaun (20 Dec 2011)

coffeejo said:


> 'specially for Admin...


 
'specially for Jo ...


----------



## dellzeqq (20 Dec 2011)

Admin said:


> Sorted.
> 
> *For the techies:* The Shorewall firewall has been installed on the server and the Baidu IP ranges have been blacklisted for the TCP protocol (on port 80).
> Shaun


I thought it might have been port 80!


----------



## jonesy (20 Dec 2011)

Slowing down again, another evil spider? Ungoliant maybe?


----------



## Lisa21 (20 Dec 2011)

I dont wanna be on here if its full of spiders


----------



## Shaun (20 Dec 2011)

Google showing us some "love" ...  ... all part and parcel of changing the software - the spiders notice the different structure, and because we're a popular site they come along en-mass to update their listings.

Google isn't as aggressive as others, but can still overwhelm your server a little.

It'll calm down over the coming weeks and months, and in the meantime if it gets to be too regular I'll cap the resources on the web server and limit the number of simultaneous requests it can serve.

Cheers,
Shaun


----------



## Lisa21 (20 Dec 2011)

Admin said:


> Google showing us some "love" ...  ... all part and parcel of changing the software - the spiders notice the different structure, and because we're a popular site they come along en-mass to update their listings.
> 
> Google isn't as aggressive as others, but can still overwhelm your server a little.
> 
> ...


Thanx Shaun, I was going to ask you if you could do that


(tbh, you lost me after "cap")


----------



## coffeejo (20 Dec 2011)

Lisa21 said:


> I dont wanna be on here if its full of spiders


Would it be better or worse if it was clowns?


----------



## Lisa21 (20 Dec 2011)

coffeejo said:


> Would it be better or worse if it was clowns?


Ooh thats not an easy one to answer
Dammit girl, you'll be giving me nightmares.............


----------



## coffeejo (20 Dec 2011)

Clowns with eight legs?
Or spiders with red noses?


----------



## potsy (20 Dec 2011)




----------



## Lisa21 (20 Dec 2011)

*OH FFS *


----------



## MattHB (20 Dec 2011)




----------



## ColinJ (20 Dec 2011)

I noticed that the ratio of guests to members often used to be 3:1 to 4:1 but now it seems to be about 2:1. Do spiders count as guests? If so, you've killed a lot of 'guests'!


----------



## Shaun (20 Dec 2011)

Yes, they do - however there are lots of them, and they seem to love the new software.

Technically they can download more pages per minute because of the smaller page footprint, and re-index much quicker; they've just all decided to do it all at once ... lol 

It'll settle down, and hopefully next year I can a shiney new server with more umph ...


----------

