Hmm, I am at home on leave from work and they are having a major issue with the web connection. The connection is run over a 2mb leased line, with verizon (worldcom) as the ISP. What seems to be happening at the moment is the connection is running at 100% utilisation, this has been confirmed by the ISP who have provided us with stats on the line showing maximum bandwidth utilisation.

Now keeping in mind that I am annual leave, and must be dedicated.

Via a very slow (due to the problem) connection, I have enabled logging on the firewall but can see nothing obvious, I have reviewed the websense proxy logs and can see nothing unusual.

We need to try and find the source of the problem, our AV software is full upto date, all the computers and servers are patched bang upto date. The firewall log shows nothing, so where do I start. They are trying to get someone with a packet sniffer which they are going to put on the internet side of the firewall to try and identify whats going on. The only issue I see with that is all the traffic will be to/from the firewall, so locating the source that way may be tricky.

Firewall is a Juniper SSG-520 running with a twin in failover mode.

Edited 17 November, 200817 yr by mannyo

My manager has hooked up ethereal to the internet side of the firewall and can see a lot of traffic.

I used Ethereal to capture data and see most traffic source to be 80.239.205.XX range which belong to AKAMAI.NET and trying to google this company but due to slow network its painful. Also I have spotted high FTP traffic in both directions to 15.216.110.140 and don't know who that maybe.

15.216.110.140 is a HP FTP server, and the other addresses I am not sure of. We are going to see later this evening when the users have gone home what the traffic looks like.

Malware backdoor?

Malware backdoor?

Not sure, all our internet traffic is routed through the latest version of websense which has an updated database thats supposed to filter out the crap.

Verizon say that most of the traffic seems to be inbound, which would also explain the sluggish outbound connection and even slower attempt at remote controlling my office pc via logmein.

We host a number of publicly accessible servers on the connection, including our main website. The website itself does not generate a lot of traffic, and we have not had a problem like this before even though nothing has changed. On the connection we have,

2 web servers,

Web email via OWA,

1 Netilla appliance for home workers to access the lan,

and 2 SMTP servers that are tied down with regard to routing and relaying.

Could it be simple utilisation?

Have you got a lot of guys in the office running internet radio or god forbid they've got a a machine pulling torrents 24/7.

It wouldn't take all that much to max out a 2Mb line.

We've got a netilla box, it can have session issues that slow things down. Have you tried another browser? Our guys have had some success using Opera rather than IE or FF.

Any update on this?

Akamai is a content provider... iirc they provide everything from ad services to p2p/file seeding media streaming etc...

Sounds like someone is trying to stream something lol..

15.216.110.140 =

DNS Stuff: DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.

IP address: 15.216.110.140

Reverse DNS: mercury.com.br.

Reverse DNS authenticity: [Verified]

ASN: 71

ASN Name: HP-INTERNET-AS (Hewlett-Packard Company)

IP range connectivity: 1

Registrar (per ASN): RIPE

Country (per IP registrar): US [united States]

Country Currency: USD [united States Dollars]

Country IP Range: 15.0.0.0 to 15.255.255.255

Country fraud profile: Normal

City (per outside source): Unknown

Country (per outside source): US [united States]

Private (internal) IP? No

IP address registrar: whois.arin.net

Known Proxy? No

Link for WHOIS: 15.216.110.140

On the Domain

% Copyright © Nic.br

% The use of the data below is only permitted as described in

% full by the terms of use (Registro .br - Terms of Use),

% being prohibited its distribution, comercialization or

% reproduction, in particular, to use it for advertising or

% any similar purpose.

% [74.53.59.137] 2008-11-19 14:35:15 (BRST -02:00)

% Permission denied. For more information, contact *****@registro.br.

% Security and mail abuse issues should also be addressed to

% cert.br, CERT.br -- Centro de Estudos, Resposta e Tratamento de Incidentes de Segurança no Brasil, respectivelly to ****@cert.br

% and **********@cert.br

%

% whois.registro.br accepts only direct match queries. Types

% of queries are: domain (.br), ticket, provider, ID, CIDR

% block, IP and ASN.

Can't tell if it's dodgy or not but unles you have dealings in Brazil it doesn't look great.

Might be worth running some spyware scans on the local machines if ethereal will ID which desktops the traffic is coming to.

I'll know some more next week, when back in the office. I'll be away until the weekend now so web use will be limited for me.

The ISP is now sending someone onsite Thursday, to do a more in depth analysis using more profesional tools. Hopefully they wont break anything, their support in this has been excellent and I would hope so for more than 1K per month.

Edited 19 November, 200817 yr by mannyo

Its going slow at the moment, so at least they will have something to analyse.

Block the p2p ports and if you have it use packet inspection to inspections on port 80 to check the item is a valid HTTP request and not something else running over the normal ports.

Another possible option is staff streaming radio via iplayer or similar.

If you can put a linux box on the outgoing link as a transparent pass through you can run IPTraff on it which will show you tye packets, types sources etc which could be useful