squid: Things to look at if websites are hanging!

 
People occasionally ask why site X doesn’t work through their Squid proxy but it does when going direct. Almost every one of those problems can be explained by three phenomena:

• ECN (Explicit Congestion Notification)
• PMTUD (Path MTU Discovery)
• TCP Window Scaling

 
Explicit Congestion Notification is a method for a router or gateway to signal end-points of a TCP connection that the path is reaching congestion and they should back off a little. ECN uses two bits from the TCP ToS byte to identify if the endpoints are ECN capable and to signal congestion. Unfortunately many older firewalls developed before ECN became main-stream do a variety of silly things when faced with ECN bits – the worst behaviour being simply dropping the packet on the floor. The pre-ECN specification noted those bits were “Not specified” but various firewall vendors took that to mean “must both be zero” and treat non-zero bits there as “invalid traffic.”
 
Disabling ECN is platform specific and is an operating system parameter – Squid doesn’t have the ability to disable ECN for specific connections. Just use Google to determine how to disable ECN.
 
Path MTU discovery is a muddy issue. PMTUD is a method for determining when a router/gateway between two IP endpoints has an MTU smaller than the endpoints. For example – if a home PC and server has an MTU of 1500 but the ISP broadband has an MTU of 1480 (as they’re tunneling broadband traffic around their network.) PMTUD specifies that hosts should set the “Don’t Fragment” bit on traffic and routers/gateways should then return an ICMP “Fragmentation Required” message when faced with a packet too large to forward. The sending host receives the ICMP packet, drops its MTU for that IP destination, and resends the packet.
 
This requires a number of things work:

Firstly, that the router/gateway actually sends the ICMP packet (which doesn’t always happen for various reasons – one of the more popular is a slightly misconfigured network resulting in MTU mismatches at Layer 2; if a packet is dropped by a Layer 2 MTU mismatch no ICMP (which is Layer 3/4) will be generated.)
 
Secondly, that ICMP is not filtered. Unfortunately a number of firewalls and servers have been configured treat ALL ICMP as hostile and drop the traffic. ICMP is a part of the IP infrastructure and is necessary for correct behaviour, but this doesn’t seem to have penetrated the heads of various firewall administrators of various large websites.
 
Resolving issues relating to PMTUD can be a bit difficult. The correct thing to do is to ensure you allow ICMP to/from your proxy servers, clients and origin servers (which holds true for reverse and forward proxies.) If you have to filter ICMP then filter out the potentially hazardous stuff (echo request/reply, port/host unreachable) but leave the important infrastructure stuff (like “fragmentation required.”) If you’re that stuck, consider dropping the default TCP MSS on your server.
 
Dropping the TCP MSS is definitely not changing the MTU of the interface. The MTU of your interface(s) should be the same as all other hosts on that network. Most operating systems have the ability to add routes which override the MSS (Maximum Segment Size) for all connections to/from that particular host.
 
Under Linux you can use the mss route flag to do this; eg
 
route add default gw X.X.X.X mss 1200
 
Note that the MSS is not the same as the IP payload. The MSS is the TCP payload and options. For example, if you have a 1500 byte TCP frame, that will be 20 bytes IP header, 20 bytes TCP header, (say) 20 bytes TCP options and the rest (1440 bytes) for payload. The MSS is the payload (1440 bytes) and the options (20 bytes) = 1460 bytes.
 
You could just turn off PMTUD and hope that the routers in the path will fragment the packets for you. I don’t do this so I’m not sure how, or how effective this will be.
 
Finally, TCP Window Scaling is a TCP option which allows TCP to transmit in units larger than the default window size (which is a 16 bit number, so 65535 bytes.) No, this doesn’t mean the packet is 65535 bytes in size; it means it’ll transmit 65535 bytes of data in MSS sized segments before waiting for an ACK.
 
The TCP WSS option tells the TCP stack how many bits to shift the window size “right” to calculate the true window size. A WSS of 0 shifts the window size to the right by 0 bits, giving the same as normal. A WSS of 1 shifts the window size to the right by 1 bits, giving you a real window size of up to 131072 bytes in 2 byte increments. A WSS of 2 shifts the window size by 2 bits, giving you a window size of 262144 in 4 byte increments, and so on.
 
This is done so higher TCP throughputs are possible over higher latency links. Higher latency being more than a handful of milliseconds (say 30ms.) Ie, most of the world.
 
Unfortunately, firewalls strike again. Some firewalls don’t understand the TCP WSS option and will do the most brain damaged thing possible – they’ll simply zero the whole WSS option. This has a horrible side-effect: both sides have sent TCP WSS options, and see the other side sending a TCP WSS option (of 0!), so they assume the option they sent is perfectly fine to use. Unfortunately they send an option of non-zero, and expect the window sizes in TCP frames to be interpreted with an option of zero, and bad things happen. This mostly shows up as slow/stalling TCP connections.
 
For now, the best thing to do is simply disable TCP WSS. Again, its operating system dependant and you’ll want to use Google to find out how to do it. You can use route flags in some operating systems (such as Linux) to set the maximum TCP window size for destinations if you wish to fine-tune things.
 
link
 

Shared folders not downloaded via Sync client?

 

by Ionican » Tue Nov 27, 2012 1:44 pm

ok – after scouring the developer boards I found this courtesy of ForestMountain (http://bugs.owncloud.org/thebuggenie/owncloud/issues/oc-755):

“I have found that you can change this

Linux (Fedora 16): /home//.local/share/data/ownCloud/folders/ownCloud

Windows7: c:usersApp DataLocalownCloudfoldersownCloud

Edit the ownCloud file with a normal texteditor. Writing targetPath=/ will reflect the localpath”

 
link
 

Create RAID1 with two usb flash drives (16GB :) ) for owncloud data

 

1. # cfdisk /dev/sdb # select FD partition type for the new partition
2. # cfdisk /dev/sdc # select FD partition type for the new partition
3. # mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
4. # mkfs.ext3 /dev/md0
5. # mount /dev/md0 /media/raid1_owncloud

6. # nano -w /etc/fstab :
   
   /dev/md0 /media/raid1_owncloud ext3 noatime,rw 0 0

7. # nano -w /etc/mdadm.conf :
   
   ARRAY /dev/md0 devices=/dev/sdb1,/dev/sdc1 level=1 num-devices=2 auto=yes

 

• Informations on existing array:

  # mdadm --detail /dev/md0
  # mdadm --monitor --scan --daemonize --mail=jdoe@somemail.com

• Assembly existing array,

  # mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1

• Stopping mdadm,

  mdadm -S /dev/md0

 

cron.daily backup owncloud

 

cat /etc/cron.daily/backup_owncloud

#!/bin/bash

oggi=$(date '+%d_%m_%y')
/bin/tar czf /home/.../backup_owncloud/backup_"$oggi".tar.gz /media/raid1_owncloud/

#pulizia:
/usr/bin/find /home/.../backup_owncloud/ -mtime +2 -exec rm {} \;

 

apache ssl certificate

 

• # openssl req -new -x509 -days 365 -sha512 -newkey rsa:2048 \
  -nodes -keyout server.key -out server.crt \
  -subj '/O=Company/OU=Dept/CN=www...'

• cat /etc/apache2/sites-enabled/ssl 
      
              DocumentRoot /var/www/ 
              ServerName 192.x.x.x
          
              Options Indexes MultiViews FollowSymLinks
              AllowOverride all
              Order deny,allow 
              Deny from all
              Allow from all
          
              SSLEngine on
              SSLCertificateFile /etc/ssl/certs/cert.crt
              SSLCertificateKeyFile /etc/ssl/private/server.key
      
  

 

Mega Security #Failed

8. Our service may automatically delete a piece of data you upload or give someone else access to where it determines that that data is an exact duplicate of original data already on our service. In that case, you will access that original data.

link
 

 

linux network traffic per applications

 

Network Bandwidth Usage Monitor

NetHogs is a small ‘net top’ tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process. NetHogs does not rely on a special kernel module to be loaded. If there’s suddenly a lot of network traffic, you can fire up NetHogs and immediately see which PID is causing this. This makes it easy to indentify programs that have gone wild and are suddenly taking up your bandwidth.