Will Partain's work blog

I can't believe how hard the following problem is turning out to be.

We are a small company with people hither and yon (for various reasons) who need to see/work-on the same "documents". In a normal Windows Explorer kinda way (not some over-there-in-the-browser kinda way) and with (near-)LOCAL DISK speed. (I do not want to hear the word 'WebDAV' ever again.)

Yes, they always need to see the "latest" version at all times. (No, they do not want to click an 'Update' button.)

They need to be told "Whoa! Two of you are trying to write that thing at once!"

Different (changing) groups of people need to see different groups of documents. Only finance people can see finance documents, etc.

An "admin" needs to be able to login, get some oversight of things, and change stuff, e.g. shut off the person who just left the company.

Windows, Macs and Linux, please. (OK, OK... we're flexible.)

I would have thought the above would be the FIRST THING any "cloud storage" wannabee might turn his hand to.

Obvious "solutions": NONE (that I've found). More as I find it.

I attempted an April Fools' Day gag at work, and FAILED. The failure is IT-instructive.

Now, I don't like April 1 jokes that are apparent just from the title; but I erred too far on the side of obscurity.

A few weeks ago, we had email chat at work about some of the weirdness you see on your router if Skype is running. I mused that I would prefer that people who do not use Skype should turn it off. Each time I was reminded that Skype was a major communication tool within Verilab.

So last Thursday I "announced" that a Skype ban would be rolled out starting with people "from the middle of the alphabet". People who don't even work in that office were "banned". To summarize:

• The "ban" was high-handed and made entirely without consultation.

• No alternative was proposed, making the "ban" extremely business-hostile.

• The "analysis" on which the "ban" was based -- we occasionally see bursts of UDP packets for a minute or two -- was, errmm... "limited" to say the least.

• "Starting in the middle of the alphabet" -- say what?

• Summary: COMPLETELY INSANE.

What's interesting is that a COMPLETELY INSANE proposal from the IT guy is taken as quite normal. I was asked polite questions, and people seemed to be trying to adapt. I don't know whether my colleagues intended to ignore me, work around me, or what. (Of course, it may be a double-bluff and the joke's on me.)

Perhaps we see so much COMPLETELY INSANE stuff in all the organizations we bump up against in everyday life that it seems normal for the IT guy to trash the company's communication structure.

I am in the army of Linux-only people who "deals with" friends' Windows PCs, thus making Microsoft look better than they deserve and Bill Gates richer thanks to no effort on his part.

Microsoft's apparent attitude to this sullen army of volunteer helpers: No good deed goes unpunished.

Case in point. Some friends' PC died; they got a new one; the disk out of the old one was fine and readily sprang to life in a USB external enclosure. They'd like to move their old stuff over to the new machine.

That would be a few minutes' easy work under Linux (but this is Windows).

Ah! But what's this?! "Windows Easy Transfer" [WET] (under Windows 7) -- "Helps you transfer personal files, e-mail, data, files, media, and settings from your old computer to the new one." Fantastic! And it can read from an external USB drive -- glory be!

Sadly, you have to have "prepared" for the "Easy Transfer" by running WET on the old machine first -- presumably before the puff of smoke and the acrid smell in the air.

SIGH. And does WET have any sort of second-best fallback option? e.g. "We can't transfer your user accounts, but we can at least move some documents over for you?" (You know, the equivalent of a one-line shell script.)

No, of course not. (What, exactly, do all of those "programmers" in Redmond do all day every day?)

So I'm doing Windows Tedious Transfer [WTT]. Copying files/folders around, deleting Obviously Useless things (e.g. cookie files), and so on. I.e. doing a third-rate job taking eight times as long.

No good deed goes unpunished.

I wrote about how to arrange disks on a host that needs to support RH-style virtual machines (VMs) in a flexible way.

And what should you do on the guest (the VM itself)?

Well, following the cited scheme, the VM will be handed an LVM "logical volume" to use.

The VM will treat those bits as a disk (as it would if you created a big file and handed that over, instead). Your only real choice is to partition that virtual disk, and use the resulting partitions in the usual ways.

If you know exactly how to use the VM's disk space, and "it will never change", then create fixed partition(s), slap filesystem(s) on them, and you're done.

If, however, we again assume a need for "flexibility", then LVM is again a good idea -- yes, LVM(guest)-within-LVM(host).

I have been told (but not tried myself) that LVM on the host can "see" the inner-LVM stuff (the guest's) and do things with it, e.g. snapshot it.

(Apparently important: the volume group(s) on the guest must have different names from the volume group(s) on the host.)

But the main virtue of LVM-within-LVM is that it's relatively simple to increase the "disk space" on the guest...

Stop the VM: virsh shutdown guest

On the host, extend the logical volume that is the guest's "disk":

lvextend -L20G /dev/vg0/guest-disk

Restart the VM: virsh start guest

The guest's "disk" just got magically bigger. Use fdisk (or equiv) to make a new "partition" out of the extra space, giving it the LVM type (8e).

Make the new "partition" an LVM "physical volume" (PV):

pvcreate /dev/xvda3

Add the PV to the VM's "volume group" (VG):

vgextend guestvg0 /dev/xvda3

Now you can make one of the "inner" (guest) "logical volumes" (LVs) bigger:

lvextend /dev/guestvg0/lvroot ...

Finally, the filesystem sitting on that LV will need resizing:

resize2fs /dev/guestvg0/lvroot

If you're going to use a Linux box for Red Hat-style virtualization (Xen/KVM), how should you use/partition your disk(s)? Here's one answer.

First: disks are cheap enough that I always use two, and software-RAID-1 them together (mirroring). Plan to do that.

Second: Though I'm no fan of LVM (the Logical Volume Manager) -- extra complication for no gain -- it's a good choice for "host for a bunch of VMs (virtual machines)". I'm assuming you don't know exactly which VMs of what size, so the flexibility of LVM is actually useful.

Also, as disks get larger, it's harder to predict in advance how you'll end up using all that space. So, again, the flexibility of LVM may benefit.

Now, I don't use LVM for the basic filesystems (/boot and / [root]). Their sizes are predictable and, when It Goes Horribly Wrong (TM), you'll be really glad not to have LVM in the way. So I partition my raw disks thusly:

Filesystem  Size  Used for
/dev/md0    300M  /boot
/dev/md1     16G  /
swap         ???
/dev/md2    rest  of the disk(s)

That /dev/md2 goes straight into LVM. Make a PV (Physical Volume):

pvcreate /dev/md2

And create a Volume Group (let's call it vg0) with it:

vgcreate -s 32M vg0 /dev/md2

(That -s 32M is not particularly well-informed.)

Now you can make LVs (Logical Volumes) to your heart's content...

lvcreate -n vm001-disk --size 10G vg0

... and feed them to your VM-creating software. (Or make things for your host machine, e.g. a varying-sized /var.)

A word on LVM-on-RAID... LVM has a mirroring facility of its own, so why use software RAID1?

First, I'm already paying the software-RAID tax with my non-LVM partitions (/boot and /).

Second: For me, software RAID is familiar and has been reliable.

Third: Software RAID-1 is less scary in a crisis. I like things like: You can take a disk that is half of a RAID-1 mirror, throw it in an enclosure, and mount it as a filesystem, no problem.

Fourth: The LVM mirroring guff seemed more complicated and/or easier to get wrong. (Just unfamiliarity, maybe.)

Further to my previous battles with /bin/rm, something new-to-me that might help: ionice -- "get/set program I/O scheduling class and priority". (Linux-only, I think.)

Running ionice -c3 /bin/rm ... should mean that rm "will only get disk time when no other program has asked for disk I/O..." (man page). That sounds about right.

Of course, ionice will not stop seek storms, nor will it leave the disk heads in a place convenient for other programs that are trying to do Real Work (TM).

NB: I have no empirical evidence that ionice actually helps in cases where /bin/rm renders a machine otherwise useless.

Who woulda thunk it? I have a new-in-the-last-year Linux box, large-ish disk (750GB), mostly one big ext3 filesystem, and -- OK, shoot me... -- a bunch of many-rsync'd-hard-links directories therein.

I made the disastrous choice to remove some of those directories, you know... /bin/rm -rf *.OLD.

Two weeks later, the job isn't finished.

The job probably would be finished if I had exclusive use of the machine. But the other users get stroppy when I render the machine useless to them with my exotic Power Tool, /bin/rm. The load average runs to 5, the CPU romps up to 98+% IO waiting, and the other users start calling my number. Not in a nice way.

I would have thought that removing files-and-folders was about the most settled/debugged/worked-out part of Linux. How naive I obviously am.

My situation, or one very like it, was discussed on the Linux kernel list: "Very poor ext3 write performance on big filesystems?" (Probably elsewhere, too.) You can read it for yourself.

The first thing I did was make a script rm-and-sleep; not rocket science:

#!/bin/sh
/bin/rm ${1+"$@"}
sleep 5

I can then run that with something like...

find . -type f -print | xargs -n 50 rm-and-sleep

... and be sure I am not eating all of the system all of the time. (Just seeing the script makes me want to weep... decades of computing progress, and we're doing this?)

One thing suggested in the discussion cited above is to remove things in inode order. I ended up using cut rather than awk in Chris Mason's script, giving:

find . \( \! -type d \) -printf "%i %p\n" \
| sort -n \
| cut -d ' ' -f 1 --complement \
| xargs -n 500 rm-and-sleep

Guff to deal with spaces-in-filenames and nice and so forth -- omitted for brevity.

Ugh. Ugh. Ugh.

Weirdest network symptom for me in a long time... An ssh -v to one of our remote hosts would hang every time with:

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

The only hint in the apparently-most-relevant online discussion is that it might have something to do with some network interface's MTU (Maximum Transmission Unit).

Lacking any better ideas, we changed the MTU on the SSH server (with ifconfig eth0 mtu 512) -- note: a value wildly lower than what should be required -- and it worked! SSH connections could happen.

Except: only just barely. The first time the remote end tried to send any amount of data -- e.g. a VNC screen refresh -- the connection would wedge, and that would be that. (I cannot explain this data point: I would expect the very low MTU setting to ensure that no big packets ever got out, and so no wedging could happen on that account.)

On the weak assumption that our problem was something to do with MTUs, I proceeded to learn a fair bit more than I wanted to know.

Every network link has an MTU, and there are on-the-order-of a dozen links between Here and There in any network interaction. An MTU of 1500 is standard for Ethernet (modulo Jumbo Frames), but you get MTUs in the high 1400s for various network technologies (1500 minus "some bytes for our special stuff"). Needing an MTU under 1000 means "something is not right".

What happens if you send a packet that is too big to make the next network hop? In theory, the network device is allowed to "fragment" the packet, and send on the pieces. I believe this sort of malarkey is frowned on, and many (most? (all?)) TCP packets are sent with a DF (Do Not Fragment) flag.

So what happens with a DF-flagged too-big packet? Well, the device at the edge of the too-small-MTU chasm is supposed to send back an ICMP packet telling of the problem, so that the originator can re-send in smaller bundles.

A whole bunch of things can go wrong. The device might simply fail to send the required ICMP packet. Any of the network devices on the way back to the originator might choose to drop the ICMP packet ("Security, you know."). The host firewall -- or the one on the last-hop consumer router -- stands a high chance of Doing The Wrong Thing.

Net effect: Big packets go, but then just drop off the end of the world. Re-sending doesn't help because, odds are, they will follow the same route and drop no less ignominiously. Eventually the connection will time out.

Your packets are said to have fallen into an MTU black hole.

Along the way, I learned that you can probe network MTUs by sending various-sized ping packets. (As in most networking problems ping and traceroute are your friends.) In Linux-speak, you type...

ping -M do -c 2 -s <n> remote.example.com

...varying <n>. So, for instance, if

ping -M do -c 2 -s 1465 remote.example.com

balks with Frag needed and DF set, but

ping -M do -c 2 -s 1464 remote.example.com

does not, then 1464+28=1492 is the least MTU between here and there. (ping has 28 bytes overhead which you need to add back in.)

Back to our own sad case. Besides trying our fabulously-low MTU of 512, we did all the other typical things, namely power-cycling all the networking doodahs around the office. In desperation, we even did a Linux reboot (a mistake). Nothing helped.

But when we woke up next morning and tried our stuff, everything was fine. Completely baffling.

Got a small router in a faraway office that has a web interface? How to get at it (without turning on the "allow access from public Internet" thing)?

Let's say it's 192.168.0.1 in the faraway office. So: SSH in with tunnel...

ssh -L10080:192.168.0.1:80 me@faraway.example.com

Now use your own web browser to visit http://localhost:10080/; it should be your faraway router's web interface.

Just another use of Fabulous SSH Tunneling...

There are a few Unix programs that will find duplicate files; none is great enough to merit any link love.

(A similar tool I use quite a bit is hardlink, which doesn't tell you about duplicate files; it just hard links them together. Which can be good.)

What I've really wanted, though, is a tool to which I could say "See that pile #1 of directories over there? And that pile #2 over there? Tell me what files are in pile #1 that aren't in pile #2." Or vice versa. Or maybe "the files that are in both piles".

What I don't want to know is that there are four copies of a file in piles #1 and #2 -- because they might all four be in pile #1.

For the special case of "files arranged identically in various directories" (e.g. if those directories are backup copies), here's something that comes close.

Over pile #1 of directories, run...

find <dirs> -type f -print | xargs sha1sum > my-file-list

The output, my-file-list, might need massaging.

Now go over to pile #2 of directories and run...

sha1sum -c my-file-list

The goal is: like-named identical files should be flagged up as 'OK', and the others as something else.

Let's say you want to delete those like-named identical files. It would then be something like (WARNING: UNTESTED)...

sha1sum -c my-file-list | egrep ': OK$' \
| sed -e 's/: OK$//'    | xargs -r /bin/rm

(I always grow such scripts incrementally, and by hand. You'll have to anyway if you've got filenames-with-spaces, or other fun.)