Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Saturday, December 8, 2012

Handling file names with spaces in shell scripts

I decided to write this post after looking on the net for the solution to this problem. There are many recommendations but most of them are ether incorrect or involve convoluted methods like creating temporary files.
Disclaimer: I am by no means a shell scripting guru so take my advice with the grain of salt and try it for yourself. This information is relevant to bash, I do not use other shells.

The problem

You want to pass a number of files to your shell script and process each file in some way. Obvious approach is to use for loop and place processing of the file inside of the loop, something like this:

files=$*
for file in $files ; do
  process-file "$file"
done


Note the quotes around the $file variable. These are used to ensure that if the file name has some spaces, the file name is processed correctly.  Placing double quotes around the variable will ensure that spaces in it will be treated as a part of a single file name rather than multiple file names separated by spaces. If none of your file names contain spaces, the code above is perfectly workable.
The for command breaks list in the files variable using field separator which normally includes space. So in the code snipped above file will be assigned portions of the file names if the file names contains spaces. One solution recommended on the net is to modify field separator to exclude space, but this creates other problems.

The solution

The actual solution is very simple and based on often overlooked statement in the bash man page. If you omit in statement, the for command executes list once for each positional parameter that is set. So reworking above snipped as indicated below gives you the code which works correctly:


for file ; do
  process-file "$file"
done


The snipped above worked in all of my scripts. The only issue might arise when you do not pass file names on the command line. In this case function can be used or positional parameters may be reassigned using set command.

Sunday, November 25, 2012

Pogoplug Arch Linux glibc update error

I have not updated Arch Linux on my Pogoplug for a period of time. When I finally tried to update it, I encountered dreaded glibc: /lib exists in filesystem error. I have tried all resources that I could find and was not able to update glibc. I was able to update all packages including linux by making symlink to usr/lib but not the glibc itself.

The funny thing is that I already had later glibc which was recommended on the archWiki site, but nothing worked. At the end I used -f flag on pacman command line and that broke the system. Do not use --force flag as it WILL break your system.
Fortunately for me I keep all important data on a drive which is separate from my system drive. I unmounted the data drive and  re-installed linux following original instructions. This time it went smooth as I already discovered and documented all pitfalls.

The best outcome without upgrading glibc would be achieved by following  these instructions. You will get linux installation error but everything else will install correctly. At this point you should uninstall linux but do not reboot! Next make symbolic link:
ln -s /usr/lib/modules /lib/modules
After this reinstall linux and it should install without any errors. Now you can reboot. You should have an updated system but with older glibc. If you know how to update glibc please post it in the comments.

After killing my system with --force, I simply deleted all data from my system drive and reinstalled from scratch. This took me less time then I spend on doing all of the experiments to fix the glibc error. The rolling distribution is not so "rolling" after all. You can get it in to a state in which it is not upgradable by simply lapsing your upgrades for a period of time.

Monday, July 9, 2012

Checking root filesystem on ArchLinux

My shiny new ArchLinux on the Pogoplug was complaining about root filesystem not being clean. I tried usual approach of placing /forcefsck file in the root folder and rebooting. Although the file was removed at the next boot the filesystem check still complained.

I went to the IRC to the #arch-linux-arm channel which is recommended on the website, but no one appears to know how to do this on the running system. The only recommendation that I got, was to take out the drive and put it in to another Linux box. Well, this being a USB flash drive makes it possible to do so. But imagine an internal disk and this quickly becomes less than practical solution.

So I went to the "Internets" and did a search. As expected someone already asked this question and got a multitude of answers which helped me to put together a procedure. So here it goes:
  • Execute following command:
    fuser -v -m /
    This will give you a list of all processes having files open on the filesystem. Look for the processes that have F or f flags set. For example:
    root        182 .rce. dhcpcd
    root        197 frce. crond
    root        219 .rce. sshd
    ntp         226 Frce. ntpd
    So it looks like the crond and the ntpd have files open.
  • Shutdown these processes by using following command:
    rc.d stop ntpd
    Shutdown all of the processes that have files open on the filesystem.
  • Now you can remount filesystem using following command:
    mount -r -o remount  / 
    If you correctly shutdown all of the processes remount should complete without any error.
  • At this point you should be able to do a filesystem check:
    e2fsck /dev/sda1 or e2fsck -y /dev/sda1 
I hope that you found this post helpful and it saved you some time.

Saturday, July 7, 2012

Pogoplug after a few days

I had this device for several days and so far it works well. I replaced the original Pogoplug software with ArchLinuxArm. The box is much snappier than my original Cobalt Qube 2 with NetBSD 5.2. Of course it is much faster machine and with double the memory. The size of the device is also small and the case while funky looking is OK. I only wish it would open more easily so I could take a look inside.

ArchLinux appears to be well made distribution with the package manager that tracks dependencies. Startup scripts are easy to understand, unlike the latest Ubuntu. I am not sure why they keep changing startup scripts every year. I installed Python and Samba to run MyMedia server and to backup my data from the Windows PC. MyMedia server runs very well on this hardware. The menus come up fast and the videos and images served quickly enough. Over all it is very good box for the purpose.

I am getting another enclosure and will install Linux on the hard drive instead of the USB stick it is on now. I want to try to compile some software and to see how well this box would work for unix development. It probably would not be a speed daemon but it could work.

At this point I would recommend this device to anyone capable of installing Linux himself.

Wednesday, July 4, 2012

Linux on Pogoplug

I was tempted by the Pogoplug Family deal and got myself a Pogoplug  classic for $29.95. They advertise it as a "free device" with Pogoplug Family but in reality you are buying the device for the above sum. Don't forget to cancel the service after receiving the device. Otherwise it would be more expensive than actually buying the same device outright.
After getting the device I evaluated it for about a day. I got the grey version of the Pogoplug Classic with Marvel Chip, 256MB of ram and 128MB of flash. The device itself is snappy and works reasonably well. It recognized my USB hard drive and went to work on it, apparently indexing the content. I measured transfer speed and using my old router with 100Mbit Ethernet, the transfer speed is around 9MB/s which is not bad. The device is silent and consumes very little power.
The companion software on the other hand is terrible. It is very much useless for backup as one can not control what is being backed up. You can only specify the top directory and everything under that directory is backed up. I thought about using the drive feature to backup with Microsoft Sync Toy but somehow it makes for a very slow backups. This device appears to be made for a completely computer illiterate person who never used any NAS devices. It also requires connection to the Pogoplug servers and I would imagine opens files on your "cloud" to snooping by the company. Over all I found the "Pogoplug Cloud" concept is sorely lacking in the implementation department. It does not hold a candle to the simplicity of Dropbox or Google Drive.

I actually got this device to replace my very old Cobalt Qube microserver which is showing its age. In that role Pogoplug Classic performs really well.
Most people appear to have good luck with the Arch Linux Arm and that is what I decided to try. I used a Centon 4GB flash drive as my root drive and WD drive in USB enclosure as my data drive. Installation of Arch Linux Arm is very simple and described on the web page. The Pogoplug Classic model number is POGO-E02 which is one of the supported devices.
The installation went smoothly and the box appeared operational but it would not acquire  IP address. Due to the bug in the Ethernet driver and my old router the device kept disconnecting and reconnecting to the network. After some search I found the solution. If your Pogoplug appears to boot but fails to acquire IP address add following line to the dhcpcd.conf file : nolink. Linux, samba and python occupy around 740MB on the flash drive. I did not install git but instead use my custom copy of the MyMedia server.

Now my Pogoplug is booting Linux and can act as NAS device with up to 4 USB drives and can run MyMedia server for Roku. It is much snappier than my old MIPS box but obviously not as fast as my desktop. It is worth $30 in my opinion if you can install your own Linux on the device. Similar devices would easy cost north of $50.

If I decide later to change my mind and use the "Pogoplug Cloud", reverting back is as simple as removing the USB flash drive and rebooting the device. The original system is preserved in the flash along with the original uBoot bootloader.