andLinux with Hardy Heron

andLinux is  built on top of co-linux (co operative linux) and basically runs side by side with Windows. andLinux packages the whole thing better (coLinux bundled with Xming and a nice systray app allowing you to launch Linux apps right in windows).

Here's details on getting off the ground - and the reason that I have this post is that though andLinux comes with an installer application, it still needs some amount of fiddling under the hood to make it work. This post is just to make sure I can go through the process again when the time comes

1. When installing andlinux, choose the COFS option for making your hard drive visible in Linux


2. Install with the command line option to launch andLinux (do not install it as a service just yet)


3. Post installation, tweak andlinux's network setup - set up a couple of virtual TAP adapter . You will have to tweak things both on the linux side and on the windows side. Basically, you create a 2 TAP adapters - one is a loopback and another for sharing your LAN connection. Your wireless network is shared via Slirp (doesnt need a TAP adapter setup).


4. Keep in mind a gotcha - slirp wont allow you to ping - so if you have only slirp working, then try a wget www.google.com to check if you have network connectivity.


5. Start the andlinux server (or if its already running) make sure that your c drive is shared - on the bash prompt you should be able to do ls /mnt/windows


6. do a apt-get update to update your package list. run an update. As of this time, the only prebuilt images on andlinux.org is gutsy.


7. do a apt-get install update-manager-core


8. run do-release-upgrade - and you should see apt running and updating your system to hardy.

Compact Ubuntu

I've always hated the fact that on Ubuntu with the default themes, there's far too much space wasted. The buttons are too tall, the treeview wastes too much space so that if you're on eclipse or some other ide, you see a precious few items on the screen.

I've been trying to tweak it to no end - even looking to see if there are any ~/.gtkrc-2.0 tweaks. Found a few links such as this Making Eclipse look good on Linux - Max's blog - however, didn't really satisfy my need.

And so it stayed until today when I came across Clearlooks Compact Gnome Theme.

I love it - one more for my list of must-haves!

Enjoy symlinks and hardlinks on NTFS

Can't believe I didnt come across this before - if you've gotten used taming your hdd by creating links to folders and have been annoyed with the lack of symlinks and hardlinks on NTFS, then despair no more. I've been using Mark Russinovich's (of sysinternals fame) tool - junction.exe all this while and though it works great, have always wanted something that would integrate with Explorer too. For an in-depth discussion - read http://shell-shocked.org/article.php?id=284 Anyways, I'm extremely happy with NTFS Link - this will surely go into my list of "Must have tools - install immediately on a new machine" list :-)

Upgrade blues - upgrading to Firefox 3 final from Firefox RC 3

As evident from other posts here - have been keenly waiting for the FF 3 final. Imagine my surprise when the "Check updates" didnt find an upgrade! (I'm on FF3 rc3).

Anyway, so off I went to Mozilla.org and downloaded a copy of the final - and did my bit towards FF download day. Happily installed it - all defaults as usual. Install told me that it was installing into the same location as my current installation (c:\program files\mozilla firefox 3 beta 1 - that's where my FF3 install have been going  - all the way from b1 to b5 and then from rc1 to rc3 - so no surprise).

Well, installation completed successfully, and I started FF 3 - but my title bar still says Build 2008052906 - even the file version has the same build ID.

Something's up - don't know what yet - but has anyone else had a similar experience?

Desultory Monday...

This entry was posted using Its all text on Firefox 3.0 RC2 on Ubuntu Hardy heron, with emacs 23 snapshot as the editor. I love it :-)

Well, Its all Text is great if you hate typing into webforms with textboxes that make editing such a big pain in the butt.

Its great to see that Its All text has been updated to work with FF 3.0 now. The fun would be to see if this works on Windows with cygwin emacs as the editor. Had problems the last time I tried that - but that's been sometime ago now.

Today's been a desultory Monday. Spent sometime getting emacs snapshot with pretty fonts on my hardy. Its beautiful.

The next thing has been mostly scratching my head on hadoop. What I'd like to do is parse an access log and generate multiple outputs - ie single input of gobs of web access logs and multiple outputs - with say requests by country, popular pages, % of client browser and so on.

1. parse web log


2. pull out remote ips and use geo ips to find the originating country


3. pull out user agent field and figure out browser distribution.


4. Filter the requested resource and pull out only pages - find pages by popularity

Now there seem to be quite a number of ways of doing this -

• Code the whole thing in Java - and this is where I'm getting into analysis paralysis.
  Look at ways to generate multiple outputs from MapRed and then use Job and JobControl to setup the pipeline.


• Use Pig - Pig examples on the Pig overview page seem to suggest that this should be trivial with Pig.


• Use Cascading - seems to be doing the same thing - will need to do this in JRuby or Groovy though.

Will post an update once I get through the java route

VPN into Windows VPN Server from Ubuntu *Hardy* Intrepid

** Update 2008/11/17 **: Networkmanager is broken in intrepid. To get it  working had to install network manager from ppa as given here - http://www.ubuntu-forums.com/showpost.php?s=e0d93c09b8c340976477456593ac4cf7&p=6094870&postcount=5

Ok - this was easy - and while there's some resources on google, I had to figure out a few itty bitty things for my work VPN setup.

install

• network-manager-pptp


• pptp-linux

Restart network manager with

killall nm-applet
sudo /etc/init.d/dbus restart
nm-applet --sm-disable &

Configure VPN settings

Click on the network manager applet and click on VPN connections

1. Create a new VPN connection


2. Ensure that you select Refuse CHAP  in the authentication tab.


3. In the routing tab, you can give netmasks that need to go through VPN - for my work network, I have: 10.10.5.0/24 172.16.106.0/24

That's it. Now click on the Network applet, and connect to your VPN. In the authentication dialog, use <domain>\username and your windows domain password.

Scalability principles: Lessons from eBay

InfoQ: Scalability Best Practices: Lessons from eBay

Great article on Ebay scalability principles.

 

Drip...

Drip..

IMG_3370_crop, originally uploaded by Raghu Rajagopalan.

Drip....

Water droplets - dipping a toe in macro photography

he he - so I have a canon S3 IS - got it last year since it allows enough manual control while also having family friendly thingies like video :). Also, with the chdk hack, the S3 IS is good enough for me to experiment.

So, one of these long time itches has been to take a water droplet splash - you know, the immensely close up snaps where you see a single drop splashing...

Here's the snaps after two evenings of trial and error (mostly errors though) - feeling quite smug myself :)

Free subversion hosting - What's the best?

All - I've just signed up for an Assembla account - these folks provide free subversion hosting with a 500 meg space and unlimited spaces.

Will see how it goes.

Firefox 3 beta 5 released. Yahoo Mail is still broken.

Firefox 3 Beta 5 release today. Release notes and downloads here.

Installed it as soon as I got to know today morning and the first thing to check was whether Yahoo Mail still crashed. Initially, Yahoo Mail seemed to work alright for all of 50 seconds - quickly moving over items in inbox caused Firefox to crash :-(

Guess will wait for some more time. I'm sure there's a bug report somewhere on this - Yahoo mail was broken on Beta 2, got fixed in Beta 3, then was broken in Beta 4  and is still broken on Beta 5.

Will wait for it to be fixed - Any idea if this is a firefox issue or a Yahoo! issue? Seems odd that script can cause the browser to crash so badly.

Hardy heron - first impressions

He he :-) - finally got Ubuntu Hardy heron beta on my home and work laptop. first impressions below:

1. Wubi install from within windows is easy and works great. If after setting up so many boxes, I can go on and on about it, I'm sure that its great help for anyone who's on Windoze. I mean, the barrier to entry has never gone down so much.

2. I guess once you've installed via Wubi and configured your system to your liking, you can uninstall and take an image that you finall install to a dedicated partition - isn't that just awesome.

3. Comes installed with Firefox 3b4 -which is awesome. Given that FF crashes badly on yahoo, this might be a bummer for many people. Should probably have some first time customization that will let you install Opera.

4. Installation is super fast - took about 10 mins for wubi to install, reboot once, finish installation and reboot again. Grub default to Last selected would probably be a better idea.

The not so good

1. Wifi doesnt work out of the box - didn't on my Dell Inspiron 1501 or the Dell Latitude D620. Its the ye olde broadcom problem. This is really the BIGGEST turn off. Hope it will get fixed by the time the final release is out. Meanwhile, had to jump through hoops getting ndiswrapper in. I didn't go the broadcom fwcutter way since that only allows a 802.11b connection from what I read. I'm still not sure what fixed the issue - irrespective, I had to update the system and then things started working like a charm.

2. Compiz configuration isnt installed by default. If this is your first time on Ubuntu and you've come this way to see the awesome 3D desktop, then this is a bummer. Finding out what you need to do is a pain too.

I think that's all there is to it. Its great once wifi starts working normally.

Gnuplot, dstat - easy graphing on Linux

Recently, started fiddling around with how to monitor and graph performance data on linux boxes. Other than the usual tools like top and vmstat, which are either interactive (top) or too textual to do anything much.

First off, vmstat, doesnt lend itself well to graphing without additional scripts to lay out the data so tools like gnuplot can be used. Secondly, and more seriously, it doesn't include a timestamp in the output.

Looking around a bit found that dstat seems to be a good replacement to vmstat (and iostat) - and the generated data is consumable with gnuplot.

Here's a quick example of generating graphs for CPU user, system and idle times

dstat -tc 5 500 > dstat.raw

now fire up gnuplot and go ahead and plot it

gnuplot> set xdata time gnuplot> set timefmt "%s" gnuplot> set format x "%M:%S" gnuplot> plot "dstat.raw" using 1:2 title "User" with lines, "dstat.raw" using 1:3 title "Sys" with lines, "dstat.raw" using 1:4 title "Idle" using lines

To make gnuplot generat an output file, you need

gnuplot> set term png

gnuplot> set output "dstat.png"

gnuplot> replot



And you're done. here's the graph generated on my machine. There's loads more that you can do - and admittedly, you can do everything by dumping your file to excel. However, that doesn't lend itself well to a completely automated process. When you're doing performance testing and such like, you will likely repeat this enough number of times. Not having to do it manually helps big time!

Working with huge XML files - tools of the trade.

XMLStarlet is great for slicing and dicing huge XML files. Had a run in recently - had a 80 Mb XML file in a single line :D. Guess what, most editors that I tried balked and fell over. This was on a 2Gig Core2 Duo machine.

XMLSpy, vi, emacs, notepad++ all died - and trying to do something with a 80 Gig XML where the 80 gigs are on a single line isnt much fun. So the first order of business was to pretty print the XML. XMLstarlet worked great -

xmlstarlet fo file.xml > output.xml

and you're done.

The next order of business was that we needed to validate the XML document against a schema. Our first attempt was with Sun's multi schema validator (MSV). MSV does not validate the whole document but instead stops after a certain number of failures. So, MSV - out, XMLStarlet in. XMLStarlet can validate documents again W3C schema, DTD  or a RELAXNG schema.

xmlstarlet val --err --xsd schema.xsd input.xml >  errors.txt

And presto! - you get an error report that you can slice and dice with sed/awk or anything else at all.

XMLStarlet also allows you to write Xpaths to query the xml - however, I found the syntax too weird and round about. A better alternative is a perl based solutions - XSH2 - a command line xml editing shell. You can install it under cygwin and it supports basic command pipelining and redirection.

So go ahead and launch XSH. At your cygwin prompt

[~]xsh
---------------------------------------
 xsh - XML Editing Shell version 2.1.1
---------------------------------------

Copyright (c) 2002 Petr Pajas.
This is free software, you may use it and distribute it under
either the GNU GPL Version 2, or under the Perl Artistic License.
Using terminal type: Term::ReadLine::Gnu
Hint: Type `help' or `help | less' to get more help.
$scratch/>

Now, lets load up our document, type

$scratch/>$x:=open formatted.xml

Your prompt changes to

$x/>

So go ahead and try a few xpaths

$x/> ls /path/to/node

and XSH prints out the matching nodes. Now what if you need to create a document fragment of nodes matching a certain xpath? Piece of cake - do ahead

$x/> ls /path/to/node | tee fragment.xml

XSH2 has many, many more features - but this should be good enough to get you off the ground.

Yahoo! mail fixed for Firefox 3 beta 2

Used to get an error on opening Yahoo mail beta in Firefox 3 beta 2 - and had to switch to the plain 'ole yahoo mail. Here's the bug report

Was pleasantly surprised today morning to see that Yahoo! mail beta now works properly in FF3b2. Thanks!

Pesky little bash quoting problem

Have to admit it - this happens every time I sit down to write a some shell script that manipulates paths on windows (where path names often end up with spaces). Soon I find my nifty little script running into problems when it doesn't handle spaces properly and I find myself reading up on bash quoting rules once again...

Anyway, so this post is mostly for self reference :) and to put down some simple rules in the hope that writing it down will help committing it to memory.

The latest (mis)adventure was to make irfanview run under wine and a little script to allow irfanview to open a file provided on the command line. Irfanview being a windoze executable, its necessary to cd to the folder and then pass the file as argument to irfanview. Trivial isn't it....until I found that the script fell over when it got a path liek /path/to/a folder with spaces/image.jpeg.

#! /bin/bash
DIRNAME=$(dirname "$1")           # double quotes necessary - since $1 could have embedded spaces

FILENAME=$(basename "$1")
echo $DIRNAME
echo $FILENAME
cd "$DIRNAME"                               # once more, double quotes necessart
irfanview $FILENAME

Golden Rule

When passing a path as argument, always enclose in double quotes.

Firefox 3 Beta 2 on Ubuntu Gutsy

I'm having weird problems with firefox 3b2 on ubuntu gutsy - and as far as I can tell, I seem to be the only one. Did not find anything similar on ubuntu forums too.

Installed firefox 3 beta 2 from Mozilla to /usr/lib/firefox3b2 folder and created

lrwxrwxrwx 1 root root 27 2007-12-30 23:44 /usr/bin/firefox-3b2 -> /usr/lib/firefox3b2/firefox

When I launch firefox3b2, I get firefox alright, however, in the location bar if I type in a url and press Enter, nothing happens - absolutely nothing at all. I have to go and click the green arrow for the browser to open the URL. The search box is even weirder - neither the Enter key works nor does the mouse!

I'm at a loss - and nor can I find any similar experiences on forums etc - ideas welcome :D

SOLVED 01/20:Backed up my .mozilla folder and started firefox 3 b2 - no problems now :D

HOWTO: Access your machine from the internet without a static IP

For machines to be accessible on the internet, usually you need a static IP that's leased from your ISP so that when someone types in your IP address, so that packets can be routed over to your machine. However, getting a static ip is costly and for the most part, internet users have dynamic IP address that the ISP allocates each time an end user connects to the internet. Since the ip address keeps changing on each connection, there's no straightforward way to connect to the machine without knowing the IP address that's been allocated - or so it was at least till Dynamic DNS came along (it isnt new - has been around for ages, but for some reason isn't that well known)

Typically, when you type in www.google.com in your browser, your machine performs a DNS (Domain name service) lookup with the DNS servers from your ISP to find out the IP address corresponding to www.google.com. With DDNS (dynamic DNS) this is made to work with your dynamically allocated IP address also. Here's how it works

1. Register with a DDNS service provider. Service provider provide free accounts for personal use - go to www.dyndns.org


2. Once you've created your account, go ahead and set up your hostname. DDNS service providers will have some domains that you can choose from and you get to choose the host part. For a fee, you can also use a domain name of your choice.


3. If your set up has a router at your end, check your router administration page if it supports dynamic DNS. If it does, you need to enter the hostname, account and password. Everytime your router connects to the internet, it sends an update notification to the DDNS service notifying the new IP obtained from your ISP. The DDNS service takes care of sending update notifications to routers on the internet.


4. If you dont have a router, then download the DDNS client software from the service provider. Most DDNS providers have windows, mac and linux clients. These run on your machine and do the same thing - notify the DDNS service provider of your new IP whenever you establish a connection with your ISP.


5. If you've got all this set up, then you can reach your machine from the net - try ping <your host name>

If you're running Linux/Ubuntu, make sure your're running SSH service and try ssh <your host name>. If you have a router setup, then you will need an additional step - basically the DDNS name refers to your router IP - and not the machine behind the router that you wish to reach. You will also need to make sure that your machine has a static IP from your router. To set up your router, go to your router administration page.

1. Go to the LAN section and give a range of IPs outside of the static IP. Most routers have lan addresses like 192.168.x.y - 192.168.x.z. If you want your host to have an IP address of 192.168.1.100, then give a LAN range that does not include this IP - say 192.168.1.110 - 192.168.1.200.


2. Save and reboot your router.


3. Now go to your network settings and enter your static IP (192.168.1.100), netmask 255.255.255.255, gateway (usually 192.168.1.1).


4. Go to your router administration page and look for a section like virtual server - your router will allow you to forward packets received on a particular port to a host and port within your LAN. You will have to enter the external port (we'll use 22), the internal machine to forward (192.168.1.100) and the port to forward to (22). With this in place, any packets received on port 22 (ssh) on your router will be forwarded to the 192.168.1.100 machine on the ssh port.


5. Save and reboot your router.


6. Give it a spin.

From a different machine (or from the same one -doesnt matter), try out ssh <your host> and you should be able to login to your machine - via the internet.

Back in circulation

I'm on vacation in Bangalore, and guess what - fixing my home computer. Mostly things like lost drivers, screen resolution, cruft in the drives - its an old machine - a P4/512 Meg, but good enough for surfing the net.

Did a few fun things in the midst, and its been ages since I've added anything to this blog. Will summarize for now and put in longer posts with more details in cases someone's interested.

1. Fixed my windows C drive which was running out of space - used trusty old windirstat for that.


2. Set up wifi at home with ADSL modem from BSNL - MT800. Again, wasn't as straightforward as I'd thought.


3. Replaced old pcq linux 2006 with ubuntu gutsy - without losing stuff :D. Need to have /home in a separate partition, but otherwise this is a breeze.


4. Having fun with compiz-fusion. Its great - however, the documentation isnt easily locatable/consumable enough for first timers (me).


5. Set up DNS caching proxy on my linux box - has improved my net/web experience a 100 fold. Was a piece of cake too.


6. Set up Dynamic DNS and remote SSH access to my box - this has been the single most important utility/maintainance action.

More later.