Hardy heron - first impressions

He he :-) - finally got Ubuntu Hardy heron beta on my home and work laptop. first impressions below:

1. Wubi install from within windows is easy and works great. If after setting up so many boxes, I can go on and on about it, I'm sure that its great help for anyone who's on Windoze. I mean, the barrier to entry has never gone down so much.

2. I guess once you've installed via Wubi and configured your system to your liking, you can uninstall and take an image that you finall install to a dedicated partition - isn't that just awesome.

3. Comes installed with Firefox 3b4 -which is awesome. Given that FF crashes badly on yahoo, this might be a bummer for many people. Should probably have some first time customization that will let you install Opera.

4. Installation is super fast - took about 10 mins for wubi to install, reboot once, finish installation and reboot again. Grub default to Last selected would probably be a better idea.

The not so good

1. Wifi doesnt work out of the box - didn't on my Dell Inspiron 1501 or the Dell Latitude D620. Its the ye olde broadcom problem. This is really the BIGGEST turn off. Hope it will get fixed by the time the final release is out. Meanwhile, had to jump through hoops getting ndiswrapper in. I didn't go the broadcom fwcutter way since that only allows a 802.11b connection from what I read. I'm still not sure what fixed the issue - irrespective, I had to update the system and then things started working like a charm.

2. Compiz configuration isnt installed by default. If this is your first time on Ubuntu and you've come this way to see the awesome 3D desktop, then this is a bummer. Finding out what you need to do is a pain too.

I think that's all there is to it. Its great once wifi starts working normally.

Gnuplot, dstat - easy graphing on Linux

Recently, started fiddling around with how to monitor and graph performance data on linux boxes. Other than the usual tools like top and vmstat, which are either interactive (top) or too textual to do anything much.

First off, vmstat, doesnt lend itself well to graphing without additional scripts to lay out the data so tools like gnuplot can be used. Secondly, and more seriously, it doesn't include a timestamp in the output.

Looking around a bit found that dstat seems to be a good replacement to vmstat (and iostat) - and the generated data is consumable with gnuplot.

Here's a quick example of generating graphs for CPU user, system and idle times

dstat -tc 5 500 > dstat.raw

now fire up gnuplot and go ahead and plot it

gnuplot> set xdata time gnuplot> set timefmt "%s" gnuplot> set format x "%M:%S" gnuplot> plot "dstat.raw" using 1:2 title "User" with lines, "dstat.raw" using 1:3 title "Sys" with lines, "dstat.raw" using 1:4 title "Idle" using lines

To make gnuplot generat an output file, you need

gnuplot> set term png

gnuplot> set output "dstat.png"

gnuplot> replot



And you're done. here's the graph generated on my machine. There's loads more that you can do - and admittedly, you can do everything by dumping your file to excel. However, that doesn't lend itself well to a completely automated process. When you're doing performance testing and such like, you will likely repeat this enough number of times. Not having to do it manually helps big time!

Working with huge XML files - tools of the trade.

XMLStarlet is great for slicing and dicing huge XML files. Had a run in recently - had a 80 Mb XML file in a single line :D. Guess what, most editors that I tried balked and fell over. This was on a 2Gig Core2 Duo machine.

XMLSpy, vi, emacs, notepad++ all died - and trying to do something with a 80 Gig XML where the 80 gigs are on a single line isnt much fun. So the first order of business was to pretty print the XML. XMLstarlet worked great -

xmlstarlet fo file.xml > output.xml

and you're done.

The next order of business was that we needed to validate the XML document against a schema. Our first attempt was with Sun's multi schema validator (MSV). MSV does not validate the whole document but instead stops after a certain number of failures. So, MSV - out, XMLStarlet in. XMLStarlet can validate documents again W3C schema, DTD  or a RELAXNG schema.

xmlstarlet val --err --xsd schema.xsd input.xml >  errors.txt

And presto! - you get an error report that you can slice and dice with sed/awk or anything else at all.

XMLStarlet also allows you to write Xpaths to query the xml - however, I found the syntax too weird and round about. A better alternative is a perl based solutions - XSH2 - a command line xml editing shell. You can install it under cygwin and it supports basic command pipelining and redirection.

So go ahead and launch XSH. At your cygwin prompt

[~]xsh
---------------------------------------
 xsh - XML Editing Shell version 2.1.1
---------------------------------------

Copyright (c) 2002 Petr Pajas.
This is free software, you may use it and distribute it under
either the GNU GPL Version 2, or under the Perl Artistic License.
Using terminal type: Term::ReadLine::Gnu
Hint: Type `help' or `help | less' to get more help.
$scratch/>

Now, lets load up our document, type

$scratch/>$x:=open formatted.xml

Your prompt changes to

$x/>

So go ahead and try a few xpaths

$x/> ls /path/to/node

and XSH prints out the matching nodes. Now what if you need to create a document fragment of nodes matching a certain xpath? Piece of cake - do ahead

$x/> ls /path/to/node | tee fragment.xml

XSH2 has many, many more features - but this should be good enough to get you off the ground.