Friday, March 28, 2008

Gnuplot, dstat - easy graphing on Linux

Recently, started fiddling around with how to monitor and graph performance data on linux boxes. Other than the usual tools like top and vmstat, which are either interactive (top) or too textual to do anything much.

First off, vmstat, doesnt lend itself well to graphing without additional scripts to lay out the data so tools like gnuplot can be used. Secondly, and more seriously, it doesn't include a timestamp in the output.

Looking around a bit found that dstat seems to be a good replacement to vmstat (and iostat) - and the generated data is consumable with gnuplot.

Here's a quick example of generating graphs for CPU user, system and idle times
dstat -tc 5 500 > dstat.raw

now fire up gnuplot and go ahead and plot it
gnuplot> set xdata time gnuplot> set timefmt "%s" gnuplot> set format x "%M:%S" gnuplot> plot "dstat.raw" using 1:2 title "User" with lines, "dstat.raw" using 1:3 title "Sys" with lines, "dstat.raw" using 1:4 title "Idle" using lines

To make gnuplot generat an output file, you need

gnuplot> set term png

gnuplot> set output "dstat.png"

gnuplot> replot

dstat png - User, system and Idle times

And you're done. here's the graph generated on my machine. There's loads more that you can do - and admittedly, you can do everything by dumping your file to excel. However, that doesn't lend itself well to a completely automated process. When you're doing performance testing and such like, you will likely repeat this enough number of times. Not having to do it manually helps big time!

Thursday, March 27, 2008

Working with huge XML files - tools of the trade.

XMLStarlet is great for slicing and dicing huge XML files. Had a run in recently - had a 80 Mb XML file in a single line :D. Guess what, most editors that I tried balked and fell over. This was on a 2Gig Core2 Duo machine.

XMLSpy, vi, emacs, notepad++ all died - and trying to do something with a 80 Gig XML where the 80 gigs are on a single line isnt much fun. So the first order of business was to pretty print the XML. XMLstarlet worked great -
xmlstarlet fo file.xml > output.xml

and you're done.

The next order of business was that we needed to validate the XML document against a schema. Our first attempt was with Sun's multi schema validator (MSV). MSV does not validate the whole document but instead stops after a certain number of failures. So, MSV - out, XMLStarlet in. XMLStarlet can validate documents again W3C schema, DTD  or a RELAXNG schema.
xmlstarlet val --err --xsd schema.xsd input.xml >  errors.txt

And presto! - you get an error report that you can slice and dice with sed/awk or anything else at all.

XMLStarlet also allows you to write Xpaths to query the xml - however, I found the syntax too weird and round about. A better alternative is a perl based solutions - XSH2 - a command line xml editing shell. You can install it under cygwin and it supports basic command pipelining and redirection.

So go ahead and launch XSH. At your cygwin prompt
[~]xsh
---------------------------------------
 xsh - XML Editing Shell version 2.1.1
---------------------------------------

Copyright (c) 2002 Petr Pajas.
This is free software, you may use it and distribute it under
either the GNU GPL Version 2, or under the Perl Artistic License.
Using terminal type: Term::ReadLine::Gnu
Hint: Type `help' or `help | less' to get more help.
$scratch/>

Now, lets load up our document, type
$scratch/>$x:=open formatted.xml

Your prompt changes to
$x/>

So go ahead and try a few xpaths
$x/> ls /path/to/node

and XSH prints out the matching nodes. Now what if you need to create a document fragment of nodes matching a certain xpath? Piece of cake - do ahead
$x/> ls /path/to/node | tee fragment.xml

XSH2 has many, many more features - but this should be good enough to get you off the ground.

Saturday, February 09, 2008

Yahoo! mail fixed for Firefox 3 beta 2

Used to get an error on opening Yahoo mail beta in Firefox 3 beta 2 - and had to switch to the plain 'ole yahoo mail. Here's the bug report

Was pleasantly surprised today morning to see that Yahoo! mail beta now works properly in FF3b2. Thanks!

Wednesday, January 23, 2008

Pesky little bash quoting problem

Have to admit it - this happens every time I sit down to write a some shell script that manipulates paths on windows (where path names often end up with spaces). Soon I find my nifty little script running into problems when it doesn't handle spaces properly and I find myself reading up on bash quoting rules once again...

Anyway, so this post is mostly for self reference :) and to put down some simple rules in the hope that writing it down will help committing it to memory.

The latest (mis)adventure was to make irfanview run under wine and a little script to allow irfanview to open a file provided on the command line. Irfanview being a windoze executable, its necessary to cd to the folder and then pass the file as argument to irfanview. Trivial isn't it....until I found that the script fell over when it got a path liek /path/to/a folder with spaces/image.jpeg.

#! /bin/bash
DIRNAME=$(dirname "$1")           # double quotes necessary - since $1 could have embedded spaces

FILENAME=$(basename "$1")
echo $DIRNAME
echo $FILENAME
cd "$DIRNAME"                               # once more, double quotes necessart
irfanview $FILENAME

Golden Rule

When passing a path as argument, always enclose in double quotes.

Thursday, January 10, 2008

Firefox 3 Beta 2 on Ubuntu Gutsy

I'm having weird problems with firefox 3b2 on ubuntu gutsy - and as far as I can tell, I seem to be the only one. Did not find anything similar on ubuntu forums too.

Installed firefox 3 beta 2 from Mozilla to /usr/lib/firefox3b2 folder and created
lrwxrwxrwx 1 root root 27 2007-12-30 23:44 /usr/bin/firefox-3b2 -> /usr/lib/firefox3b2/firefox

When I launch firefox3b2, I get firefox alright, however, in the location bar if I type in a url and press Enter, nothing happens - absolutely nothing at all. I have to go and click the green arrow for the browser to open the URL. The search box is even weirder - neither the Enter key works nor does the mouse!

I'm at a loss - and nor can I find any similar experiences on forums etc - ideas welcome :D

SOLVED 01/20:Backed up my .mozilla folder and started firefox 3 b2 - no problems now :D

HOWTO: Access your machine from the internet without a static IP

For machines to be accessible on the internet, usually you need a static IP that's leased from your ISP so that when someone types in your IP address, so that packets can be routed over to your machine. However, getting a static ip is costly and for the most part, internet users have dynamic IP address that the ISP allocates each time an end user connects to the internet. Since the ip address keeps changing on each connection, there's no straightforward way to connect to the machine without knowing the IP address that's been allocated - or so it was at least till Dynamic DNS came along (it isnt new - has been around for ages, but for some reason isn't that well known)

Typically, when you type in www.google.com in your browser, your machine performs a DNS (Domain name service) lookup with the DNS servers from your ISP to find out the IP address corresponding to www.google.com. With DDNS (dynamic DNS) this is made to work with your dynamically allocated IP address also. Here's how it works

  1. Register with a DDNS service provider. Service provider provide free accounts for personal use - go to www.dyndns.org

  2. Once you've created your account, go ahead and set up your hostname. DDNS service providers will have some domains that you can choose from and you get to choose the host part. For a fee, you can also use a domain name of your choice.

  3. If your set up has a router at your end, check your router administration page if it supports dynamic DNS. If it does, you need to enter the hostname, account and password. Everytime your router connects to the internet, it sends an update notification to the DDNS service notifying the new IP obtained from your ISP. The DDNS service takes care of sending update notifications to routers on the internet.

  4. If you dont have a router, then download the DDNS client software from the service provider. Most DDNS providers have windows, mac and linux clients. These run on your machine and do the same thing - notify the DDNS service provider of your new IP whenever you establish a connection with your ISP.

  5. If you've got all this set up, then you can reach your machine from the net - try ping <your host name>


If you're running Linux/Ubuntu, make sure your're running SSH service and try ssh <your host name>. If you have a router setup, then you will need an additional step - basically the DDNS name refers to your router IP - and not the machine behind the router that you wish to reach. You will also need to make sure that your machine has a static IP from your router. To set up your router, go to your router administration page.

  1. Go to the LAN section and give a range of IPs outside of the static IP. Most routers have lan addresses like 192.168.x.y - 192.168.x.z. If you want your host to have an IP address of 192.168.1.100, then give a LAN range that does not include this IP - say 192.168.1.110 - 192.168.1.200.

  2. Save and reboot your router.

  3. Now go to your network settings and enter your static IP (192.168.1.100), netmask 255.255.255.255, gateway (usually 192.168.1.1).

  4. Go to your router administration page and look for a section like virtual server - your router will allow you to forward packets received on a particular port to a host and port within your LAN. You will have to enter the external port (we'll use 22), the internal machine to forward (192.168.1.100) and the port to forward to (22). With this in place, any packets received on port 22 (ssh) on your router will be forwarded to the 192.168.1.100 machine on the ssh port.

  5. Save and reboot your router.

  6. Give it a spin.


From a different machine (or from the same one -doesnt matter), try out ssh <your host> and you should be able to login to your machine - via the internet.

Thursday, January 03, 2008

Back in circulation

I'm on vacation in Bangalore, and guess what - fixing my home computer. Mostly things like lost drivers, screen resolution, cruft in the drives - its an old machine - a P4/512 Meg, but good enough for surfing the net.

Did a few fun things in the midst, and its been ages since I've added anything to this blog. Will summarize for now and put in longer posts with more details in cases someone's interested.

  1. Fixed my windows C drive which was running out of space - used trusty old windirstat for that.

  2. Set up wifi at home with ADSL modem from BSNL - MT800. Again, wasn't as straightforward as I'd thought.

  3. Replaced old pcq linux 2006 with ubuntu gutsy - without losing stuff :D. Need to have /home in a separate partition, but otherwise this is a breeze.

  4. Having fun with compiz-fusion. Its great - however, the documentation isnt easily locatable/consumable enough for first timers (me).

  5. Set up DNS caching proxy on my linux box - has improved my net/web experience a 100 fold. Was a piece of cake too.

  6. Set up Dynamic DNS and remote SSH access to my box - this has been the single most important utility/maintainance action.


More later.

Tuesday, July 03, 2007

Sluggish Firefox - and what a hog!

Okay - for the past few days I've been irritated with the browsing experience at home. Pages (google reader, Yahoo mail etc seemed inexplicably slower than before - but they'd load alright - just seemed that teeny weeny bit slower that's enough to leave you suspicious).

I first suspected my ISP (verizon) for frequent dropped connections (saw the DSL modem lights reset a couple of times a day), then my wifi modem (not a high end one), then spyware/malware. So after the usual barrage of tests - wifi interference sources/anti virus/anti spyware/cable tests etc, I still hadn't nailed it.

Finally, used procexp ( you can use plain 'ole Task manager too - this is just flashier) and saw that FF had 330 MB of RAM with 3 or 4 tabs. Also, just plain clicking on a text box was slow, typing into a text field would echo characters after a noticeable delay - so this was definitely a browser problem.

When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.


-- Sherlock Holmes


Ain't that an apt quote??? I love the addons I have and probably I have one too many. This page on problematic addons  was a life-saver - after disabling a bunch of infrequently used addons (stumble upon toolbar, google toolbar, browser sync, adblock filterset g, foxy tunes and some more), I'm back in browsing heaven. The only addons I have now enabled are




  • diigo toolbar

  • all in one gestures

  • Adblock

  • Flashgot

  • Piclens


What a relief!

BTW, addons are also the latest attack vector. So be wary of who you let into your browser!

Thursday, June 28, 2007

Piclens - full screen slideshows with flickr (and others)

discovered Piclens

Its a great addin for firefox - and integrates with Flickr to give you full screen slideshows a'la Picasa slideshows on your machine!! Its a bit tricky to figure out how to get it to work - Just hover any picture on any page  and click on the blue bubbly overlay button that appears

Python, cygwin, TurboGears, mysql hell

Ok - here goes - I've always liked python, though definitely a noob. I was interested in python on the web and after a bit of googling, seems like TurboGears is the way to go.

First things first - decided to use mysql as the database (already have it on my machine and didn't want to install one more database (postgresql/sqlite). Now it turns out that MySQL doesnt have a cygwin package. More googling - mysql server can't run on cygwin due to something to do with pthreads. You can compile the mysql client on cygwin though.

That's what I decided to do - grabbed the Linux tar.gz source from mysql.com, got it into a directory and ran ./configure --without-server, followed by make && make install. All went through fine -other than the fact that it was time consuming and pretty boring (more so since I had to download and install gcc, bintools first in cygwin)

I thought I'd got through the hard part and what remained was to install python-MySQLdb package. Off I went to

easy_install MySQLdb

No luck there - package build failed with missing library -lmysqlclient_r. Turns out that the 'thread-safe' version of mysql client (mysqlclient_r) is preferred but the mysql build doesnt build this by default. What a shame!!!

Anyway, so I wasnt going to redo the whole mysql client library build again - more README files and googling later, grabbed mysql-python-1.2.2 tarball, got it into a folder, edited site.cfg and changed 'threadsafe=false'. The next run on python setup.py build worked properly, with the python mysql linking against the non threadsafe mysqlclient library.

Think troubles are over yet? No way.

Off I went to test - started the python interpreter and did a import MySQLdb, and got a Permission Denied in some 'egg' file!!! What the heck are these egg files anyway. Well, I did'nt have much of a clue and more google later got educated that these are install packages used by the easy install system. The more I looked, the more it seems that the easy install is anything but easy :(. Anyway, this one had me floored - since I couldnt get to the line of source where the error was and had no clue how to view the contents of an 'egg' (they're zips - but I didnt know that and very helpfully there's hardly anyplace where they say that they're zips with the extension of egg!!! Baaah!! - why couldn't they just use .zip?)

More and more hard googling - and this time the info's really sketchy till eventually found a post from a guy who asked the exact same question. Guess what, the easy_install system unzips the eggs to some folder (pointed by PYTHON_EGG_CACHE env var) and there I needed to do a chmod a+x on the _mysql.dll. Well so I did echo $PYTHON_EGG_CACHE and the var isnt set!!! Admittedly at this point, I'm not looking sharp either - what started out as a quick spin has become a quagmire of installation issues - but I'll be damned if I let it sink me!!! Eventually, had the Eureka moment and checked ~/.python-eggs and sure enough found the truant _mysql.dll. Quick chmod a+x and presto - import MySQLdb worked!!! YAHOOOOOOOOOOOOOOOOOOOO!!!!

And now back to where I started - went back to turbogears, did a tg-admin quickstart, setup a mysql database and started with python start-testproject.py. Guess what - no luck yet - turns out that mysql cant client to my windows server with a socket.

More googling - and this time its really desperate - and more enlightenment - the windows mysqld doesnt do unix sockets - how do I force tcp/ip? Simple - use 127.0.0.1 as hostname in the connection settings instead of localhost!! Finally something that was easy to fix. Finally after 8 hours of on and off hacking away at installation issues I'm glad to see a turbogears web page.

Bottomline: Python's great, from the looks of it, turbo gears seems well designed. Mysql is a great database - a cygwin native server would be great - or atleast a client package. But if one has to run through all these hoops just go get a 'quick spin' then adoption's going to be difficult.

I havent tried RoR - but has someone tried a similar thing on cygwin (cygwin, ruby, RoR, mysql)? How does the experience compare - is it any easier to get off the ground?

Thursday, May 24, 2007

Maven Emma plugin - filters dont work

Using v0.5 of the Maven emma plugin. Found that if you specify an include/exclude filter, no classes are instrumented....On digging a little deeper under the covers, it seems that there's a bug with the Maven emma plugin - I've edited the plugin's jelly script and using it here.

Posted the bug and the fix to Maven's sourceforge developer forum. You can find it here

The fixed plugin.jelly's here - Maven emma plugin 0.5 fixed

Friday, May 04, 2007

Building a FC6 server for Java development

It has come upon me to build a FC6 server with a Java build environment that is going to serve as the build machine. This is the first time I'm coming to Fedora - always having used Debian in the past.

  1. OS installation was smooth. I installed with KDE since some folks prefer the GUI.

  2. During the installation, I chose Java Development - and all the gcj components got installed. Then post installation, I downloaded JDK from Sun and installed it too. The (gcj) tomcat of course didnt work very well in this mess. Fixing it was easy - just fired up yum via Add or Remove Programs and removed gcj - which removed all the other gcj tools and libraries too.

  3. Setting the host name - unexpectedly, had trouble doing this - finally used system-config-network.

  4. Environment variables - again a little digging around to figure out how to set environment variables so that they are effective for all users on the system - essentially JAVA_HOME, JRE_HOME, CATALINA_HOME, MAVEN_HOME etc. Finally found that this is best set in /etc/environment where it works for all users.

  5. Downloaded java tools and libraries - housed tomcat and maven under /usr/lib/java/maven-xxx and /usr/lib/java/apache-tomcat-x.xx and created symlinks. Placed symlinks to tomcat's startup.sh and shutdown.sh in /usr/bin. Placing a symlink to maven doesnt work - but an alias works as well. To make it work for all users, simply put it in /etc/profile
    alias maven=/usr/lib/java/maven/bin/maven


  6. Created a group java and added users to the group. Set permissions on the maven installation folder so that java group has write access (this is so that maven plugin:download for additional plugins works properly and can write to the maven plugins folder)


That's all there is to it. Just make sure that you do everything as the 'application' user - dont do it with your account or the root account.

Tuesday, May 01, 2007

Ant - Debugging classpath

Here's a nice tip on debugging classpaths in Ant: http://www.javalobby.org/java/forums/t71033.html

Essentially,
<?xml version="1.0"?>
<project name="project" default="default">
<property name="lib" value="web/WEB-INF/lib"/>
<property name="src" value="src"/>
<property name="dist" value="dist"/>

<path id="classpath">
<fileset dir="${lib}">
<include name="**/*.jar"/>
</fileset>
</path>

<target name="default" description="--> description">
<javac srcdir="${src}" destdir="${dist}">
<classpath refid="classpath"/>
</javac>
</target>

</project>

Sunday, April 22, 2007

Cobertura vs EMMA

This is a rant against the Cobertura maven integration - and to a certain extent on Cobertura also - and hopefully an objective one ;). Hopefully this'll help someone select between the two big (open source) coverage tools out there.

We've been using Cobertura at work till now and its done its job nicely - the reports look great and the maven 1.x integration, while not neat, is functional. While we knew that someday we were going to have to merge coverage data and have a single report across multiple test methods (junit, selenium tests and manual), cobertura documentation stated that this was possible and so we weren't really bothered.

Thought I'd give it a whirl and set it up - and that's when the trouble started. Atleast, with the maven integration.

First of all - there's no way to just instrument code. You can generate the report (which will instrument classes and run the tests) but if you just want to instrument classes so that the final deployable contains instrumented classes, its a no go with the maven plugin tools.

Obviously, no point giving it up there - so thought I'd just include the ant tasks and go the ant way in my maven goal. Turns out that there's no 'plugin init' kind of goal that can be called post build:start that will set up dependencies and import the cobertura ant tasks. You have to do it all yourself.  Fine - went that way too - so now my maven.xml uses the cobertura ant tasks and finally I'm able to generate an instrumented build. YIPPPPPPPEEEEEE.... or wait...lets' just make sure that this thing works...

Does it?

Turns out - no - it doesnt - so I dropped the WAR into tomcat and accessed the login page of the application and then shut down tomcat nicely. There's even a cobertura.ser created in the tomcat bin folder and I'm thinking that probably this will all work together finally...

So I go ahead, tweak my Maven.xml further with a coverage task that will merge the data from the junit runs and servlet container runs. Turn the switch on... and lo and behold...Exception reading the merged data file. Back to google and after hunting around for sometime found this Bug while merging reports

So finally I'm ready to give up cobertura and give Emma a try...and it couldnt have been better...

1. Goals are nicely setup

2. You can init the emma system with the emma:init goal and then use ant tasks if you want flexibility for doing things like merging reports.

3. The merging works :))

One sticky issue that I did come up  with was that the for the same source and test cases the coverage reported by cobertura and emma differ widely. With Cobertura, we were at 40% coverage while with EMMA, the number's up to 60% coverage - and while EMMA has some literature on how it does things - I'd be glad if someone did explain why or how the reported numbers could be so different for the same base code and unit test suite?

Tuesday, April 17, 2007

Tulip festival at Skagit county

We visited the Tulip festival at Skagit valley. Its hard to describe in words - and a picture's worth a thousand words - so here's a bunch of pics - let them do the talking :).

Anyway, had a lovely weekend driving up to Mount Vernon with friends and had a memorable weekend.







TulipFestival

Saturday, March 17, 2007

Ninotech Path Copy 4

Ninotech Path Copy 4
Novell Cool Solutions: Cool Tool

Was looking for a shell extension to copy filenames from explorer. Thought there'd be a dime a dozen - and it turns out that
a) they have 20 other things that I dont want.
b) Its trial ware.

Finally after searching high and low, came across this and it's GREAT!


powered by performancing firefox

Thursday, March 15, 2007

Production Eclipse Configuration

Part of the game of getting code out that conforms to coding standards without having to go through the "One Big spring cleaning" exercise is configuring and centralizing the tools.

At a minimum, it helps to have a central Eclipse installation that has all the tools configured and setup - so each person doesn't have to do it. Alternatively, there should be a reference set of plugins and their configuration files available in the source control repository.  Here's my eclipse configuration

Wednesday, March 14, 2007

Tips for using Eclipse

http://www.aspectprogrammer.org/blogs/adrian/2006/02/tips_for_using.html

Great Article - full of very very useful tips!

Firefox - not so obvious search

I discovered this - totally by accident.

In Firefox, you can use / (forward slash) to search for text OR links - and if its a link, then you can follow the link just by clicking Enter!  That is way cool - especially when you're on a laptop and cant/dont have your mouse around. I've always find it a pain to do a "link find"  - ' (single quote) and a text find - ctrl-f separately...Never knew that there was a shortcut that did both and what more - its way natural if you're used to 'less'!!

To complete the keys for Firefox mouseless browsing,

  • space -  page down

  • shift - space - page up

Sunday, February 25, 2007

Performancing and Wordpress - initial imression - very nice!


Performancing | Firefox Add-ons | Mozilla Corporation
Performancing for Firefox is a full featured blog editor that sits right in your Firefox browser and lets you post to your blog easily. You can drag and drop formatted text from the page you happen to be browsing, and take notes as well as post to your blog.


powered by performancing firefox

Saturday, February 24, 2007

Diigo - a hidden gem

If you use del.icio.us for online bookmarking and like google's notepad for online research, then suggest that you take a look at Diigo's offering. It's all those rolled in one and then some.

I've been using it for quite some time now (after trying out del.icio.us and google notepad) and I'm not budging! Recently I was doing a bunch of research online and diigo excelled!

  1. Online bookmarking - will also simultaneously post to your del.icio.account

  2. Firefox toolbar

  3. tagging

  4. Web clipping

  5. Web annotation


Now to "hidden" - I've been on Diigo more than 8 months or so now - and yet I'm still to meet someone who knew about Diigo!!
Diigo rocks!

Imaging the tenth dimension

Amazing flash movie!

Great flash movie explaining dimensions 4 (time), 5, 6 all the way uptil 10. And why they stop after 10!

I've never come across a clearer explanation ever and the great visualization does the trick.

Instant messenger client utopia

GAIM...

I hate running 3 different chat clients (yahoo, msn, gtalk) on my machine - and that's just 3x memory, startup programs and useless adversting too many.

And I hate msn and yahoo's advertising that's built into their clients...esp on MSN which insists on opening a mini page on login...

Use GAIM - and do yourself a favour. Of course that's if you don't need the very specialized features :)).

Installation is a breeze and use plugins you like and avoid the bloat....The best plugins I like are guifications and text replace. Lastly, if you keep moving between machines as much as I do, then its a breeze to retain your settings between all your machines.

Friday, April 29, 2005

XSLT Analog to sysouts

I'm on a high. I've been stuck with this problem of trying to understand a HUGE xslt that operates on an even bigger XML. I sorely needed something that will let me trace through the xslt execution to understand the flow.

Tried a couple of IDEs - Stylus Studio (free edition) and Marrowsoft XSelerator. Stylus studio did a graceful exit, Xselerator went purple in the face and died a gruesome death :-(

Hmm... so after sometime I was wondering if I could annotate the XSL output with information on the templates matched it would atleast help partway. I was thinking of perl/C#/regular expressions and then suddenly the penny dropped "for each xsl:template node, include a comment with the template match/mode" - Hang on!!! looks like that sounds like a job for XSLT....

Anyway, there are a couple of quirks - the first one you hit will be when you try to output a template like this


<xsl:stylesheet version="1.0"  xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >
<xsl:template match="xsl:stylesheet" >
<!-- generate an output xsl:stylesheet node -->
<xsl:stylesheet></xsl:stylesheet>
</xsl:template>
</xsl:stylesheet>


Oops! The XSLT processor cribs (and with good reason too)! It doesn't know which xsl:template is for the current stylesheet vs which is intended to be output to the result document.There are a couple of approaches around this. One is to use xsl:element like this

    <xml:namespace prefix = xsl />
    <xsl:element name="xsl:template">
    </xsl:element>
  

But this results in enormously wordy documents. Thankfully there's a neater way out. You use something called . Basically, what it does is that it allows you to use a dummy namespace in your xslt. You set up the dummy namespace (let's say genxsl) to map to a real namespace in the result document (xsl). Then you basically use the dummy namespace in your XSLT. However, when generating output, the processor will replace all references to the dummy namespace in the result document with references to the real namespace. For ex.


<xsl:stylesheet version="1.0"  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:gen="http://www.w3.org/1999/XSL/Transform/2">
    <xsl:namespace-alias stylesheet-prefix="gen" result-prefix="xsl"/>
    <xsl:template match="xsl:stylesheet">
        <gen:stylesheet>
            <xsl:for-each select="@*">
                <xsl:attribute name="{name(.)}">
                    <xsl:value-of select="."/>
                </xsl:attribute>
            </xsl:for-each>
            <xsl:apply-templates></xsl:apply-templates>
            <xsl:if test="not(xsl:template[@name='pseudo-xpath-to-current-node'])" &gt;
                <xsl:text>
                    &amp;#10;&amp;#10;
                </xsl:text> <xsl:copy-of select="document('')/xsl:stylesheet/xsl:template[ @name='pseudo-xpath-to-current-node']"/>
                <xsl:text>
                    &amp;#10;&amp;#10;
                </xsl:text>
            </xsl:if>
        </gen:stylesheet>
    </xsl:template>
</xsl:stylesheet>



Note the usage of xsl:namespace-alias and the code for generating an xsl:stylesheet element in the result document.

I've included my efforts here - along with a simple books.xml, a books.xsl which generates a table and finally an instrument.xsl that instruments books.xsl to generate an instrumented version. Transforming books.xml with the instrumented xslt generates output that annotated with custom nodes that highlight which template got called when.

After I was mostly done with the code, I came across an article in IBM developerWorks which discusses the same topic. Rather than cover the same material again, you can find the article here. Stuff that's different is that I generate custom nodes (which I thought would be useful to view in XML IDE which allow a hierarchical display). I've also shamelessly borrowed the code to generate the Xpath of the node (part of what you see in the snippet).


I'm on a high. I've been stuck with this problem o...

I'm on a high. I've been stuck with this problem of trying to understand a HUGE xslt that operates on an even bigger XML. I sorely needed something that will let me trace through the xslt execution to understand the flow.

Tried a couple of IDEs - Stylus Studio (free edition) and Marrowsoft XSelerator. Stylus studio did a graceful exit, Xselerator went purple in the face and died a gruesome death :-(

Hmm... so after sometime I was wondering if I could annotate the XSL output with information on the templates matched it would atleast help partway. I was thinking of perl/C#/regular expressions and then suddenly the penny dropped "for each xsl:template node, include a comment with the template match/mode" - Hang on!!! looks like that sounds like a job for XSLT....

Anyway, there are a couple of quirks - the first one you hit will be when you try to output a template like this
<xsl:stylesheet version="1.0"  xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >

<xsl:template match="xsl:stylesheet" >

	<!-- generate an output xsl:stylesheet node -->

		<xsl:stylesheet>

		</xsl:stylesheet>

	</xsl:template>

</xsl:stylesheet>

Oops! The XSLT processor cribs (and with good reason too)! It doesn't know which xsl:template is for the current stylesheet vs which is intended to be output to the result document.There are a couple of approaches around this. One is to use xsl:element like this
    <xml:namespace prefix = xsl />

	<xsl:element name="xsl:template"></xsl:element>

But this results in enormously wordy documents. Thankfully there's a neater way out. You use something called . Basically, what it does is that it allows you to use a dummy namespace in your xslt. You set up the dummy namespace (let's say genxsl) to map to a real namespace in the result document (xsl). Then you basically use the dummy namespace in your XSLT. However, when generating output, the processor will replace all references to the dummy namespace in the result document with references to the real namespace. For ex.
<xsl:stylesheet version="1.0"

	xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

	xmlns:gen="http://www.w3.org/1999/XSL/Transform/2">

<xsl:namespace-alias stylesheet-prefix="gen" result-prefix="xsl"/>

<xsl:template match="xsl:stylesheet">

	<gen:stylesheet>

		<xsl:for-each select="@*">

		<xsl:attribute name="{name(.)}">

		<xsl:value-of select="."/>

		</xsl:attribute>

	</xsl:for-each>

	<xsl:apply-templates></xsl:apply-templates>

	<xsl:if test="not(xsl:template[@name='pseudo-xpath-to-current-node'])" >

	<xsl:text></xsl:text>

	<xsl:copy-of select="document('')/xsl:stylesheet/xsl:template[ @name='pseudo-xpath-to-current-node']"/>

	<xsl:text></xsl:text>

	</xsl:if>

</gen:stylesheet>

</xsl:template>

Note the usage of xsl:namespace-alias and the code for generating an xsl:stylesheet element in the result document.

I've included my efforts here - along with a simple books.xml, a books.xsl which generates a table and finally an instrument.xsl that instruments books.xsl to generate an instrumented version. Transforming books.xml with the instrumented xslt generates output that annotated with custom nodes that highlight which template got called when.

After I was mostly done with the code, I came across an article in IBM developerWorks which discusses the same topic. Rather than cover the same material again, you can find the article here. Stuff that's different is that I generate custom nodes (which I thought would be useful to view in XML IDE which allow a hierarchical display). I've also shamelessly borrowed the code to generate the Xpath of the node (part of what you see in the snippet).

Thursday, April 14, 2005


I love the premise of Test Driven Development - I've even used it a few times in the line of duty ;-) (I do have to admit, I've been naughty and left out the step of  seeing the test case fail a few times )...Anyway, I end up working on web applications more often than not and while you can use TDD for your class libraries, a web app is a totally different animal. The fact that you can use TDD for class libraries makes the whole thing even more frustrating - you have a bit that works for sure (the class libraries with their tests) and then you hit this piece (aspx) on which you dont have the same level of confidence.


I've been working around it making sure that pages generate nice logs, so that during development, whenever I find a sticky piece, I put in an additional log statement. This thing works but at best is a poor cousin to automated testing ala nunit.


Enter NunitAsp - it promises to do for web applications what nunit does to class libraries - pretty stiff goal indeed! I've looked at this piece about an year ago for a similar project but had to decide against its use after going through the feature list. As a result, though I understand the aims, I havent got my hands into it. These days' I'm planning to do a dive-deep into it - just to make a more diligent evaluation if it'll actually work.


I love the premise of Test Driven Development -...


I love the premise of Test Driven Development - I've even used it a few times in the line of duty ;-) (I do have to admit, I've been naughty and left out the step of  seeing the test case fail a few times )...Anyway, I end up working on web applications more often than not and while you can use TDD for your class libraries, a web app is a totally different animal. The fact that you can use TDD for class libraries makes the whole thing even more frustrating - you have a bit that works for sure (the class libraries with their tests) and then you hit this piece (aspx) on which you dont have the same level of confidence.


I've been working around it making sure that pages generate nice logs, so that during development, whenever I find a sticky piece, I put in an additional log statement. This thing works but at best is a poor cousin to automated testing ala nunit.


Enter NunitAsp - it promises to do for web applications what nunit does to class libraries - pretty stiff goal indeed! I've looked at this piece about an year ago for a similar project but had to decide against its use after going through the feature list. As a result, though I understand the aims, I havent got my hands into it. These days' I'm planning to do a dive-deep into it - just to make a more diligent evaluation if it'll actually work.


Monday, April 11, 2005


For the most part, I find ASP.NET far more easier to use than Java. But the ONE BIG THING where I've found ASP.NET sorely lacking is in the support for page templates.


Page templates, if you need to brush up, allow you to define common layout and contents for a web site. Furthermore, once defined, its easy to change the layout and or move your default items around the place.


Basically, what you need is to be able to define a template page with the different areas (header, left pane, main content, footer etc). So the template page controls what is shown where. In addition, you also define the default content for all these areas.


Now each page in the application just overrides the content for the main area (assuming that the defaults are fine for the rest of it). WOW!!!


Java's had this quite some time - Jakarta Struts has something called Tiles which does exactly this.


For .NET, as I mentioned, the need's going to be fulfilled with v2.0 of ASP.NET. Meanwhile, if you feel the idea's great and there's no point in waiting for v2.0, release, do take a look at MasterPage as www.asp.net control gallery. Do note that since the asp.net team has released this control, there's a good chance that most of the features will end up in asp.net 2.0.


There are a few shortcomings of the control though - you'll get a hang of them if you read the posts. Paul Wilson has a version which overcomes these - and best of all, he releases the control with source :). You can find it here.


For the most part, I find ASP.NET far more easi...


For the most part, I find ASP.NET far more easier to use than Java. But the ONE BIG THING where I've found ASP.NET sorely lacking is in the support for page templates.


Page templates, if you need to brush up, allow you to define common layout and contents for a web site. Furthermore, once defined, its easy to change the layout and or move your default items around the place.


Basically, what you need is to be able to define a template page with the different areas (header, left pane, main content, footer etc). So the template page controls what is shown where. In addition, you also define the default content for all these areas.


Now each page in the application just overrides the content for the main area (assuming that the defaults are fine for the rest of it). WOW!!!


Java's had this quite some time - Jakarta Struts has something called Tiles which does exactly this.


For .NET, as I mentioned, the need's going to be fulfilled with v2.0 of ASP.NET. Meanwhile, if you feel the idea's great and there's no point in waiting for v2.0, release, do take a look at MasterPage as www.asp.net control gallery. Do note that since the asp.net team has released this control, there's a good chance that most of the features will end up in asp.net 2.0.


There are a few shortcomings of the control though - you'll get a hang of them if you read the posts. Paul Wilson has a version which overcomes these - and best of all, he releases the control with source :). You can find it here.


Friday, April 08, 2005


The ASP.NET validation summary is great for displaying all the errors in the page. It would be nice though to be able to use a validation summary to display errors that occur on the server side.


A typical scenario is when you have a page that does a search and then displays the results of the search. For the case where no results are found, it would be nice to be able to display the results in our custom validation summary control. It takes away the need to handle the display of the error message in a validation summary control.


This is what you can do about it - implement the IValidator class




public
class CustomErrorMessage:IValidator
    {
        privatestring message;
        public CustomErrorMessage()
        {
            //
            // TODO: Add constructor logic here
            //
        }
        #region IValidator Members

        publicvoid Validate()
        {
            // TODO: Add CustomErrorMessage.Validate implementation
        }

        publicbool IsValid
        {
            get
            {
                returnfalse;
            }
            set
            {
                
            }
        }

        publicstring ErrorMessage
        {
            get
            {
                
                return message;
            }
            set
            {
                this.message = value;
            }
        }

        #endregion
    }


And here's code to use the validator at runtime in response to a server side error.



CustomErrorMessage msg = new CustomErrorMessage();
msg.ErrorMessage = "No rows found";
ServerErrors.Visible =true;
ValidationSummary1.Visible = false;
Page.Validators.Add(msg);

Page.Validate();



The ASP.NET validation summary is great for displaying all the errors in the page. It would be nice though to be able to use a validation summary to display errors that occur on the server side.


A typical scenario is when you have a page that does a search and then displays the results of the search. For the case where no results are found, it would be nice to be able to display the results in our custom validation summary control. It takes away the need to handle the display of the error message in a validation summary control.


This is what you can do about it - implement the IValidator class




public
class CustomErrorMessage:IValidator
    {
        privatestring message;
        public CustomErrorMessage()
        {
            //
            // TODO: Add constructor logic here
            //
        }
        #region IValidator Members

        publicvoid Validate()
        {
            // TODO: Add CustomErrorMessage.Validate implementation
        }

        publicbool IsValid
        {
            get
            {
                returnfalse;
            }
            set
            {
                
            }
        }

        publicstring ErrorMessage
        {
            get
            {
                
                return message;
            }
            set
            {
                this.message = value;
            }
        }

        #endregion
    }



Been tinkering with getting a nice paging algorithm out. To get a basic hang of the problem, do take a look at





Many ASP developers create their own search engines that return back the results to the web browser one page at a time



Do take a look at the second query given. I've modified it a little bit so that you can sort by a given field and removed a bit of the cruft (the au_lname like '%A%' bit). Here the table used is called Pager - with a column called Name.


Some of my bare bones requirements for a paging system are:


1. Should allow sorting


2. Should not impose any requirements on the table schema/ resultset.


3. Should be done on SQL Server as much as possible. Definitely not default paging that results in all rows being sent to the middle layer.


4. Ideally, should not require dynamic queries. (Though note that this conflicts with 1 & 3 as these two requirements almost make dynamic queries mandatory).


5. Should not use temp tables.


declare @pagenum int
declare @pageSize int


set rowcount  @pagesize


select * 
   from Pager P
where
    (select count(*)
    from Pager P2
    where P2.Name <= P.name) > @pagesize * @pagenum
order by
       p.name


 


The ASP.NET validation summary is great for dis...


The ASP.NET validation summary is great for displaying all the errors in the page. It would be nice though to be able to use a validation summary to display errors that occur on the server side.


A typical scenario is when you have a page that does a search and then displays the results of the search. For the case where no results are found, it would be nice to be able to display the results in our custom validation summary control. It takes away the need to handle the display of the error message in a validation summary control.


This is what you can do about it - implement the IValidator class




public
class CustomErrorMessage:IValidator
    {
        privatestring message;
        public CustomErrorMessage()
        {
            //
            // TODO: Add constructor logic here
            //
        }
        #region IValidator Members

        publicvoid Validate()
        {
            // TODO: Add CustomErrorMessage.Validate implementation
        }

        publicbool IsValid
        {
            get
            {
                returnfalse;
            }
            set
            {
                
            }
        }

        publicstring ErrorMessage
        {
            get
            {
                
                return message;
            }
            set
            {
                this.message = value;
            }
        }

        #endregion
    }


And here's code to use the validator at runtime in response to a server side error.



CustomErrorMessage msg = new CustomErrorMessage();
msg.ErrorMessage = "No rows found";
ServerErrors.Visible =true;
ValidationSummary1.Visible = false;
Page.Validators.Add(msg);

Page.Validate();


The ASP.NET validation summary is great for dis...


The ASP.NET validation summary is great for displaying all the errors in the page. It would be nice though to be able to use a validation summary to display errors that occur on the server side.


A typical scenario is when you have a page that does a search and then displays the results of the search. For the case where no results are found, it would be nice to be able to display the results in our custom validation summary control. It takes away the need to handle the display of the error message in a validation summary control.


This is what you can do about it - implement the IValidator class




public
class CustomErrorMessage:IValidator
    {
        privatestring message;
        public CustomErrorMessage()
        {
            //
            // TODO: Add constructor logic here
            //
        }
        #region IValidator Members

        publicvoid Validate()
        {
            // TODO: Add CustomErrorMessage.Validate implementation
        }

        publicbool IsValid
        {
            get
            {
                returnfalse;
            }
            set
            {
                
            }
        }

        publicstring ErrorMessage
        {
            get
            {
                
                return message;
            }
            set
            {
                this.message = value;
            }
        }

        #endregion
    }


Been tinkering with getting a nice paging algor...


Been tinkering with getting a nice paging algorithm out. To get a basic hang of the problem, do take a look at





Many ASP developers create their own search engines that return back the results to the web browser one page at a time



Do take a look at the second query given. I've modified it a little bit so that you can sort by a given field and removed a bit of the cruft (the au_lname like '%A%' bit). Here the table used is called Pager - with a column called Name.


Some of my bare bones requirements for a paging system are:


1. Should allow sorting


2. Should not impose any requirements on the table schema/ resultset.


3. Should be done on SQL Server as much as possible. Definitely not default paging that results in all rows being sent to the middle layer.


4. Ideally, should not require dynamic queries. (Though note that this conflicts with 1 & 3 as these two requirements almost make dynamic queries mandatory).


5. Should not use temp tables.


declare @pagenum int
declare @pageSize int


set rowcount  @pagesize


select * 
   from Pager P
where
    (select count(*)
    from Pager P2
    where P2.Name <= P.name) > @pagesize * @pagenum
order by
       p.name


 


Monday, March 28, 2005


Have been pulling my hair out trying to make commandbar buttons go in VB Addins for Office 2000. There are a couple of gotchas here


1. Any references to buttons that you've added are not valid by the time you get to the disconnection method. There's a MSDN KB article on this and the workaround is to use FindControl to get a new reference to the button.


2. The above technique works for Powerpoint and Excel. With ms word, the button is more stubborn. Turns out that it updates the normal template. So in a word addin, in addition to calling delete on the button, you should also save the normal template - see below..




btn.Delete
m_Application.NormalTemplate.Save



Have been pulling my hair out trying to make co...


Have been pulling my hair out trying to make commandbar buttons go in VB Addins for Office 2000. There are a couple of gotchas here


1. Any references to buttons that you've added are not valid by the time you get to the disconnection method. There's a MSDN KB article on this and the workaround is to use FindControl to get a new reference to the button.


2. The above technique works for Powerpoint and Excel. With ms word, the button is more stubborn. Turns out that it updates the normal template. So in a word addin, in addition to calling delete on the button, you should also save the normal template - see below..




btn.Delete
m_Application.NormalTemplate.Save



Thursday, March 24, 2005

BSNL Broadband..

I finally got a broadband connection from BSNL last week. T'was heady stuff for the first few days - was getting 40 Kbytes/sec downloads - which translates to around 320 Kbps - more than the advertised 256Kbps.

And then it all had to come down eventually!!! For the past two days, the downloads are going at 3 Kbytes/sec... So today I called their help desk and as expected, got no real help other than getting a complaint registered. I was told that the problem will be fixed 'soon' - with no details being provided on how soon is 'soon' :-)

Anyway, thankfully the connections a bit better now - though still slow...getting 100 Kbps - which is way behind the advertised speed.

So its wait and watch down here.

BSNL Broadband..

I finally got a broadband connection from BSNL last week. T'was heady stuff for the first few days - was getting 40 Kbytes/sec downloads - which translates to around 320 Kbps - more than the advertised 256Kbps.

And then it all had to come down eventually!!! For the past two days, the downloads are going at 3 Kbytes/sec... So today I called their help desk and as expected, got no real help other than getting a complaint registered. I was told that the problem will be fixed 'soon' - with no details being provided on how soon is 'soon' :-)

Anyway, thankfully the connections a bit better now - though still slow...getting 100 Kbps - which is way behind the advertised speed.

So its wait and watch down here.

Thursday, March 10, 2005


Am learning how to call web services asynchronously....here are some good resources to start off with.



Accessing an XML Web Service Asynchronously in Managed Code


Communicating with XML Web Services Asynchronously


I would actually like to make asynch calls from vb (sic!)...well that's quite some way off..


 


These should help

1.  Integrating Web Services and COM Components using the SOAP Toolkit - Article

2.  Migrating from the Microsoft SOAP Toolkit to the .NET Framework - Article

3.  Calling .NET Assemblies and Web Services from VB6 - Article

From:
Thanks, Simon

Wednesday, March 09, 2005

Editing documents in sharepoint library in Office 2000



"Edit
In" functionality is only available for Office XP and Office 2003 on
the client, however I saw this workaround from a now unknown poster and


Joel Faigin

was kind enough to confirm that it works.


Â

"Office 2000 is not supported to edit documents within the Document Library under the "All Documents" view.



But here is a work around to get the same exact function from Office 2000 as you do in Office 2003.

This is for working with Windows 2000 and Office 2000.

1. Open your particular document library.

2. Make sure "All Documents" view is selected.

3. Check Out the file (do not open the file just check it out)

4. Click "Explorer View"

5. Open the document you checked out and then edit and save the document.

6. Now return to the "All Documents" view.

7. Check In the document, and make your selections as well as enter your comments.


8. Your changes should now be shown, as well as version history if enabled."


from:

http://wss.collutions.com/Lists/FAQ/DispForm.aspx?ID=308

Coaxing IE to open office documents in Application windows

Found this great tip on preventing IE from opening office documents.
  1. Launch Windows Explorer.
  2. Select “Folder Options...“ from the Tools menu.
  3. Select the “File Types“ tab.
  4. Scroll down the list of “Registered file types“ until you find “DOC - Microsoft Word Document“ and select it.
  5. Click on the “Advanced...“ button at the bottom of the dialog.
  6. Uncheck the “Browse in same window“ checkbox.

from:

http://blogs.msdn.com/gusperez/archive/2004/04/28/122768.aspx


Editing documents in sharepoint library in Office 2000



"Edit
In" functionality is only available for Office XP and Office 2003 on
the client, however I saw this workaround from a now unknown poster and


Joel Faigin

was kind enough to confirm that it works.


Â

"Office 2000 is not supported to edit documents within the Document Library under the "All Documents" view.



But here is a work around to get the same exact function from Office 2000 as you do in Office 2003.

This is for working with Windows 2000 and Office 2000.

1. Open your particular document library.

2. Make sure "All Documents" view is selected.

3. Check Out the file (do not open the file just check it out)

4. Click "Explorer View"

5. Open the document you checked out and then edit and save the document.

6. Now return to the "All Documents" view.

7. Check In the document, and make your selections as well as enter your comments.


8. Your changes should now be shown, as well as version history if enabled."


from:

http://wss.collutions.com/Lists/FAQ/DispForm.aspx?ID=308

Obituary - My VS.NET 2003 debugger

VS.NET 2003 debugger died unexpectedly today. I get an access denied
error with a suggestion to add myself to Debugger Users group or be an
Administrator.


I'm already the administrator and also part of Debugger users group.


Googling a bit, found a collection of entries on M K Park's blog

Feature Request: Environment variable expansion in CMAB

I've been using CMAB for sometime now and the only big crib that I have
against it is that it doesnt support environment variables for
specifying things like the path of the actual configuration file [One
of the projects that does is Log4Net]


I've looked at the code and supporting environment variable expansion shoudlnt be too difficult to do.


I'll post a suggestion in and see what happens.

Enterprise Library

Probably I've been living in a cave...but looks like we've got a release of the enterprise library.


"Summary

The patterns & practices Enterprise Library is a library
of application blocks designed to assist developers with common
enterprise development challenges. Application blocks are a type of
guidance, provided as source code that can be used "as is," extended,
or modified by developers to use on enterprise development projects.
Enterprise Library features new and updated versions of application
blocks that were previously available as stand-alone application
blocks. All Enterprise Library application blocks have been updated
with a particular focus on consistency, extensibility, ease of use, and
integration."

I've just got a copy of it. The cribs are

A. The download locations for the old Application Blocks is still in
place. They don't link to the Enterprise library page either. (Though
GDN does have a note pointing users to the Enterprise library workspace)

B. I've yet to look at the Blocks in detail, but the changes are big and not very helpful.

a. Namespaces have changed.

b. There's a new configuration tool

c. The documentation's been updated but there's precious little for people migrating from older versions.

d. [Configuration Application Block] <Crib>
Name value collections are not supported and so are XML Serializer. One
of the beauties of the original release was that you could integrate it
with the standard classes seamlessly. Probably, this one does too, but
I'll need to look it up.

Coaxing IE to open office documents in Application windows

Found this great tip on preventing IE from opening office documents.
  1. Launch Windows Explorer.
  2. Select “Folder Options...“ from the Tools menu.
  3. Select the “File Types“ tab.
  4. Scroll down the list of “Registered file types“ until you find “DOC - Microsoft Word Document“ and select it.
  5. Click on the “Advanced...“ button at the bottom of the dialog.
  6. Uncheck the “Browse in same window“ checkbox.

from:

http://blogs.msdn.com/gusperez/archive/2004/04/28/122768.aspx


Obituary - My VS.NET 2003 debugger

VS.NET 2003 debugger died unexpectedly today. I get an access denied
error with a suggestion to add myself to Debugger Users group or be an
Administrator.


I'm already the administrator and also part of Debugger users group.


Googling a bit, found a collection of entries on M K Park's blog

Feature Request: Environment variable expansion in CMAB

I've been using CMAB for sometime now and the only big crib that I have
against it is that it doesnt support environment variables for
specifying things like the path of the actual configuration file [One
of the projects that does is Log4Net]


I've looked at the code and supporting environment variable expansion shoudlnt be too difficult to do.


I'll post a suggestion in and see what happens.

Enterprise Library

Probably I've been living in a cave...but looks like we've got a release of the enterprise library.


"Summary

The patterns & practices Enterprise Library is a library
of application blocks designed to assist developers with common
enterprise development challenges. Application blocks are a type of
guidance, provided as source code that can be used "as is," extended,
or modified by developers to use on enterprise development projects.
Enterprise Library features new and updated versions of application
blocks that were previously available as stand-alone application
blocks. All Enterprise Library application blocks have been updated
with a particular focus on consistency, extensibility, ease of use, and
integration."

I've just got a copy of it. The cribs are

A. The download locations for the old Application Blocks is still in
place. They don't link to the Enterprise library page either. (Though
GDN does have a note pointing users to the Enterprise library workspace)

B. I've yet to look at the Blocks in detail, but the changes are big and not very helpful.

a. Namespaces have changed.

b. There's a new configuration tool

c. The documentation's been updated but there's precious little for people migrating from older versions.

d. [Configuration Application Block] <Crib>
Name value collections are not supported and so are XML Serializer. One
of the beauties of the original release was that you could integrate it
with the standard classes seamlessly. Probably, this one does too, but
I'll need to look it up.

Monday, March 07, 2005

Sharepoint Web Services and NTLM Authentication

I noticed this strange behaviour and thought its worth a post - till I find a solution/explanation.


Scenario: I have a custom web service running under sharepoint. This means that the .asmx is in "c:\Program Files\Common Files\Microsoft Shared\web server extensions\60\ISAPI" and the binaries in the bin folder under it. The url of the web service is



http://aditi277/XTeam/spaces/outlooksynch/_vti_bin/OfflineServices.asmx


The _vti_bin has NTLM authentication enabled.


When I paste the url in a browser, I go straight through to the page without any authentication! (I've set "Prompt for username and password" in IE).


In the web service, Context.Request.IsAuthenticated returns False and Context.User.Identity.Name returns an empty string. This is as expected.


I have a small program that connects to the web service. In the program I'm setting the credentials for the web service.


The fun starts when I create an SPSite object in the web service code. The SPWeb.CurrentUser.ToString() returns the username of the credentials set on the client!!! How the heck is the identity being passed on to Sharepoint? Or is it a bug in sharepoint that it doesnt set the Context properties? I'm stumped!!!


Â


Sharepoint Web Services and NTLM Authentication

I noticed this strange behaviour and thought its worth a post - till I find a solution/explanation.


Scenario: I have a custom web service running under sharepoint. This means that the .asmx is in "c:\Program Files\Common Files\Microsoft Shared\web server extensions\60\ISAPI" and the binaries in the bin folder under it. The url of the web service is



http://aditi277/XTeam/spaces/outlooksynch/_vti_bin/OfflineServices.asmx


The _vti_bin has NTLM authentication enabled.


When I paste the url in a browser, I go straight through to the page without any authentication! (I've set "Prompt for username and password" in IE).


In the web service, Context.Request.IsAuthenticated returns False and Context.User.Identity.Name returns an empty string. This is as expected.


I have a small program that connects to the web service. In the program I'm setting the credentials for the web service.


The fun starts when I create an SPSite object in the web service code. The SPWeb.CurrentUser.ToString() returns the username of the credentials set on the client!!! How the heck is the identity being passed on to Sharepoint? Or is it a bug in sharepoint that it doesnt set the Context properties? I'm stumped!!!


Â


Friday, March 04, 2005

I N K S C A P E

Just discovered something great - Inkscape - go grab a copy and enjoy...


Â


Logging in ASP.NET - Web Services (contd.)

I'm almost through with a decent logging implementation for a Web service running under Windows Sharepoint services. Here are some hard earned lessons:


1. I could not use Global.asax for a sharepoint hosted web service. The Sharepoint web services Global.asax derives from stssoap and when I included the code in a C# script block, the code never got invoked.


2. Also, the concept of an application is not useful in the case of web service - since a single web might be used to host multiple services. So, now, I've moved all the Log4Net initialization code to the ASMX code behind ctor. I've also got a SetContext() method that sets the user name, application (service) name. I'm also mulling about including the last few characters from the session key - for cases where the same user is logged on from multiple machines.


3. I need to do some more fine tuning - like enabling the connection pooling for the log4net connection string.


4. Another great lesson learned - and this is for all the times when you've got all the bits of log4net up and running and yet can't seem to get the logs...Use the log4net Debug release with the DbgView utility from sysinternals. Log4net fails silently on errors and just doesn't give you a clue as to what happened. For all those times when something utterly silly like a typo in your config brings all your logging to a standstill, keep DbgView running side by side. If Log4net hits a problem, it'll get logged in the dbgview console. :)


I N K S C A P E

Just discovered something great - Inkscape - go grab a copy and enjoy...


Â


Logging in ASP.NET - Web Services (contd.)

I'm almost through with a decent logging implementation for a Web service running under Windows Sharepoint services. Here are some hard earned lessons:


1. I could not use Global.asax for a sharepoint hosted web service. The Sharepoint web services Global.asax derives from stssoap and when I included the code in a C# script block, the code never got invoked.


2. Also, the concept of an application is not useful in the case of web service - since a single web might be used to host multiple services. So, now, I've moved all the Log4Net initialization code to the ASMX code behind ctor. I've also got a SetContext() method that sets the user name, application (service) name. I'm also mulling about including the last few characters from the session key - for cases where the same user is logged on from multiple machines.


3. I need to do some more fine tuning - like enabling the connection pooling for the log4net connection string.


4. Another great lesson learned - and this is for all the times when you've got all the bits of log4net up and running and yet can't seem to get the logs...Use the log4net Debug release with the DbgView utility from sysinternals. Log4net fails silently on errors and just doesn't give you a clue as to what happened. For all those times when something utterly silly like a typo in your config brings all your logging to a standstill, keep DbgView running side by side. If Log4net hits a problem, it'll get logged in the dbgview console. :)


Thursday, March 03, 2005

Logging in ASP.NET

I've been trying to get decent logs on web application/web service. Its critical in cases where you dont have remote debugging access on the server.


I've been using Log4Net for over an year now and it's saved me more than once. It's the first time that I've tried to use it in a web scenario. There's just a couple of things to keep in mind:


Usually the worker process runs under the ASPNET account - this doesnt have rights to logon and doesnt have an associated console. Meaning console appender is out of the reckoning.


You could use FileAppender (the usual choice on single user applications), however, as multiple requests are served simultaneously, the logs will most probably quite unreadable.


The best choice is the ADONetAppender - you log everything in the DB and then use SQL to get a trace. Here we come across the next problem - how do you get the user information for each log entry?


The solution lies in using log4net's Mapped Diagnostic Context - MDC. Its basically a set of name value pairs that you can render using a pattern layout %X{name} in the log4net config file.


Here's the Log4Net.config file for reference

<>

Note the lines where the insert statement has an the username parameter and the additional param tag is added to the parameter list. You'll need to create a matching column definition in your table.


The next issue to be tackled is the log4net initialization and setting the username for the request.


The log4net system can be initialized the Application_onStart event and you can set the username in the PreRequestHandlerExecute event - this event is called just before the request is handed off for processing to the HttpModule and hence the best place to set the username. Relevant portions of Global.asax are shown below.


Â



protected
void Application_Start(Object sender, EventArgs e)
{
   ConfigureLog4Net();          Â
   Debug.WriteLine("In Application Start");
}

privatevoid ConfigureLog4Net()
{
   string filepath = Server.MapPath("log4net.config");
   Debug.WriteLine(filepath);
   FileInfo fs = new FileInfo(filepath);
   log4net.Config.DOMConfigurator.ConfigureAndWatch(fs);
   Debug.WriteLine("Finished Log4net configuration");
}

protectedvoid Application_PreRequestHandlerExecute(Object sender, EventArgs e)
{
       Debug.WriteLine("In PreRequestHandlerExecute");
   if (!LogManager.GetLoggerRepository().Configured)
   {
           Debug.WriteLine("Log4net not configured, calling configure");
           ConfigureLog4Net();
   }
   Debug.WriteLine(Context.User.Identity.Name);
   log4net.MDC.Set("user",Context.User.Identity.Name);
}


Logging in ASP.NET

I've been trying to get decent logs on web application/web service. Its critical in cases where you dont have remote debugging access on the server.


I've been using Log4Net for over an year now and it's saved me more than once. It's the first time that I've tried to use it in a web scenario. There's just a couple of things to keep in mind:


Usually the worker process runs under the ASPNET account - this doesnt have rights to logon and doesnt have an associated console. Meaning console appender is out of the reckoning.


You could use FileAppender (the usual choice on single user applications), however, as multiple requests are served simultaneously, the logs will most probably quite unreadable.


The best choice is the ADONetAppender - you log everything in the DB and then use SQL to get a trace. Here we come across the next problem - how do you get the user information for each log entry?


The solution lies in using log4net's Mapped Diagnostic Context - MDC. Its basically a set of name value pairs that you can render using a pattern layout %X{name} in the log4net config file.


Here's the Log4Net.config file for reference

>

Note the lines where the insert statement has an the username parameter and the additional param tag is added to the parameter list. You'll need to create a matching column definition in your table.


The next issue to be tackled is the log4net initialization and setting the username for the request.


The log4net system can be initialized the Application_onStart event and you can set the username in the PreRequestHandlerExecute event - this event is called just before the request is handed off for processing to the HttpModule and hence the best place to set the username. Relevant portions of Global.asax are shown below.


Â



protected
void Application_Start(Object sender, EventArgs e)
{
   ConfigureLog4Net();          Â
   Debug.WriteLine("In Application Start");
}

privatevoid ConfigureLog4Net()
{
   string filepath = Server.MapPath("log4net.config");
   Debug.WriteLine(filepath);
   FileInfo fs = new FileInfo(filepath);
   log4net.Config.DOMConfigurator.ConfigureAndWatch(fs);
   Debug.WriteLine("Finished Log4net configuration");
}

protectedvoid Application_PreRequestHandlerExecute(Object sender, EventArgs e)
{
       Debug.WriteLine("In PreRequestHandlerExecute");
   if (!LogManager.GetLoggerRepository().Configured)
   {
           Debug.WriteLine("Log4net not configured, calling configure");
           ConfigureLog4Net();
   }
   Debug.WriteLine(Context.User.Identity.Name);
   log4net.MDC.Set("user",Context.User.Identity.Name);
}