Proxy PAC file for work

Proxy pac file to switch proxy based on whether I’m connected to OpenVPN or not.

Basically, OpenVPN IP is 10.8.0.6 and at home I have 192.168.1.xxx - so if either of those two are available, then I bypass proxy.

function FindProxyForURL(url, host)
{
    // firefox - Ctrl-Shift-J Chrome/Chromium: chrome://net-internals/#events
    var debug = true;
    var log = function() {
        if (debug) {
            var args = Array.prototype.slice.call(arguments);
            alert(args.join(" "));
        }
    }
    var bypassProxy = false;
    log("Checking for availability of myIpAddressEx:", typeof(myIpAddressEx) == "function");
    // firefox doesn't have myIpAddressEx - but it does seem to return the
    // right IP
    if (typeof(myIpAddressEx) != "function") {
        log("myIpAddress returns: ", myIpAddress());
        bypassProxy = myIpAddress() == '10.8.0.6' || myIpAddress().startsWith("192.168.1.");
    } else {
        // chromium seems to pick the first IP - in my case, a virtualbox HOST
        // only IP. So check in all IPs returned by myIpAddressEx
        var ips = myIpAddressEx().split(";");
        log("myIpAddressEx returns", myIpAddressEx());
        for(i=0; i< ips.length; i++) {
            if (ips[i] == "10.8.0.6" || ips[i].startsWith("192.168.1.")) {
                bypassProxy = true;
                break;
            }
        }
    }
    log("Can bypass proxy: ", bypassProxy);
    if (bypassProxy)
    {
        log ("DIRECT")
        return "DIRECT";
    }
    else
    {
        log( "PROXY")
        return "PROXY 10.98.2.8:8080";
    }
}

Put it someplace and in IE configure it with a file:// url

I really, really hate our proxy configuration at work!

Also, proxy pac file support is really arcane…​ however, it does work without additional configuration/addons - so there’s that.

Easyblogger 1.2.4 released to PYPI

Just pushed out a minor update v1.2.4 to Easyblogger

This release now let’s you get blog posts by url

# get a post by url
$ easyblogger get -u "https://blog.rraghur.in/2015/06/js-development-2015.html"
658290074469501527,JS development - 2015,http://blog.rraghur.in/2015/06/js-development-2015.html

AsciiDoc vs Markdown

I’ve been a markdown user almost exclusively - more due to github and stackoverflow popularizing the format. Markdown is great for lightweight markup and with excellent tools like Pandoc, it’s a joy to produce good looking technical docs. Suffice to say that I’ve been taken with markdown and even use it to produce word documents and such.

While Markdown is great, it has it’s share of niggles - try to go beyond the basics and want some advanced features and you get into trouble. Want tables? Or want to include diagrams generated from ascii art? It’s all a hit or miss depending on which Markdown implementation you’re using. Ditto for things like nested lists and lists with sub paras and whole lot of other niggles where each different Markdown implementation may end up behaving differently. There have been some efforts to have a rigourous spec but I don’t think things went anywhere with CommonMark - atleast, I don’t see it used much anywhere.

For the latter problem of wanting to include diagrams, I wrote mermaid-filter and while it’s super useful, it works only with mermaid. If you want to use graphviz or something like that, sorry - you’re out of luck.

I’ve resigned myself to Pandoc’s version of markdown - since I end up using pandoc almost always to convert to other formats.

Wanting a better lightweight markup format, I spent sometime looking at alternatives this weekend and came upon asciidoc - and I’m just wowed! It pretty much ticks all the boxes in terms of all the niggles I have with Markdown and it’s not so different from markdown either so that transitioning to AsciiDoc is a non issue.

AsciiDoc has the following over Markdown:

1. Supports advanced formatting natively.
   
   1. Native code formatting with pluggable syntax highlighters.
      
   2. code call outs - oh so useful.
      
   3. Cross references
      
   4. Tables
      
2. Rigourous spec
   
3. Smart, natively extensible
   
   1. For ex: want diagrams? A whole bunch are supported
      

It’s miles ahead of Markdown, IMO - so much so that I’m jumping ship :)

So, in a rush of enthusiasm, I’ve updated EasyBlogger with asciidoc support and pushed a release out to PyPI. This also means that I would probably not need mermaid-filter anymore - but hey, you should see better looking posts from me here.

A C# Trie implementation

What’s a trie?

A trie is a datastructure that’s optimized for prefix retrieval. It’s basically a prefix tree with each node being the letters seen till that point. For example - A trie with apple

Now if we add apply to the tree above, we get:

What if we add something without a common prefix - say boat

I needed a trie recently and was checking Nuget for available libraries - surprisingly there aren’t too many datastructure libraries available for .net on nuget. I found VDS.Common which seems to be most popular at 36K downloads.

Anyway, after using VDS.Common and getting around 700ms search time for distinct search results over a dataset, I thought I’d give writing my own a shot to see if I could do better.

Here’s the base interface:

 public interface ITrie<in TK, TV>
    {
        IEnumerable<TV> Find(IEnumerable<TK> keyprefix);
        void Add(IEnumerable<TK> key, TV value);
    }

Next, we model a trie node - this keeps links to child nodes and at the leaf level, a reference to the data we want to store in the trie.

 class TrieNode<TK, TV>
    {
        private TK keyBit;
        private TV value;
        private IDictionary<TK, TrieNode<TK, TV>> children;

        public TrieNode(TC ch, TV value)
        {
            this.keyBit = ch;
            this.value = value;
            this.children = new Dictionary<TK, TrieNode<TK, TV>>();
        }

        public IDictionary<TK, TrieNode<TK, TV>> Children
        {
            get { return children; }
            set { children = value; }
        }

        public void Add(IEnumerable<TK> keySeq, TV value)
        {
            var node = this;
            foreach (var ch in keySeq)
            {
                if (node.children.ContainsKey(ch))
                {
                    node = node.children[ch];
                }
                else
                {
                    var newNode= new TrieNode<TK, TV>(ch, default(TV));
                    node.children[ch] = newNode;
                    node = newNode;
                }
            }
            node.value = value;

        }

        public IEnumerable<TV> GetValues()
        {
            var stack = new Stack<TrieNode<TK, TV>>();
            stack.Push(this);
            while (stack.Count > 0)
            {
                var node = stack.Pop();
                if (node.value != null)
                    yield return node.value;
                foreach (var kv in node.Children.Reverse())
                {
                    stack.Push(kv.Value);
                }

            }
        }
    }

Finally, let’s have a concrete implementation of ITrie

    public class Trie<TK, TV> : ITrie<TK, TV>
    {
        private TrieNode<TK, TV> root;

        public Trie()
        {
            this.root = new TrieNode<TK, TV>(ch: default(TK), value: default(TV));
        }

        public IEnumerable<TV> Find(IEnumerable<TK> keyprefix)
        {
            var node = this.root;
            foreach (var ch in keyprefix)
            {
                if (node.Children.ContainsKey(ch))
                {
                    node = node.Children[ch];
                }
                else
                {
                    return Enumerable.Empty<TV>();
                }

            }
            return node.GetValues();
        }

        public void Add(IEnumerable<TK> key, TV value)
        {
            this.root.Add(key, value);
        }
    }

On the same dataset, I get < 500ms over VDS.Common. Admittedly, this is less full featured, but I can live with that.

EasyBlogger now supports pandoc filters

I just pushed version 1.1.0 of EasyBlogger to pypi. Notable changes are that it now exposes ability to set pandoc filters.

Pandoc filters basically let you write plugins that can affect the document conversion. For example, my filter - mermaid-filter will let you embed diagrams in text using mermaid syntax and then when you convert a markdown document, generate a .png file on the fly, upload it to imgur and embed the link in the generated HTML file - so that your final HTML shows the image and not a fenced code block!

A quick example - with markdown like this

~~~mermaid
graph TD
    A --> B
~~~

note that you must specify that it’s a mermaid syntax with ~~~mermaid as the start of your code block

You get an image like this in your blog:

In fact, even this blog post was generated from mermaid source itself :)

Goes without saying that this isn’t limited to mermaid-filter - you can use any of the other pandoc filters that are available to generate/filter your blog markup!

Hope you like it as much as I do!

Feature branches vs feature flags

Of late, we’ve been working with more and more customers adopt a more DevOps friendly culture and engineering practices. While customers love what they hear about DevOps, esp that it lets you ship features sooner out of the door and shorten feedback cycles, my experience with customer organizations has been somewhat a mixed bag in terms of true on the ground adoption.

A recent experience that has gotten me thinking is about feature branches vs feature flagging. On a large distributed software engineering effort, we moved a bunch of services from a monolithic svn repo (single project) to multiple git repos which publish nuget packages of components that get consumed elsewhere. We also suggested creating feature branches for each ticket/work item and then merging often. This works well if changes are limited to a single service - however, when you have changes across services that need to be coordinated, it’s not sufficient on it’s own.

One of the things that modular repos led to is that the teams were struggling with shipping code in sprints even after a decent time given for adjustment to process changes. On two week sprints, multiple sprint teams were working on different features that all affected the same set of services - towards, the end of the sprint, there’d be a mad rush to merge in PRs - and once one team merged their PRs, the others would have to pull in changes from master and deal with merge conflicts at the fag end of the sprint. This usually led to a lot of stress at the last minute with stories being slipped over to the next sprint.

One of the factors causing this was that it was a legacy system where services were coupled with a single database at the bottom (oops!)- this made cross functional features affect the same set of services over and over again, leading to contention and merge hells. However, even without that particular wrinkle, cross functional features that affect multiple services are harder to ship completely within a single sprint since a feature release must coordinate changes across multiple service repositories. Add automation, unit testing, QA validation and so on, and very soon, 2 weeks doesn’t seem enough. When that happens, you end up with feature branches sitting out on dependent repositories that cannot be merged into master since the feature isn’t done and now someone has to keep the branch in sync with master as new features and bug fixes get merged into master. Added to that, since the feature branch has been sitting out of master for a longer period of time, merging is more complicated since the changes are more.

Reflecting back, I feel at least for cross functional features, having feature flags is an absolute must - this would have allowed teams to break down feature development into multiple sprints and merge enabling services/apis into master without worrying about shipping incomplete code. It also would have reduced the burden of keeping branches in sync. OTOH, it would have increased the burden of keeping feature flags in check and ensuring that they don’t run wild. Furthermore, it’s conceivable that you’d need a central way of managing sets of feature flags across multiple services… so that a single end user functionality can be toggled by affecting a composite feature flag rather than twiddling a bunch of flags across different services individually.

Looking for feature flag libraries, there don’t seem to be enough good open source options in the .NET world. On Java however, there seem to be a few good candidates - with ff4j leading the pack.

In summary, microservices are great - however, when you’re migrating legacy code, think long and hard about how you’d like to handle cross functional changes that affect multiple services without impacting your agility to ship. Feature branches are only part of the solution - consider feature flagging as well so that you can build cross functional features in stages. Also, keep in mind that Feature flags aren’t free - they add their own burden - but with a good engg governance around limiting the number of feature flags and how they’re phased out, you should be in good hands.

Neovim on Windows

If you’re a fan of Vim, then no doubt you’ve heard of Neovim - in fact, neovim has been a key driver in much of the changes that have made it into the latest version of vim (async, job control etc).

Neovim works extremely well as a daily driver on Linux - but for Windows support has been lagging. I’ve been using a fork from contributor equalsraf who’s been behind a lot of the windows support.

Last week, the main neovim project gained windows CI builds on appveyor. The instructions for installation were also updated on the wiki - so obviously, I was quick to try it. The windows appveyor builds also bundle the frontend - neovim-qt - however, there seems to be an issue that not all of neovim-qt runtime files are being included correctly in the build. Due to this, one cannot set GuiFont and so on and this is a bummer. Anyway, it’s easy to fix - just get the [neovim-qt binaries] from equalsraf’s appveyor and unzip them over.

Since neovim is now on master and changes are coming in fast and thick, I wrote a little powershell to update my neovim installation. Here it goes for your enjoyment (you need powershell v5 for this)

UPDATE - 2017-03-01 - The bug in question https://github.com/neovim/neovim/issues/6145 is now fixed in Neovim windows builds. So in the script below you can leave out pulling down neovim-qt separately

param (
    [string] $loc="d:\utils\"
      )
wget "https://ci.appveyor.com/api/projects/neovim/neovim/artifacts/build/Neovim.zip?branch=master&job=Configuration%3A%20MINGW_64" `
    -OutFile "$env:TEMP\neovim.zip"
wget "https://ci.appveyor.com/api/projects/equalsraf/neovim-qt/artifacts/neovim-qt.zip?branch=master&job=Environment%3A%20SCRIPT%3Dcontrib%5Cappveyor-msvc.bat" `
    -OutFile "$env:TEMP\neovim-qt.zip"

expand-archive -Path "$env:TEMP\neovim.zip" -DestinationPath $loc -Force
expand-archive -Path "$env:TEMP\neovim-qt.zip" -DestinationPath "$loc\neovim" -Force