Downloading over an unreliable connection with Wget

This is a part rant, part tip - so bear with me... My broadband connection absolutely sucks over the past week. I upgraded from 2Mbps with a download limit to a 4Mbps with unlimited downloads and since then it has been nothing but trouble... Damn BSNL!! I've probably registered about 30 odd complaints with them to no avail. If there was a Nobel for bad customer service, BSNL would probably win it by a mile. Some examples:

1. They'll call to find out what the complaint it and even when I explain what's happening, they hardly hear me out at all.
2. Either they call up and say 'We have fixed it at the Exchange' and nothing has changed
3. They automatically close the complaints :)

Guess they find it too troublesome that someone who's paying for broadband actually expects the said broadband connection to work reliably!

Anyway, Airtel doesn't seem to be any better - they need 10 days to set up a connection and when I was on the phone with them, they didn't seem too interested in increasing their customer count by 1 :).

I also tried calling an ISP called YouBroadband after searching some of the Bangalore forums for good ISP providers. They promised a call in 24 hours to confirm if they have coverage in my area and it was feasible for them to set up the connection and that was 48 hours ago!

At work, I've heard good things about ACTBroadband and they have some ads in TOI as well, but they said they don't have coverage in my area :(.

So how do you download

Today I needed to download something and doing it from the browser failed each time since my DSL connection would blink out in between!

After ranting and raving and writing the first part above and still mentally screaming at BSNL, decided to do something about it... Time for trusty old wget - surely, it'll have something?

Turns out that guess was a 100% on the money... it took a few tries experimenting with different options, but finally worked like a charm

wget -t0 --waitretry=5 -c -T5 url
# where
# -t0 - unlimited retries
# --waitretry - seconds to wait between retries
# -c resume partially downloaded files
# -T5 - set all timeouts to 5 seconds. Timeouts here are connect timeout, read timeout and dns timeout

Single Page Apps

We released the Scheduler service (cloud hosted cron that does webhooks) on the 18th of Jan. It was our first release (still in beta) and you can sign up for it via the Windows Azure store as an addon. Upcoming release will have a full portal and the ability to register without going via the Windows Azure portal.

We've been building the user portal to the Scheduler service as a Single Page app (SPA) and I wanted to share a some background and insights we've gained.

SPA overview

To review, an SPA is a web app contained in a single page - where 'pages' are nothing but divs being shown/hidden based on the state of the app and user navigation.

The benefits are that you never have a full page refresh at all - essentially, page loads are instantaneous and data is retrieved and shown via AJAX calls. From a UX standpoint, this delivers a 'speedier' experience, since you never see the 'static' portions of your page reload when you navigate around.

All that speediness is great but the downsides are equally important.

SPA - Challenges

1. Navigation - SPA's by nature break the normal navigation mechanism of the browser. Normally, you click a link, it launches off a request and would update the url on the address bar. The response is then fetched and painted. In an SPA however, a link click is trapped in JS and the state is changed and you show a different div (with a background AJAX req being launched).
   This breaks Back/Forward navigation and since the URL doesn't change, bookmarkability is also broken to boot.
2. SEO - SEO also breaks because links are associated with JS script and most bots cannot follow such links.

Now, none of this is really new. Gmail was probably the first well known SPA implementation and that's been around since 2004. What's changed is tha t now there are better tools and frameworks for writing SPAs. So how do you get around the problems?

1. Back/Forward nav and Bookmarkability: SPA's use hash fragment navigation - links contain hash fragments. According to the original HTTP standard, hash fragments are for within page navigation and hence while the browser will update the address on the address bar and push an entry into the history stack, it will not make a request to the server. Client side routing can listen for changes to the location hash and manipulate the DOM to show the right 'section' of the page.


2. SEO - Google (and later Bing) support crawling SPA websites provided the links are formatted specifically. See

Why we went the SPA way

When we started out with the Portal, we needed to take some decisions around how to go about it

1. Scheduler REST service is a developer focused offering and the primary interaction for our users is the API interface itself. While the portal will have a Scheduler management features, this is really to give our users a 'manual' interface to scheduler. The other important use case for the portal is when you want to see history of a task's execution. Given that the API was primary, we wanted to build the UI using the APIs to dogfood our API early and often.
2. It just made sense to have the UI consume the APIs so that we weren't re-writing the same capabilities again just to support the UI.
3. Getting the portal to work across devices was important. In that sense, going with an approach that reduces page loads makes sense.
4. We wanted public pages to be SEO friendly - so the SPA experience kicks in only after you login.
5. Bookmarkability is important and it should be easy to paste/share links within the app.

Tools and frameworks

We evaluated different frameworks for building the SPA. We wrote a thin slice of the portal - a few public pages, a Social login page and a couple of logged in pages for navigation and bookmarkability.

1. KO+ approach - I'm calling this KO+ as it's KO is just a library for MVVM binding and we needed a bunch of other libraries for managing other aspects of the SPA.
   • Knockout.js - for MVVM binding
   • Sammy.js - Client side routing
   • Require.js - script dependency management.
   • Jquery - general DOM manipulation when we needed it.
2. Angular.js - Google's Angular.js is a full suite SPA framework that handles all the aspects of SPA

We chose the KO+ approach as there was knowledge and experience on KO in the team. The learning curve's also lesser since each library can be tackled at a time. While Angular offers a full fledged SPA framework, it also comes with more complexity to be grappled with and understood - essentially, the 'Angular' way of building apps.

That said, once you get over the initial learning curve of Angular, it does have a pleasant experience and you don't have to deal with integration issues that come up when using different libraries. We had prior experience on KO on the team so it just made sense to pick it given our timelines.

I'll post an update once we have it out of the door and ready for public consumption.