Tuesday, April 2, 2013

A Better Storage API

Modern browsers have these very handy shortcuts to persistence called localStorage and sessionStorage.  While they're not recommended for everything due to some feature shortage, they are extremely useful for a modern webapp developer.  They are far better than cookies.

There's just one problem: the API is still really lame. Yes, it's a big jump up from cookies, but that's not saying much.  It's verbose, only works with strings and is decidedly lacking in features and convenience.  This is exactly the kind of wheel that i cannot resist reinventing.

I actually wrote store.js well over two years ago, in its first incarnation.  It has come a long way since then and has been well used here at ESHA Research.  However, it has never yet seen the "light of day" out there in open source.  Now, you can fork it on GitHub!


store.js handles JSON transparently, provides a very rich API, and is even extensible (with a few nice extensions and a few crazy ones already available in the repo). It supports both concise and explicit function calls to suit your needs (e.g. store('foo') === store.get('foo')).  It even supports namespacing your data with ease.

Check out the README on the github repo for documentation.  I think you'll like it. :)

Tuesday, December 18, 2012

Interface vs. Implementation

What is more valuable: interface or implementation?

This is the most fundamental question in software development. Actually, it might be the most important question in most engineering endeavors.

I'm here to tell you that it is interface. Implementation is essential, of course, as nothing works without it. And good, fast, reliable implementations are immensely valuable. But while you can have interface without implementation (at least in software), you cannot have implementation without interface. Further, your project will always be judged first and foremost on its interface. This is as true of a simple command line utility as it is of a cutting edge video game. Even you, no matter how efficient and clever your implementation is, are more likely to judge your own software mostly by how simple/natural/intuitive/beautiful you consider the interface to be.

There are exceptions, yes. Tasks that are especially greedy for cycles or memory, big data, and the like. Preference in such things will tilt heavily toward performance. But not absolutely. Once competitors are all within "acceptable" performance standards--whatever those may be--the judgement refers immediately back to interface, with few exceptions.

And interface is harder. Interface is art. Implementation is science. Implementation starts (and usually ends) with a simple pass/fail test: does it work? If need be, you can move on to improving the robustness or performance of the implementation. But these are still simple tests. Does it break on unexpected input? Does this change make it faster or slower? Use more memory or less? Implementation skill is a straightforward matter of knowledge, education, and test code. There are reliable rules, clear choices. Good tests guarantee good implementations.

Interface is about intuition and aesthetics, balancing multiple dimensions of competing concerns. There are no guarantees and no rules, only probabilities and guidelines. There are no tests that can be written to pass judgement on an interface, only users can do that. Even when you are the only user, confident interface design can be a challenge; you are a moving, changing target. What is simple for you today becomes inflexible tomorrow. What is flexible becomes unmaintainable. What is concise becomes conflicting. What is explicit becomes burdensome. What is convenient becomes constricting. What is clear becomes confusing. Assumptions are dangerous, but overdesign can cost you both now and then.

Uncomfortable interfaces are THE impetus for coders to reinvent the wheel, over and over again. If there is a problem with the implementation or its performance, most will submit a patch for it. "Wait," you say, "they can't always get patches accepted. Might that not motivate them to reinvent?" You forget: the patch submission process (or lack thereof) is a feature of any project's interface. Interface is not just the API, it's the entire surface area of the project: name, website, license, documentation, developers, active community, and API. In every case, you'll find an interface is ultimately the problem that created the competitor, though implementation problems do, of course, contribute.

So, if interface is both more important and harder, the moral of the story is...

Prioritize it!

Make it a point to learn the strengths and weaknesses of different strategies for naming and structuring all aspects of a project. Never let implementation dictate, block, overwhelm, or otherwise hinder the interface. Expect problems to arise, both initially and as a result of your solutions to those problems. Never forget KISS, YAGNI, and the like. Never treat such acronymisms (ha!) as anything other than helpful guidelines. Don't write any tests (implementation!) until you have put in some time planning your interface or your interface will be better for testing than actual use. Strive to make your interfaces self-explanatory, intuitive, convenient, simple, flexible, robust, well-documented, consistent, extensible, logical, and familiar. Realize that many of those goals often become opposites when mired in the particulars of the real world. Balance is not always possible. Sacrifices must sometimes be made.

Good luck.

Monday, August 6, 2012

Browser (In)Security

...is a myth for client-side code. I'm not just talking about the fact that all your source code is right there for anyone to see (uglified or not). Your active memory is all available to every javascript debugger on the planet. Your every command to or from the server can be intercepted and/or spoofed. You can't trust anything on the client.

Yet i still see people talk about client security as important. Sanitizing inputs, not executing scripts added via innerHTML, blocking cross origin resource loading, hiding variables in closures.  None of these things actually stop a malicious user (or innocent user with a compromised system) from doing anything and everything they want in the browser.  Seriously, anything they want.

Granted, every one of these things make it harder, and making life harder for the bad guys is a good thing.  I'm not opposed to any of these measures. But while useful and good, none of these browser "security" measures actually make your client-side app safe. The only thing you can actually secure is the server.

Never rely upon your client code to validate, sanitize, prevent or secure anything of importance.  There is no security in the browser. Once it's off your server, it is out of your hands. End of story. Anyone suggesting otherwise is likewise not to be trusted. :)

Monday, July 23, 2012

Dependency

The Java world has Maven and Ivy.  Tools meant to untangle dependency webs, to get you what you need, when you need it and put it where you want it.  I've seen a few attempts over the years to adapt Maven to handle javascript dependencies.  They were abandoned unfinished.  I haven't looked much in the last year.

Javascript development is immature.  Like Java 10 years ago, you generally need to manage the resources manually.  It's up to you to juggle version control, version numbers and untangle the web of dependency.  Once you have your libraries in their proper versions and understand what order you need them in, there are tools to help you execute what you decide/know.  RequireJS, yepnode.js, and Grunt interest me the most so far, but they all have different priorities and all get involved late in the dependency management problem. They help you execute deployment and development, but only after you have manually resolved all your dependencies.

Programmatic resolution is hard. To do it well requires organized public repositories, which require standardized naming, versioning, and dependency declaration. Not to mention bandwidth. The person and/or organization that takes it upon themselves to get this rolling in the javascript world has a lot of work cut out for them. But plenty of reward as well.

Ender does programmatic resolution, and it's package.json could probably be made to do more, but it only seems to run just one level deep, when a deep, tangled dependency web is the inevitable reality of any modular application development. It also neglects script loading, test running, support for app code, and stands as a competitor to other frameworks, not a complement.

Dependency management in the javascript world is harder than Java because javascript doesn't require compilation. Compilation funnels all Java devs through a common build step, making a natural insertion point for a dependency management tool. Javascript lacks that chokepoint. Testing, linting, minimizing, concatenating--all of these have some potential but are still inconsistently used or only used in production deployment, not development. And both deployment and development themselves often are embedded within build/deploy processes for whatever other platforms are being used to build the server side of the web app. Any javascript build process ultimately needs to work as a subprocess of those builds.

Javascript/HTML5 as a platform is still wild, untamed by the build tools and processes that smooth out other application development environments.  What i would really love to have in taming it is a dependency management tool that works with these features:

  • Repository support - public would be great, easy tools or conventions for private ones are acceptable. CDNs only get us part-way there; wheel authors need to be able to post libraries.
  • Standards for declaring dependency info - to optimize performance and untangle the web this should support/require:
    • library name and version
    • CDN URLs for the library
    • global variables exposed by the library
    • the libraries dependencies
      • name
      • version (range)
      • when needed (execution, document-ready, window-load, optional, etc)
    • license/copyright info
  • Conditional script loading (for polyfills)
  • Easy, declarative switching between development (test and lint, but no minimization or concatenation) and deployment (minimize and maybe concatenate)
  • Plugins for easy integrating with other build tools (Ant/Maven/etc)
Support for non-browser javascript and things like passing modules into each other (to avoid global exposure) are bonuses.  I don't care about them currently, but other people will.

A large portion of the code seems to be out there.  Grunt build process, Ender dependency resolution, yepnope script loading, and you're already well on your way. If you build it, they will come. Well, i certainly will! I'd built it myself, but there are only so many hours in a day.


#javascript

Monday, July 16, 2012

Readable Code

Code needs to be readable.  Readable code is debuggable, re-inventable and extendable.  But not everyone seems to know what makes code readable.

Occasionally, programmers will use comments to explain their code.  Comments certainly can help make a section of code understandable, assuming the comments are intelligible and up to date with the code.  But they in no way make code itself readable.

    Comments should only ever be used to say things that the code itself cannot be made to say.

In other words, you can use comments to explain why a piece code is there, but not to explain what the code is doing.  Yes, there are legitimate exceptions to that explanation, but again, those all fall under the first statement.

Mindful of the value of self-documenting code, many programmers create code like this:

AddressMapper.prototype.setInitialLocation = function(initialLocation) {
    this.initialLocation = initialLocation;
};
AddressMapper.prototype.setContainerElement = function(containerElement) {
    this.containerElement = containerElement;
};
AddressMapper.prototype.renderMapWithEffect = function(effect) {
    //magic!
};
// and so on...

After all, this is self-documenting, understandable code.  No one can accuse them of being opaque or obscure, right?  Beware the evils of premature optimization, right?  YUI 2 was like this, so it's ok, right? No, it most certainly is not OK.  And YUI 2 made me want to cry.

While long, descriptive names are indeed much better than inscrutable acronyms and abbreviations, they do not always produce code that can be called readable.  The example above, is unreadable because it is repetitive, mind-numbing, finger-cramping blather. When code gets repetitive like this, eyes gloss over and even stupid bugs find it easy to hide. Granted, they are likely to do that no matter what, but code like this does not help. And it gets worse, when you create APIs like this, the app code using it ends up just as obnoxious:

var addressMapper = new AddressMapper();
addressMapper.setInitialLocation('Portland, OR');
addressMapper.setContainerElement($('#portlandMap')[0]);
addressMapper.renderMapWithEffect(AddressMapper.Effect.SLIDE_DOWN);

Verbosity is contagious. It spreads. It infects. It sucks the will to live.

When reinventing a wheel, you are not designing an independent feature.  You are creating an API.  Yes, the wheel must be able to roll and so the raw, unadorned functionality of the code is your priority.  But ugly wheels make for ugly vehicles, and no one wants those.

Readable wheel code must avoid needless repetition and be designed to encourage readable app code.  Now, imagine this fictional plugin as a jQuery plugin:

var map = $.fn.map = function(address, effect) {
    return this.each(function() {
       render(this, address, effectFunctions[effect]);
    });
};
function render(el, address, effectFn) {
    // magic!
};
var effectFunctions = {
    slideDown: function(){...},
    ...
};

Which leads to user code like this:

$('#portlandMap').map('Portland, OR', 'slideDown');

<div id="#portlandMap"></div>

The repetition has dropped to tolerable levels, but it's not exactly a picture of readability.  Let's take it a step further and follow all the Rules for Reinvention:

var map = $.fn.map = function() {
    return this.each(map.render);
};

map.render = function(element) {
    var $el = $(element),
        address = $el.attr('address'),
        effectFn = map.effects[$el.attr('effect')];
    // magic!
;
map.effects = {
    slideDown: function(){...},
    ...
};

$(document).ready(function() {
   $('map').map();
});

<map address="Portland, OR" effect="slideDown"/>

At this point, the plugin code is not only more readable, but the app code has become pretty much ideal--declarative, minimal and extremely readable.  In the end, this should always matter most.  Sometimes, for whatever reason, there will be an irreducible amount of complex, ugly code.  In such cases, you should always strive to absorb that mess into your library and keep your app code clean and neat.

                           A readable API takes priority over readable internal code.

In fact, the lower "level" a library runs at, the less readability matters (and the more test code does!)  Since wheel code is just one small step away from app code, your wheels should still be highly readable wherever possible.

Wednesday, July 11, 2012

More Fun With Custom Events - state.js

My web apps these days are typically a single page, perhaps excepting a login page or some help content.  One page, composed from many files--during development, at least--does most of the work.  This design obviously began with the usual problems of lacking browser history control and bookmark-ability.  Looking at solutions others made, as you might expect, prompted me to reinvent another wheel.

Lo, state.js:

This enables browser history and bookmarks, by shoving simple key/value pairs into either the query string or the hash, as supported.  Keeping everything in the hash for all browsers would have been simpler, but then it would not survive a server-side login redirection process, undermining the ability to effectively bookmark a gated app.

state.js keeps track of the key/value pairs in memory, in order to notify listeners when those values are changed by the browser/user.  You can register a listener directly with state('foo', listenFn) or just watch for 'foo.state' events on the document.  This latter feature is part of the most interesting feature of state.js: it can be used entirely via custom events.  You do not need to ever directly reference the 'state' function in your app code or other js libs.  Granted, if you want soft dependency like this, it is easy to just have your config code wait for document ready and check if 'state' is available, but as state.js is such an event-oriented bit of code to begin with, i found it useful, amusing, and trivial to add an event-based API.

As an example, this:

state('foo', function(val, oldVal) {
    console.log('foo changed from ',oldVal,' to ',val);
});
state('foo', 'bar');// set foo to bar

is equivalent to this:

$(document).on('foo.state', function(val, oldVal) {
    console.log('foo changed from '+oldVal+' to '+val);
})
.trigger('state', ['foo','bar']);// set foo to bar

And here's the other useful features:

var all = state();// get a copy of current state
var foo = state('foo');// get current 'foo' state
$(document).trigger('back');// tell the browser to go back a page
$(document).trigger('forward'); // === history.forward()

As you can probably guess, this event-based API comes in quite handy when using trigger.js.  Now your page's controls can control state values and shift browser history:

<button trigger="nextPg state['pg',2]">Next</button>
<button trigger="back">Back</button>

Friday, July 6, 2012

HTML Validators Considered Harmful

There are these things out there called "HTML validators".  The last time i tried one was about 12 years ago.  There's a reason i haven't gone back.  It's not that i don't believe in standards; i do.  It's not that i'm sloppy or sneer at perfectionism; i don't!  I don't use HTML validation, because it doesn't work, won't work and shouldn't work.

HTML validation doesn't work because the browsers have never even come close to uniform standards coherence.  So the validators are testing markup against pure fiction, an idealistic but flawed dream of W3C hegemony.

HTML validation won't work because both HTML and browser versions are toast.  Over and done.  HTML has officially been declared what it always was: a fuzzy target.  Browsers have begun a version number war.  It is accepted dogma in web app development that you are doing it wrong when you check the user-agent string for any browser or version number.  Feature detection is the new game, and javascript is now used to emit so much markup into the document that HTML validators are a hopeless cause.

HTML validation shouldn't work, because it has always been worse than useless. It is constricting, a bane to innovation.  Why should app developers and wheel reinventors be restricted to a generic set of elements and attributes?  What good does it accomplish to place such a limit?  Like many other languages, HTML/XML is designed to be extended.  That creative ability should not be left only to the W3C and browser vendors.

In practical terms, browsers are spectacular at one thing, above all else: ignoring things.  They ignore extra whitespace, unrecognized elements, custom attributes, and even unclosed tags in many cases.  They do this so well out of necessity. Standard specifications have never been met.  They also need not only a high degree of backward compatibility, but forward compatibility as well.

Browsers are designed for new things.  Be creative!


The only valid kind of HTML validation is syntax validation.  All i ever want to know is if it is parse-able.  Really, that's it!