Archive for the ‘Uncategorized’ Category

Your Application Probably Doesn’t Need a Database

This post originally appeared on Dzone here.

Ok, I’m being a little facetious here, but it’s certainly my default position. Developers seem to love to integrate databases into their applications without a thought for the requirements, despite the fact they end up moaning a lot about the database (and, if you’re in a big organization, the database team). It just seems to be the default position — “I am building an application so I will need a database and I will need Spring and I will need MQ and I will need <Insert defaults here>”. This is the wrong way of thinking.

People love habits. Once we’ve fallen into them it’s nearly impossible to escape, and countless books have been written to try and help. But as the old saying goes, the first step is admitting you have a problem. You’re addicted to Hibernate and Postgres. Sure everyone else is too, but that doesn’t make it right.

Let me say off the bat that databases aren’t inherently a bad thing. I don’t dislike them in the way I do Spring for example. Databases can be great and wonderful and exceptionally useful.  It’s just that it has become the default position, when in fact there are a lot of applications where a big, separate database isn’t necessarily needed.

So What’s the Alternative?

No Database!

RAM is a wonderful thing. It’s cheap. It’s incredibly fast. The more of your applications data you have in memory the faster it’s going to run. I can’t emphasise how much easier it is to store data in memory.

There’s no need to have a complex object mapping layer. There’s no need to deal with transactions. You just have data in Objects and it’s so easy to work with.

Obviously this won’t work for every application, but it will work for more than you think. The criteria:

  • Stateless service.
  • You must be able to bootstrap any data from other sources at startup.
  • Assuming you aren’t able to have a big cluster of these apps for failover purpose, you need to be dealing with information small enough that the bootstrap process is quick.

This obviously works well for aggregating services that sit atop other data sources, a surprisingly common workflow which is going to become more common with the increasing popularity of microservices.

Phone a Friend

If you’re dealing with a fairly large amount of data and it requires processing in your application, you can quickly get to a point where start-up times are huge if bootstrapping from external sources. In a resource constrained environment, this can be a big problem.

It may also be impossible to bootstrap the data from downstream all the time. In finance for example a lot of data is generated for “Start of Day (SOD)” from downstreams, but offer no intraday sources. This means that if the application goes down intraday there would be no way to restart it without some sort of data store.

But still, put down the database to start with. In this system, we have a dump of data at the beginning of the day, and we then receive updates throughout the day. This is still stateless; each instance of the application will be identical as there is no state that is unique to each node.

If the application goes down we need a source of the updates we’ve received since start of day then you need to phone a friend.

Unless there’s a catastrophic failure, as long as you have a single node up it can provide the missing data to it’s peers. The mechanism is up to you; HTTP, MQ, file dump, whatever floats your boat. It’s just a way to bootstrap a new node.  Everything’s still kept in memory.

There is obviously the risk that all nodes die and you can’t get your state from anywhere else.  If this could be a problem in your system, why not try local files?

Files

Files are awesome. They’re well understood, can be human readable, have a great set of APIs around them and are fast. It is relatively simple to write an application’s state out to file, particularly if you’re using an event sourced system — you don’t need to edit “records”, you just have a timeline of events which you can replay in.

Files are also a great way to speed up the recovery process; instead of sending a huge dataset over the wire from a peer, you can bootstrap from local files up to the point of failure, then just request updates from your peers for after that point.

This really does work in production. One of the systems I worked on was designed for speed and used an event log file for data recovery. It was super fast and quick to recover, and we never had to interact with the database team which was a huge bonus.

SQLite

Perhaps you’re really unwilling to give up on SQL. Maybe you don’t want to have to write and maintain your own serialization (I’ll admit, it can be a burden if not done well). Don’t rush back to your centralized DB! SQLite is a brilliant, super lightweight SQL implementation which stores your database locally in flat files. It’s super fast, it’s in very active development, and has a whole heap of benefits (read more here). It doesn’t even run as a separate process; just include the dependency in your project and start using it.  You get to have all the semantics of a database, but without dealing with anything remote.

None of These Apply to My System!

I’ll admit that not every system can get away with a local file for storage. Some genuinely do need global access and consistency, in which case I encourage you to go crazy. But, if you’re designing a new system (or you’re struggling with your current one), don’t instantly jump to Postgres or MySQL. You could save yourself a lot of time and pain, as well as speeding up your application, by designing your system to store data in memory or on flat file.

It’s all so quiet

Hello!
The blog hasn’t been silent because I’m lazy or busy; A few months ago I became a Zone Leader for Dzone.com.  This has me churning 3 articles a week out over there.  It’s been an interesting experience on many levels; primarily seeing just how angry people on the internet can get when someone expresses an opinion, but also seeing how posts I though were amazing have had relatively low hits, and posts I thought wouldn’t interest people have gone on to be super successes.

I’ve now got permission to repost my articles here, which I’m going to start doing with my favourite ones.  You can see them all along with comments and likes by heading to this link though.

Below are 3 of my most “popular” articles so far which I’ve enjoyed writing; they’ve had a ton of hits and/or a lot of comments.

 

Exceptions in Java: You’re (Probably) Doing It Wrong

This has gotten really popular since the editor changed the title to this slightly more provocative one.  People simply won’t let checked exceptions go! Fortunately there’s also lots of people that do understand it and have offered support.

Upgrade Your Code Conventions

A list of some things that I do differently on my code bases to most people.  It’s generated some good discussion, and it’s also found some good trolls.  The one with the most opponents is not using @Override, which in fairness I’m not that passionate about.  I’m glad that hopefully people have reevaluated their coding.

Disposable Architecture: The Tiny Microservice

Not so many comments but a lot of views.

 

Accessing Amazons Product API using Clojure

I recently embarked upon a new project with some friends called Swirl (swrl.co). We decided to write it in Clojure; we’re all Java developers by trade but wanted to give clojure a go as a bunch of smart people we know keep saying it’s amazing.  I’ve definitely formed some opinions, but that’s for another day (or over a beer).

The purpose of swirl is to store your recommendations. Nothing annoys me more when I finish reading a book and I can’t remember who recommended it to me in the first place. Now, if I recommend something to someone I do it through swirl, and they can then respond on the site so I know they’ve actually watched/read/listened to whatever I recommended.

The first port of call was Amazon integration.  Amazon offers an API (The Product Advertising API) to search the site for products and get the details back (and wrapped as an affiliate link).  What I quickly learnt was that the API is horrible to work with for 2 reasons.

1) No JSON option.  Anger!!

2) Every call must include a signature, which is an MD5 hash of all the other arguments and is time-sensitive.

For a read only product API the signature seems massively overkill.  This wouldn’t be such a big issue if we’d been using Java, but as I was grappling with Clojure syntax for the first time it meant I got stuck for a long time.  There’s default implementations for most languages, so I wanted to make my implementation available for anyone else to use.

The code is available on gist at https://gist.github.com/samberic/27439e09ee336cf3dc61.

The two key methods here are search-book, which uses the API to search through all of amazons catalog to return a series of results, and get-book which retrieves the details of a specific book by ASIN.  The url creation (e.g. the tricky bit) is done in search-url.

(defn search-url [bookname]
  (createEncryptedUrl (assoc (timestamp params) :Keywords bookname :Operation "ItemSearch" :SearchIndex "Books"
                                                )))

There are a bunch of parameters I’ve hard coded- amazon key and associate tag, service and version (no need to change these) and ResponseGroup.  ResponseGroup lets you choose what you want returned from your search, and you must have at least one.

I merge this standard list with the current timestamp, the book to be searched for, and the operation and search index.  If you want to search for things other than books, you can simply change the search index. Once we have a full map of parameters we can use this to create the signature.

 

The signature is made of an alphabetically-sorted (hence why params is a sorted-map) list of all parameters joined using “&”, and properly encoded (this was a real gotcha: I had to do string replace on +/%20 in as neither form-encode or url-encode got the job done properly. Hack!), preceded with “GET\nwebservices.amazon.com\n/onca/xml\n”, which you then put through a “RFC 2104-compliant HMAC with the SHA256 hash” using your amazon private key.

 

After much bashing of head against desk, the amazing Dan Flower pointed me to Buddy, a clojure security library we already had in the project anyway which took care of the hashing in a clear and concise way.

I hope this helps someone out somewhere.

How to “Maker” you tests clear

Tests should be your primary means of documentation in a system. I like to think we’ve all moved passed the idea of using comments in code (except for APIs), and most of us know to strive for clean, self documenting code. But for me tests are the easiest and most powerful way of documenting a section of code. If I’m looking at code and having a WTF moment, I should be able to flip to the tests and see what the intended behaviour was.

The problem is that, despite good intentions, writing clear and well documented tests is hard. Really hard. Particularly if you’re working with legacy code which has tight coupling, making it hard to split out the bits that you care about.

I recently discovered a test when going through a system which looked not unlike this:

@Test
public void bankTransferWillIncreaseDestinationBankAccount() throws Exception {
    BankAccount bankAccount = new BankAccount(20.0);
    final BankTransfer mock = mockery.mock(ConcreteBankTransfer.class);

    mockery.checking(new Expectations(){{
         allowing(mock).accountFrom(); will(returnValue(“accFrom”));
         allowing(mock).accountTo(); will(returnValue(“accTo”));
         allowing(mock).name(); will(returnValue(“Jon Smith”));
         allowing(mock).transferId(); will(returnValue(“143NMd24”));
         allowing(mock).ccy(); will(returnValue(null));
         allowing(mock).overdraftLimit();will(returnValue(200.0));
         oneOf(mock).amount(); will(returnValue(20.0));
     }
     });
    bankAccount.apply(mock);
    assertThat(bankAccount.amount(), is(40.0));
}

This is a dramatic reconstruction of the actual code. No programmers were hurt in the creation of this code.

The thing is, there were three tests in the class, which all had this vomit of “allowing”. This code smells badly. Mocks should be used to mock behaviour. In this case the BankTransfer object is to all intents and purposes a  POJO. Bad behaviour.

But the bigger issue here is that it’s impossible to see what’s going on. Does the fact ccy is null matter? What about the transfer ID? Does the overdraft have something to do with it?

This test is woefully unclear. Even if we remove the mock abuse and replace with a real implementation it doesn’t help.

@Test
public void bankTransferWillIncreaseDestinationBankAccount() throws Exception {
    BankAccount bankAccount = new BankAccount(20.0);
    final BankTransfer mock = new ConcreteBankTransfer(“accFrom”, “accTo”, null, 20.0, 200.0, “Jon Smith”, “143NMd24”);

    bankAccount.apply(mock);
    assertThat(bankAccount.amount(), is(40.0));
}

Sure there’s less code, but I have no idea what each field in the constructor means. If anything it’s less clear now which fields matter.

This is where Maker’s come in. A Maker allows you to construct an object whilst saying “I don’t care about any values except these specific ones”. Let’s look at the final code before showing the implementation.

private Maker aBankTransfer = a(BankTransferMaker.BankTransfer);

@Test
public void bankTransferWillIncreaseDestinationBankAccount() throws Exception {
    BankAccount bankAccount = new BankAccount(20.0);
    BankTransfer bankTransfer = make(aBankTransfer.but(with(BankTransferMaker.amount, 20.0)));
    bankAccount.apply(bankTransfer);
    assertThat(bankAccount.amount(), is(40.0));
}

For me this is infinitely clearer. The code does exactly as it says; it creates a BankTransfer (and we don’t care what the looks like), but we specify that it must have an amount of 20.0 as this is the value that we care about for our test. Very terse, clear and also reusable. Anywhere that needs a BankTransfer object can reuse this.

To use Maker, you need to import Nat Pryce’s “make-it-easy” (Maven details at http://mvnrepository.com/artifact/com.natpryce/make-it-easy). Then it’s simply a matter of creating your Maker.

There’s a fair amount of boilerplate code, and it can be quite tiresome to build a Maker. As a result you may want to think carefully before starting to use them everywhere.

public class BankTransferMaker {

    public static final Property<BankTransfer, String> accountFrom = newProperty();
    public static final Property<BankTransfer, String> accountTo = newProperty();
    public static final Property<BankTransfer, String> name = newProperty();
    public static final Property<BankTransfer, String> transferId = newProperty();
    public static final Property<BankTransfer, Double> overdraftLimit = newProperty();
    public static final Property<BankTransfer, Double> amount = newProperty();
    public static final Property<BankTransfer, ConcreteBankTransfer.Currency> ccy = newProperty();

    public static final Instantiator BankTransfer = new Instantiator() {

       @Override public BankTransfer instantiate(PropertyLookup lookup) {

            BankTransfer bankTransfer = new ConcreteBankTransfer(
                lookup.valueOf(accountFrom, random(5)),
                lookup.valueOf(accountTo, random(5)),
                lookup.valueOf(ccy, new ConcreteBankTransfer.Currency(random(3))),
                lookup.valueOf(amount, nextDouble(0,100000)),
                lookup.valueOf(overdraftLimit, nextDouble(0,100000)),
                lookup.valueOf(name, random(10)),
                lookup.valueOf(transferId, random(10)));
  
            return bankTransfer;
           }
       };
}

For each constructor parameter we need we create a Property value, which we have to type correctly. This is where a lot of the frustration can come from, as you have to manually build up this mapping.

We then create an Instantiator; this is how our object is actually created. You can set the default values to whatever you want; I’ve used apache-commons to plug random values in, because I really don’t care what’s there.

When it comes to creating test objects you can then modify any and all of these with the builder-style pattern seen in my original code base. We can change multiple values too, like so:

make(aBankTransfer.but(
with(BankTransferMaker.amount, 20.0),
with(BankTransferMaker.transferId, “Octopus”)));

It’s a really nice way to give clear visibility to which values matter in your test.

Fighting information obesity

Hi. My name’s Sam, I’m addicted to information and suffer from information obesity. I’ve always been a big fan of information, but I’d say I only started to gorge myself in the last year or so.  My brain is overweight with the volume of information I’m putting in it. I don’t even enjoy it anymore but I couldn’t stop.  However just recently I’ve been kicking the habit, and you can too!

Firstly, you need to know the symptoms of Information Obesity so you can self diagnose. Do you

  • Wake up in the morning, turn your alarm off, and read your email/facebook/twitter straight away?
  • Spend all of your public transport journeys immersed reading social networks/feedly feeds/the interwebz?
  • Open your email/social push notifications AS SOON AS YOU SEE THEM
  • Constantly check your phone so you can open your notifications AS SOON AS YOU SEE THEM
  • Feel like your brain is constantly slightly fuzzy?

Chances are you’re suffering from Information Obesity.  I too was suffering this, but don’t worry. You too can be cured.

I highly recommend reading this Gizmodo article on turning your smartphone into a dumbphone; it was the inspiration for my experiment.  I genuinely just found that at any given moment I already had my phone out checking for facebook updates or twitter posts or emails. God forbid I let an email go unread for several minutes!  So I decided to try out a dumbphone diet, although not quite as extreme as Gizmodo.

  • All push notifications were switched off, except for messaging apps (whatsapp etc)
  • All social type apps (fb, twitter, email) moved to the last screen on my iPhone
  • No checking any of those things on the phone; only on proper laptops

I set off with the goal of managing a week, and I’m now punching over 2 weeks.  And it’s such a relief! I encourage you to give it a go.  Things I have discovered

  • On the first day I really, really struggled to not check my email.  Addiction level difficult (ex smoker preaching here).  This went through the roof when I saw I had over 20 or so emails on the little red notification on the mail app (by accident).  I managed to make it till 5PM at which point I gave in.  And my biggest discovery was that not a single one of those emails was of importance.  I’d been lucky enough to have been retweeted by Scott Hanselman after promoting his email list (which is epic, btw), hence the massive (for me) dump of emails.  
  • And so with that, I realised that life goes on without me.  No email needs answering desperately within 6 hours. It became easier. I moved email off to another screen to reduce the chance of accidentally looking at it, and it makes for fairly smooth sailing
  • I assigned time in the morning before work to sit and read/reply to emails.  This was really cathartic, and meant I actually responded to emails as opposed to letting them hang around for a week and started the day clean.  Pro Tip! I read somewhere on twitter about starting the day with a glass of cold water (supposedly better than coffee).  Can highly recommend it. 
  • I’d also sit down after work to go through what’s going on.  Confining my email and socials to these period allowed my brain to breath
  • Commuting is really interesting without being locked into a smartphone.  It allowed me to think.  As cheesy as it sounds, I’d gotten to the point where I was so busy reading the entire internet, I didn’t get chance to think about the day ahead, and to just let my brain go free for a bit. It’s brilliant!

I have to admit after the end of week 1 I’ve not been as stringent on these rules, but things are definitely better now.  I miss a lot of twitter reading, but it’s nothing I can’t live without.

So, if you think your brain is fat and overweight with the volume of information you’re taking in, why not put it on a diet?  Try it for a week! 

Corkboard: Online image pasting/collage making

So this weekend inspired by a project a couple of friends are doing I decided I fancied hacking around with the HTML5 paste API.  This quickly and rapidly turned into Corkboard, which you can see at http://pastey.samatkinson.com.  Here’s a corkboard of some of the photos  of me playing recently to satisfy my massive ego.

All in all I’d guess it’s probably taken somewhere between 6-8 hours, of which most was due to my inability to RTFM and the fact Javascript hates me.  Most of the project is just mashing up a bunch of pre existing ingredients.

STRd6 imgpaste plugin

When googling around for how to do it, most of the stackoverflow esque threads had been jumped on by STRd6 to point to the plugin he wrote, and  I salute him, because it works a treat. You can find it at here.  It binds to the paste command, checks it it’s an image, and if it is calls your callback with the file and the img encoded as a dataurl (particularly awesome/handy).  In no time I was pasting images to the top left of a blank page with a bit of jquery append() fun.

JQuery UI draggable and resizable

You can find these here and here respectively.  They should pretty much work out the box which is nice.  However, I wanted more than just images; If you mouse over an image you can remove it from corkboard. This was some CSS/JS magic lightly stolen from a friends project, but meant wrapping the image in a div.  I wasted a lot of time trying to figure out how to make the img resize at the same time as the div.  If you RTFM you’ll discover this wonder:

$pasteimage
                        .draggable()
                        .resizable({
                            alsoResize: $pasteimage.find('img'),
                            aspectRatio: true
                        }
                )

Pay special attention to alsoResize. This is the key to the gold.

To avoid manually generating the HTML at this point I created a simple template in handlebars.

Bootstrap

So now we can drag and resize images, but it’s fugly.  Along comes bootstrap, the HTML5 UI starter package from the folks at Twitter. HIstorically I’ve always used foundation for making my sites pretty, but I wanted to branch out.  I’ve not done anything advanced, just a nav bar with drop down and a little form; The form inside nav bar situation is hard to do in foundation, so I was relieved to see it’s built in in bootstrap.  I had my usual pebcak issues of incorrect ordering/import of CSS and JS, but once up and running, it was very easy to use, and I think overall looks nicer than foundation.

Save as an image

So one of the nice features about corkboard is that you can save it to your local machine as an image. This was a mashup of 2 plugins; HTML2Canvas andCanvas2Image. The first traverses the DOM of whatever element you pass it and renders it out as a Canvas; I imagine this may have issues if you’re doing anything very funky in your code but it worked out the box on this. Canvas2Image then gives you a whole bunch of images for turning a canvas into an image, and for downloading it.  Both plugins worked seemlessly, pretty much first time. Very impressive.

html2canvas($('#theFunStuff'), {
                onrendered: function (canvas) {
                   Canvas2Image.saveAsPNG(canvas);
                }
            })

And that’s basically the whole thing. There’s tons more I want to do (being able to raise/lower layers, rotating, persistance), but that’s for another day.  There’s also some issues on HTML2Canvas2Image depending on where the images are when the save happens I need to look into.