Running Webpack on AWS Lambda

AWS Lambda, but more importantly the serverless ecosystem, is changing the way we can create and think our softwares on the cloud.
With serverless we can really focus on coding our service more than worrying about the infrastructure. Serverless is not useful in any occasion but there is a wide range of situation where using it becomes handy and very powerful.

For instance, let’s think for a moment our automation build pipeline: what would you say if I can tell you that you won’t need to tight them with a specific tool like Jenkins or Bamboo but you should use Jenkins or similar as luncher running multiple Lambda functions, in parallel or in sequence, levering the power of the cloud?

I can give you a concrete example, yesterday night I was doing a spike in order to generate a Javascript bundle on the cloud with Webpack.
Therefore I invested some time creating an AWS Lambda that executes Webpack for bundling a simple example that contains lodash and some ES6 code like this one:

import _ from ‘lodash’;
function component () {
       var element = document.createElement(‘div’);
       element.innerHTML = _.join([‘Hello’,’webpack’], ‘ ‘);
       return element;
}
document.body.appendChild(component());

This is an example that you can find in the webpack official website, I used that just for the sake of the demo.
What we want to do now is the possibility to bundle this ES6 code and its library to a unique Javascript file that could be used inside our hypothetic application or website, mimicking what a step of a build pipeline would do.
Obviously you could run any other executables inside AWS Lambda, I choose Webpack because was the one used in my working place.

AWS Lambda at the rescue

If you create an automation pipeline on the cloud and maybe you don’t have many devops in your team or company, you should spend some time learning AWS Lambda, it could help out in these kind of activities.

What is AWS Lambda? Long story short: it’s a stateless docker container that is maintained by AWS where you can focus on writing the business logic of your activity more than thinking on the infrastructure.
Too good for being true? Yeah, you are right, Lambda has some limitations:

Information retrieved from Amazon documentation in March 2017
Information retrieved from Amazon documentation in March 2017
Information retrieved from Amazon documentation in March 2017

More information regarding the limits are available in the Lambda documentation website

But still the amount of things you can do with it is pretty impressive!

So, going back to our main goal, we want to bundle our Javascript project with Webpack inside Lambda, how can we achieve that?

First thing first, I created a git repository where you can find a Javascript project to simply use inside a AWS Lambda function. In this way you won’t need to create a custom project and you can focus more on the AWS side.
There are few points that I’d like to highlight in this simple project because usually are the ones that you can waste your time:

  1. Lambda functions can save temporary files inside the /tmp/ folder (bear in mind that you are running your code inside a docker container!).
    If you try to save somewhere else you will receive an error trying executing it.
  2. With Lambda you can run executables or node CLI tools like NPM or Webpack just uploading them inside your Lambda environment and referring to them with relative path.
  3. AWS Lambda runs for not more than 300 seconds therefore if you have a complex operation you could split it up in different chained Lambda functions that are triggered in sequence.
    This should help you in debugging your operations too.

In the project I set up the webpack config file in this way:

var path = require('path');
module.exports = {
   entry: './app/index.js',
   output: {
      filename: 'bundle.js',
      path: path.resolve('/tmp/')
   }
};

As you can see I’m saving my bundle in the tmp folder because is the only one with write permissions (remember the capacity limit of 512MB for that folder).

Then I created an index.js file where I describe my Lambda function:

var spawn = require('child_process').spawn;
var fs = require('fs');


exports.handler = (event, context, callback) => {
   var wp = spawn('./node_modules/.bin/webpack', ['--config', 'webpack.config.js']);

   wp.stdout.on('data', function(data){
     console.log('stdout: ' + data);
   });

   wp.stderr.on('data', function(err){
     context.fail("writeFile failed: " + err);
   });


   wp.on('close', (code) => {
     fs.readFile('/tmp/bundle.js', 'utf8', function (err, data) {
         if (err) context.fail("read file failed: " + err);
         context.succeed(data);
     });
   });
};

Very simple code here, I’m using Node, as you can see, but potentially you could use Python or Java (these 3 languages are supported by AWS Lambda at the moment), up to you peeking your favourite.
I import the spawn method in order to run webpack and once it has finished I read the content inside the Javascript bundle file created by Webpack in the tmp folder and I return it via context.succeed method.
Context is an object, always available inside a Lambda function, that will allow you to interact with Lambda for retrieving some useful information or, like in this case, to define when the function succeed or failed.

Now we are ready to upload the application in an AWS Lambda function.
In order to do that you will need to zip your project files (not the parent folder but just the files) and upload then in AWS.
If you didn’t install the project dependencies after cloning the repository, you should do it before uploading it to AWS.

Select and compress only the files inside your project not the parent folder 

Inside your AWS console, after selecting Lambda service, you should be able to create a new function (as far as I know not all the regions are supporting AWS Lambda).
Choose your favorite language, in my case Node 4.3, and define the basic information like “function name”, “description” and so on.
Then instead of writing the Lambda function code inside the editor provided by AWS, open the dropdown and select Upload a ZIP file

Select upload a ZIP file

Then setup the handler, role and advanced settings in this way

it’s important set at least 30 seconds as Timeout period

The important part will be setting up the docker container where the Lambda is going to be executed with enough memory size and with a decent timeout because we are running an executable therefore we don’t want to block the execution due to a Lambda timeout.
If for any reason, you need to increase the 300 seconds soft limit set by default, you will need to contact Amazon and ask to increase it.
Another important information to remember is when the Lambda is not active for a certain amount of time (it’s estimated to 5 mins), the container used by your code will be re-used for other Lambda functions.
Therefore when you will trigger your function again it will be recreated (cold Lambda), instead if the Lambda function runs several times in few mins (warm Lambda) we will have better performances because the container will be already available and live to execute a new activity again.

Now if you want to test your Lambda function, you will need to click the button test and you should have an output similar to this one:

you can see easily the output produced by this Lambda function that is the content inside the Javascript bundle created by Webpack

If you want to test live the Lambda I created you can trigger it from this link

Where to go from here

Obviously the example described is very basic and it works mainly with the project I created, but it is useful to know also how you could expand this example:

  1. AWS Lambda functions accept arguments passed when we trigger them, therefore potentially you could upload your project files in S3 and trigger the Lambda function directly after the upload.
    In fact Lambda can be triggered by several cloud software in AWS like DynamoDB, SNS and so on; S3 is present in the list.
  2. In order to expose the Lambda externally you will need to connect it via API Gateway, another tool provided by AWS.
    In the example I shared above I configured API Gateway to trigger my Lambda function when someone is calling a specific URL.
  3. The fastest way, and my personal recommendation, to work with AWS Lambda is via a CLI tool like Serverless Framework, you won’t need to configure manually API Gateway and your Lambda environment because Serverless Framework will provide a boilerplate to work with.
    On top of that it will allow you to test your Lambda functions locally without uploading them every time on AWS.
    There are many other CLI tools available but at the moment Serverless is the most complete and documented with a large community behind it.
  4. Recently Amazon added the possibility to set environment variables for Lambda functions, therefore if your project requires them you will have the possibility to configure easily via AWS console or inside Serverless framework configuration file.
  5. If you don’t want to upload a CLI tool with the node_modules folder, you can create an executable with all static dependencies and upload just that file inside the ZIP file.
    For Node I found a tool that works pretty well with Webpack and NPM called EncloseJS.
  6. Remember to not abuse the power of the serverless ecosystem but understand the pros and the cons before starting using it because in some cases it’s definitely not the right choice
  7. An important consideration of the combination API Gateway + Lambda is that could work with HTTP2 protocol out of the box and you can use Cloudfront to cache your responses (also if they are dynamic) with just few configurations to set in the API Gateway console.
  8. With Lambda you pay just what you use, therefore if you use it for cron services, automation pipelines, triggering databse backup operations or similar, you could end up savings quite few dollars compare to an EC2 instance.
Advertisements

Visual Studio Code extensions demystified

When I tried for the first time Visual Studio Code on my Mac I remained quite impressed about its performances.
The investment Microsoft did during the last few years on this editor is really remarkable, considering also that it’s an open source software and not a commercial one.
As you know with Visual Studio Code you can create your own extensions and then share with the community inside the marketplace.
This for me was just an interesting and quick pet project before going back the my reactive studies, but it is worth to share it

I created a simple extensions for retrieving all the annotations in my Javascript projects grouping per categories inside the output panel or in a markdown file.

vscode-annotations-panel

You can download the extension called vscode-annotations directly from the marketplace or inside the extensions panel in Visual Studio Code editor.
If you want instead take a look to the source, feel free to clone the project from Github.

 

extensionpanel

First steps

If you wanna quickly start working on an extension, there is a Yeoman generator provided by the Visual Studio Code team that will create the folder structure and the necessary files for publishing your extension later on.
In order to use it just run these commands in your terminal window:

npm install -g yo generator-code
yo code

During the generation, the interactive generator will ask if you prefer working with Typescript or pure Javascript, in my case I picked the latter one.
After that you will have your project ready and you can start to have fun with Visual Studio Code!
if you prefer start with the classic Hello World project feel free to check Microsoft tutorial.

In my annotations extension what I’ve done is just providing 3 commands available in the command palette (CMD+SHIFT+P or View > Command Palette) :

. output all the annotations in the file opened inside the editor
. output all the annotations in a specific project
. export all the annotations in a specific project to a Markdown file

The first two will create an output panel inside the editor showing the annotations present inside a specific file or an entire workspace, the third one will create a markdown file with all the annotations for a specific project.

When you want to create a command inside the command palette, you need to set it up in few files, the first one is the package.json:

"activationEvents": [
     "onCommand:extension.getAnnotations",
     "onCommand:extension.getAllAnnotations",
     "onCommand:extension.createAnnotationsOutput"
 ]

and then in the commands array:

"contributes": {
    "commands": [
    {
        "command": "extension.getAnnotations",
        "title": "ANNOTATIONS: check current file"
    },
    {
        "command": "extension.createAnnotationsOutput",
        "title": "ANNOTATIONS: export markdown file"
    },
    {
        "command": "extension.getAllAnnotations",
        "title": "ANNOTATIONS: check current project"
    }]
 }

so in the commands array we are just defining the label that will be inside the command palette and the action that should be triggered when the user selects a specific command.
Then we will need to add each of them in the extension.js file (created by the scaffolder) inside the activate method that will be triggered once the editor will have loaded your extension:

vscode.commands.registerCommand('extension.getAnnotations', function () {
    // extension code here
 });

Just with these few lines of code you can see the expected results of having your commands present in the palette

vscode-annotations-palette

Microsoft is providing a well documented APIs for interacting with the editor, obviously, because it’s based on Electron bear in mind that you can also use Node.js APIs for extending the functionalities of your extension, for instance to create a file or interacting with the operating system.

Working with the workspace

When you want to interact with the editor manipulating files or printing inside the embedded console you need to deal with the workspace APIs.
In order to do that you need to become confident with a couple of objects of the vscode library:

  • window
  • workspace

With window and workspace you can handle end to end the editor UI and the project selected.
Window object is mainly use to understand what’s happening inside a file meanwhile an user is editing it.
You can also use the window object for showing notification or error messages or change the status bar

With the workspace object instead, you will be able to manage all the interactions that are happening inside the menu or editor interface.
Workspace object is useful when you want to iterate trough the project files or if you need to understand which files are currently open in the editor and when they will be closed for instance.

In my extension I used these 2 objects for showing a notification to the user:

vscode.window.showErrorMessage('There aren\'t javascript files in the open project!');

for interacting with the output panel:

vscode.window.createOutputChannel(outputWin_NAME);

[....]

outputWin.appendLine(`FILE -> file://${doc.fileName}`);
outputWin.appendLine("-----------------------------------------");
outputWin.appendLine(getBody(data, OUTPUT_PANEL_CONFIG))
outputWin.appendLine(OUTPUT_PANEL_CONFIG.newline);

outputWin.show(true);

and for iterating and opening a javascript file present inside a proejct:

vscode.workspace.openTextDocument(file.path)

[...]

vscode.workspace.findFiles('**/*.js', '**/node_modules/**', 1000).then(onFilesRetrieved)

Debugging an extension

Considering you are developing an extension for an editor you can easily debug what you are doing simply running the extension debug mode (F5 or fn+F5 from your macbook).

screen-shot-2017-01-31-at-04-06-39

Few suggestions regarding the debug mode:

  • console.dir doesn’t work, console.log will substitute what console.dir does if you are inspecting an object but not an array!
  • when an error occurs it’s not very self-explained (kudos to Facebook for the react native errors handling, best implementation ever!) so you will need to follow the stack trace as usual

Publishing an extension

Last part of this brief post will be related to the submission of your extension to the Visual Studio Code marketplace.
Also in this case Microsoft did a good job creating an extensive guide on how to do that, few suggestions also in this case:

  • in order to submit an extension in the marketplace you will need to create a Microsoft and a Visual Studio Team Service accounts
  • when you create the Personal Access Token for publishing your extension, bear in mind to set access to all accounts and all scope otherwise you could end up with a 401 or 404 error when you try to publish the extension
    screen-shot-2017-01-31-at-04-26-47
  • vsce command line tool is pretty good in order to create a publisher identity and super fast to publish an application on the marketplace.
    Considering that is a CLI tool you can also automate few part of the publishing process (increasing release number for instance) adding a scripts in your package.json
  • to make your extension more accessible in the marketplace, remember to add the keywords array inside the package.json with meaningful words and the appropriate category, at the moment there are the following categories available:
    Debuggers
    Extension Packs
    Formatters
    Keymaps
    Languages
    Linters
    Snippets
    Themes
    Other

    screen-shot-2017-01-31-at-22-14-31

Wrap up

There could be tons of other things to do and to discover for developing a Visual Studio Code extension but I think that could be a good recap of the lessons learnt for creating one that you could use along with the Microsoft guide.

HTTP2: the good, the bad and the ugly

I spent last few weeks investigating on HTTP2, the successor of HTTP1.1 and I’d like to share my findings and thoughts in this post.

Let’s start saying that if the question you have in mind at this point is: “Can I really use it today, not only for experiments but also in production?”
My answer would be: “YES, you can!”

First of all, I’d like to share with you the browsers implementation status for this protocol

screen-shot-2016-09-06-at-23-00-23

As you can see from the screenshot taken from caniuse.com it’s definitely well supported on the latest version of the major browsers with some caveats obviously.

If you are not convinced yet, please check this website with one of the browsers that currently supports HTTP2 and look how fast to load is!
I’d suggest to install the HTTP2 indicator Chrome extension to discover how many web apps or online services are using this protocol:

screen-shot-2016-09-07-at-21-41-09

Not yet convince?! OK let’s move to a deeper analysis then!

HTTP2 is a binary protocol with a multiplexing requests method implemented, that means all the browser requests will be handled asynchronously.

This massive change will increase drastically the performance of your application.
Considering at the moment a browser can download simultaneously a maximum of 5 resources per domain (let’s avoid talking about “resource sharding” for now), with HTTP2 we will be able to request all the resources and render them when the browser will accomplish their download, check this demo made with Go Lang for a proper comparison between the 2 protocols and check also the Network panel in the Chrome Dev Tools or Firefox dev tools in order to understand how the 2 protocols differ.

The Good

HTTP2 has really few rules in order to be implemented:

  • it works ONLY with https protocol (therefore you need a valid SSL certificate)
  • it’s backward compatible, so if a browser or a device where your application is running, don’t support HTTP2 it will fall back to HTTP1.1
  • it comes with great performance improvements out-of-the-box
  • it doesn’t require to do anything on the client side but on the server side for a basic implementation
  • few new interesting features will allow to speed up the load of your web project in a way that is not even imaginable with HTTP1.1 implementation

Despite the short list, HTTP2 is bringing a substantial change to the internet ecosystem.
One of my favourite feature is the server PUSH where a server can pass a link header specifying what the browser should download in advance before starting to parse entirely the HTML document.
In this case, we can educate the browser to download several resources like images, css or even javascript files before the engine recognise them inside the DOM, providing a better user experience to our web apps and/or games.

The Bad

There is still plenty of works to do in order to have a great penetration of this protocol, few specs are still on going (read the next paragraph: the ugly) and probably it will take quite few months before we will see a lot of services moving to this new protocol.

A part from the high level overview of the downsides, let’s look what will change on the technical side.

Considering that HTTP2 is not restrict on the amount of requests a browser is doing in order to download resources few techniques for optimising our websites will need to be reviewed or even removed from our pipeline.
Delivering all the application inside a unique javascript file won’t have any benefit with HTTP2, so we need to move our logic downloading only what we need when we need it.
Knowing that downloading large files won’t be a problem we could use sprites instead of several small images to handle the icons of our website.
Probably the different tools like Grunt, Gulp or Webpack will need to review their strategies or update their plugin in order to provide real value to this new project pipeline.

The Ugly

Google Chrome protocol implementation!
Chrome is my favorite browser and I use it extensively, in particular, when I need to debug a specific script or I need to gather metrics from a specific behavior of a web app.
At the moment it’s the only browser that requires HTTP2 server negotiation via ALPN (Application-Layer Protocol Negotiation) that basically is an extension allowing the application layer to negotiate which protocol will be used within the TLS connection.

Considering that OpenSSL integrates ALPN only from version 1.0.2, we won’t be able to enable HTTP2 protocol support for Chrome (from build 51 and above) if we don’t configure our server correctly.
For instance, on Linux OS, only Ubuntu from version 16.04 has that OpenSSL version installed by default, for all the other major Linux version you will either install the newer version manually or you’ll need to wait for the next major OS release.

I’d suggest reading carefully the article that describes this “issue” on ngnix blog before you start to configure your server for Chrome.

Wrap up

HTTP2 is not perfect and probably is not supported as it should be but, definitely, could improve (drastically in certain cases) your web project performance.
A lot of “big players” are already using HTTP2 protocols in production (Instagram, Twitter or Facebook for instance) and the results are remarkable.

Why not starting catching up with the future today?

Hot and Cold observables

One important thing to understand when you are working with Reactive Programming is the difference between hot and cold observables.
Let’s start with the definition of both types:

Cold Observables

Cold observables start running upon subscription: the observable sequence only starts pushing values to the observers when subscribe is called.

Values are also not shared among subscribers.

Hot Observables

When an observer subscribes to a hot observable sequence, it will get all values in the stream that are emitted after it subscribes.

The hot observable sequence is shared among all subscribers, and each subscriber is pushed the next value in the sequence.

Let’s see now how these 2 different observables are used across a sample application.

Using the same application I started to describe in my previous blog post, we can spot in the main class some code that is commented.
The first part represent how a cold observable works, so if you try to have the following code inside App.js you can immediately understand what’s going on when an observer subscribes to a cold observable:

let engine = new BallsCallSystem(); 
let t1 = new Ticket("t1", engine.ballStream); 

setTimeout(()=>{
    let t2 = new Ticket("t2", engine.ballStream);
}, 5000);

engine.startGame();

Our game engine instead has the following code in order to share the balls called across the tickets:

let stream = Rx.Observable
    .interval(INTERVAL)
    .map((value) => {return numbersToCall[value]})
    .take(TOTAL_CALLS);
 
this.published = stream.publish();

As you can see when we start the game there is only 1 ticket bought by the user, then after 5 seconds the user bought a new ticket and in theory we should check if in that ticket all the numbers previously called are present or not.

coldObservables
But with cold observables when an observer subscribe, it won’t be able to receive the previous data but just the data since when it subscribed, completely broken the main purpose of our bingo game… we definitely need something different in order to work with it, potentially something that won’t completely change our game logic but that could be useful to fix the issue…

That’s where the hot observables come in!
The cool part of hot observables is that you can define how much data you want to buffer inside the memory, sometimes you need to store all of them (like in the bingo game) sometimes you need only the last 4-5 occurrences, but as you will see changing from a cold to an hot observables it’s a matter of 1 line of code!

Inside the same git repo, you can find a class called BallsCallSystemReplay, this class implement an hot observable, let’s see how much the code changed inside our game engine:

this.stream = Rx.Observable
   .interval(INTERVAL)
   .map((value) => {return numbersToCall[value]})
   .take(TOTAL_CALLS)
   .shareReplay();

Basically we removed the publish method after creating the stream (that was useful if you want to control when the observable starts to stream data) and we added shareReplay that is a method that transform a cold observable to an hot one sharing all the data pass trough the stream every time a new observer subscribe to it.

So now if we change the code in App.js in this way:

let engine = new BallsCallSystemReplay();
let t1 = new Ticket("t1", engine.ballStream);
 
setTimeout(function() {
    let t2 = new Ticket("t2", engine.ballStream);
}, 5000);

Now you can see that when the second ticket is bought after the game started, we’re going anyway to check all the values called by the engine without doing a massive refactor of our game logic! Below you can see when the second ticket subscribe to the hot observable, the first thing is going to receive are all the previous values called by the engine and then is receiving the data with a certain interval following the logic of the game.

hotObservables

Reactive Programming with RxJS

In the past 6 months I spent quite few time trying to understand Reactive Programming and how it could help me on my daily job.
So I’d like to share in this post a quick example made with RxJS just to show you how Reactive Programming could help when you are handling asynchronous data streams.

If you are not familiar at all with these concepts I’d suggest to watch first my presentation on Communicating Sequential Process and Reactive Programming with RxJS (free registration) or check the slides below.

For this example I thought to create a basic bingo system that I think is a good asynchronous application example that fits perfectly with the Reactive Programming culture.
I won’t introduce in this blog post concepts like hot and cold observables, iterator pattern or observer pattern mainly because all these theoretical information are present in the webinar and the slides previously mentioned. 

You can clone the project repository directly from my git account.

Let’s start talking a little bit about the engine, basically a bingo system is composed by an engine where the numbers are called every few seconds and shared with the users in order to validate in which ticket bought by the user the number called is present.
For this purpose working with observables will facilitate the communication and the information flow between the engine and the ticket objects.
In BallsCallSystem class, after setting up the object creating few constant that we’re going to use inside the core engine, we’re going to implement the core functionality of the engine:

let stream = Rx.Observable
              .interval(INTERVAL)
              .map((value) => {return numbersToCall[value]})
              .take(TOTAL_CALLS);

These few lines of code are expressing the following intents:

  1. we create an observable (Rx.Observable)
  2. that every few milliseconds (interval method)
  3. iterate trough the interval values (incremental value from 0 to N) and return a value retrieved from the array numbersToCall (function described inside the map method)
  4. and after a certain amount of iteration we need to close the observable because the game is ended (take method) so all the observer will stop to execute their code

If we compare with an imperative programming implementation made with CSP (communicating sequential processes) I’ll end up having something similar to this one:

this.int = setInterval(this.sendData.bind(this), 3000);
[....]
sendData(){
   var val = this.numbersToCall[this.counter];
   console.log("ball called", val);
 
   csp.putAsync(this.channel, val);
   this.counter++;
 
   if(this.counter > this.numbersToCall.length){
      clearInterval(this.int);
      this.channel.close();
      console.log("GAME OVER");
   }
 }

As you can see I needed to express each single action I wanted to do in order to obtain the core functionality of my bingo system.
These 2 implementations are both solving exactly the same problem but as you can see the reactive implementation is way less verbose and easy to read than the imperative one where I’ve control of anything is happening inside the algorithm but at the same time I don’t really have a specific reason to do it.

Moving ahead with the reactive example, when we create an observable that streams data we always need an observer to retrieve these data.
So now let’s jump to Ticket class and see how we can validate against a ticket the numbers called by the engine

First of all we pass the observable via injection to a Ticket object:

let t2 = new Ticket("t2", engine.ballStream);

Then, inside the Ticket class we subscribe to the observable and we handle the different cases inside the stream (when we receive data, when an error occurs and when the stream will be terminated):

obs.subscribe(this.onData.bind(this), this.onError.bind(this), this.onComplete.bind(this));
onData(value){
    console.log("number called", value, this.tid);
    if(this.nums.indexOf(value) >= 0){
       this.totalNumsCalled.push(value);
       console.log(value + " is present in ticket " + this.tid);
    } 
 }
 
 onError(err){
    console.log("stream error:", err);
 }
 
 onComplete(){
    console.log("total numbers called in " + this.tid + ": " + this.totalNumsCalled.length);
    console.log(this.totalNumsCalled);
 }

Also here you can notice the simplicity of an implementation, for instance if we are working with React it will be very easy to handle the state of an hypothetical Ticket component and create a resilient and well structured view where each stream state is handled correctly.

An interesting benefit provided by reactive programming is for sure the simplicity and the modularity at the same time how our implementations are working.
I would really recommend to spend sometime watching the webinar in order to get the first approach to Reactive Programming and to understand better the purpose of the example described briefly above.

Hapi.js and MongoDB

During the Fullstack conference I saw a small project made with Hapi.js during a talk, so I decided to invest some time working with Hapi.js in order to investigate how easy it was create a Node.js application with this framework.

I’ve to admit, this is a framework really well done, with a plugin system that give you a lot of flexibility when you are creating your server side applications and with a decent community that provides a lot of useful information and plugins in order to speed up the projects development.

When I started to read the only book available on this framework I was impressed about the simplicity, the consideration behind the framework but more important I was impressed where Hapi.js was used for the first time.
The first enterprise app made with this framework was released during Black Friday on Walmart ecommerce. The results were amazing!
In fact one of the main contributor of this open source framework is Walmart labs, that means a big organisation with real problems to solve; definitely a good starting point!

Express vs Hapi.js

If you are asking why not express, I can reply with few arguments:

  • express is a super light and general purpose framework that works perfectly for small – medium size application.
  • hapi.js was built on top of express at the beginning but then they move away in order to create something more solid and with more built in functionalities, a framework should speed up your productivity and not giving you a structure to follow.
  • express is code base instead hapi.js is configuration base (with a lot of flexibility of course)
  • express uses middleware, hapi.js uses plugins
  • hapi.js is built with testing and security in mind!

Hapi.js

Let’s start saying working with this framework is incredibly easy when you understand the few concepts you need to know in order to create a Node project.

I created a sample project where I’ve integrated a mongo database, exposing few end points in order to add a new document inside a mongo collection, update a specific document, retrieve all documents available inside the database and  retrieving all the details of a selected document.

Inside the git repo you can find also the front end code (books.html in the project root) in Vanilla Javascript, mainly because if you are passionate about React or Angular or any other front end library, you’ll be able to understand the integration without any particular framework knowledge.

What I’m going to describe now will be how I’ve structured the server side code with Hapi.js.

In order to create a server in Hapi.js you just need few lines of code:

let server = new Hapi.Server();
server.connection();
server.start((err) => console.log('Server started at:', server.info.uri));

As you can see in the example (src/index.js) I’ve created the server in the first few lines after the require statements and I started the server (server.start) after the registration of the mongoDB plugin, but one step per time.

After creating the server object, I’ve defined my routes with server.route method.
The route method will allow you to set just 1 route with an object or several routes creating an array of objects.
Each route should contain the method parameter where you’ll define the method to reach the path, you can also set a wildcard (*) so any method will be accepted in order to retrieve that path.
Obviously then you have to set the route path, bear in mind you have to start always with slash (/) in order to define correctly the path.
The path accepts also variables inside curly brackets as you can see in the last route of my example: path: ‘/bookdetails/{id}’.

Last but not least you need to define what’s going to happen when a client is requesting that particular path specifying the handler property.
Handler expects a function with 2 parameters: request and reply.

This is a basic route implementation:

{
   method: 'GET',
   path: '/allbooks',
   handler: (request, reply) => { ... }
}

When you structure a real application, and not an example like this one, you can wrap the handler property inside the config property.
Config accepts an object that will become your controller for that route.
So as you can see it’s really up to you pick up the right design solution for your project, it could be inline because it’s a small project or a PoC rather than an external module because you have a large project where you want to structure properly your code in a MVC fashion way (we’ll see that in next blog post ;-)).
In my example I created the config property also because you can then use an awesome library called JOI in order to validate the data received from the client application.
Validate data with JOI is really simple:

validate: {
   payload: {
      title: Joi.string().required(),
      author: Joi.string().required(),
      pages: Joi.number().required(),
      category: Joi.string().required()
   }
}

In my example for instance I checked if I receive the correct amount of arguments (required()) and in the right format (string() or number()).

MongoDB plugin

Now that we have understood how to create a simple server with Hapi.js let’s go in deep on the Hapi.js plugin system, the most important part of this framework.
You can find several plugins created by the community, and on the official website you can find also a tutorial that explains step by step how to create a custom plugin for hapi.js.

In my example I used the hapi-mongodb plugin that allows me to connect a mongo database with my node.js application.
If you are more familiar with mongoose you can always use the mongoose plugin for Hapi.js.
One important thing to bear in mind of an Hapi.js plugin is that when it’s registered will be accessible from any handler method via request.server.plugins, so it’s injected automatically from the framework in order to facilitate the development flow.
So the first thing to do in order to use our mongodb plugin on our application is register it:

server.register({
   register: MongoDB,
   options: DBConfig.opts
}, (err) => {
   if (err) {
      console.error(err);
      throw err;
   }

   server.start((err) => console.log('Server started at:', server.info.uri));
});

As you can see I need just to specify which plugin I want to use in the register method and its configuration.
This is an example of the configuration you need to specify in order to connect your MongoDB instance with the application:

module.exports = {
   opts: {
      "url": "mongodb://username:password@id.mongolab.com:port/collection-name",       
      "settings": {          
         "db": {             
            "native_parser": false         
         }
      }    
   }
}

In my case the configuration is an external object where I specified the mongo database URL and the settings.
If you want a quick and free solution to use mongoDB on the cloud I can suggest mongolab, when you register you’ll have 500mb of data for free per account, so for testing purpose is really the perfect cloud service!
Last but not least, when the plugin registration happened I can start my server.

When I need to use your plugin inside any handler function I’ll be able to retrieve my plugin in this way:

var db = request.server.plugins['hapi-mongodb'].db;

In my sample application, I was able to create few cases: add a new document (addbook route), retrieve all the books (allbooks route) and the details of a specific book (bookdetails route).

Screen Shot 2015-12-04 at 23.44.38

If you want to update a record in mongo, remember to use update method over insert method, because, if correctly handled, update method will check inside your database if there are any other occurrences and if there is one it will update that occurrence otherwise it will create a new document inside the mongo collection.
Below an extract of this technique, where you specify in the first object the key for searching an item, then the object to replace with and last object you need to add is an object with upsert set to true (by default is false) that will allow you to create the new document if it doesn’t exist in your collection:

db.collection('books').updateOne({"title": request.payload.title}, dbDoc, {upsert: true}, (err, result) => {
    if(err) return reply(Boom.internal('Internal MongoDB error', err));
    return reply(result);
});

SAMPLE PROJECT GITHUB REPOSITORY

Resources

If you are interested to go more in deep about Hapi.js, I’d suggest to take a look to the official website or to the book currently available.
An interesting news is that there are other few books that will be published soon regarding Hapi.js:

that usually means Hapi js is getting adopt from several companies and developers and definitely it’s a good sign for the health of the framework.

Wrap up

In this post I shared with you a quick introduction to Hapi.js framework and his peculiarities.
If you’ve enjoyed please let me know what you would interested on so I’ll be able to prepare other posts with the topics you prefer.
Probably the next one will be on the different template systems (handlebars, react…) or about universal application (or isomorphic application as you prefer to call them) or a test drive of few plugins to use in Hapi.js web applications.

Anyway I’ll wait for your input as well 😀

2015 the birth of London JS community

We are very close to the end of 2015 and it’s time give a look back to the first 11 months of this year understanding what I’ve accomplished and trying to get some resolutions for the new year as well!

2015 for my professional life  was a year of changes: I moved to a new job, I learnt new programming paradigms (reactive and functional programming), I met a lot of incredible and talented  people and many many other things.

Probably the biggest changes (or old loves?) are the creation of London JavaScript community and my returning as speaker on technical talks after a couple of years of inactivity.

Last May I started a new adventure creating a new JavaScript community in London; I spent a month to understand how and why I could do that properly.
I had the possibility in the past to be staff member of the largest Actionscript Italian community and it was really a great experience.
I was young, with a lot of passion, not much experience and the community was the perfect place for growing properly, meeting new friends, learning from the different people, trying to solve technical puzzle everyday and having fun why not!

Starting again this experience in a new country it was a real challenge for me, but this time, with way more experience than the first time, clear ideas and a touch of madness.

In 2013 I had the opportunity to spend few months in Silicon Valley and I went to several meetup events and conferences from San Francisco to San Jose.
What I was really impressed of that environment was for sure the vibe I was able to breathe in any of these events.
People from all over the world that help each others, facilitating the connections between individuals, creating opportunities and sharing knowledge.
I was astonished about this way of doing community and when I moved from Italy to London I spent the first 18 months going to different events trying to retrieve the same experience and vibe.

When I started the London Javascript community my main goal was definitely recreating that vibe in this great city where the best developers all over the world are working in interesting projects with a lots of challenges to solve.
That’s why I decided to do that, filling a gap present in London communities where there were many and strong vertical framework communities (like React and Angular one), but not a strong and general JavaScript community.

After 8 months I’ve to admit I’m very happy about the results that this community was able to generate:

  • ~1400 members
  • 7 live events
  • 1 code lab for half day
  • 2 webinars
  • an average of 60 people per event
  • Listed as O’Reilly Community partner, Google Community and Skillsmatter Community
  • Community Partner of great events like: FullStack conference and Dot.JS conference
  • more than 1900 tweets & retweets with more than 500 followers

I had also the opportunity to meet great speakers, amazing developers and passionate people.

What this community is giving me back is really invaluable and really hard to find in similar activities too, I’m really grateful of any single moment spent organising any event.

What about the resolutions for 2016?

Recently I started to gather the interest from several companies that are asking me to organise meetup events in their offices and that means all the hard work done is paying this community back!

I contacted few speakers from California and next year I’m organising a couple of webinars with speakers from Silicon Valley and directly from Google office in Mountain View.

I’m already in contact with few great speakers ready to share with the community tips and tricks on different topics like webpack, jspm, angular, webrtc, react and ES6! Keep an eye on the community page in order to discover more about these events.

Last but not least, I’m working right now on the official community website that will be a SPA website with socials integration and the possibility to subscribe to a technical newsletter where I’m going to share the best articles, tutorials and events on the web.

Obviously these are already in the roadmap but I’m really open to listen what the JavaScript community is looking for! So don’t waste this opportunity and share your ideas on how to grow and make more special OUR community!