Backendjs Documentation

##Table of contents

Backend platform for node.js

General purpose backend framework. The primary goal is to have a scalable platform for running and managing node.js servers for Web services implementation.

This framework only covers the lower portion of the Web services system: node.js processes, HTTP servers, basic API functinality, database access, caching, messaging between processes, metrics and monitoring, a library of tools for developing node.js servers.

For the UI and presentation layer there are no restrictions what to use as long as it can run on top of the Express server.

Features:

Check out the Documentation for more details.

Installation

To install the module with all optional dependencies if they are available in the system

npm install backendjs

This may take some time because of downloading and compiling required dependencies like ImageMagick. They are not required in all applications but still part of the core of the system to be available once needed.

To install from the git

 npm install git+https://github.com/vseryakov/backendjs.git

or simply

 npm install vseryakov/backendjs

Quick start

Configuration

Almost everything in the backend is configurable using a config files, config database or DNS. The whole principle behind it that once deployed in production, even quick restart are impossible to do so there should be a way to push config changes to the processes without restarting.

Every module defines a set of config parameters that defines the behavior of the code, due to single threaded nature of the node.js, it is simple to update any config parameter to a new value so the code can operate differently. To achieve this the code must be written in a special way, like driven by configuration which can be changed at any time.

All configuration goes through the configuration process that checks all inputs and produces valid output which is applied to the module variables. Config file or database table with configuration can be loaded on demand or periodically, for example all local config files are watched for modification and reloaded automaticlaly, the config database is loaded periodically which is defined by another config parameter.

Backend runtime

When the backendjs server starts it spawns several processes that perform different tasks.

There are 2 major tasks of the backend that can be run at the same time or in any combination:

These features can be run standalone or under the guard of the monitor which tracks all running processes and restarted any failed ones.

This is the typical output from the ps command on Linux server:

ec2-user    891  0.0  0.6 1071632 49504 ?  Ssl  14:33   0:01 bkjs: monitor
ec2-user    899  0.0  0.6 1073844 52892 ?  Sl   14:33   0:01 bkjs: master
ec2-user    908  0.0  0.8 1081020 68780 ?  Sl   14:33   0:02 bkjs: server
ec2-user    917  0.0  0.7 1072820 59008 ?  Sl   14:33   0:01 bkjs: web
ec2-user    919  0.0  0.7 1072820 60792 ?  Sl   14:33   0:02 bkjs: web
ec2-user    921  0.0  0.7 1072120 40721 ?  Sl   14:33   0:02 bkjs: worker

To enable any task a command line parameter must be provided, it cannot be specified in the config file. The bkjs utility supports several commands that simplify running the backend in different modes.

Application structure

The main puspose of the backendjs is to provide API to access the data, the data can be stored in the database or some other way but the access to that data will be over HTTP and returned back as JSON. This is default functionality but any custom application may return data in whatever format is required.

Basically the backendjs is a Web server with ability to perform data processing using local or remote jobs which can be scheduled similar to Unix cron.

The principle behind the system is that nowadays the API services just return data which Web apps or mobiles apps can render to the user without the backend involved. It does not mean this is simple gateway between the database, in many cases it is but if special processing of the data is needed before sending it to the user, it is possible to do and backendjs provides many convenient helpers and tools for it.

When the API layer is initialized, the api module contains app object which is an Express server.

Special module/namespace app is designated to be used for application development/extension. This module is available the same way as the api or core which makes it easy to refer and extend with additional methods and structures.

The typical structure of a backendjs application is the following (created by the bkjs init-app command):

    var bkjs = require('backendjs');
    var api = bkjs.api;
    var app = bkjs.app;
    var db = bkjs.db;

    app.listArg = [];

    // Define the module config parameters
    core.describeArgs('app', [
        { name: "list-arg", array: 1, type: "list", descr: "List of words" },
        { name: "int-arg", type: "int", descr: "An integer parameter" },
     ]);

    // Describe the tables or data models, all DB pools will use it, the master or shell
    // process only creates new tables, workers just use the existing tables
    db.describeTables({
         ...
    });

     // Optionally customize the Express environment, setup MVC routes or else, `api.app` is the Express server
    app.configureMiddleware = function(options, callback)
    {
       ...
       callback()
    }

    // Register API endpoints, i.e. url callbacks
    app.configureWeb = function(options, callback)
    {
        api.app.get('/some/api/endpoint', function(req, res) {
          // to return an error, the message will be translated with internal i18n module if locales
          // are loaded and the request requires it
          api.sendReply(res, err);
          // or with custom status and message, explicitely translated
          api.sendReply(res, 404, res.__("not found"));

          // with config check
          if (app.intArg > 5) ...
          if (app.listArg.indexOf(req.query.name) > -1) ...

          // to send data back with optional postprocessing hooks
          api.sendJSON(req, err, data);
          // or simply
          res.json(data);
        });
        ...
        callback();
    }

    // Optionally register post processing of the returned data from the default calls
    api.registerPostProcess('', /^\/account\/([a-z\/]+)$/, function(req, res, rows) { ... });
     ...

    // Optionally register access permissions callbacks
    api.registerAccessCheck('', /^\/test\/list$/, function(req, status, callback) { ...  });
    api.registerPreProcess('', /^\/test\/list$/, function(req, status, callback) { ...  });
     ...
    bkjs.server.start();

Except the app.configureWeb and server.start() all other functions are optional, they are here for the sake of completness of the example. Also because running the backend involves more than just running web server many things can be setup using the configuration options like common access permissions, configuration of the cron jobs so the amount of code to be written to have fully functionaning production API server is not that much, basically only request endpoint callbacks must be provided in the application.

As with any node.js application, node modules are the way to build and extend the functionality, backendjs does not restrict how the application is structured.

Modules

Another way to add functionality to the backend is via external modules specific to the backend, these modules are loaded on startup from the backend home subdirectory modules/ and from the backendjs package directory for core modules. The format is the same as for regular node.js modules and only top level .js files are loaded on the backend startup.

By default no modules are loaded except bk_accounts|bk_icons, it must be configured by the -allow-modules config parameter.

The modules are managed per process role, by default server and master processes do not load any modules at all to keep them small and because they monitor workers the less code they have the better.

The shell process loads all modules, it is configured with .+.

To enable any module to be loaded in any process it can be configured by using a role in the config parameter:

  // Global modules except server and master
  -allow-modules '.+'

  // Master modules
  -allow-modules-master 'bk_accounts|bk_debug'

Once loaded they have the same access to the backend as the rest of the code, the only difference is that they reside in the backend home and can be shipped regardless of the npm, node modules and other env setup. These modules are exposed in the core.modules the same way as all other core submodules methods.

Let's assume the modules/ contains file facebook.js which implements custom FB logic:

     var bkjs = require("backendjs");
     var fb = {
     }
     module.exports = fb;

     fb.configureWeb = function(options, callback) {
       ...
     }

     fb.makeRequest = function(options, callback) {
       ...
     }

This is the main app code:

    var bkjs = require("backendjs");
    var core = bkjs.core;

    // Using facebook module in the main app
    api.app.get("some url", function(req, res) {

       core.modules.facebook.makeRequest({}, function(err, data) {
          ...
       });
    });

    bkj.server.start()

Database schema definition

The backend support multiple databases and provides the same db layer for access. Common operations are supported and all other specific usage can be achieved by using SQL directly or other query language supported by any particular database. The database operations supported in the unified way provide simple actions like db.get, db.put, db.update, db.del, db.select. The db.query method provides generic access to the database driver and executes given query directly by the db driver, it can be SQL or other driver specific query request.

Before the tables can be queried the schema must be defined and created, the backend db layer provides simple functions to do it:

        db.describeTables({
           album: {
               id: { primary: 1 },                         // Primary key for an album
               name: { pub: 1 },                           // Album name, public column
               mtime: { type: "now" },                     // Modification timestamp
           },
           photo: {
               album_id: { primary: 1 },                   // Combined primary key
               id: { primary: 1 },                         // consiting of album and photo id
               name: { pub: 1, index: 1 },                 // Photo name or description, public column with the index for faster search
               mtime: { type: "now" }
           }
        });

Each database may restrict how the schema is defined and used, the db layer does not provide an artificial layer hiding all specifics, it just provides the same API and syntax, for example, DynamoDB tables must have only hash primary key or combined hash and range key, so when creating table to be used with DynamoDB, only one or two columns can be marked with primary property while for SQL databases the composite primary key can consist of more than 2 columns.

The backendjs always creates several tables in the configured database pools by default, these tables are required to support default API functionality and some are required for backend opertions. Refer below for the Javascript modules documenttion that described which tables are created by default. In the custom applications the db.describeTables method can modify columns in the default table and add more columns if needed.

For example, to make age and some other columns in the accounts table public and visible by other users with additional columns the following can be done in the api.initApplication method. It will extend the bk_account table and the application can use new columns the same way as the already existing columns. Using the birthday column we make 'age' property automatically calculated and visible in the result, this is done by the internal method api.processAccountRow which is registered as post process callback for the bk_account table. The computed property age will be returned because it is not present in the table definition and all properties not defined and configured are passed as is.

The cleanup of the public columns is done by the api.sendJSON which is used by all API routes when ready to send data back to the client. If any postprocess hooks are registered and return data itself then it is the hook responsibility to cleanup non-public columns.

    db.describeTables({
        bk_account: {
            gender: { pub: 1 },
            birthday: {},
            ssn: {},
            salary: { type: "int" },
            occupation: {},
            home_phone: {},
            work_phone: {},
        });

    app.configureWeb = function(options, callback)
    {
       db.setProcessRow("post", "bk_account", this.processAccountRow);
       ...
       callback();
    }
    app.processAccountRow = function(req, row, options)
    {
       if (row.birthday) row.age = Math.floor((Date.now() - core.toDate(row.birthday))/(86400000*365));
    }

To define tables inside a module just provide a tables property in the module object, it will be picked up by database initialization automatically.

var mod = {
    name: "billing",
    tables: {
       invoices: {
          id: { type: "int", primary: 1 },
          name: {},
          price: { type: "real" },
          mtime: { type: "now" }
       }
    }
}
module.exports = mod;

// Run db setup once all the DB pools are configured, for example produce dynamic icon property
// for each record retrieved
mod.configureModule = function(options, callback)
{
    db.setProcessRows("post", "invoices", function(req, row, opts) {
       if (row.id) row.icon = "/images/" + row.id + ".png";
    });
    callback();
}

API requests handling

All methods will put input parameters in the req.query, GET or POST.

One way to verify input values is to use lib.toParams, only specified parameters will be returned and converted according to the type or ignored.

Example:

   var params = {
      test1: { id: { type: "text" },
               count: { type: "int" },
               email: { regexp: /^[^@]+@[^@]+$/ }
      }
   };

   api.app.all("/endpoint/test1", function(req, res) {
      var query = lib.toParams(req.query, params.test1);
      ...
   });

Example of TODO application

Here is an example how to create simple TODO application using any database supported by the backend. It supports basic operations like add/update/delete a record, show all records.

Create a file named app.js with the code below.

    var bkjs = require('backendjs');
    var api = bkjs.api;
    var lib = bkjs.lib;
    var app = bkjs.app;
    var db = bkjs.db;

    // Describe the table to store todo records
    db.describeTables({
       todo: {
           id: { type: "uuid", primary: 1 },  // Store unique task id
           due: {},                           // Due date
           name: {},                          // Short task name
           descr: {},                         // Full description
           mtime: { type: "now" }             // Last update time in ms
       }
    });

    // API routes
    app.configureWeb = function(options, callback)
    {
        api.app.get(/^\/todo\/([a-z]+)$/, function(req, res) {
           var options = api.getOptions(req);
           switch (req.params[0]) {
             case "get":
                if (!req.query.id) return api.sendReply(res, 400, "id is required");
                db.get("todo", { id: req.query.id }, options, function(err, rows) { api.sendJSON(req, err, rows); });
                break;
             case "select":
                options.noscan = 0; // Allow empty scan of the whole table if no query is given, disabled by default
                db.select("todo", req.query, options, function(err, rows) { api.sendJSON(req, err, rows); });
                break;
            case "add":
                if (!req.query.name) return api.sendReply(res, 400, "name is required");
                // By default due date is tomorrow
                if (req.query.due) req.query.due = lib.toDate(req.query.due, Date.now() + 86400000).toISOString();
                db.add("todo", req.query, options, function(err, rows) { api.sendJSON(req, err, rows); });
                break;
            case "update":
                if (!req.query.id) return api.sendReply(res, 400, "id is required");
                db.update("todo", req.query, options, function(err, rows) { api.sendJSON(req, err, rows); });
                break;
            case "del":
                if (!req.query.id) return api.sendReply(res, 400, "id is required");
                db.del("todo", { id: req.query.id }, options, function(err, rows) { api.sendJSON(req, err, rows); });
                break;
            }
        });
        callback();
     }
     bkjs.server.start();

Now run it with an option to allow API access without an account:

node app.js -log debug -web -api-allow-path /todo -db-create-tables

To use a different database, for example PostgresSQL(running localy) or DynamoDB(assuming EC2 instance), all config parametetrs can be stored in the etc/config as well

node app.js -log debug -web -api-allow-path /todo -db-pool dynamodb -db-dynamodb-pool default -db-create-tables
node app.js -log debug -web -api-allow-path /todo -db-pool pgsql -db-pgsql-pool default -db-create-tables

API commands can be executed in the browser or using curl:

curl 'http://localhost:8000/todo?name=TestTask1&descr=Descr1&due=2015-01-01`
curl 'http://localhost:8000/todo/select'

Backend directory structure

When the backend server starts and no -home argument passed in the command line the backend makes its home environment in the ~/.bkjs directory. It is also possible to set the default home using BKJS_HOME environment variable.

The backend directory structure is the following:

Cache configurations

Database layer support caching of the responses using db.getCached call, it retrieves exactly one record from the configured cache, if no record exists it will pull it from the database and on success will store it in the cache before returning to the client. When dealing with cached records, there is a special option that must be passed to all put/update/del database methods in order to clear local cache, so next time the record will be retrieved with new changes from the database and refresh the cache, that is { cached: true } can be passed in the options parameter for the db methods that may modify records with cached contents. In any case it is required to clear cache manually there is db.clearCache method for that.

Also there is a configuration option -db-caching to make any table automatically cached for all requests.

Local

If no cache is configured the local driver is used, it keeps the cache on the master process in the LRU pool and any wroker or Web process communicate with it via internal messaging provided by the cluster module. This works only for a single server.

memcached

Set ipc-cache=memcache://HOST[:PORT] that points to the host running memcached. To support multiple servrs add the option ipc-cache-options-servers=10.1.1.1,10.2.2.1:5000.

Redis

Set ipc-cache=redis://HOST[:PORT] that points to the server running Redis server.

To support more than one master Redis server in the client add additional servers in the servers parameter, ipc-cache-options-servers=10.1.1.1,10.2.2.1:5000, the client will reconnect automatically on every disconnect. To support quick failover it needs a parameter for the node-redis module (which is used by the driver) max_attempts to be a number how many attempts to reconnect before switching to another server like ipc-cache-options-max_attempts=3. Any other node-redis module parameter can be passed as well.

Cache configurations also can be passed in the url, the system supports special parameters that start with bk-, it will extract them into options automatically.

For example:

ipc-cache=redis://host1?bk-servers=host2,host3&bk-max_attempts=3
ipc-cache-backup=redis://host2
ipc-cache-backup-options-max_attempts=3

Redis Sentinel

To enable Redis Sentinel pass in the option -sentinel-servers: ipc-cache=redis://host1?bk-sentinel-servers=host1,host2.

The system will connect to the sentinel, get the master cache server and connect the cache driver to it, also it will listen constantly on sentinel events and failover to a new master autimatically. Sentinel use the regular redis module and supports all the same parameters, to pass options to the sentinel driver prefix them with sentinel-:

ipc-cache=redis://host1?bk-servers=host2,host3&bk-max_attempts=3&bk-sentinel-servers=host1,host2,host3
ipc-cache-backup=redis://host2
ipc-cache-backup-options-sentinel-servers=host1,host2
ipc-cache-backup-options-sentinel-max_attempts=5

PUB/SUB or Queue configurations

Publish/subscribe functionality allows clients to receive notifications without constantly polling for new events. A client can be anything but the backend provides some partially implemented subscription notifications for Web clients using the Long Poll. The Account API call /account/subscribe can use any pub/sub mode.

The flow of the pub/sub operations is the following:

Redis

To configure the backend to use Redis for PUB/SUB messaging set ipc-queue=redis://HOST where HOST is IP address or hostname of the single Redis server. This will use native PUB/SUB Redis feature.

Redis Queue

To configure the backend to use Redis for job processing set ipc-queue=redisq://HOST where HOST is IP address or hostname of the single Redis server. This driver implements reliable Redis queue, with visibilityTimeout config option works similar to AWS SQS.

Once configured, then all calls to jobs.submitJob will push jobs to be executed to the Redis queue, starting somewhere a backend master process with -jobs-workers 2 will launch 2 worker processes which will start pulling jobs from the queue and execute.

An example of how to perform jobs in the API routes:

   app.processAccounts = function(options, callback) {
       db.select("bk_account", { type: options.type || "user" }, function(err, rows) {
          ...
          callback();
       });
   }

   api.all("/process/accounts", function(req, res) {
       jobs.submitJob({ job: { "app.processAccounts": { type: req.query.type } } }, function(err) {
          api.sendReply(res, err);
       });
   });

RabbitMQ

To configure the backend to use RabbitMQ for messaging set ipc-queue=amqp://HOST and optionally amqp-options=JSON with options to the amqp module. Additional objects from the config JSON are used for specific AMQP functions: { queueParams: {}, subscribeParams: {}, publishParams: {} }. These will be passed to the corresponding AMQP methods: amqp.queue, amqp.queue.sibcribe, amqp.publish. See AMQP node.js module for more info.

DB

This is a simple queue implementation using the atomic UPDATE, it polls for new jobs in the table and updates the status, only who succeeds with the update takes the job and executes it. It is not effective but can be used for simple and not busy systems for more or less long jobs. The advantage is that it uses the same database and does not require additional servers.

SQS

To use AWS SQS for job processing set ipc-queue=https://sqs.amazonaws.com...., this queue system will poll SQS for new messeges on a worker and after succsesful execution will delete the message. For long running jobs it will automatically extend visibility timeout if it is configured.

Local

The local queue is implemented on the master process as a list, communication is done via local sockets between the master and workers. This is intended for a single server development pusposes only.

Security configurations

API only

This is default setup of the backend when all API requests except /account/add must provide valid signature and all HTML, Javascript, CSS and image files are available to everyone. This mode assumes that Web development will be based on 'single-page' design when only data is requested from the Web server and all rendering is done using Javascript. This is how the api.html develpers console is implemented, using JQuery-UI and Knockout.js.

To see current default config parameters run any of the following commands:

    bkjs run-backend -help | grep api-allow

    node -e 'require("backendjs").core.showHelp()'

To disable open registration in this mode just add config parameter api-deny-path=^/account/add$ or if developing an application add this in the initMiddleware

    api.initMiddleware = function(callback) {
        this.allow.splice(this.allow.indexOf('^/account/add$'), 1);
    }

Secure Web site, client verification

This is a mode when the whole Web site is secure by default, even access to the HTML files must be authenticated. In this mode the pages must defined 'Backend.session = true' during the initialization on every html page, it will enable Web sessions for the site and then no need to sign every API reauest.

The typical client Javascript verification for the html page may look like this, it will redirect to login page if needed, this assumes the default path '/public' still allowed without the signature:

   <script src="/js/jquery.js"></script>
   <link href="/css/bootstrap.css" rel="stylesheet">
   <script src="/js/bootstrap.js"></script>
   <script src="/js/knockout.js" type="text/javascript"></script>
   <script src="/js/crypto.js" type="text/javascript"></script>
   <script src="/js/bkjs.js" type="text/javascript"></script>
   <script src="/js/bkjs-bootstrap.js" type="text/javascript"></script>
   <script src="/js/bkjs-ko.js" type="text/javascript"></script>
   <script>
    $(function () {
       Bkjs.session = true;
       $(Bkjs).on("nologin", function() { window.location='/public/index.html'; });
       Bkjs.koInit();
   });
   </script>

Secure Web site, backend verification

On the backend side in your application app.js it needs more secure settings defined i.e. no html except /public will be accessible and in case of error will be redirected to the login page by the server. Note, in the login page Bkjs.session must be set to true for all html pages to work after login without singing every API request.

  1. We disable all allowed paths to the html and registration:
   app.configureMiddleware = function(options, callback) {
      this.allow.splice(this.allow.indexOf('^/$'), 1);
      this.allow.splice(this.allow.indexOf('\\.html$'), 1);
      this.allow.splice(this.allow.indexOf('^/account/add$'), 1);
      callback();
   }
  1. We define an auth callback in the app and redirect to login if the reauest has no valid signature, we check all html pages, all allowed html pages from the /public will never end up in this callback because it is called after the signature check but allowed pages are served before that:
   api.registerPreProcess('', /^\/$|\.html$/, function(req, status, callback) {
      if (status.status != 200) {
          status.status = 302;
          status.url = '/public/index.html';
      }
      callback(status);
   });

WebSockets connections

The simplest way is to configure ws-port to the same value as the HTTP port. This will run WebSockets server along the regular Web server. All requests must be properly signed with all parameters encoded as for GET requests.

Example:

    wscat --connect ws://localhost:8000
    connected (press CTRL+C to quit)
    > /account/get
    < {
        "status": 400,
        "message": "Invalid request: no host provided"
      }
    >

Versioning

There is no ready to use support for different versions of API at the same because there is no just one solution that satifies all applications. But there are tools ready to use that will allow to implement such versioning system in the backend. Some examples are provided below:

          api.all(/\/domain\/(get|put|del)/, function(req, res) {
              var options = api.getOptions(req);
              var cmd = req.params[0];
              if (options.appBuild) cmd += "/" + options.appBuild;
              switch (cmd) {
              case "get":
                  break;

              case "get/2015-01-01":
                  break;

              case "put":
                  break;

              case "put/2015-02-01":
                  break;

              case "del"
                  break;
              }
          });
        var options = api.getOptions(req);
        var version = lib.toVersion(options.appVersion);
        switch (req.params[0]) {
        case "get":
            if (version < lib.toVersion("1.2.5")) {
                res.json({ id: 1, name: "name", description: "descr" });
                break;
            }
            if (version < lib.toVersion("1.1")) {
                res.json([id, name]);
                break;
            }
            res.json({ id: 1, name: "name", descr: "descr" });
            break;
        }

The actual implementation can be modularized, split into functions, controllers.... there are no restrictions how to build the working backend code, the backend just provides all necessary information for the middleware modules.

The backend provisioning utility: bkjs

The purpose of the bkjs shell script is to act as a helper tool in configuring and managing the backend environment and as well to be used in operations on production systems. It is not required for the backend operations and provided as a convenience tool which is used in the backend development and can be useful for others running or testing the backend.

Running without arguments will bring help screen with description of all available commands.

The tool is multi-command utility where the first argument is the command to be executed with optional additional arguments if needed. On Linux, when started the bkjs tries to load and source the following config files:

    /etc/sysconfig/bkjs
    $BKJS_HOME/etc/profile

Any of the following config files can redefine any environmnt variable thus pointing to the correct backend environment directory or customize the running environment, these should be regular shell scripts using bash syntax.

Most common used commands are:

Deployment use cases

AWS instance setup with node and backendjs

Here is the example how to setup new custom AWS server, it is not required and completely optional but bkjs provies some helpful commands that may simplify new image configuration.

AWS instance as an appliance

To make an API appliance by using the backendjs on the AWS instance as user ec2-user with the backend in the user home

NOTE: if running behind a Load balancer and actual IP address is needed set Express option in the command line -api-express-options {"trust%20proxy":1}. In the config file replacing spaces with %20 is not required.

AWS Beanstalk deployment

As with any node.js module, the backendjs app can be packaged into zip file according to AWS docs and deployed the same way as any other node.js app. Inside the app package etc/config file can be setup for any external connections.

AWS Provisioning examples

Note: on OS X laptop the -aws-sdk-profile uc when AWS credentials are in the ~/.aws/credentials.

Make an AMI

On the running machine which will be used for an image:

bksh -aws-create-image -no-reboot

Use an instance by tag for an image:

bksh -aws-create-image -no-reboot -instance-id `bkjs show-instances -name api -fmt id | head -1`

Launch instances when not using AutoScaling Groups

When launching from an EC2 instance no need to specify any AWS credentials.

Launch Configurations

bksh -aws-create-launch-config -config-name elasticsearch -aws-sdk-profile uc -instance-type m3.large -update-groups -bkjs-cmd stop-service -bkjs-cmd init-logwatcher -bkjs-cmd "init-elasticsearch-service -memsize 50" -device /dev/xvda:gp2:16 -dry-run

Copy Autoscaling launch configs after new AMI is created

bksh -aws-create-launch-config -config-name jobs -aws-sdk-profile uc -update-groups -dry-run
bksh -aws-create-launch-config -config-name api -aws-sdk-profile uc -update-groups -dry-run

Update Route53 with all IPs from running instances

bksh -aws-set-route53 -name elasticsearch.ec-internal -filter elasticsearch

Proxy mode

By default the Web proceses spawned by the server are load balanced using default cluster module which relies on the OS to do scheduling. On Linux with node 0.10 this is proven not to work properly due to the kernel keeping the context switches to a minimum thus resulting in one process to be very busy while the others idle. Node versions 4 and above perform round-robin by default.

For such case the Backendjs implements the proxy mode by setting proxy-port config paremeter to any number above 1000, this will be the initial port for the web processes to listen for incoming requests, for example if use -proxy-port 3000 and launch 2 web processes they will listen on ports 3000 and 3001. The main server process will start internal HTTP proxy and will perform round-robin load balancing the incoming requests between the web proceses by forwarding them to the web processes over TCP and then returning the responses back to the clients.

Configure HTTP port

The first thing when deploying the backend into production is to change API HTTP port, by default is is 8000, but we would want port 80 so regardless how the environment is setup it is ultimatley 2 ways to specify the port for HTTP server to use:

Backend framework development (Mac OS X, developers)

Design considerations

While creating Backendjs there were many questions and issues to be considered, some i was able to implement, some still not. Below are the thoughts that might be useful when desining, developing or choosing the API platform:

API endpoints provided by the backend

All API endpoints are optional and can be disabled or replaced easily. By default the naming convention is:

 /namespace/command[/subname[/subcommand]]

Any HTTP methods can be used because its the command in the URL that defines the operation. The payload can be urlencoded query parameters or JSON or any other format supported by any particular endpoint. This makes the backend universal and usable with any environment, not just a Web browser. Request signature can be passed in the query so it does not require HTTP headers at all.

Authentication and sessions

Signature

All requests to the API server must be signed with account login/secret pair.

The resulting signature is sent as HTTP header bk-signature or in the header specified by the api-signature-name config parameter.

For JSON content type, the method must be POST and no query parameters specified, instead everything should be inside the JSON object which is placed in the body of the request. For additional safety, SHA1 checksum of the JSON paylod can be calculated and passed in the signature, this is the only way to ensure the body is not modified when not using query parameters.

See web/js/bkjs.js function Bkjs.createSignature or api.js function api.createSignature for the Javascript implementations.

There is also native iOS implementation Bkjs.m.

Authentication API

Accounts

The accounts API manages accounts and authentication, it provides basic user account features with common fields like email, name, address.

This is implemented by the accounts module from the core. To enable accounts functionality specify -allow-modules=bk_accounts.

Health enquiry

When running with AWS load balancer there should be a url that a load balancer polls all the time and this must be very quick and lightweight request. For this purpose there is an API endpoint /ping that just responds with status 200. It is not open by default, the allow-path or other way to allow non-authenticted access needs to be configured. This is to be able to control how pinging can be perform in the apps in cae it is not simple open access.

Public Images endpoint

This endpoint can server any icon uploaded to the server for any account, it is supposed to be a non-secure method, i.e. no authentication will be performed and no signagture will be needed once it is confgiured which prefix can be public using api-allow or api-allow-path config parameters.

The format of the endpoint is:

Icons

The icons API provides ability for an account to store icons of different types. Each account keeps its own icons separate form other accounts, within the account icons can be separated by prefix which is just a namespace assigned to the icons set, for example to keep messages icons separate from albums, or use prefix for each separate album. Within the prefix icons can be assigned with unique type which can be any string.

Prefix and type can consist from alphabetical characters and numbers, dots, underscores and dashes: [a-z0-9._-]. This means, they are identificators, not real titles or names, a special mapping between prefix/type and album titles for example needs to be created separately.

The supposed usage for type is to concatenate common identifiers first with more specific to form unique icon type which later can be queried by prefix or exactly by icon type. For example album id can be prefixed first, then sequential con number like album1:icon1, album1:icon2.... then retrieving all icons for an album would be only query with album1: prefix.

The is implemented by the icons module from the core. To enable this functionality specify -allow-modules=bk_icons.

File API

The file API provides ability to store and retrieve files. The operations are similar to the Icon API.

This is implemented by the files module from the core. To enable this functionality specify -allow-modules=bk_files.

Connections

The connections API maintains two tables bk_connection and bk_reference for links between accounts of any type. bk_connection table maintains my links, i.e. when i make explicit connection to other account, and bk_reference table is automatically updated with reference for that other account that i made a connection with it. No direct operations on bk_reference is allowed.

This is implemented by the connections module from the core. To enable this functionality specify -allow-modules=bk_connections.

Locations

The location API maintains a table bk_location with geolocation coordinates for accounts and allows searching it by distance. The configuration parameter min-distance defines the radius for the smallest bounding box in km containing single location, radius searches will combine neighboring boxes of this size to cover the whole area with the given distance request, also this affects the length of geohash keys stored in the bk_location table. By default min-distance is 5 km which means all geohashes in bk_location table will have geohash of size 4. Once min-distance is set it cannot be changed without rebuilding the bk_location table with new geohash size.

The location search is implemented by using geohash as a primary key in the bk_location table with the account id as the second part of the primary key, for DynamoDB this is the range key. When request comes for all matches for the location for example 37.7, -122.4, the search that is executed looks like this:

This is implemented by the locations module from the core. To enable this functionality specify allow-modules=bk_locations.

Messages

The messaging API allows sending and receiving messages between accounts, it supports text and images. All new messages arrive into the bk_messsage table, the inbox. The client may keep messages there as new, delete or archive them. Archiving means transfering messages into the bk_archive table. All sent messages are kept in the bk_sent table.

This is implemented by the messages module from the core. To enable this functionality specify -allow-modules=bk_messages.

Counters

The counters API maintains realtime counters for every account records, the counters record may contain many different counter columns for different purposes and is always cached with whatever cache service is used, by default it is cached by the Web server process on every machine. Web worker processes ask the master Web server process for the cached records thus only one copy of the cache per machine even in the case of multiple CPU cores.

This is implemented by the counters module from the core. To enable this functionality specify -allow-modules=bk_counters|bk_accounts.

Data

The data API is a generic way to access any table in the database with common operations, as oppose to the any specific APIs above this API only deals with one table and one record without maintaining any other features like auto counters, cache...

Because it exposes the whole database to anybody who has a login it is a good idea to disable this endpoint in the production or provide access callback that verifies who can access it.

This is implemented by the data module from the core.

Pages

The pages API provides a simple Wiki like system with Markdown formatting. It keeps all pages in the database table bk_pages and exposes an API to manage and render pages.

The pages support public mode, all pages with pub set to true will be returning without an account, this must be enabled with api-allow-path=^/pages/(get|select|show) to work.

All .md files will be rendered into html automatically if there is not _raw=1 query parameter and pages view exists (api-pages-view=pages.html by default).

This is implemented by the pages module from the core. To enable this functionality specify -allow-modules=bk_accounts.

System API

The system API returns information about the backend statistics, allows provisioning and configuration commands and other internal maintenance functions. By default is is open for access to all users but same security considerations apply here as for the Data API.

This is implemented by the system module from the core. To enable this functionality specify -allow-modules=accounts.

Author

Vlad Seryakov

Check out the Documentation for more details.

Configuration parameters

Module: API

Module: API_ACCOUNTS

Module: API_AUTH

Module: API_FILES

Module: API_ICONS

Module: API_PROXY

Module: API_STATISTICS

Module: APP

Module: AWS

Module: AWS_DYNAMODB

Example:

      ddbCreateTable('users', { id: 'S', mtime: 'N', name: 'S'},
                              { keys: ["id", "name"],
                                local: { mtime: { mtime: "HASH" } },
                                global: { name: { name: 'HASH', ProvisionedThroughput: { ReadCapacityUnits: 50 } } },
                                projections: { mtime: ['gender','age'],
                                               name: ['name','gender'] },
                                readCapacity: 10,
                                writeCapacity: 10 });

Module: AWS_EC2

Module: AWS_S3

Module: AWS_SNS

Module: CORE

In the main app.js just load it and the rest will be done automatically, i.e. routes will be created ...

   var mymod = require("./mymod.js");

Running the shell will make the object mymod available

   ./app.sh -shell
   > mymod
     {}

Module: CORE_UTILS

Module: DB

The following databases are supported with the basic db API methods: Sqlite, PostgreSQL, MySQL, DynamoDB, MongoDB, Elasticsearch, Cassandra, Redis, LMDB, LevelDB, Riak, CouchDB

All these drivers fully support all methods and operations, some natively, some with emulation in the user space except Redis driver cannot perform sorting due to using Hash items for records, sorting can be done in memory but with pagination it is not possible so this part must be mentioned specifically. But the rest of the operations on top of Redis are fully supported which makes it a good candidate to use for in-memory tables like sessions with the same database API, later moving to other database will not require any application code changes.

Multiple connections of the same type can be opened, just add -n suffix to all database config parameters where n is a number, referer to such pools in the code as poolN.

Example:

      db-pgsql-pool = postgresql://locahost/backend
      db-pgsql-pool-1 = postgresql://localhost/billing
      db-pgsql-pool-max-1 = 100

      in the Javascript:

      db.select("bills", { status: "ok" }, { pool: "pgsql1" }, lib.log)

Pass MongoDB options directly: db.create("test_table", { id: { primary: 1, type: "int", mongodb: { w: 1, capped: true, max: 100, size: 100 } }, type: { primary: 1, pub: 1 }, name: { index: 1, pub: 1, mongodb: { sparse: true, min: 2, max: 5 } } });

Example

  db.setProcessRow("post", "bk_account", function(req, row, opts) {
      if (row.birthday) row.age = Math.floor((Date.now() - lib.toDate(row.birthday))/(86400000*365));
  });

  db.setProcessRow("post", "bk_icon", function(req, row, opts) {
      if (row.type == "private" && row.id != opts.account.id) return true;
  });

Module: DB_DYNAMODB

Module: DB_ELASTICSEARCH

Module: DB_SQL

Module: DB_SQLITE

Module: HTTP_GET

Module: IPC

Example

ipc.get(["my:key1", "my:key2"], function(err, data) { console.log(data) });
ipc.get("my:key", function(err, data) { console.log(data) });
ipc.get("my:counter", { set: 10 }, function(err, data) { console.log(data) });
ipc.get("*", { mapName: "my:map" }, function(err, data) { console.log(data) });
ipc.get("key1", { mapName: "my:map" }, function(err, data) { console.log(data) });
ipc.get(["key1", "key2"], { mapName: "my:map" }, function(err, data) { console.log(data) });

Module: IPC_CLIENT

Module: IPC_LOCAL

Module: IPC_MEMCACHE

Module: IPC_REDIS

Module: IPC_SQS

Module: JOBS

Module: LIB

Module: LOGGER

Module: METRICS

Module: MSG

Module: MSG_APN

Module: MSG_FCM

Module: MSG_GCM

Module: SERVER

Module: BK_ACCOUNT

Module: BK_CONNECTION

Module: BK_COUNTER

Module: BK_DATA

Module: BK_FILE

Module: BK_ICON

Module: BK_LOCATION

Geo grid for browsing is a list from the center all rows up, this means that records with smaller distances will appear again when geohash range is big: center left01 right01 left02 right02 top1 left11 right11 left12 right12 bottom2 left21 right21 left22 right22 top3 left31 right31 left32 right32 bottom4 left41 right41 left42 right42

Module: BK_MESSAGE

Module: BK_SHELL

Module: BK_SHELL_AWS

Module: BK_SHELL_DB

Module: BK_STATUS

Module: BK_SYSTEM

Module: DB_CASSANDRA

Module: DB_COUCHDB

Module: DB_LMDB

Module: DB_MONGODB

Module: DB_MYSQL

Module: DB_PGSQL

Module: DB_REDIS

Module: DB_RIAK

Module: IPC_AMQP

Module: IPC_DB

Module: IPC_HAZELCAST

Module: MSG_SNS