A cron job with nodejs, sails and flickr

Introduction

I am writing a flickr API image extractor app.

Technologies

• Nodejs with Sails framework
• Flickr API (of course)

Pre-Requisite Background

• You should have experience using Sails framework and nodejs.
• You should have a new sails project ready to put code into.

New Project Startup Steps

• sails new flickrAccess - To create a new flickrAccess sails project
• npm install flickrAPI - nodejs library to connect with flickr: https://www.npmjs.com/package/flickrapi

Loading the App Like a Process Without a Web Server on Sails

Sails is primarily a web framework, so the usual 'sails lift' command also creates and opens a web server. For our purposes, this will not be needed, and in fact could be a security concern making the code accessible through http.

We have two options, both involve using default node command to start the app with 'node <filename>', instead of doing sails lift on command / shell prompt. (Sidebar: I develop on windows and host on linux, so still I am going to be referring to the prompt as a command prompt or cmd)

Add the below code to the top of our app. This is one of the first things we write in our flickr-get-interesting.js file (that is what I am calling the root app code):

var Sails = require('sails');

Difference between Sails Lift / Load

Whether you use a Lift / Load has to do with just one simple difference: Whether you need a web server or not. Lift will create and start a web server.

Sails.lift({
  log: {
    level: 'silent'
  }
}, function() {

console.log('sails started..do commands');

process.stdin.destroy();
});

OR

Sails.load({
   log:    {level:'verbose'}    
  },     function(err, sails) {

console.log('sails started..do commands');

});

We will use Load obviously since we don't require a server, and we won't need to modify the default log level setting as well, so all we need is:

Sails.load (function(err, sails){

console.log('sails started..do commands');

});

At this point, if you execute this node flickr-get-interesting, you should see the log we are printing.

sails started..do commands

The Flickr API

At this point, you need to have a flickr account, api_key, etc., Which you can do here: https://www.flickr.com/services/apps/create/

Required

• flickr api_key from flickr App Garden
• flickr secret

A Utility Class for Our Utility Methods

Let's create a flickrUtils.js file with the below code:

var util = require('util');

module.exports = {

    api_key: "9d520e52f32630e1469231xxxxxxxxxx",  //replace with your api_key
    secret: "3e23bd445xxxxxxxx"   // replace with your api secret

};

The above code will ensure we can access our api_key / secret with something like myUtils.api_key if we mention this with a require on our flickr-get-interesting.js the main code file.

Initiating flickr API

flickr is one of those APIs which only needs api_key and secret and once initiated, you query it using the API methods almost like sails blueprint API.

Let's add all our require's into our main file and initiate flickr.

Steps

1. Start sails with a load - without a web server
2. Initiate flickr API with the api_key and secret and get an object to use the flickr API methods from there on.

//flickr-get-interesting.js

var util = require('util');
var Sails = require('sails');
var flickrUtils = require('./flickrUtils.js');

var oFlickr;   //using an external scoped variable for the api object, 
               //so we can use it in our other util methods without passing back-n-forth.

Sails.load(function(err, sails) {

    console.log('sails started..do commands');

    var Flickr = require("flickrapi"),
        flickrOptions = {
            api_key: flickrUtils.api_key,
            secret: flickrUtils.secret
        };    

    Flickr.tokenOnly(flickrOptions, function(error, flickr) {

        oFlickr = flickr;
    });
});

Before you run this, ensure all your require'd files are in the root directory on your sails project (- Remember: we are not starting a web server or creating assets to load from a different directory.. unlike a regular sails web project).

When you run it, flickr API takes quite some time to initiate.. like 2-3 minutes even (not sure, but I believe it downloads some flickr library files on first initiation... after which almost every other time, the initiation will be much quicker).

Ah! If you didn't notice: we missed a part: "how do you run it ?"

• .. not with sails lift
• use the below command on your command/shell prompt at the root directory of your sails project

node flickr-get-interesting.js

process.exit();

We did not add the process.exit(); line in our code, so flickr initiates and sails waits with a loaded app.

As we are not expecting it to run forever (we expect to run it like a cron job every few minutes), we will add the code process.exit();

Get Interesting

flickr has an 'interesting' attribute in photos and an API method to get all the recent interesting photos.

We will add a call to this, and we will move our process.exit(); into the Ajax success/error blocks of this method, so our process doesn't exit before response is received from flickr.

Flickr.tokenOnly(flickrOptions, function(error, flickr) {

        oFlickr = flickr;

        // we can now use "flickr" as our API object,
        // but we can only call public methods and access public data

        //flickr.photos.getRecent({
        oFlickr.interestingness.getList({
                page: 1,   
                per_page: 1    //change this appropriately later after debugging 
                               //to get more of the interesting images
            },
            function(err, result) {
                if (err) {
                    throw new Error(err);

                    finished();
                }

                //print the result object to console.log
                 console.log(util.inspect(result, {
                    showHidden: false,
                    depth: null
                }));

                // do something with result

                 finished();
            });
        });
});

function finished(){
    process.exit();
}

In the same way, as above, you could call many different flickr methods. API reference is at https://www.flickr.com/services/api/.

The below revisions of the app, and subsequent source code versions behave different from console app.

Revision 2

What Is in This ReVision

1. Refactor our code into different files like a professional project should be.
2. Store flickr images in an array and process them.

Refactor

flickrUtils.js

We will write our flickr utility methods here.

process.js

Create a process.js and create all flickr API calls as wrapper methods into this.

Now to be easy to work on the flickr object from different methods, we will put the flickr object in a variable on our process object like below.

As all flickr methods are asynchronous, and to avoid writing code into the response events on flickr calls, we will create a model/object structure to store values received from flickr methods into models. You could simply use JavaScript array objects... but we are going to do this as a model in flickr.

promises

To simplify the async calls, its success and failures, we are going to use promises from the Q library.

We use Q promise library within a loop at one place which is explained here.

Revision 3

What Is in This Revision

We will continue use sails, but with a web API for flickrAccess.

• The 3^rd revision of flickrAccess is to make a web API with a model on the backend.
• The web API will be called from a browser instance and set as a cron job, just like PHP cron jobs.
• We will create an API in sails to record data to a mongo DB.

Custom Model in Sails

Sails will come with /api/models folder by default... for loading models in a structure.

We will create a custom model in this directory.

Create fPhoto.js with the below structure in /api/models.

After this, when you start sails, you will see this message.. you may choose 1 for safe, as we are not immediately using this model in connection with a database. The migrate option makes sense only in connection with a database... more here on migrate settings. And ah! yes, that does mean sails models were not designed to be used without databases (at least as of this writing on 16^th May, 2015)... Also the migrate option clearly doesn't have options to be offline. So using a custom model for entity storage may be considered like pushing for using sails and you could simply use a normal array object.

Excuse my interruption, but it looks like this app does not have a project-wide "migrate" setting configured yet.
(perhaps this is the first time you're lifting it with models?)

And to avoid the above message (as it will block app execution until user entry), make the setting for 'migrate' permanently under /config/models.js.

Next Steps

We can improve on this by adding code to store values into a database.

As a Cron job

And to setup as a cron job, you would setup the below command on a cron with path to the filename:

node /myproject folder path/flickr-get-interesting.js