Madcap Idea Part 3 : Bringing Play Back End Into The Fold + Some Basic Streaming

Last Time

Last time we looked at bringing in the Inversify.js JS IOC container. This time we will bring a Play Framework (scala based MVC web framework) into the fold, and shall be using the front end we have been working on so far to be the front end for the Play Framework back end.

PreAmble

Just as a reminder this is part of my ongoing set of posts which I talk about here :

https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

• WebPack
• React.js
• React Router
• TypeScript
• Babel.js
• Akka
• Scala
• Play (Scala Http Stack)
• MySql
• SBT
• Kafka
• Kafka Streams

What Is The Play Framework?

The Play Framework is a Scala based MVC (model view controller) type web application framework. As such it has in built mechanisms for things typical of a MVC web framework (certainly if you have done any ASP MVC . NET you would find it very familiar).

So we have the typical MVC concerns covered by the Play Framework

• Controllers
• Actions
• Routing
• Model binding
• JSON support
• View engine

Thing is I will not be doing any actual HTML in the Play Framework back end code, I want to do all of that using the previously covered webpack/typescript/react starter code I have shown so far. Rather I will be using the Play Framework as a API backend, where we will simply be using various controllers as endpoint to accept/serve JSON, and Event streamed data. All the actual front end work/routing will be done via webpack and React.

There are still some very appealing parts in Play that I did want to make use of, such as:

• It is Scala, which means when I come to integrate Kafka / Kafka Streams it will be ready to do so
• It uses Akka which I wanted to use. I also want to use Akka streams, which Play also supports
• Play controllers lend themselves quite nicely to being able to create a fairly simple REST API
• It can be used fairly easily to serve static files (think of these as the final artifacts that come out of the webpack generation pipeline). So things like minimized CSS / JS etc etc

So hopefully you can see that using Play Framework still made a lot of sense, even if we will only end up using 1/2 of what it has to offer. To be honest the idea of using controllers for a REST API is something that is done in ASP MVC .NET all time either by using of actual controllers or by using the WebApi.

Ok so now that we know what we will be using Play Framework for, how about we dive into the code for this post.

Play Framework Basics

Lets start by looking at the bare bones structure of a Play Framework application, which looks like this (I am using IntelliJ IDEA as my IDE)

Lets talk a bit about each of these folders

app

This folder would hold controllers/views (I have added the Entities folder there that is not part of a Play Framework application requirements). Inside the controllers folder you would find controllers, and inside the views folder you would find views. For the final app there will be no views folder, I simply kept that in this screenshot to talk about what a standard Play Framework application looks like

conf

This folder contains the configuration for the Play Framework application. This would include the special routes file, and any other application specific configuration would might have.

Lets just spend a minute having a look at the Play Framework routes file, which you can read more about here : https://www.playframework.com/documentation/2.5.x/ScalaRouting

The routes file has its own DSL, that is responsible for matching a give route with a controller + controller action. The controller action that matches the route is ultimately responsible for servicing the http request. I think the DSL shown in routes file below is pretty self explanatory with perhaps the exception of the assets based routes.

All assets based http requests (ie ones that start with /assets for example http://localhost:9000/assets/images/favicon.png would actually be routed through to a special controller called Assets. You dont see any code for this one, its part of the Play Framework application codebase. This special Assets inbuilt play controller is responsible for serving up static data files which it expects to find in the public folder. So for example our initial request of http://localhost:9000/assets/images/favicon.png would get translated into this file (relative path from project root) /public/images/favicon.png. As I say this is handled for you by the special Assets built in controller.

The only other funky part to the Assets based route is that it uses a *file in its route. Which essentially boils down to the play framework being able match a multi-part path. Which we actually just saw with the example above http://localhost:9000/assets/images/favicon.png , see how that contains not only the file name, but also a directory of images. The Assets controller + routing is able to deal with that path just fine.

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~

# Home page

GET        /                                   controllers.HomeController.index()

GET        /scala/comet/liveClock              controllers.ScalaCometController.streamClock()
GET        /scala/comet/kick                   controllers.ScalaCometController.kickRandomTime()

# Map static resources from the /public folder to the /assets URL path
GET        /assets/*file                       controllers.Assets.at(path="/public", file)

Ok so moving on to the rest of the standard folders that come with a Play Framework application

public

This is where you will need to put any static content that you wish to be served. Obviously views (if you use that part of play) will be within in the app/views folder. Like I say I am not using the views aspect of Play so you will not be seeing any views in my views folder. I instead want to let webpack et all generate my routing, web page etc etc. I do however want to serve bundles so later on I will be showing you how my webpack generated bundles fit in with the Play Framework ecco system.

target

Since this is a scala based project we get the standard scala based folders, and target is one of them, that has the generated/compiled code in it.

SBT

It is worth pointing out that my Play Framework application is an SBT based project, as such there is an SBT aspect to it, which largely boils down to these files

Project [root-build] / plugs.sbt file

This file adds Play as a plugin for the SBT project

// The Lightbend repository
resolvers += Resolver.typesafeRepo("releases")

// Use the Play sbt plugin for Play projects
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.5.14")

build.sbt

This is the main SBT build file for the project. This is where all our external dependencies are brought in etc etc (standard SBT stuff)

import play.sbt._
import sbt.Keys._
import sbt._

name := "play-streaming-scala"

version := "1.0-SNAPSHOT"

scalaVersion := "2.11.11"

lazy val root = (project in file(".")).enablePlugins(play.sbt.PlayScala)

javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")

initialize := {
  val _ = initialize.value
  if (sys.props("java.specification.version") != "1.8")
    sys.error("Java 8 is required for this project.")
}

So I think that covers the basics of a standard Play Framework application, the remainder of this post will look at the following aspects

• How do we provide a stream based request, that allows server sent events to be sent back to JavaScript from server side code
• How do we integrate the webpack based front end we have been playing with in previous posts

How do provide a stream based request (server sent events)

One of the main ingredients of the application we have chosen to build is the ability to stream data back to the JavaScript in response to incoming Kafka data. Ok we have not got to the Kafka part yet, but we know we will need some way of pushing data from the Play Framework application code to the front end JavaScript code. So how do we do that?

Well there are several parts to this, so lets just go through them in turn

Play streaming route

There are 2 routes for the example stream for this post

• The actual stream endpoint route itself
• An endpoint that allows us to produce a new value for the Akka Stream that is exposes in the stream endpoint route

Here is the full routing code for these 2 routes

GET        /scala/comet/liveClock              controllers.ScalaCometController.streamClock()
GET        /scala/comet/kick                   controllers.ScalaCometController.kickRandomTime()

Play stream provider

The job of creating the stream is handled by the ScalaCometController where it uses Akka Streams to provide the stream itself. The clever part is what parts of the Akka Streams API we used.

I have chosen to use MergeHub and BroadCastHub, which act as Fan-In and Fan-Out stages respectively. This allows us to have many producers and many consumers that are able to push and listen to the stream.

I have also supplied a route kickRandomTime() which will produce a new random JSON value that will either be a Location or a Resident domain object entity converted to JSON.

Another thing to note is how we handle stream errors. We use ideas borrowed from regular Akka where we use a strategy to manage the stream

Here is the full code to the ScalaCometController

package controllers

import java.time.ZonedDateTime
import java.time.format.DateTimeFormatter
import javax.inject.{Inject, Singleton}

import akka.actor.ActorSystem
import akka.stream.{ActorMaterializer, ActorMaterializerSettings, Materializer, Supervision}
import akka.stream.scaladsl.{BroadcastHub, Keep, MergeHub, Source}
import play.api.http.ContentTypes
import play.api.libs.Comet
import play.api.libs.json._
import play.api.mvc.{Controller, _}
import Entities._
import Entities.JsonFormatters._

import scala.concurrent.ExecutionContext


object flipper
{
  var current: Boolean = false
}


@Singleton
class ScalaCometController @Inject()
  (
    implicit actorSystem: ActorSystem,
    ec: ExecutionContext
  )
  extends Controller  {

  //Error handling for streams
  //http://doc.akka.io/docs/akka/2.5.2/scala/stream/stream-error.html
  val decider: Supervision.Decider = {
    case _                      => Supervision.Restart
  }

  implicit val mat = ActorMaterializer(
    ActorMaterializerSettings(actorSystem).withSupervisionStrategy(decider))


  val (sink, source) =
    MergeHub.source[JsValue](perProducerBufferSize = 16)
      .toMat(BroadcastHub.sink(bufferSize = 256))(Keep.both)
      .run()

  def streamClock() = Action {
    Ok.chunked(source via Comet.json("parent.clockChanged")).as(ContentTypes.HTML)
  }

  def kickRandomTime() = Action {

    var finalJsonValue:JsValue=null

    flipper.current = !flipper.current
    if(flipper.current) {
      finalJsonValue = Json.toJson(Location(1.0,1.0))
    } else {
      val rand = DateTimeFormatter.ofPattern("ss mm HH").format(ZonedDateTime.now().minusSeconds(scala.util.Random.nextInt))
      finalJsonValue = Json.toJson(Resident("Hove", 12, Some(rand)))
    }
    Source.single(finalJsonValue).runWith(sink)
    Ok(s"We sent '$finalJsonValue'")
  }

}

Client side Comet frame

The Play Framework has support for web sockets, and we could have used a web socket, but there are times when you dont want to initiate some comms, you just want notifications of something that happened on the server. Luckily the Play Framework has support for this too using Comet, which you read more about here : https://www.playframework.com/documentation/2.5.x/ScalaComet

What this really boils down to is have a IFRAME that is permanently part of the rendered HTML (forever-iframe technique) that is linked up to a comet play url. For me this is like this

<iframe id="comet" src="/scala/comet/liveClock"></iframe>

Where it can be seen that this uses the route

GET        /scala/comet/liveClock              controllers.ScalaCometController.streamClock()

Which we saw above

Client side RX comet listener

So we have now seen how the back end produces a stream of differing JSON object (ok toggles between 2 fixed JSON objects), and how we use the forever-iframe idea with a comet based Play route. So what about the message coming into the JavaScript what does that look like?

Well here is the relevant code, where we wrap the code we want to run in a self executing function. There are a couple of points here of note

Ok.chunked(source via Comet.json("parent.clockChanged")).as(ContentTypes.HTML))

• We create a top level window function called clockChanged which ties up with the back end comet route code (
• We create a custom event that we dispatch via the window
• We use that custom event to create an Rx Observable (I want this separation as long term the Rx code will become a service that is injected into the props of a React component from the IOC container)

import Rx from 'rx';  


(function () {

    var evt;

    window['clockChanged'] = function (incomingJsonPayload) {
        evt = new CustomEvent('onClockChanged', { detail: incomingJsonPayload });
        window.dispatchEvent(evt);
    }

    var source = Rx.Observable.fromEvent(window, 'onClockChanged');

    var subscription = source.subscribe(
        function (x) {
            console.log('RX saw onClockChanged');
            console.log('RX x = ', x.detail);
        },
        function (err) {
            console.log('Error: %s', err);
        },
        function () {
            console.log('Completed');
        });

} ());

And that is all there is to that part. As I say this is demo code and will be moved about a bit over time, but it does show the moving parts ok.

How do integrate our existing webpack based front end with Play

Up until now I have avoided talking about the view, and Play and the code from the webpack based front end we have seen in the previous posts. Things is we can do views in play but I was kind of keen to do all my front end work in TypeScript/react/webpack. So I wanted play to merely serve up my front end code from those tools. What that means is really serving up the final JS/CSS bundles and index.html produced by webpack and the awesome HtmlWebpackPlugin (that uses a template and injects the right script bundles). So can this webpack based workflow work ok with Play. I thought about it a bit, and came up with this.

Making sure webpack generates bundles to correct place

The first thing to do was to include the code in the Play application at some location. For me this as follows:

You want to avoid putting to much stuff under the public folder in Play as Play treats these files as static files, that can be cached etc etc, so 100MB of node modules should defo not go under public if you can help it.

Then from there we need to ensure the bundles get built to the correct location, which for my setup and my Play application meant the public folder

Making sure bundles are prefixed

The other thing I needed to do was make sure the webpack output was setup to have the correct public path.

To do the 2 things above, I changed my webpack config file to this

With these changes in place this is what I now get for my index.html page in the public\dist folder

<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8" />
        <title>Hello React!</title>

       
		<link href="/assets/dist/vendor.bundle.994d68871d44c4fc440c.css" rel="stylesheet">
		<link href="/assets/dist/indexCss.bundle.994d68871d44c4fc440c.css" rel="stylesheet"></head>
    <body>
         <div>
             <iframe id="comet" src="/scala/comet/liveClock"></iframe>
         </div>
        <div id="example"></div>
        <!-- Main -->
		<script type="text/javascript" src="/assets/dist/vendor.bundle.994d68871d44c4fc440c.js"></script>
		<script type="text/javascript" src="/assets/dist/index.bundle.994d68871d44c4fc440c.js"></script>
	</body>
</html>

See how the bundles point to the assets folder, which we now know Play will serve using the AssetsController.

Serving the initial webpack generated HTML page

The last piece of the puzzle is how do we get a standard play route to serve up this index.html file? Well here is how

package controllers


import javax.inject.Inject

import play.api.mvc.{Action, Controller}

class HomeController @Inject() (environment: play.api.Environment)
  extends Controller {

  def index() = Action {
    val fullpath = s"${environment.rootPath}\\public\\dist\\index.html"
    val htmlContents = scala.io.Source.fromFile(fullpath).mkString
    Ok(htmlContents).as("text/html")
  }

}

Quite cunning no, we just read the file as text, and serve the string contents as HTML. Where the route for this is still using the standard Play route

GET        /                                   controllers.HomeController.index()

And bam, we now have a streaming Akka/Play/Webpack app working. Neato

A Demo

Now that I have talked about most of the parts, lets see a little demo of it working, so we need to load our site up (which is the GET / route above)

Our ugly but functional, react page is shown. Not very exciting, lets see the console (clear it), and load up the great REST tool Postman, and poke the stream simulation endpoint, and watch our Rx JavaScript code write out to the console

Here is the console after a few Send clicks in postman for the http://localhost:9000/scala/comet/kick route

Conclusion

So that is all I wanted to say this time. I think it has worked out fairly well. Next time we will be looking at adding react-routing into the fold, and create some routes, and dummy place holder pages which we develop along the way (after we do the post on prototyping the screens of course)