Google App Engine - Go Tutorial

Introduction

Welcome to Google App Engine! Creating an App Engine application is easy, and only takes a few minutes. And it's free to start: upload your app and share it with users right away, at no charge and with no commitment required.

Google App Engine applications can be written in the Go, Java, Python or PHP programming languages. This tutorial covers Go. If you would prefer to use Java, Python or PHP to build your applications, see the Java, Python 2.7 or PHP guides.

In this tutorial, you will learn how to:

• build an App Engine application using Go,
• use the Go http package to serve web pages,
• use the App Engine datastore with the Go datastore API,
• integrate an App Engine application with Google Accounts for user authentication,
• use Go's html/template package with your app, and
• upload your app to App Engine.

By the end of the tutorial, you will have implemented a working application, a simple guest book that lets users post messages to a public message board.

To get started developing Google App Engine applications, you download and set up the App Engine software development kit.

The Development Environment

You develop and upload Go applications for Google App Engine using the App Engine Go software development kit (SDK).

The Go SDK includes a web server application that simulates the App Engine environment, including a local version of the datastore, Google Accounts, and the ability to fetch URLs and send email directly from your computer using the App Engine APIs. The Go SDK uses slightly modified versions of the development tools from the Python SDK, and will run on any Intel-based Mac OS X, Linux or Windows computer with Python 2.7.

If necessary, download and install Python 2.7 for your platform from the Python web site. Most Mac OS X users already have Python 2.7 installed. If you have issues with the Python tools, please ensure you have Python 2.7 installed.

Download and install the App Engine SDK for Go for your operating system.

For this tutorial, you will use two commands from the goapp tool in the SDK:

• goapp serve, for running a local development server
• goapp deploy, for uploading your app to App Engine

You can find these commands in the go_appengine directory of the zip archive.

To simplify development and deployment, consider adding this directory to your PATH environment variable. This can be done by adding the following line to your $HOME/.profile, $HOME/.bashrc, or equivalent:

export PATH=/path/to/go_appengine:$PATH

The local development environment lets you develop and test complete App Engine applications before showing them to the world. Let's write some code.

Hello, World!

Go App Engine applications communicate with the outside world via a web server compatible with Go's http package. This makes writing Go App Engine applications very similar to writing stand-alone Go web applications.

Let's begin by implementing a tiny application that displays a short message.

Creating a Simple HTTP Handler

Create a directory named myapp. All files for this application reside in this directory.

Inside the myapp directory, create a file named hello.go, and give it the following contents:

package hello

import (
    "fmt"
    "net/http"
)

func init() {
    http.HandleFunc("/", handler)
}

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprint(w, "Hello, world!")
}

This Go package responds to any request by sending a response containing the message Hello, world!.

Note: When writing a stand-alone Go program we would place this code in package main. The Go App Engine Runtime provides a special main package, so you should put HTTP handler code in a package of your choice (in this case, hello).

Creating the Configuration File

An App Engine application has a configuration file called app.yaml. Among other things, this file tells the App Engine service which runtime to use and which URLs should be handled by our Go program.

Inside the myapp directory, create a file named app.yaml with the following contents:

application: helloworld
version: 1
runtime: go
api_version: go1

handlers:
- url: /.*
  script: _go_app

From top to bottom, this configuration file says the following about this application:

• The application identifier is helloworld. When you register your application with App Engine later in this tutorial, you will select a unique identifier, and update this value. This value can be anything during development. For now, leave it set to helloworld.
• This is version number 1 of this application's code. If you adjust this before uploading new versions of your application software, App Engine will retain previous versions, and let you roll back to a previous version using the administrative console.
• This code runs in the go runtime environment, with API version go1.
• Every request to a URL whose path matches the regular expression /.* (all URLs) should be handled by the Go program. The _go_app value is a magic string recognized by the development web server; it is ignored by the production App Engine servers.

Note: the Go SDK does things differently than the Python and Java SDKs: all Go packages for a given app are built into a single executable, and request dispatch is handled by the Go program itself. This is why we call http.HandleFunc inside the init function to associate our handler with the web root ("/"). However, you may still use the app.yaml file to configure paths that serve static files or require special permissions.

The syntax of this file is YAML. For a complete list of configuration options, see the Go Application Configuration page.

Testing the Application

With the hello package and configuration file mapping every URL to the Go program, the application is complete. You can now test it with the web server included with the App Engine SDK.

Check that you have everything in its right place. The application's directory structure should look like this:

myapp/
  app.yaml
  hello.go

Run the following command, giving it the path to the myapp directory, to compile your app and start the development web server:

/path/to/go_appengine/goapp serve myapp/

You may drop the /path/to/go_appengine/ if you added it to your PATH, as suggested earlier. You can also omit the path altogether if myapp is your current directory, so the command is simply:

goapp serve

The web server is now running, listening for requests on port 8080. Test the application by visiting the following URL in your web browser:

• http://localhost:8080/

For more information about running the development web server, including how to change which port it uses, see the Development Server reference, or run goapp help serve.

Iterative Development

The development app server knows to watch for changes in your file. As you update your source, it recompiles them and relaunches your local app. There's no need to restart goapp.

Try it now: leave the web server running, then edit hello.go to change Hello, world! to something else. Reload http://localhost:8080/ to see the change.

To shut down the web server, make sure the terminal window is active, then press Control-C (or the appropriate "break" key for your console).

Leave the web server running for the rest of this tutorial. If you need to stop it, you can restart it again by running the command above.

You now have a complete App Engine application! You could deploy this simple greeting right now and share it with users worldwide. But before we deploy it, let's add some features.

Using the Users Service

Google App Engine provides several useful services based on Google infrastructure. One example is the Users service, which lets your application integrate with Google user accounts. With the Users service, your users can employ the Google accounts they already have to sign in to your application.

The Users service makes it easy to personalize this application's greeting.

Using Users

Edit myapp/hello.go again, and replace its contents with the following:

package hello

import (
    "fmt"
    "net/http"

    "appengine"
    "appengine/user"
)

func init() {
    http.HandleFunc("/", handler)
}

func handler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)
    u := user.Current(c)
    if u == nil {
        url, err := user.LoginURL(c, r.URL.String())
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        w.Header().Set("Location", url)
        w.WriteHeader(http.StatusFound)
        return
    }
    fmt.Fprintf(w, "Hello, %v!", u)
}

Reload the page in your browser. Your application presents you with a link that, when followed, will redirect you to the local version of the Google sign-in page suitable for testing your application. You can enter any username you'd like in this screen, and your application will see a fake user.User value based on that username.

When your application is running on App Engine, users will be directed to the Google Accounts sign-in page, then redirected back to your application after successfully signing in or creating an account.

The Users API

Let's take a closer look at the new pieces:

// Create a new context.
c := appengine.NewContext(r)

The appengine.NewContext function returns an appengine.Context value associated with the current request. This is an opaque value used by many functions in the Go App Engine SDK to communicate with the App Engine service.

// Get the current user.
u := user.Current(c)

If the user is already signed in to your application, user.Current returns a pointer to a user.User value for the user. Otherwise, it returns nil.

if u == nil {
    url, err := user.LoginURL(c, r.URL.String())
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    w.Header().Set("Location", url)
    w.WriteHeader(http.StatusFound)
    return
}

If the user has not signed in, redirect the user's browser to the Google account sign-in screen by setting a Location header and returning an HTTP status code of 302 "Found". The redirection includes the URL to this page (r.URL.String()) so the Google account sign-in mechanism will send the user back here after signing in or registering for a new account.

The user.LoginURL function returns an error value as its second argument. Though an error is unlikely to occur here, it is good practice to check it and display an error user, if appropriate (in this case, with the http.Error helper).

// Display an output message with user name.
fmt.Fprintf(w, "Hello, %v!", u)

If the user has signed in, display a personalized message using the name associated with the user's account. In this case, the fmt.Fprintf function calls *user.User's String method to get the user's name in string form.

For more information about the Users API, see the Users reference.

Note: for specific paths that should be served only to logged in users, use the login: required directive in the app.yaml file. See the app.yaml reference for details.

Our application can now greet visiting users by name. Let's add a feature that will let users greet each other.

Handling Forms

If we want users to be able to post their own greetings, we need a way to process information submitted by the user with a web form. The Go http package makes processing form data easy.

Handling Web Forms

Replace the contents of myapp/hello.go with the following:

package hello

import (
    "fmt"
    "html/template"
    "net/http"
)

func init() {
    http.HandleFunc("/", root)
    http.HandleFunc("/sign", sign)
}

func root(w http.ResponseWriter, r *http.Request) {
    fmt.Fprint(w, guestbookForm)
}

const guestbookForm = `
<html>
  <body>
    <form action="/sign" method="post">
      <div><textarea name="content" rows="3" cols="60"></textarea></div>
      <div><input type="submit" value="Sign Guestbook"></div>
    </form>
  </body>
</html>
`

func sign(w http.ResponseWriter, r *http.Request) {
    err := signTemplate.Execute(w, r.FormValue("content"))
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
    }
}

var signTemplate = template.Must(template.New("sign").Parse(signTemplateHTML))

const signTemplateHTML = `
<html>
  <body>
    <p>You wrote:</p>
    <pre>{{.}}</pre>
  </body>
</html>
`

Reload the page to see the form, then try submitting a message.

This version has two handlers: the path / is mapped to root, which displays a web form. The path /sign is mapped to sign, which displays the data submitted by the web form.

The sign function gets the form data by calling r.FormValue and passes it to signTemplate.Execute that writes the rendered template to the http.ResponseWriter. In the template code, the content is automatically filtered to escape HTML special characters. The automatic escaping is a property of the html/template package, as distinct from text/template.

Now that we can collect information from the user, we need a place to put it and a way to get it back.

Using the Datastore

Storing data in a scalable web application can be tricky. A user could be interacting with any of dozens of web servers at a given time, and the user's next request could go to a different web server than the one that handled the previous request. A web server may depend on data that is spread out across dozens of machines, possibly in different locations around the world.

Thanks to Google App Engine, you don't have to worry about any of that. App Engine's infrastructure takes care of all of the distribution, replication and load balancing of data behind a simple API—and you get a powerful query engine as well.

App Engine's data repository, the High Replication Datastore (HRD), uses the Paxos algorithm to replicate data across multiple datacenters. Data is written to the Datastore in objects known as entities. Each entity has a key that uniquely identifies it. An entity can optionally designate another entity as its parent; the first entity is a child of the parent entity. The entities in the Datastore thus form a hierarchically structured space similar to the directory structure of a file system. An entity's parent, parent's parent, and so on recursively, are its ancestors; its children, children's children, and so on, are its descendants. An entity without a parent is a root entity.

The Datastore is extremely resilient in the face of catastrophic failure, but its consistency guarantees may differ from what you're familiar with. Entities descended from a common ancestor are said to belong to the same entity group; the common ancestor's key is the group's parent key, which serves to identify the entire group. Queries over a single entity group, called ancestor queries, refer to the parent key instead of a specific entity's key. Entity groups are a unit of both consistency and transactionality: whereas queries over multiple entity groups may return stale, eventually consistent results, those limited to a single entity group always return up-to-date, strongly consistent results.

The code samples in this guide organize related entities into entity groups, and use ancestor queries on those entity groups to return strongly consistent results. In the example code comments, we highlight some ways this might affect the design of your application. For more detailed information, see Structuring Data for Strong Consistency.

Storing the Submitted Greetings

For the guestbook application, we want to store greetings posted by users. Each greeting includes the author's name, the message content, and the date and time the message was posted so we can display messages in chronological order.

To represent this data we create a Go struct named Greeting:

type Greeting struct {
        Author  string
        Content string
        Date    time.Time
}

Now that we have a data type for greetings, the application can create new Greeting values and put them into the datastore. The new version of the sign handler does just that:

func sign(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)
        g := Greeting{
                Content: r.FormValue("content"),
                Date:    time.Now(),
        }
        if u := user.Current(c); u != nil {
                g.Author = u.String()
        }
        // We set the same parent key on every Greeting entity to ensure each Greeting
        // is in the same entity group. Queries across the single entity group
        // will be consistent. However, the write rate to a single entity group
        // should be limited to ~1/second.
        key := datastore.NewIncompleteKey(c, "Greeting", guestbookKey(c))
        _, err := datastore.Put(c, key, &g)
        if err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
                return
        }
        http.Redirect(w, r, "/", http.StatusFound)
}

This creates a new Greeting value, setting its Author field to the current user, its Content field with the data posted by the user, and its Date field to the current time.

Finally, datastore.Put saves our new value to the datastore. We pass it a new, incomplete key so that the datastore will create a new key for this record automatically.

Because querying in the High Replication Datastore is strongly consistent only within entity groups, we assign all of one book's greetings to the same entity group in this example by setting the same parent for each greeting. This means a user will always see a greeting immediately after it was written. However, the rate at which you can write to the same entity group is limited to 1 write to the entity group per second. When you design a real application you'll need to keep this fact in mind. By using services such as Memcache, you can mitigate the chance that a user won't see fresh results when querying across entity groups immediately after a write.

Retrieving the Stored Greetings With datastore.Query

The datastore package provides a Query type for querying the datastore and iterating over the results.

The new version of the root handler queries the datastore for greetings:

func root(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)
        // Ancestor queries, as shown here, are strongly consistent with the High
        // Replication Datastore. Queries that span entity groups are eventually
        // consistent. If we omitted the .Ancestor from this query there would be
        // a slight chance that Greeting that had just been written would not
        // show up in a query.
        q := datastore.NewQuery("Greeting").Ancestor(guestbookKey(c)).Order("-Date").Limit(10)
        greetings := make([]Greeting, 0, 10)
        if _, err := q.GetAll(c, &greetings); err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
                return
        }
        if err := guestbookTemplate.Execute(w, greetings); err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
        }
}

First the function constructs a Query value that requests Greeting objects that are descendants of the root guestbook key, in Date-descending order, with a limit of 10 objects.

// Create the query.
q := datastore.NewQuery("Greeting").Ancestor(guestbookKey(c)).Order("-Date").Limit(10)

Then it calls q.GetAll(c, &greetings), which runs the query and appends the query results to the greetings slice.

greetings := make([]Greeting, 0, 10)
if _, err := q.GetAll(c, &greetings); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
}

Finally, the guestbookTemplate.Execute function renders an HTML page containing these greetings and writes it out to the http.ResponseWriter. For more details on the templating language, see the text/template package documentation. Note that here we use the html/template, a package that wraps text/template and automatically escapes content in HTML templates, preventing a class of script injection attacks.

For a complete description of the Datastore API, see the Datastore reference.

Clearing the Development Server Datastore

The development web server uses a local version of the datastore for testing your application, using temporary files. The data persists as long as the temporary files exist, and the web server does not reset these files unless you ask it to do so.

If you want the development server to erase its datastore prior to starting up, see the Development Server reference, which explains the datastore configuration options for the development server.

A Complete Example Using the Datastore

Here is a new version of myapp/hello.go that stores greetings in the datastore. The rest of this page discusses the new pieces.

package guestbook

import (
        "html/template"
        "net/http"
        "time"

        "appengine"
        "appengine/datastore"
        "appengine/user"
)

type Greeting struct {
        Author  string
        Content string
        Date    time.Time
}

func init() {
        http.HandleFunc("/", root)
        http.HandleFunc("/sign", sign)
}

// guestbookKey returns the key used for all guestbook entries.
func guestbookKey(c appengine.Context) *datastore.Key {
        // The string "default_guestbook" here could be varied to have multiple guestbooks.
        return datastore.NewKey(c, "Guestbook", "default_guestbook", 0, nil)
}

func root(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)
        // Ancestor queries, as shown here, are strongly consistent with the High
        // Replication Datastore. Queries that span entity groups are eventually
        // consistent. If we omitted the .Ancestor from this query there would be
        // a slight chance that Greeting that had just been written would not
        // show up in a query.
        q := datastore.NewQuery("Greeting").Ancestor(guestbookKey(c)).Order("-Date").Limit(10)
        greetings := make([]Greeting, 0, 10)
        if _, err := q.GetAll(c, &greetings); err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
                return
        }
        if err := guestbookTemplate.Execute(w, greetings); err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
        }
}

var guestbookTemplate = template.Must(template.New("book").Parse(`
<html>
  <head>
    <title>Go Guestbook</title>
  </head>
  <body>
    {{range .}}
      {{with .Author}}
        <p><b>{{.}}</b> wrote:</p>
      {{else}}
        <p>An anonymous person wrote:</p>
      {{end}}
      <pre>{{.Content}}</pre>
    {{end}}
    <form action="/sign" method="post">
      <div><textarea name="content" rows="3" cols="60"></textarea></div>
      <div><input type="submit" value="Sign Guestbook"></div>
    </form>
  </body>
</html>
`))

func sign(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)
        g := Greeting{
                Content: r.FormValue("content"),
                Date:    time.Now(),
        }
        if u := user.Current(c); u != nil {
                g.Author = u.String()
        }
        // We set the same parent key on every Greeting entity to ensure each Greeting
        // is in the same entity group. Queries across the single entity group
        // will be consistent. However, the write rate to a single entity group
        // should be limited to ~1/second.
        key := datastore.NewIncompleteKey(c, "Greeting", guestbookKey(c))
        _, err := datastore.Put(c, key, &g)
        if err != nil {
                http.Error(w, err.Error(), http.StatusInternalServerError)
                return
        }
        http.Redirect(w, r, "/", http.StatusFound)
}

Replace myapp/hello.go with this, then reload http://localhost:8080/ in your browser. Post a few messages to verify that messages get stored and displayed correctly.

Warning! Exercising the queries in your application locally causes App Engine to create or update index.yaml. If index.yaml is missing or incomplete, you will see index errors when your uploaded application executes queries for which the necessary indexes have not been specified. To avoid missing index errors in production, always test new queries at least once locally before uploading your application. See Go Datastore Index Configuration for more information.

A Word About Datastore Indexes

Every query in the App Engine Datastore is computed from one or more indexes—tables that map ordered property values to entity keys. This is how App Engine is able to serve results quickly regardless of the size of your application's Datastore. Many queries can be computed from the builtin indexes, but for queries that are more complex the Datastore requires a custom index. Without a custom index, the Datastore can't execute these queries efficiently.

For example, our guestbook application above filters by guestbook entries and orders by date, using an ancestor query and a sort order. This requires a custom index to be specified in your application's index.yaml file. You can edit this file manually or, as noted in the warning box earlier on this page, you can take care of it automatically by running the queries in your application locally. Once the index is defined in index.yaml, uploading your application will also upload your custom index information.

The definition for the query in your index.yaml file looks like this:

indexes:
- kind: Greeting
  ancestor: yes
  properties:
  - name: Date
    direction: desc

You can read all about Datastore indexes in the Datastore Indexes page. You can read about the proper specification for your index.yaml file in Go Datastore Index Configuration.

We now have a working guest book application that authenticates users using Google accounts, lets them submit messages, and displays messages other users have left. Because App Engine handles scaling automatically, we will not need to restructure our application as it gets popular.

Uploading Your Application

You create and manage App Engine applications using the Google Developers Console. Once you have registered an application ID for your application, you upload it to your website using goapp deploy.

Note: Application IDs must begin with a letter. Once you register an application ID, you can delete it, but you can't re-register that same application ID after it has been deleted. You can skip these next steps if you don't want to register an ID at this time.

Note: If you have an App Engine Premier account, you can specify that your new application should reside in the European Union rather than the United States. Developers that do not have a Premier account need to fill out this form and enable billing for applications that should reside in the European Union.

Hosting applications in the European Union is especially useful if your users are closer to Europe than to the United States. There is less network latency and the End User Content will be stored at rest in the European Union. You must specify this location by clicking the "Edit" link in the "Location Options" section when you register the application; you cannot change it later.

Registering the Application

You create and manage App Engine applications from the Developers Console at this URL:

https://console.developers.google.com/

Sign in to App Engine using your Google account. If you do not have a Google account, you can create a Google account with an email address and password.

Note: You may have already created a project using the Google Developers Console. If this is the case, you do not have to create a new application. Your project has a title and an ID. In the instructions that follow, the project title and ID can be used wherever an application title and ID are mentioned. They are the same thing.

To create a new application, click the "Create an Application" button. Follow the instructions to register an application ID, a name unique to this application.

If you elect to use the free appspot.com domain name, the full URL for the application will be http://your_app_id.appspot.com/. You may also purchase a top-level domain name for your app, or use one that you have already registered.

If you have an App Engine Premier account, you can specify that your new application should reside in the European Union rather than the United States. This is especially useful if your application's users are closer to Europe than to the United States. There is less network latency and the End User Content will be stored at rest in the European Union. You must specify this location when you register the application; you cannot change it later. Click the Edit link in the Location Options section; select a location option, either United States or European Union.

Edit the app.yaml file, then change the value of the application: setting from helloworld to your registered application ID.

Uploading the Application

To upload your finished application to Google App Engine, run the command:

goapp deploy myapp/

Enter your Google username and password at the prompts.

If you see compilation errors, fix the source and rerun goapp deploy; it won't launch (or update) your app until compilation is successful.

Checking Your Application State

After your application is uploaded, its Datastore Indexes will be automatically generated. This operation may take some time, and any visitors to your site will receive a DatastoreNeedIndexException until the indexes have been built. You can monitor the progress of the operation by visiting the App Engine console, selecting your application, and then selecting the Datastore Indexes link.

Accessing Your Application

You can now see your application running on App Engine. If you set up a free appspot.com domain name, the URL for your website begins with your application ID:

http://your_app_id.appspot.com

Congratulations!

You have completed this tutorial. For more information on the subjects covered here, see the rest of the App Engine documentation.

Except as otherwise noted, the code samples of this page is licensed under the Apache 2.0 License.