ErrZure - Self-managed error catcher

Links 

• Challenge 2, Blog: http://errzure.azurewebsites.net/
• Challenge 3, Service: http://errzureservice.azurewebsites.net/
  • UserName: admin ; Password: dicker 
• Challenge 3, Service Source Code (Open Source): tbd 
• Challenge 4, Service: http://errbitserverv2.cloudapp.net:3000/
  • Username: errbit@errbit.example.com Password: password
  • Be fair and don't delete / resolve Errors if you are not a reviewer!
• Challenge 4, OpenSource ErrBit Notifier for Windows Phone: https://github.com/K2DaC/ErrBitWP
   
• Challange 5  

QuickJump

• Second Challenge: Build a website 
• Third Challenge: Using SQL on Azure 
• Fourth challenge, VM 

Background - Introduction 

I've published a couple of apps for different platforms in the past 2 years. Android, Windows Phone, Windows 8, and Blackberry10. I've made some of them in my spare time, some of them for the company I work for. I've made some of them completely by myself, some of them in a bigger team. Some were made by agencies, we just deployed and published them. But I'm not only a developer. I'm also a User. I'm using apps all day long. But no matter if its Twitter on Android, the AppStore App on the iPad or my own creation on Windows Phone.  There will always be this one edge-case nobody has tested. But there is one thing which is even worse then having a bug in the app. Not knowing it!

ErrZure will be a self-managed error storage. All your uncaught exceptions will be logged in the cloud.

Why don't you just use ...?

Marketplaces build in solutions 

Some Marketplaces are logging exceptions automatically. Windows 8 does something in this direction. Android also. But sometimes you just send your apk¹⁾  to beta testers via email. Sometimes you publish them in 3rd party stores like Yandex in Russia. And what happens with exceptions when there is no internet connection? Do you really know this?

^{1) File extension of Android-Apps} 

Analytic Tools 

Both analytic tools I've used, Google Analytics and Flurry Analytics, have build in solutions to log exceptions. But there are two problems with that.

It's a huge overhead, if you are not interested in Analytics. The results are not always helpful.

Google Analytics 

ImageSource: Link  

Flurry 

ImageSource: Link

With this information at least i know that something is wrong.

Existing Services 

BugSense 

BugSense indeed is exactly what we want. They offer 500 "Free exceptions". The more bugs you have, the more you pay. Lots of big companies are using their service. If you need a easy solution, this might be a good idea. They focus on Apps: Support for iOS, Android, Windows Phone & HTML5 Apps

ImageSource: BugSense 

Airbrake   

Airbrake looks also very professional.  They focus on Ruby on Rails and iOS. With 3rd party support for nearly everything, this is also nice way to let other people do the work for us.

ImageSource: AirBrake  

How about some open source solutions?   

ErrBit  

ErrBit is the (more or less) open source clone of Airbrake. It's made with Ruby on Rails, and MongoDB stores all the exceptions. ErrBit will be used in one of the challenges (Fourth Challenge: Virtual Machines.) to have a nice compare to our own service we want to create in this challenge.

ImageSource: gitHub 

What can Azure do for me?

Azure removes the pain of setting up servers and services, databases and web sites. At the same time it removes the pain of fixed contracts you have to sign in classic storage solutions when scaling up or down.

Challenges (first thoughts) 

First Challenge: Getting Started

Signing up for a free account  was as easy as expected. A credit card is needed,  but will not be charged in the first 3 month. There is no risk: if a limit is reached (e.g.: 25GB outgoind data / month), the service will be disabled until the next month. You should change your subscription from free to paid, if you plan to go into production mode.

Writing this article was harder, English is not my native language, it's the third language i had to learn (Polish, German, English).  I hope you forgive me some of my grammatical mistakes.

Second Challenge: Build a website 

The final "product" should be completely open source. Therefore, in this challenge we'll prepare a Wordpress blog which can be used after this competition to announce updates and exchange with potential users.
This Blog will be used during the contest to announce changes, progress etc. (next to this article)

Third Challenge: Using SQL on Azure   

We will write the backend solution for our service in this challenge. More details will follow on time. However, the project will not be as complicated as BugSense or ErrBit.  (Time, Money, Staff). There will be also some mobile code / mobile integration already in this challenge to test everything. This mobile code will not be part of the fifth challenge, there will be something else.

Fourth Challenge: Virtual Machines

In this challenge we will install and run the open source solution: ErrBit in a Linux Ubuntu VM. We will install Ruby, Rails, mongoDB and all other requirements to do this.

Fifth Challenge: Mobile access 

In the fifth challenge we will create a mobile app to check our logged exceptions on the go. If there will be some time left, we will manipulate our service from the third solution to make our service ready to send push-notifications to our app if a new error occurs (only once per error)

Challenges (Project Overview)

The System

In this project, we will use Android devices to force some exception. We will also use already existing solutions to catch the exceptions and transfer them to our service. This has two reasons.

1. Our Service will support everything which is also supported by ErrBit and AirBrake (not only mobile, but also Ruby on Rails, php, erlang, .NET) 
2. We'll save some time in the second challenge, which might be the biggest in this project.

We will use ErrBit Notifier by from Matteo Piotto to catch, prepare and transmit our exceptions. He forked it from James Smith who created the origin AirBrake Notifyer for Android.
The Code of this Notifiers should not be taken into account for this challenge!

The best part about them is the super easy implementation.

ErrbitNotifier.register(this, "errbit.domain.com", 
  "your-api-key-goes-here");

We want to keep this, so our service should handle this implementation and provide us a API-key for the app, but we will check the database structure from errbit later in the challenge.

All the "notifiers" have a strict url structure:

ErrbitNotifier.errbit_endpoint = 
  "http://" + endpoint + "/notifier_api/v2/notices";

We have to keep this, to support all the already existing notifiers!

UI Wireframes 

Our service in challenge 3 will be the most difficult part. Here are some raw drafts of the UI.

Mobile Client Wireframe (Challenge 5)  

Data structure 

This project should be compatible to all currenty available ErrBit Error Notifiers. Therefor this project need the same DataStructure.
ErrBit and AirBrake are using XML.
This might not be the best solution, since xml always has some overhead if you compare it to json, but it's a limitation we have to deal with.

AirBrake and ErrBit are both using the following xml structure

<?xml version="1.0" encoding="UTF-8"?>
<notice version="2.0">
   <api-key>ff0621fe7eaad6d6f4afbfaf3635fe79</api-key>
   <notifier>
      <name>Android Errbit Notifier</name>
      <version>0.2.0</version>
      <url>http://welaika.com</url>
   </notifier>
   <error>
      <class>java.lang.RuntimeException</class>
      <message>[1.0] java.lang.RuntimeException: Unable to start activity 
        ComponentInfo{com.errbit.testapp/com.errbit.testapp.MainActivity}: 
        java.lang.NullPointerException</message>
      <backtrace>
         <line method="android.app.ActivityThread.performLaunchActivity" 
           file="ActivityThread.java" number="2121" />
         <line method="android.app.ActivityThread.handleLaunchActivity" 
           file="ActivityThread.java" number="2146" />
         <line method="android.app.ActivityThread.access$700" 
           file="ActivityThread.java" number="140" />
         <line method="android.app.ActivityThread$H.handleMessage" 
           file="ActivityThread.java" number="1238" />
         <line method="android.os.Handler.dispatchMessage" 
           file="Handler.java" number="99" />
         <line method="android.os.Looper.loop" 
           file="Looper.java" number="137" />
         <line method="android.app.ActivityThread.main" 
           file="ActivityThread.java" number="4947" />
         <line method="java.lang.reflect.Method.invokeNative" 
           file="Method.java" number="-2" />
         <line method="java.lang.reflect.Method.invoke" 
           file="Method.java" number="511" />
         <line method="com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run" 
           file="ZygoteInit.java" number="1038" />
         <line method="com.android.internal.os.ZygoteInit.main" 
           file="ZygoteInit.java" number="805" />
         <line method="dalvik.system.NativeStart.main" 
           file="NativeStart.java" number="-2" />
         <line file="### CAUSED BY ###: java.lang.NullPointerException" number="" />
         <line method="com.errbit.testapp.MainActivity.onCreate" 
           file="MainActivity.java" number="17" />
         <line method="android.app.Activity.performCreate" 
           file="Activity.java" number="5207" />
         <line method="android.app.Instrumentation.callActivityOnCreate" 
           file="Instrumentation.java" number="1094" />
         <line method="android.app.ActivityThread.performLaunchActivity" 
           file="ActivityThread.java" number="2085" />
         <line method="android.app.ActivityThread.handleLaunchActivity" 
           file="ActivityThread.java" number="2146" />
         <line method="android.app.ActivityThread.access$700" 
           file="ActivityThread.java" number="140" />
         <line method="android.app.ActivityThread$H.handleMessage" 
           file="ActivityThread.java" number="1238" />
         <line method="android.os.Handler.dispatchMessage" 
           file="Handler.java" number="99" />
         <line method="android.os.Looper.loop" 
           file="Looper.java" number="137" />
         <line method="android.app.ActivityThread.main" 
           file="ActivityThread.java" number="4947" />
         <line method="java.lang.reflect.Method.invokeNative" 
           file="Method.java" number="-2" />
         <line method="java.lang.reflect.Method.invoke" 
           file="Method.java" number="511" />
         <line method="com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run" 
           file="ZygoteInit.java" number="1038" />
         <line method="com.android.internal.os.ZygoteInit.main" 
           file="ZygoteInit.java" number="805" />
         <line method="dalvik.system.NativeStart.main" 
           file="NativeStart.java" number="-2" />
      </backtrace>
   </error>
   <request>
      <url />
      <component />
      <action />
      <cgi-data>
         <var key="Manufacturer">samsung</var>
         <var key="Device">GT-N5100</var>
         <var key="Brand">samsung</var>
         <var key="Android Version">4.1.2</var>
         <var key="App Version">1.0</var>
      </cgi-data>
   </request>
   <server-environment>
      <environment-name>production</environment-name>
      <app-version>1.0</app-version>
   </server-environment>
</notice> 

The root Element is a "notice". Each notice has:

1. API Key
2. Error 
   • Class
   • Message
   • Backtrace
3. Notifier
4. Additional Request Parameters
5. Server-environment 

Second Challenge: Build a website 

The main idea was to set up a blog which can be used for the open source project after all the challenges.

Azure has some pre-configured solutions, including wordpress, which perfectly fits our need.

Second Challenge: Wordpress 

To set up Wordpress, we first need to login at https://manage.windowsazure.com and select the "Website" Tab from the menu on the left. Use the "New" Button in the left bottom corner, the start with the setup.

This will bring up a menu from the bottom. Select "FROM GALLERY"

Azure will show all the pre-configured solutions. From CMS solutions, image gallerys to wiki. We will stick to Wordpress.
Setting up  Wordpress is a 3 Step process. 2 are actually needed to install Wordpress. The last step is setting up things like admin user, password, title and eMail.

• Step 1: Select url, region and possible Database (MySQL)
• Step 2: Select Database region / name
• Step 3: Config your Wordpress (user, password)

The Blog is now available unter http://errzure.azurewebsites.net 

Second Challenge: Custom website with git

Setting up Wordpress in Azure was just to easy too call it a Challenge. We have to find a new one.
This time we will use git, write some php code connect to the database and hit the frist limit of the free Azure trail.
After pressing the "NEW" - Button once again, we'll select the "Custom Create" Option

Once again a guide will popup an help us setting up everything correctly.  Make sure to check the "Publish from source control" checkbox.

After using the arrow to continue with the setup Azure will pupup a warning:

Because we have no plans to release something "important" here, nor do we plan to use the MySQL database, we could go back and disable the database completely.

But we can also use the already existing one, which was created during the WordPress installation. (Avoid this in production if not really needed!! This could be dangerous.)

In the last step, we are able to choose from different "source controls". Maybe "Dropbox" should not be called a source control, but git definitely is one.

Azure should ask you for your git credentials. Select a username and a password (remember them! You will need you pw every time you want to publish / deploy your code)

Our first setup is done. Going to our selected URL from the first step, we should get the welcome message.

To get all information about setting up git and how to use it, select your website, and use the "Deployments" Tab.

We will switch to the console now and setup everything to easily deploy our "website". I was doing all this using Ubuntu / Linux, but it's also possible to use GIT in Windows or Mac. We will run a bunch of GIT / Linux commands now:

1. Create a new folder for our projects and switch to it 
   • mkdir phpErrbit
   • cd  phpErrbit 
2. Clone our project and switch to our project folder
   • git clone https://podkowik@errzurephp.scm.azurewebsites.net/ErrZurePhP.git 
   • cd ErrZurePhP  
3.  Create a new index.php file and add some php code with nano
   • nano index.php 
4. Add our file to the git index, commit our change and setup our remote repository
   • git add .
   • git commit -m "check for db connection"
   • git remote add azure https://podkowik@errzurephp.scm.azurewebsites.net/ErrZurePhP.git
5.  The last step will push everything to our remote Azure server and deploy our new code.

The PHP code in index.php will create a connection to our MySQL server

<?php
// Create connection
$con=mysqli_connect("eu-cdbr-azure-north-a.cloudapp.net",
  "USERNAME","PASSWORD","DB");
// Check connection
if (mysqli_connect_errno($con))
  {
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
  }else{
   echo "Connected";
  } 
?> 

Going back to our Azure management website, we should have triggered a new build in the deployments tab. You can easily check here all deployments and simply rollback if you recognize a bug in your current version

Going to our project url in the browser should display the "connected" string (if everything is fine!)

The Danger Part 1 

Using one Database in different projects, which are not connected to each other, is very dangerous. If you host services for different clients on the same Database, it could end up in a disaster. Wordpress is smart enough to name all their database tables wp_XXX, e.g.: wp_users. So we could avoid using the same tablenames by having our own structure, like ez_users. But we could simply access also the wordpress user database. Imagine having a online-store with credit card information etc. on one website and a very unstable release of WHATEVER on another one. Both connected to the same database ... We would also fail setting up another instance of Wordpress here (with git).

We will do a small tests here. We create a new file "print.php" in our project and add some code to it, to find all tables and their names in the database. I found the code here: Link

<?php
/****************
* File: displaytables.php
* Date: 1.13.2009
* Author: design1online.com, LLC
* Purpose: display all table structure for a specific database
****************/
//connection variables
$host = "localhost";
$database = "your_db_name";
$user = "your_username";
$pass = "your_pass";
//connection to the database
mysql_connect($host, $user, $pass)
or die ('cannot connect to the database: ' . mysql_error());
//select the database
mysql_select_db($database)
or die ('cannot select database: ' . mysql_error());
//loop to show all the tables and fields
$loop = mysql_query("SHOW tables FROM $database")
or die ('cannot select tables');
while($table = mysql_fetch_array($loop))
{
    echo "
        <table cellpadding=\"2\" cellspacing=\"2\" 
          border=\"0\" width=\"75%\">
            <tr bgcolor=\"#666666\">
                <td colspan=\"5\" align=\"center\"><b>
                  ;<font color=\"#FFFFFF\">" . $table[0] . "</font></td>
            </tr>
            <tr>
                <td>Field</td>
                <td>Type</td>
                <td>Key</td>
                <td>Default</td>
                <td>Extra</td>
            </tr>";
    $i = 0; //row counter
    $row = mysql_query("SHOW columns FROM " . $table[0])
    or die ('cannot select table fields');
    while ($col = mysql_fetch_array($row))
    {
        echo "<tr";
        if ($i % 2 == 0)
            echo " bgcolor=\"#CCCCCC\"";
        echo ">
            <td>" . $col[0] . "</td>
            <td>" . $col[1] . "</td>
            <td>" . $col[2] . "</td>
            <td>" . $col[3] . "</td>
            <td>" . $col[4] . "</td>
        </tr>";
        $i++;
    } //end row loop
    echo "</table><br/><br/>";
} //end table loop
?> 

The result is exactly what we've expected. All our wordpress tables popup there:

I think this screenshot illustrates the potential risk once more.

The Danger, Part 2

The second part of this small problem-analysis will be the more "dangerous" part. We've linked our database in two project. However, the second project was just made to play a little bit around and explore git functionality in Azure. We want to delete it again.

If you select the delete button in your overview, you get this small confirm-dialog

Here you have to be 100 % sure that you remember, if the database was linked somewhere else or not. I've checked the checkbox careless in my first attempt, which destroyed my Wordpress installation. The MySQL database of both projects was gone just by selecting one checkbox. No additional warning about a possible additional linked project.

Second Challenge: Custom website with git, sql and some (wanted) errors

For our third project, we need to setup everything the same we did in our second project, but we will choose SQL instead of MySQL. Before connecting to the database using php, we want to get some more information about the current PHP setup.

<?php 
      phpinfo();
?>   

But let's put a syntax-error in here, let's use php_info();. Of course, this does not work. But we want to see the log files, some more information would be nice!

We navigate to

https://errzurephponsql.scm.azurewebsites.net

Azure wants a username and password, use your git credentials. They will also work for our Wordpress installation, but I'm note sure what happens, when you never set them up! Try yourself!

You will end up in Kudu, which can be very useful.

• The link Environment variables will show you all the, you guessed it, Environment variables you may need to setup some projects.  
• The link Diagnostic Dump will let you download a LOT of log files. All your git-deployments, all error logs, everything in a zip file. Here we also have a php_errors.log file included, which shows
  "Call to undefined function php_info() on line 2.  
• The link Diagnostic Log will popup a new windows showing a live-stream of logfiles. Everything which is done now (e.g. : git push) will be shown here.

To test the Diagnostic Log, we will deploy a new file, test.php, and we connect to the SQL Server via PHP.

<?php
$serverName = "BLAURL.database.windows.net, 1433"; 
//serverName\instanceName, portNumber (default is 1433)

$connectionInfo = array( "Database"=>"DATABASE", 
  "UID"=>"USERNAME", "PWD"=>"PASSWORD");
$conn = sqlsrv_connect( $serverName, $connectionInfo);
if( $conn ) {
     echo "Connection established.<br />";
}else{
     echo "Connection could not be established.<br />";
     die( print_r( sqlsrv_errors(), true));
}
?> 

Connecting from php to (ms-) SQL is a little bit different then connecting to MySQL, but this should not be part of this article. Of course the Connection works and we get a "Connection established." opening our test.php in the browser. But here is what happens in the Live-Log: 

As we've used MS-SQL, we can also create tables using the "Manage" button in our SQL-Tab

However, this ends up in a Page which is using Silverlight. The Silverlight Plugin for Linux is not working correctly, so we end up on a white page. All we can do is right-clicking the page and check our plugin, but it's not working:

Lessens learned from Challenge 2 

1. Very easy installation of configured popular systems like wordpress
2. very good support of git and other source code versioning
3. secure use of the free-trail. Warnings instead of pay-requests when hitting limits.
4. linked resources can be evil 
5. Parts of Azure website require Silverlight, which is bad for Linux users

Third Challenge: Using SQL on Azure

Initial consideration

In this Challenge we will create the main Service for ErrZure. ErrZure will be contently  written in PHP, MySQL will be used as the technology to store our data. Using this two technologies has a couple of reasons.

1. ErrBit, the service we want to rewrite from scratch, is written in Ruby on Rails. We will not use the same technology
2. ASP.NET works perfectly on Azure, but i would limit our service. ErrBit is already very limited (Rails, MongoDB, ...). We need something which runs nearly everywhere!
3. Haven't used PHP for years. Third reason is definitely: Having fun and refreshing knowledge! 
4. Microsoft's WebMatrix tools supports php development and the "Starter Site" provides a lot of benefits.  

Getting Started 

Once we've setup a new Website on the Azure Management WebSite, we will switch to WebMatrix and create a new PHP project from the templates. The "Starter Site" already brings mysql connections and a set of typical operations like: Login, Logout, Sessions, User management and a lot more.

The following screenshot illustrates the "Starter Site" project Structure:

From this moment on we can switch to our favorite web-browser and check the progress by navigating to:   http://localhost:26579/

The initial WebSite already looks quite good

Analyzing the Code

The most important changes will affect the create tables function. The default Site comes with 4 Tables: users, roles, users_in_roles and pages. Here's the code from the database.php file

function create_tables($databaseConnection)
{
    $query_users = "CREATE TABLE IF NOT EXISTS users (id INT NOT 
      NULL AUTO_INCREMENT, username VARCHAR(50), password CHAR(40), PRIMARY KEY (id))";
    $databaseConnection->query($query_users);

    $query_roles = "CREATE TABLE IF NOT EXISTS roles 
      (id INT NOT NULL, name VARCHAR(50), PRIMARY KEY (id))";
    $databaseConnection->query($query_roles);

    $query_users_in_roles = "CREATE TABLE IF NOT EXISTS users_in_roles 
      (id INT NOT NULL AUTO_INCREMENT, user_id INT NOT NULL, role_id INT NOT NULL, ";
    $query_users_in_roles .= " PRIMARY KEY (id), FOREIGN KEY (user_id) 
      REFERENCES users(id), FOREIGN KEY (role_id) REFERENCES roles(id))";
    $databaseConnection->query($query_users_in_roles);

    $query_pages = "CREATE TABLE IF NOT EXISTS pages (id INT NOT 
      NULL AUTO_INCREMENT, menulabel VARCHAR(50), content TEXT, PRIMARY KEY (id))";
    $databaseConnection->query($query_pages);
}

Of course we don't have / need pages. But we have Apps and Errors, so we need to change the database setup here:

$query_apps = "CREATE TABLE IF NOT EXISTS apps (
  id INT NOT NULL AUTO_INCREMENT, appkey VARCHAR(50), appname TEXT, PRIMARY KEY (id))";
$databaseConnection->query($query_apps);

$query_error = "CREATE TABLE IF NOT EXISTS error (id INT NOT NULL 
  AUTO_INCREMENT, appkey VARCHAR(50), errorclass TEXT, errormessage TEXT, 
  errormanufacturer VARCHAR(20), errordevice VARCHAR(20), errorbrand VARCHAR(50), 
  errorandroidversion VARCHAR(10), errorappversion VARCHAR(10), 
  errorbacktrace TEXT,  submissiondate DATETIME, PRIMARY KEY (id))";
$databaseConnection->query($query_error); 

Here's an overview of the new Database Tables

Creating the API

Once we have our Database tables set up, we need to create the API to receive the crash reports. As already explained, we want to be compatible to all existing "Error Notifiers" for ErrBit / AirBrake, so we have to adopt the behaviors.

1. Using the same URL structure : http:/.../notifier_api/v2/notices 
2. Using the same XML layout (see introduction) and being able to parse it correctly. 

For the url structure we just need to create the needed folders and a index.php file in it. PHP will use the index.php file by default if no file is provided, so this will work quite well.

Parsing XML in PHP is also no big deal. Here's a code snippet from index.php file which illustrates how to handle XML in PHP:

$appkey = array_shift($xml->xpath('/notice/api-key'));
$errorclass = array_shift($xml->xpath('/notice/error/class'));
$errormessage = array_shift($xml->xpath('/notice/error/message'));
$errormanufacturer = array_shift($xml->xpath('/notice/request/cgi-data/var[@key = "Manufacturer"]'));
$errordevice = array_shift($xml->xpath('/notice/request/cgi-data/var[@key = "Device"]'));
$errorbrand = array_shift($xml->xpath('/notice/request/cgi-data/var[@key = "Brand"]'));  

Finally we just need to store everything into the database.

$query = "INSERT INTO error (appkey, errorclass, 
  errormessage, errormanufacturer, errordevice, errorbrand, errorandroidversion, 
  errorappversion, errorbacktrace, submissiondate) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
	
$statement = $databaseConnection->prepare($query);
$statement->bind_param('ssssssssss', $appkey, $errorclass, $errormessage, 
  $errormanufacturer, $errordevice, $errorbrand, $errorandroidversion, 
  $errorappversion, $errorbacktraceToDB, $submissiondate);
$statement->execute();
$statement->store_result(); 

Website

The Default template still provides us the possibility to add, remove and edit pages. Since we don't have pages, but Apps, we just remove everything which contains the word "Page" and create a couple of new files to be able to Add / Edit / View and Delete Apps. We also want to be able to see the crash reports, so we need some more logic here too.

Adding Apps

Creating new Apps should be as easy as possible. We just need the possibility to enter a AppName. The AppKey should be automatically generated.

The following code-snippet will create a unique ID for us:

$appkey = uniqid (rand (),true); 

Once the AppName is choosen the following php code will save the app in our database

if (isset($_POST['submit']))
{
    $appname = $_POST['app-name'];
    $query = "INSERT INTO apps (appname, appkey) VALUES (?, ?)";

    $statement = $databaseConnection->prepare($query);
    $statement->bind_param('ss', $appname, $appkey);
    $statement->execute();
    $statement->store_result();

    if ($statement->error)
    {
        die('Database query failed: ' . $statement->error);
    }

    $creationWasSuccessful = $statement->affected_rows == 1 ? true : false;
    if ($creationWasSuccessful)
    {
        header ("Location: index.php");
    }
    else
    {
        echo 'Failed adding new app';
    }
}  

Once created, we should see all our apps on the main overview page.

We just need to adjust our apps.php file to get all apps and the amount of reported errors

<table class='apps'>
<thead>
    <tr>

        <td class='name' >Name</td>
        <td class='count'>Count</td> 
    </tr>
</thead>
<tbody>
<?php

$statement = $databaseConnection->prepare("select apps.appkey, 
  apps.appname, count(error.id) as count from apps LEFT JOIN error ON 
  apps.appkey=error.appkey group by apps.appkey  ORDER BY count desc;");
$statement->

Submit a Crash Report using Android-ErrBit Notifier

To test our service we setup a blank android app in eclipse, download the ErrBit Notifier from github and set it up in our project with this one line of code:

ErrbitNotifier.register(this, "errzureservice.azurewebsites.net", "1262151a1ea8dcc1088.55035136"); 

The first Parameter, "this", is needed to make sure ErrbitNotifier can access the hole context of the System (e.g.: Save an exception to a file, to make sure it's not lost when no internet connection is available). The second parameter if our endpoint, here our new created azure website. Finally, the third parameter is our unique App Key,  generated by our service.

To test everything, we need to do two more things. Android needs permissions for nearly everything. Our App would not be able to access the internet or write files, so we need to need two lines to and AndroidManifest.xml to tell the system, we need these permissions:

<uses-permission android:name="android.permission.INTERNET" /> 
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> 

The second thing to do is to throw an Exception:

List l = null;
l.add("Hello Exception");  

At this point, everything should be fine and we should see our first error logged in our Azure service. It was tested on a Ubuntu VM with apache + php + mysql, and everything was just fine.

However, IIS server seems to have some other rules in working with PHP. After hours of fighting with logfiles and google i've found the problem.

Azure does not accept HTTP POST method in my API: http://.../notifier_api/v2/notices/. If we would change the API Endpoint to http://.../notifier_api/v2/notices/index.php, everything would work just fine.

Since i've wasted to many time with finding this issue, i'll change this after this challenge and write down all steps so others can learn from it.

But with the "new" API endpoint, everything seems to be fine:

If we click on our app name, we'll see all exceptions for this one app:

And finally, after following the link behind the error, we'll see all detailed information:

So where is the Source Code? 

I have to admit that I've underestimated this project. Using a programming language after a 2-3 year break isn't as easy as I've expected.  I've also never used PHP / MySQL on Windows. This was not 100% the same experience i had on Linux, but maybe things have just changed. And finally, there are two types of Open Source mindset:

1. Open Source everything as soon as possible. People are responsible for their own if they use my code
2. Open Source a nice product people can learn from 

I belong to the second group of people. My code works, but it's not good enough to post it as an open source project on CodeProject. I have to resign from the third challenge! Code will follow the next 1-2 weeks!

Fourth challenge, VM 

In this challenge we will setup the open source error reporting solution "ErrBit" in an Azure Ubuntu VM. We will also create an open source library, an ErrBit Notifier for Windows Phone.

I haven't found any "ready to go" libraries yet for Windows Phone, so this is a good chance to create it by ourselves.

Setting Up ErrBit

Setting up ErrBit seems to be very complicated for some people. ErrBit is written in Ruby on Rails, and requires Ruby on Rails on the machine to run it. Ruby on Rails is written in Ruby, therefore Ruby is also required. To setup and maintain Ruby on the VM easily we will also install rvm (ruby version manager). This is not required, but i would not install ruby without rvm anymore. With rvm you can have different versions of ruby installed and easily switch between them. Some Ruby on Rails web-apps have a special ruby version as a requirement. This can be a real pain, if you have to setup two or more Rails-Apps on one VM without rvm. ErrBit stores the whole amount of Data in a MongoDB (NoSQL) Database, which of course is not preinstalled. We will also install bundler to manage required Gems. A Gem in Ruby is very similar to what we call a library in all other programming languages. And bundler is something like NUGET in the .NET world. However, a Gem can also be a standalone program, so it's a little bit more then just a library. We will also install git in our VM, so we can easily clone the newest errbit version from github.

To setup our VM, we just need to login into our management center and select the VM Tab:

In this Tab we need to hit the "NEW" Button in the Bottom of the Page.

Which will bring up a menu from the bottom. Select "FROM GALLERY" to get a huge list of possible operating systems which can be used in a VM,

We are going to use Ubuntu Server 12.04 LTS as our operating system.

The LTS suffix is the short term for Long Term Support and will guaranty that we will get updates for this operating system. The support for this version, 12.04, will be available until 2017-04-26, so a quite good time.

We have to setup different things in the following 4 steps.

Release Date of our operating system (use the newest one), a name for our VM, a new username, password, the VM size, location and a dns name. All these things are self-explaining and the azure wizard will guide us the these steps.

Once we've reached the last step and created the VM, our VM tab will update and we can see our VM starting and finally, after a couple of minutes, beeing ready and running:

Connecting to the VM

Our VM is now setup, running and doing nothing. We cannot go to a web-page now, because there is no http server running yet, and even if, the port is blocked by now. To connect to our VM, we need to know on which port we should connect. SSH is typicality running on port 22, however, for security reasons it's a good idea to move this to another port, so bots who scan the internet for "typical" open ports, will skip this host. To get the correct port, we need to go to the "ENDPOINTS" Tab in our currently created VM:

We see that our SSH daemon is really running on the port 22, however, the public endpoint is redirected to the port 58317.

With this information we are now able to connect to our VM. This tutorial is written using Linux, so it's easily possible to use ssh from the console. Windows user have to download and use Putty (just one possibility, but might be the best)

With the following command, we are able to connect to our VM

ssh errbitserver.cloudapp.net -p 58317 -l ErrBitUser 

1. errbitserver.cloudapp.net is our host name, we could also use the IP address of our server
2. With -p 58317 we are specifying the port to connect. (default would be 22)
3. With -l ErrBitUser we are using another username (l for login) the we are currently using in our local operating system

Linux will complain a little bit about your attempt to connect to another machine via SSH the first time and asking you if you really want to do this. Once you've accepted this by typing yes (or just "y"), the server-fingerprint will be added to the list of known hosts and this questions will not appear again. 

Now we just need to enter the password we've choose while setting up our VM in the management center of azure and we are connected.

Getting Started on Linux  

In this section we are going to install all required apps, libs and tools we need to run our ErrBit service. This will be a complete guide, we will not skip any step, since not everyone is familiar with Linux! Whenever we are connecting to a new and fresh installation of Ubuntu, we should run an apt-get update here.

sudo apt-get update

This will not update the operating system itself or all the pieces / tools which are already installed. It will just update the references and the package list, which contains links to the newest versions of tools we might want to install.

1. apt is the short command for Advanced Packaging Tool
2. apt-get is some kind of front-end tool to GET, or Download, all kind of other tools
3. update is used to resynchronize the package index files from their sources (Quote from Wikipedia) 
4. sudo is a program which runs the following tool / program / command with root / admin rights   

Note: APT is not available on all Linux distributions after a fresh install.

Some minutes after running this command we should be up to date and able to continue

We will first install a huge amount of tools and libraries with just one command.

sudo apt-get -y install curl git-core patch 
\build-essential bison zlib1g-dev libssl-dev libxml2-dev 
\libxml2-dev sqlite3 libsqlite3-dev autotools-dev 
\libxslt1-dev libyaml-0-2 autoconf automake libreadline6-dev 
\libyaml-dev libtool

Some of them should be known

1. curl
2. git-core
3. sqlite3

... others are just dependencies, libraries or development tools, e.g.: automake.

This command however, will take some time. Fortunately we've used -y in the command, which will answer all questions during the installations with yes (questions like "do you really want to download these 25MB of data" or "git need also BLABLA to be installed, do you want to install this as well").

After some minutes we should be ready and finally able to start with the real first step

Installing rvm, the Ruby Version Manager.

We will use this command to download and run a "ready to use" rvm installer from github

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)    

It's common in linux to just put a bunch of commands together, making it sometimes impossible to understand what happens for non Linux users. In fact we are using two tools here:

1. bash, which is is a Unix shell written by Brian Fox  (You can have different shells in parallel on Linux)
2. curl, which is a command line tool to transfer files

We could also open the URL to this script in the browser and get some details about how this all works. But this should not be part of this tutorial.

This command will be finished with wise words

It will! It will! Thx Wayne!

But to make sure rvm can be used in our terminal (shell) we need to set up some more things. Everyone should have installed java at some point in his life and had to setup the PATH correctly, we need to do the same here.

We are using "nano", which is a console text editor for Linux to setup up this

nano ~/.bashrc

Once we've opened our bashrc file, we just need to add these two lines

PATH=$PATH:~/.rvm/bin

[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"

Add the lines between the first comment and the second  (comments start with # )

To close the windows we need to hit

[ctrl] + [o] to save our file ( + hit enter to overwrite the existing file)  
[ctrl] + [x] to leave the nano editor 

We should be now able to use rvm. Lets check the current version with

rvm -v 

Of course, it did not work!

Normally, changes on the path file are followed by a restart of the machine.

But we can just restart (better: reload) our terminal settings by running this command

. ~/.bashrc

Running the same rvm -v command once again should now print the following message

Finally we can install Ruby now. We are going to use ruby-1.9.2, which is a very stable version of ruby.
With:

rvm install ruby-1.9.2 

the magic will happen automatically (we just need to enter our password again)

At this point we need to understand that we've just compiled Ruby from the source. This is also the reason why it took so long (will be faster on better VMs with more CPU of course)

We should check our azure management interface and select the monitor tab for our VM. The CPU usage should be very high, compiling ruby need some power! 

To be able to use ruby, we need to enter two more commands. The third command is, once again, just the check that everything is fine

/bin/bash --login
rvm default 1.9.2
ruby -v 

With the first command we are telling our terminal to use the bash interpreter from now on. The second one tells our ruby version manager which ruby version should be used by default. This of course is 1.9.2, the only one we've installed. The third command should print the ruby version

Before starting the Ruby on Rails installation, we are going to update some gems and the gem management system itself:

rvm rubygems current

and

gem update 

will do the magic for us. Once again, this will take a while. But we don't want to get into conflicts because of old gems / libs, so this should be done always after a new ruby setup.

Finally we can run the command to install Ruby on Rails:

gem install rails

This again, takes some time.

But finally we can run rails -v to test if everything went fine:

rails -v

Everything went better then expected. We have successful installed Rails 3.2.13!

MongoDB

Unfortunately, Ubuntu does not come with the most stable source / package list for MongoDB, so we need to add our own source before running the installation.

With:

sudo nano /etc/apt/sources.list.d/10gen.list

we are going to create a file called 10get.list in the sources.list.d directory (password is needed, since we are going to change some stuff in the system)

Add these line to it:

##10gen package locationdeb 
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen 

save the file with [ctrl] + [o] and exit with [ctrl] + [x]

The 10gen repro (10gen is the company behind mongoDB) requires GPG keys. This makes their repro very secure. Just run this command to make sure you will not get any errors later.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 

Since we've changed our package sources by adding a new one to it, it might be a good idea to update the list again

sudo apt-get update  

will fetch the new data (in fact, it will check the package source list again and tell you, that they are all up to date. Just the new one will be refreshed).

Now we can finally run the command which will install the mingoDB NOSQL Database on our VM

sudo apt-get install mongodb-10gen 

As you can see in the 10th Line in the image above, the installation is using our new added source to get the latest stable version. The installation will be finished by starting the mongodb process.

Even more required tools...  

What we have done until now, is to setup a vm which is perfectly prepared for getting started with Ruby on Rails development. However, ErrBit has some more requirements. The github page tells us all the requirements, and we just follow these steps.

sudo apt-get -y install libxml2 libxml2-dev libxslt-dev libcurl4-openssl-dev 

After these libraries are installed, we finish by installing bundler

gem install bundler  

Get ErrBit and Run it! 

We need to clone ErrBit from github to our local machine. To have things clear and structured, we first create a new Folder, switch to it and clone the git-repo:

mkdir ErrBit
cd ErrBit
git clone https://github.com/K2DaC/errbit.git

To finalize everything, we switch to the git-repo called errbit, and tell bundler to get all gems which are required by errbit.

cd errbit
bundle install

We are now installing things like json parsers, http auth libraries and a lot of other things for errbit.

It's always fun to install the HTTParty Gem  

After having all requirements installed, we need to bootstrap ErrBit.

This just means we need to copy some configuration files and prepare the database (Create Indexes, etc). Remember, we are using a NOSQL database. We are not creating Tables here...

rake errbit:bootstrap  

Finally we can run the Rails Server!

script/rails server -d 

Ruby on Rails comes with a own WebServer called Thin, which can be used for debugging. This server runs on port 3000 by default. But if we try to connect to this port, we will not see our ErrBit service. The firewall is blocking our connection. We need to go to our management system of azure and open the required port:

We can now go to our web-site and open it (on port 3000)

The default username and password should be setup in the seeds.rb file. Since we haven't done this, we just use the default one which was created for us:

http://errbitserverv2.cloudapp.net:3000/ 

User : errbit@errbit.example.com
Password:  password  

We can use this to login and we see our "ready to use" solution.

ErrBit testing

Our service is up and running, but we haven't used it yet. We need some things to test and see what errbit provides to us

1. We need a new App.   

Just hit the "New App"-button in the top right and give it a name.

We can setup a lot of things here. For example we can get eMails, every time a new error is spotted. We can also use the issue-tracker, so each error (of course only the first time it's reported) will create a ticket on Redmine (Which is a Ticket / Bugtracking Tool).

But we just give our App a name : AndroidApp

After hitting the save button we get an info-page, how we could use our App now in Ruby on Rails

Airbrake.configure do |config|
   config.api_key = '52c9ca0c3b462037759ece317ba8d790'
   config.host = 'errbitserver.cloudapp.net'
   config.port = 3000
   config.secure = config.port == 443
end   

We will not use this in a Ruby on Rails application. We are going to use this in Android now (Can be used in nearly in every programming language).

First of all we need to download and include the existing ErrBit Notifier Library for ErrBit from gitHub. We setup a new android project in eclpise, link our library to it correctly and call this one method:

ErrbitNotifier.register(this, "errbitserverv2.cloudapp.net:3000","52c9ca0c3b462037759ece317ba8d790");  

Since we are going to report Errors / Crashes here, we need to force a exception here:

We just do this:

List l = null;
l.add("CRASH"); 

Now we just need to run this and see what happens. Of course our App is crashing, but we also get the result in our ErrBit Service

If we click on our App, we get some more information

And finally we can follow this error to get more information about it

Create our own Notifier for Windows Phone 7 

There are notifiers for nearly every platform / language.

1. Android
2. iOS 
3. PHP
4. ...

There is also a .NET solution, but i haven't found a real Windows Phone solution for this, which really takes care about different possible scenarios. E.g.: Crash happens without Internet Connection!

Requirements:

1. Able to report Errors to ErrBit and ErrZure
   1. Use same XML Layout
   2. Use same API-Structure
   3. HTTP Post Errors to service 
2. Save errors to local storage to be able to send them later if no internet connection is available
3. Possibility to set API-KEY / Host easily

Our goal is to initialize our library by this one line of code:

ErrBitNotify.Register("API-KEY", "ENDPOINT", this); 

That's why we need a register method wit 3 parameters

1. API KEY, to identify the App in our Server
2. The endpoint, to make it possible to use this just everywhere
3. The Application class, so we can attach to the UnhandledExceptionEvent

The method mainly prepares things, if everything is fine. But this method is also responsible for sending exceptions from the last session. On Android for example, you have as much time as you need after an exception is thrown to do all kind of stuff. Just a new thread is needed. On Windows Phone, you have just a couple of seconds. This might not be enough to send the exception to the server, so we store them first, to make sure we never lose them.

public static void Register(String apiKey, String endpoint, Application app)
{
   mApiEndpoint = "http://" + endpoint + "/notifier_api/v2/notices";
   mApiKey = apiKey;
   mAppVersion = 
     System.Reflection.Assembly.GetExecutingAssembly().FullName.Split('=')[1].Split(',')[0];
   app.UnhandledException += 
     new EventHandler<ApplicationUnhandledExceptionEventArgs>(app_UnhandledException);
   SendAllExceptionsToServer();
}

As you see we prepare the host-URL and detect the app-version. This information can save you a lot of time. Just say "FIXED" to your QA department ;). We also create a new EventHandler for the UnhandledException. Finally we try to send all exceptions to the server. In most cases, this method will do nothing (we hope so), but if you had an error in your last session, and the library was not able to transfer all information to the server, we will do it now.

Our callback for the ... does mainly two things.

1. Create the XML file in the correct format / layout
2. Transfer the Exception to the Server (or try at least)

static void app_UnhandledException(object sender, ApplicationUnhandledExceptionEventArgs e)
{
    mException = e.ExceptionObject;
    WriteXMLToFile();
    SendAllExceptionsToServer();
}

Using XML Files in Windows Phone is very straight forward. The same applies to saving files on the device.

public static void WriteXMLToFile()
{
   using (IsolatedStorageFile myIsolatedStorage = 
             IsolatedStorageFile.GetUserStoreForApplication())
   {
       if (!myIsolatedStorage.DirectoryExists(DIR))
            myIsolatedStorage.CreateDirectory(DIR);

       String time = "" + DateTime.Now.Ticks;
       String filename = mAppVersion + "-" + time + ".xml";
       using (IsolatedStorageFileStream isoStream = new IsolatedStorageFileStream(
         DIR + "\\" + filename, FileMode.Create, myIsolatedStorage))
       {
            XmlWriterSettings settings = new XmlWriterSettings();
            settings.Indent = true;
            using (XmlWriter writer = XmlWriter.Create(isoStream, settings))
            {
                List<ParsedException> list = SplitException();
                writer.WriteStartElement("notice", "");
                writer.WriteAttributeString("version", "2.0");
                writer.WriteStartElement("api-key", "");
                writer.WriteString(mApiKey);
                writer.WriteEndElement();
                writer.WriteStartElement("notifier", "");
                //More meta-info here
                writer.WriteString(mException.GetType().FullName);
                writer.WriteEndElement();
                //More details here, e.g. stacktrace ...                       
                writer.Flush();
           }
       }
   }
}

As you can see in the code above, we have a unique filename, based on DateTime.Now.Ticks. We also store them in a separate folder, so we are able to access only our files later and make sure to not have any conflicts with other apps.

After we have "handled" the exception and stored it in the device memory, we can finally transmit it to the server.

The SendAllExceptionsToServer() method just loops through all files we have in our folder, transfer one after another to our server and delete it locally.

string searchPattern = DIR + "\\*";
string[] fileNames = myIsolatedStorage.GetFileNames(searchPattern);
if (fileNames.Length > 0)
{ // DO THE REAL WORK } 

The "real work" is just this POST Request preparations:

HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(mApiEndpoint);
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/xml; charset=utf-8";
httpWebRequest.BeginGetRequestStream(result =>
{
    PostWebRequest(result, httpWebRequest, fileNames[0]);
}, null); 

With fileNames[0] we always just grab the first one and send it with this ....

private static void PostWebRequest(IAsyncResult result,
                              HttpWebRequest request,
                              string filename)
{
    Stream postStream = request.EndGetRequestStream(result);
    String post;
    using (IsolatedStorageFile myIsolatedStorage = 
      IsolatedStorageFile.GetUserStoreForApplication())
    {
        IsolatedStorageFileStream isoFileStream = 
          myIsolatedStorage.OpenFile(DIR + "//" + filename, FileMode.Open);
        using (StreamReader reader = new StreamReader(isoFileStream))
        {
            post = reader.ReadToEnd();
        }
    }
    byte[] postBytes = Encoding.UTF8.GetBytes(post);
    postStream.Write(postBytes, 0, postBytes.Length);
    postStream.Close();

    request.BeginGetResponse(res =>
    {
        GetResponseCallback(res, request, filename);
    }, null);

}

Our final callback handles the server response

private static void GetResponseCallback(IAsyncResult asynchronousResult, 
          HttpWebRequest request, string filename)
{
    HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult);
    HttpStatusCode rcode = response.StatusCode;
    if (rcode.Equals((HttpStatusCode.OK)))
    {
        using (IsolatedStorageFile myIsolatedStorage = 
                 IsolatedStorageFile.GetUserStoreForApplication())
        {
            myIsolatedStorage.DeleteFile(DIR + "\\" + filename);
        }
        SendAllExceptionsToServer(); //Keep sending files until all are submitted!
    }
} 

If everything went well, we get a 200, OK, response code from the server. (Which might not be best practice in using the http protocol, a 201, CREATED, would be better).

If this is the case, we simply delete the file we've submitted and just call SendAllExceptionsToServer() again.

To test everything, we create a new app in our ErrBit service and just force an Exception in our Windows Phone App.

The result can be seen in ErrBit: Our new created app has an error now:

If we follow the link to the next page, we see that i had fun testing all scenarios : No Internet connection, slow Internet connection, ... and I've forced 28 Errors.

The final details page provides the most useful information: the method which is finally responsible for the crash.

History

• 26 April 2013, Kickoff 
• 26 April 2013, Format   
• 27 April 2013, Added more details about the system. Fixed typo with AirBrake  
• 27 April Fixed wrong date in history  
• 28 April 2013, first UI draft  
• 30 April 2013, added XML layout  
• 30 April 2013, replaced ugly wireframes, added mobile wireframe 
• 01. May 2013, started challenge 2 
• 02. May 2013, finished challenge 2  
• 10. May 2013, added an Easter Egg
• 11. May 2013, fixed typos (thx to roschler)  
• 26. May 2013, challenge 3 
• 01.June 2013, challenge 4 
• 03.June 2013, updated challenge 4, fixed missing images, fixed typos 
• 03.June 2013, updated challenge 4, added URL to the top of this article 
• 06.June 2013, updated challenge 4, added Windows Phone Notifier  
• 09.June 2013, fixed some typos 
• 14.June 2013, changed URL of ErrBit VM / Service 
• 14.June 2013, added QuickJump + fixed links again!