API Abstractions to Accelerate the Development Process?

No matter if you are writing mobile applications, websites, desktop applications or server backends, you’ll often have to integrate multiple external APIs. A mobile application should offer the possibility to signup and login with any one of many social networks, a website is expected to have buttons for liking and commenting via social media, a desktop application should synchronize data with a cloud storage provider, ideally the user can choose which, and a server must send an email after the successful registration of a new user.

All of this requires access to the APIs of providers like Facebook, Dropbox, SendGrid and others.

The Challenge for Developers

Most providers expose their services over HTTP interfaces and often provide SDKs for diverse platforms which encapsulate the actual HTTP communication and hide it behind function calls.

Authentication can already be the first hurdle for developers. If you want to access your own account via the API, e.g. to make use of an SMS provider, it usually suffices to pass an API key along with every request. If on the other hand a developer wants to access a user’s data, a common use case e.g. with cloud storage providers, OAuth 1.0/2.0 is most commonly found. OAuth 2.0 is an evolution of OAuth 1.0 which features increased flexibility and reduced complexity due to the mandatory use of HTTPS instead of a laborious manual signing process.

Both have in common that they allow an application to gain authorized access to a user’s data without that user having to expose their username and password to the application. Instead, the user authenticates to the service and authorizes access to their data which in turn makes the service grant access to the application. Even though OAuth 2.0 is less complex and more widely used nowadays than OAuth 1.0, it still has to be understood by the developer and can be found in different implementations on the server side. In addition, there still are prominent services like e.g. Twitter which still use OAuth 1.0.

The most widely used concept to structure such APIs is without a doubt REST (Representational State Transfer) which, among other things, specifies that resources should be uniquely addressable and accessible in a structured way via URIs. Access usually happens via HTTP(S) with the HTTP method (GET, POST, PUT, DELETE, …) indicating the operation to be performed on the resource. While many APIs claim to be "RESTful", everyone seems to have a slightly different opinion of how exactly REST should be implemented, no doubt also due to the fact that REST is more of a paradigm than a detailed standard. This makes API structure inconsistent.

Data formats are different too, most often JSON or XML are used. JSON has the reputation of being more compact and readable, XML is considered more expressive. Especially in web browser environments with ubiquitous JavaScript, JSON demonstrates its advantages as it can be mapped directly to JavaScript objects. XML on the other hand allows, without extensions, the use of references, even cyclic ones, and arbitrary data types. The challenge for the developer results from some APIs using JSON while others are using XML, even for similar purposes. A few let you choose to get data in either format but this is rare.

The actual data a service returns or accepts is different as well. A simple example of this behavior can be observed with a user’s date of birth, information nearly all social networks provide. Facebook returns it as a String in the format "MM/DD/YYYY", "YYYY" or "MM/DD" depending on which parts of the date are public.

Google Plus uses "YYYY-MM-DD" and Microsoft Live returns three separate numbers for day, month and year. This means a developer will not get around getting involved with such details in order to be able to normalise the information to the format they are using internally in their application.

When it comes to documentation, not every service covers itself in glory. There are services which extensively document their HTTP interface as well as their SDKs, helpful examples included, but also those where a handful of outdated examples are as good as it gets. Only the extensive use of your favorite search engine or trial and error can help you find out how to access or input the information you need.

Last but not least the communication with these APIs will rarely ever happen without any errors. If the user provides invalid input a developer can certainly make sure that e.g. an email address is really a String and in the right format or throw an exception if that is not the case. Yet it is not possible to locally verify that the email address actually exists and that an email could actually be successfully delivered to it. This is why it is important to be able to appropriately react to error codes and messages from the server. Here as well APIs differ in their behavior. Often an error has no direct or only a very unspecific correspondence with an HTTP status code reducing their usefulness. The error description and the machine-readable error code commonly found in the body or header of the server’s answer are specific to the API.

The aforementioned issues predominantly relate to the direct interaction with the HTTP API but users of the provider SDKs will encounter most of them as well.

API abstraction to the rescue

The biggest problem lies in the difference in details found even with very similar services. Thus it seems obvious to abstract similar services against a common interface. Even without the cooperation of the services this creates a de facto standard. The more similar functions the services provide the more can be abstracted. This has a number of advantages. A developer does not have to go into each service’s documentation to learn how to accomplish a certain task but can concentrate on only one interface. The services which implement this common interface can then be treated uniformly in code, making it shorter, more compact and less error prone.

CloudStorage cs;

switch (serviceChoice) {
  case 0: cs = new Dropbox(this, "[ClientID]", "[ClientSecret]"); break;
  case 1: cs = new GoogleDrive(this, "[ClientID]", "[ClientSecret]"); break;
  case 2: cs = new Box(this, "[ClientID]", "[ClientSecret]"); break;
}

InputStream is = cs.download(path);

In addition it becomes much easier to exchange a service for another in case the one currently used loses attractivity due to changed pricing, rate limits etc. The disadvantage is clearly that it is not possible to abstract a function which is only provided by a single service without losing the advantages of uniform treatment and easy exchangeability. This may well be acceptable for developers that exclusively use common standard functionality, e.g. upload and download for Cloud Storage providers, and do not require access to more specific features.

There are multiple providers of such abstracted integration solutions which can be roughly divided into two categories: IPaaS Middleware and Abstracting SDKs.

Integration Platform as a Service

Commercial providers like Kloudless and Cloud-Elements provide a middleware solution, often called "Integration Platform as a Service" (IPaaS), which offers a uniform, abstracted interface to different third-party services. Just like the third-party services themselves, the IPaaS exposes an HTTP interface and SDKs for diverse platforms to make the interface easily accessible.

IPaaS providers use external middleware (Img. 4)

A clear perk of this approach is that developers can use the abstraction on any platform because even when there is no SDK for the platform used it is always possible to interact directly with the HTTP interface. Unfortunately the IPaaS approach also comes with a couple of disadvantages. First of all, the middleware adds another single point of failure, decreasing the robustness of the overall system. If the servers of the IPaaS provider are unreachable, all the integrations break. Moreover, all data that flows between the service and the application passes through the IPaaS provider’s servers causing unnecessary traffic especially for Cloud Storage providers resulting in additional cost the developer ends up paying for. Last but not least the IPaaS provider requires access to unencrypted user data, how else could the aforementioned date of birth be normalised. This can be clearly problematic in terms of privacy and data protection.

Abstracting SDKs

An alternative approach to IPaaS are abstracting SDKs. They are not hosted as middleware on a server but are instead directly integrated into the application as a software library.

At least from the hardware side this makes them less susceptible to service outages. Data can be encrypted using HTTPS before it leaves the device and is sent to the service directly, there are no detours.

Spring Social for Java applications using the Spring Framework is an open source project that facilitates social network access. While it does not abstract all social networks against one interface it offers a uniform look and feel for all available services and simplifies authentication with OAuth.

HybridAuth for PHP also concentrates on social networks but provides an abstracted common interface, as of now for 32 different services. This offer can be further extended with plugins, authentication happens automatically and it is also open source.

A comprehensive, commercial solution is CloudRail which also offers a free community version. With abstracting SDKs for Android, Java, Node.js, Swift and Objective-C developers can integrate services from many categories. As of now these categories are Cloud Storage, Social Networks, SMS, Email, Payment and Points of Interest. The solution features extensive documentation, abstracted APIs and simple authentication. In addition, CloudRail offers a feature called "API Change Management", providing automated API monitoring and notifications to the developer in case one of the integrated APIs changes or becomes unavailable like Dropbox announcing the shutdown of their v1 API. Due to the abstraction it suffices to update the SDK, the code needs not change.

// let social = Twitter(clientID: "[clientID]", clientSecret: "[clientSecret]")
let social = Facebook(clientID: "[clientID]", clientSecret: "[clientSecret]")
do{
  try social.postUpdateWithContent("CloudRail is awesome!!!")
} catch let error{
  print("An error: \(error)")
}

Summary

If you have to integrate many different APIs you will be faced with a range of challenges due to their diversity in authentication, structure, data formats, documentation and error reporting. Abstraction can be a remedy, there are hosted solutions and libraries that are integrated directly into applications. The former convince with platform independence, the latter offer better scalability, robustness and data security. It seems unlikely that API providers will eventually agree on a common standard that every one of them sticks to. Too big is the need to functionally distinguish one’s own product from competitors’ products and standards would hinder that. Since we can expect to see more and more APIs emerge in the market, "unified access" becomes increasingly important making above mentioned solutions evermore relevant.