WpGet: Private Wordpress Plugin Repository

• Download plugin-test (sample GPL) - 5.2 KB
• GitHub Project
• Download web-installer

Table of Contents

• 1. Introduction
• 2. Why We Need a Private Repository for Wordpress
• 3. Functional Requirements: What a Public Wordpress Repository Should Be
• 4. Technical Requirements: How a Private Repository for Wordpress Should Be Done
• 5. The Backend
  • 5.1. Multiple Backend Applications
  • 5.2. Automate Routing
  • 5.3. Automate Database Creation
  • 5.4. Automate CRUD Operation
  • 5.4. Automate Dependency Management
• 6. The Frontend
  • 6.1. UI Framework: prime NG
  • 6.2. Implement http Interceptor to Manage Authentication
  • 6.3. Dynamically Compute baseHref
  • 6.4. Decouple Configuration From Compilation
• 7. The Wordpress Plugin
• 8. Point of Interest
  • 8.1. Installation Process
  • 8.2. Build and Packaging
• 9. Conclusions
• 10. References
• 11. History

1. Introduction

This article tells about I design and build wpget, the on-premise wordpress plugin repository. This is an interesting implementation that can be used as reference for API-oriented web applications. It is based on PHP stack, using slim framework as backend, token based authentication with the angular2 frontend. I will discuss about backend and frontend design, having a deeper investigation about continuous integration, and delivery to the on-premise instances.

2. Why We Need a Private Repository for Wordpress

Wordpress is the most used CMS in the world with the most important plugin system. It can be customized with more than 5000 plugins and cover the 60% of CMS powered websites (29% of the whole websites). It's easy for a developer to create and publish a new plugin into the public repository. But what if I had to create and maintain a private plugin? I'm speaking about:

• a plugin with an high intellectual content,
• a plugin not ready for the public, maybe focused on some particular business case
• a custom super-plugin built used to inject custom plugin dependency

There are actually these opportunities into public repository:

1. Make a free plugin
2. Make a paid plugin (but placing into main repository, it is available to everyone open to buy it)
3. Install your plugin manually

First two options are the most widely adopted by public plugin sellers, while the third one is a very common practice into agency or companies that have some common plugin reused in many websites.

WpGet came to give a fourth opportunity. It is conceived as a private repository where you can push plugins and allow wordpress installation to update from that source, in addition to the standard platform. For those who come from other technology, WpGet is similar to NuGet, Maven or NPM package manager.

Here a simple diagram to explain how WPGet works.

I hosted all code on Github where you can also download the last release.

3. Functional Requirements: What a Public Wordpress Repository Should Be

In my experience, I played with many package repositories (Maven with Java, composer with PHP, Nuget with .NET and so on…) and I think in 2018, everybody uses them each day during development. However, what about Wordpress plugin? In fact, a plugin is just a zip file. A zip file with PHP file that implements features basing on Wordpress standard, but simply a zip file from a higher view. So a plugin repository can be treated as a raw zip file repository. Such purpose can be achieved using some ready product like Nexus that can stored from a versioned zip file into raw repository. While this could be ok for archiving plugin, it may find some limits in the integration with Wordpress itself because there isn’t any plugin that integrates such system. Searching into Github, there are some Wordpress projects that manage plugin using some http repository but no one seems to be so mature to be considered as a standard and no one has a UI that lets you understand what happens behind the box.

To be more clear, what we need from this project is:

• Provide updates for plugins, keeping all packages and providing to the Wordpress installation. Using token based authentication, all processes are kept secure.
• Easy to setup: Setup must be easy. Just a copy of the bundle to the server and few steps more. Plugin integration is very easy, just a class to add.
• API oriented All core features are exposed by API. This will allow you to integrate the delivery process with other tools in the company.
• UI to get control: A simple UI shows what packages are loaded and gives you the power.
• Low Server Requirements: Just a webserver with PHP and a database. For few data, using SQLite, neither database server.
• Versioning: have a system that stores and version wordpress plugins
• Integrable in development process: be able to publish wordpress plugin using a push script
• Made for humans: simply upload packages manually if you don’t like shell commands
• Multi tenant: system that allows to manage multiple repository with the same installation instance (i.e., one repository for each customer)
• Secure: a secure system that allows only trusted websites to download packages

4. Technical Requirements: How a Private Repository for Wordpress Should Be Done

The principles that drive my design were:

• employ standard technology
• API oriented application
• Backend\Frontend complete separation, SPA for the frontend
• token based authentication
• use technologies that can be compatible with Wordpress to easy installation
• to be delivered in a one-click installation package

To grant the compatibility of technology, my choice for the backend was a PHP application based on Slim Framework. This is because I would use a simple and lightweight framework as the application is designed to manage just few database entities and a very simple business logic. By the way, Slim has a lot of modules that can be integrated, so further expansion of the application will be managed without any limits. Frontend is implemented using an Angular application that is a powerful solution.

5. The Backend

5.1. Multiple Backend Applications

WpGet is mainly designed for small applications like the web agency that develop some plugins and want to provide to their customer. Basing on such scenarios, it is easy to imagine the load of application won’t be so heavy. Users don’t really need to play with UI at all times, publishing packages are not so frequent (1-2 per day), and clients will update Wordpress installation only if there are changes in plugins. We don’t have a crystal ball to tell for sure, but probably performance and load won’t be an issue on most applications of WpGet. Anyway, I wanted to design an architecture that may scale in future, in example, if we would like to publish this application in SaaS environment. While this application doesn’t use sessions or shared memory other than database and storage folder, it will be quite easy to proceed with horizontal scaling as users will grow. By the way, I also wanted to create a structure that allows to let application grow following application module usage. So, I split the monolith into three components:

• Auth: application that provides authentication
• Api: application that hosts all API for UI and CRUD operations
• catalog: application that orchestrates integration with Wordpress and publishers

This will allow to destinate the right amount of resources to the single component. Moreover, we will have three smaller applications, that have only the required dependency each one, making them faster and easy to deploy. In the regular on-premise bundle, all the applications are bundled together to simplify installation, but, playing with configuration and servers, we can simply move to the complex solution.

5.2. Automate Routing

Slim framework is a great solution and allows quickly to produce a rest endpoint in a quick and dirty way.

You can use lambda in the main application file or create invokable classes.

Examples

$app->any('/mycontrollerpath', function ($request, $response, $args) {
    // do stuff here
});

$app->any('/user', 'MyRestfulController');

class DynamicController
{
    public function __invoke($request, $response, $args)
    {
      // do stuff here 
    }
}

Meanwhile, I resigned myself to work with untyped data delivered by service without a strong contract (like WCF or Web API, to explain better…), I wanted to avoid enumerating all possible routes into main application files. The best solution I imagined was to declare some classes or annotation, then decorate controllers, then make a scan at runtime and dynamically use such information. This will have lead to something very similar to ASP.NET WebApi engine, but I preferred to find a simpler solution because:

1. this is a very out-of-standard solution
2. PHP doesn’t have any “native” application memory state so I had to add some memory state (i.e., using memcache) or load each time wasting resources.

The solution I found was simple and very efficient:

1. I create a base controller class, called DynamicController. This controller implements Invoke method and will receive all action calls, dispatching to a method with same name of the action and same http method. This is based on naming convention so you will have /mycontroller/myaction get call managed by getMyAction method inside Mycontroller class.
2. The real implementation of action methods is inside a class that implements DynamicController.
3. Inside application file, you have to add only one entry per controller:

$app->any('/repository/{action}[/{id}]', RepositoryController::class);

5.3. Automate Database Creation

Another part I would improve is about CRUD operations. I used eloquent ORM and it is a very simple framework to set up. In its simple implementation to create a rest service that exposes CRUD operation, you need:

1. Create a model class. The only required information is the name of the table.
2. All access to fields can be considered by-name as PHP members are dynamic, so if you need to set\get a field, just use it.
3. In query expression, there is a fluent syntax that takes field names in string format.
4. You can use some API to manage schema operations.

In a simple scenario, I could have just used some db script to manage schema definition, but in this case, as the application will be installed using many different databases, I want to set up a system that is not dependent on database dialect. So I declined the SQL option and start using eloquent APIs. The question is that I also don’t want to create a huge script that replicates the same boring code that checks if table exists, then create the field and so on. My solution was to:

1. create an interface that represents the table and that asks to implement to define basic information (table name and field list, for example)
2. I implemented one instance for each table telling what field I want in each table
3. I created a Manager class that loads all table classes, then applies schema changes basing on class definition

Example of Interface

 abstract class TableBase
{
    public abstract function getFieldDefinition();
    public abstract function getTableName();

    //field list
    public function getBaseFields()
    {
        return array(
            'id' =>'bigIncrements',
            'created_at'=>'timestamp',
            'updated_at'=>'timestamp',
        );
    }

    function getAllColumns()
    {
        return array_merge($this->getBaseFields(),$this->getFieldDefinition());
    }
}

Example of Implementation

 class UsersTable extends TableBase
{
    public function  getFieldDefinition()
    { 
        return array(
            'username' =>'string',
            'password' =>'string',
            'token'=>'string',
        );
    }
    
    public function getTableName()
    {
        return "user";
    }
}

Example: Manager Usage

 $um= new UpdateManager($dbSettings);

$um->addTable(new RepositoryTable());
$um->addTable(new PublishTokenTable());
$um->addTable(new UsersTable());
$um->addTable(new PackageTable());

$um->run();

5.4. Automate CRUD Operation

As routing and ORM layer are setted up, I created an EntityController that’s an abstract, generic class that manages basic CRUD operation on generic model. To add an entity, you just have to create a concrete class based on it and register the routing.

Example: Implement Entity

 class User extends \Illuminate\Database\Eloquent\Model {  
  protected $table = 'user';
}

Example: Implement Controller

 class UserController extends EntityController
{
    public function getTableDefinition()
    {            
        return new UsersTable();
    }

     public function getModel()
     {
         return '\WpGet\Models\User';
     }
}

Example: Register Startup

$app->any('/user/{action}[/{id}]', UserController::class);

All basic operations are already managed. In some case, you can override the method and do additional business logic like special validation, compute field or save related entities.

5.5. Automate Dependency Management

If there is a thing that struck me using PHP is the dependency management. The concern is not about composer and the package management that is awesome and resolved most of the problems. My complaint is about the fact you need to link the files. You will use into the main application using include statement or variants like include once, requires, etc. This is very annoying and may lead to issues when, like in our case, we will have many entry points (do you remember we decided to have three different applications?). In many PHP based solutions, they implemented some bootstrapper to simplify the script loading (i.e., drupal). In this application, I wanted to implement a simple bootstrap framework. The best approach I found in products is to define for each module or plugin the dependency list, then bootstrapper will collect and load all files basing on active plugins. In this case, as we do not have a plugin system but just three applications, I decided to avoid such overkill solution and found a simpler. I implemented a class “DependencyManager” that resolves dependency based on a list of folders or files. Each application provides the list of dependency so that dependency manager will scan file system to load all PHP files.

Example: Usage of Dependency Managment

$dm= new DependencyManager($appPath);

$dm->requireOnceFolders( array(
  'src/models',
  'src/controllers/base',
  'src/controllers',
  'src/db/base',
  'src/db',
  'src/handlers',
  'src/utils',
));

If application will grow in future and will need a real plugin system, we will need to make the dependency manager read lists from database or some configuration files.

6. The Frontend

The UI application is written in Angular (I mean the 2+ one, not AngularJS, if you was confused ;) ). Although I wanted to keep everything closer as much as possible to standards, there is some point that needs some attention and will be useful when you will design your next Angular application.

6.1. UI Framework: Prime NG

The first question was about the component framework to use. My personal selection list:

1. CoreUI
2. PrimeNG
3. Material

I think these are the best solutions the market offers if you don't want to pay. From my experience, I found the material very hard to be managed and I prefer to not use if the context doesn’t require, despite the fact that it is a very good framework. CoreUI is very complete and I love it as you can produce a nice application without having any graphic designer competence. Moreover, the fact that is based on bootstrap helps the designer a lot to look it better and customize. PrimeNG is not based on bootstrap and this brings some issues when I ask a bootstrap guy to put hands on it. Nothing about PrimeNG framework, but just the question is that nowadays everybody already knows bootstrap, not all PrimeNg. By the way, PrimeNG has the best grid for Angular 2+ so that's why in many projects, I used it mixed with CoreUI. By the way, in this case, I wanted to keep things simple reducing as much as possible dependencies, so I simply used PrimeNG for all applications.

6.2. Implement http Interceptor to Manage Authentication

This is a common issue: authentication client http calls to the server. Using token based authentication is easy to add a token into a request, question is to not do it each time. In Angular, I found two ways to do this:

• Extend HttpClient: In this way, you extend the class, override methods and you can manage an additional http header and redirection to login for not logged users.

• Use interceptor: Interceptor is like an hook that allow you to manipulate all request from the application in a single point. This wasn’t present at the beginning of Angular 2 but available since Angular 4.

There are some good reasons to prefer HttpClient extension pattern in most application. The biggest problem is about how many HttpInterceptor definition may interact together and how to manage different external destination. In our application, such issues are not so important as we send unauthenticated requests to the login and all other requests are authenticated. By the way, in a simple application, it was awesome to have a single point where manage all the stuff without asking services to use the special http client instead of the base one. So I followed Interceptor way, here are the basics to manage the token based authentication:

Example: Code From Interceptor

 @Injectable()
export class AuthInterceptor implements HttpInterceptor {

    forceLogout(): any {
        localStorage.clear();
        document.location.href= this.config.baseHref;
    } 

    intercept(req: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {      
       
        if( checkIfRequestNeedToken()) // checkIfRequestNeedToken code is omitted. 
                                       // In this case is the request to app backend
        {          
            //user null means no login done, user not null but 401 means token expired
            if (this.auth.currentUser()  ) {
             
                let url=req.url;
                return next.handle(req.clone({
                    setHeaders: {
                        'Authorization': 'Bearer '+this.auth.currentUser().token
                    }
                }))
                .catch((err, source) => {
                    if (err.status  == 401 || err.status  == 0) {
                        this.forceLogout();
                           return Observable.empty();
                       } else {
                           return Observable.throw(err);
                   }  
                   });
            }
            else 
            {
                //completely force logout
                this.forceLogout();
                return;
            }
        }
        else
        {
            return next.handle(req);
        }
    }
}

6.3. Dynamically Compute baseHref

BaseHref is a parameter that tells Angular the path relative from hostname and is required from routing. In standard application, it is easy to add a parameter to the build command basing on environment where you want to publish. This is quite easy to manage and doesn’t give you any real problem as you have basically many installation (prod, QA,test, integration) that are copies of the same. In this case, as application is an on-premise installation, I can’t know where application will be installed. It may be in the root of the web site or in a subfolder. So I decided to hook baseHref basing on customer environment settings. This path is taken from a dynamic configuration variable, set during the installation. So, when the user will install, backend compute and store baseHref that is used from UI during regular usage. Here is the piece of code that setup the parameter. Please note that you could place here any logic to define your required values:

Example: Base href Override (app,module.ts)

 export function baseHrefFactory (config: ConfigurationService)  {
  return window.location.href.substring
         (window.location.href.lastIndexOf(config.baseHref));
}
//...
 providers: [
 //...
{
    provide: APP_INITIALIZER,
    useFactory:configFactory,
    deps: [ConfigurationService,HttpClientModule],
    multi: true
},
{ provide: APP_BASE_HREF, 
  useFactory: baseHrefFactory,
  deps: [ConfigurationService,HttpClientModule],
 },
 //...

6.4. Decouple Configuration From Compilation

Usage of environments is very useful and it is awesome that this development framework takes into account the possibility that an application will be deployed in more than one place. The matter is about the way it does it: at compilation times. While in a simple project you can do many builds for each commit, producing different artifacts for each deployment environment or rebuild code during release, I cannot build application for all the installation users will be done from users. So I decided to change this approach and use dynamic settings.

Settings file is a json file placed into assets folder. For who will tell is not secure, I will remember also environment files are stored inside js file in plain text so it is the same security level. Information inside js are created at installation time and values are computed based on the environment. This will allow us to make only one build, produce an universal package, and tune parameters during startup.

7. The Wordpress Plugin

About wordpress plugin, it is needed to do a deep explanation about standard, core feature of the engine in order to explain it. This is a huge examination and a little bit off-topic from this article, that’s mostly focused on WpGet application. That’s why I decided to explain here only an high-level overview from the user side. The purpose of this chapter is explain the basics to create a new plugin compliant with WpGet specification. I’ll dedicate a separate article to the plugin architecture and about Wordpress specific issues.

As already mentioned, a Wordpress plugin is basically a zip file. The main idea is to put inside it a file in yaml format to explain what the plugin is. I chose Yaml format because it is a format that is easy to read and write and more suitable to contain textual data than XML or JSON format (just think about escape of special characters…) This manifest must have .wpget.yml name and during upload of package is read from application, no matter where it is. By the way, it is recommended to place inside the plugin directory. Note that the yml file is placed inside plugin and can be used for the plugin itself to determine the current version and to display settings. What I described until now is how a Wordpress developer can describe its plugin and load information into wpget system. Other question is how to make a plugin interact with wpget server to get updatest. As always, there are many way to do this. During analysis, I covered many of them:

1. Manually add and register a class. We can provide to the user a PHP class, already working, that can be placed into plugin itself, registered and configured to make plugin updatable. This is the easier solution on vendor side (just prepare the class and a sample module), by the way is the less scalable as each user’s plugin must copy inside the class replicating the code and each plugin need some manual interaction.

2. Create an umbrella plugin: This solution will pass the limitation of previous implementation by creating and releasing a plugin into public wordpress repository. This plugin will introduce in Wordpress the basic classes needed for integration, so no more copy and past of the class. Moreover, this will allow to manage basic settings for modules (URL of wpget, token for authentication) avoiding hard coded configurations.

3. Create an alternative plugin manager: This will allow you to connect with many wpget repository, download a full list of available packages, then install\keep update them.

I decided to start from the simple solution for the vendor as the effort requested to the user is minimum. After this validation phase, if many users will report installation, I’ll consider to implement the solution (2) to give them an easier management on the Wordpress side. About solution (3), I think it haven't any real advantages than (2) but it is in fact a much more invasive implementation.

Coming back to the actual solution, we need two steps. First, you need to add the .wpget.yml file on the root, with this content

Example: Yaml Sample

 version: 3.7.0
name: my-plugin
homepage: https://github.com/zeppaman/WpGet
upgrade_notice: >
 [Improvement] New changes made in version 4.0 were causing problem 
               at websites running on PHP version less than 5.0
author: Francesco Minà
author_profile: https://github.com/zeppaman/WpGet
requires: 4.9.4 

Other parameters are omitted, you must check for complete yaml sample attached to this article

Then you have to insert the PHP class (you can download here) and integrate into plugin file as follows:

Example: Integration Sample

// remember to set variables (es in wp-config.php file) before use this file

// // repository config data
// define( 'WPGET_REPO_SLUG','test000' );
// define( 'WPGET_PACKAGE_NAME','my-plugin' );
// define( 'WPGET_API_URL','http://localhost:3000/web/' );
// define( 'WPGET_TOKEN_READ','FnrvNuzwKodEgIqxsBctbFc2SxMncM');

// // plugin info
// // plugin filename
// define( 'WPGET_PLUGIN_FILE', 'plugin-test.php' );
// // plugin directory
// define( 'WPGET_PLUGIN_DIR', 'plugin-test' );

require_once( 'WpGetUpdater.php' );

8. Point of Interest

In the previous chapter, I explored the most interesting point of principal modules of the application. Moreover, there are some important topics to discuss not directly related with frontend or backend that I will cover here. I’m referring to the DevOps part and to the packaging\installation process.

8.1. Installation Process

As this application will be installed from user, I would create an install procedure that will do all the required operations for you. This script will:

1. check if application is already installed. If yes, simply do nothing
2. ensure all folders are present with right permission
3. create data schema in database
4. produce settings file for frontend
5. fail if one of previous steps fails

Even if it would be beautiful, this is an installation script, not an installation wizard. This means that user must follow these steps:

• get a server
• extract zip file somewhere on server disk
• make the /web folder reachable to the web
• insert database settings into settings file
• run install script

To avoid running a non configured application, I add a rewrite rule into .htacces (for Apache installation) that redirects all traffic to the install page if application is not installed. This prevents that user to stop reading the previous dotted list and stop at point (2) get confused about what to do. Installation script check for write permission and helps a lot the user to reach the right server configuration. There isn’t really much settings to do except insert database connection settings, have some directory writable (storage, logs, assets) but in my experience, I found better an application that doesn’t start until all is fine than one that doesn't work.

8.2. Build and Packaging

This is an opensource project, so code is hosted into Github (for opensource code, there are some alternatives nowadays?). This gives me some interesting opportunities for automating the process, finally, I choose circleci that’s free for public project and very easy to use. My build process simply downloads the code, deleted unused folders, builds Angular application. This is very easy but circleci runs all the process in a docker image. He provides a node image and a PHP one, but no one with both of them together. This limit can be passed by creating a new docker image with node and PHP installed. I chose the lazy solution: I use the PHP Docker image provided by circleci, but I installed node as first step of compilation. This will help me a lot in packaging phase as I have all files inside the same folder, so I just need to produce a zip and upload to circleci as a zip file. This is the configuration for circleci with the installation script.

Example: circle ci config

# PHP CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-php/ for more details
#
version: 2
jobs:
  build:
    docker:
      - image: circleci/php:7.1.5-browsers
      
    working_directory: ~/repo

    steps:
      - checkout
     
      # Download and cache dependencies
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "composer.json" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-
      - run: chmod -R 777 *
      - run: sh .circleci/build.sh
            
      - store_artifacts:
          path: /tmp/artifacts
      
      - save_cache:
          paths:
            - ./vendor
          key: v1-dependencies-{{ checksum "composer.json" }}
        
      # run tests!
     # - run: phpunitconfig.yml

Example: circle ci startup script

#install node
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php
php -r "unlink('composer-setup.php');"
php composer.phar self-update
sudo mv composer.phar /usr/local/bin/composer
composer install -n --prefer-dist
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs

#angular build
cd src-ui;
npm install;
npm run ng build --env prod;
cd ..;

#delete unused folders
rm -rf src-ui;
rm -rf .circleci;
rm -rf .git;
rm -rf .vscode; 
rm -rf web/ui/assets/settings.json

#build artifact
mkdir /tmp/artifacts;
zip -r /tmp/artifacts/web.zip .

9. Conclusions

Developing this application give me the opportunity to test all the components of a modern web application with the stack based on:

• PHP
• Slim
• Angular

Apply such kind of application in the product on premise world bring out some issues that usually we do not meet in the development process for regular business applications. This is interesting because the experience made on this project can be used as basics for new projects or simply used if application requirements need it.

10 References

• Wordpress statistic usage
• GitHub Project
• GitHub Plugin Project

11 History

• 10^th February, 2018: First release of wpget
• 14^th February, 2018: First release of this article
• 15^th February, 2018: Update binary, updated link