Html5 Image Markup

Introduction

Countless times, I needed to download an image from somewhere on the internet, open it up using some image editing application, add some markups, save the image and then use it somewhere else. For some types of websites it would be great, I think, if the website itself allowed users to annotate the image online (with both text and drawings) and then save the combining image locally, or send it to somewhere on the internet. This article introduces a jQuery plugin that implements such feature - at least in a simplified way.

Background

The idea behind the Image Markup plugin came recently in a talk with our CEO Marco Raduan at ILang Educação, when he showed us a nice Mac OS feature which allowed image annotations directly on top of the image itself through built-in OS capabilities, without the need of a separate application. This kind of direct annotation was considered a good asset for our Learning Management System (ILang), in a way that teachers could review image-scanned open-ended answers handwritten by students and annotate them while evaluating them. Such visual annotations could be persisted to database along with evaluation and grade assigned to the student assessment, in order to represent the whole process of evaluation of open-ended questions.

System Dependencies

The application will reside 100% on the client browser. It is written in JavaScript and depends on four main frameworks/libraries:

• jQuery: Unsurprisingly, since Image Markup is a jQuery plugin, it could never work without the jquery framework preloaded. The simplicity here is really the key to success.

• Paper JS: The toolbox provided by Paper JS as a vector graphics library never ceases to amaze me. PaperJS does all the heavyweight work of drawing/text on top of HTML5 canvas, which otherwise would have me spend countless hours of hard working.

• ContextMenu: The implementation of a context menu (that one which is accessed through the right mouse button) is a nice and clean way to provide a display of menu options, without the need to pollute the screen with tools and visual shortcuts. In my opinion, the context menu jQuery plugin implemented by Rodney Rehm, Christian Baartse and Addy Osmani excels by its simplicity and power customization capabilities.

• CommandManager: Drawing and text writing sometimes can lead to mistakes, so it is important to have undo and redo commands every time the user makes a modification. Alexander Brevig developed a simple and effective implementation Command pattern in JavaScript.

Using the code

As any good jQuery plugin, Image Markup only requires one line of JavaScript initialization code, and that is just for setting up the IMAGE element on top of which the CANVAS annotation layer will be built. For example:

var markup = $('#myImage').imageMarkup();

The above line will create an instance of Image Markup attached to the IMAGE having the id equals to myImage.

Users may find helpful to attach an Image Markup instance to multiple IMAGE elements. In the following case, a single instance of Image Markup will be attached to every IMAGE element inside an element with the "img-container" class:

var markup = $('.img-container img').imageMarkup();

Overriding Default Options

Some properties can be overriden in order to customize the way the plugin works. They are:

• color: the color of drawings and text elements. This can be changed once you start working.
• width: you can predefine how thick your drawings are going to be. Obviously this only applies to drawings, not to text elements.
• opacity: because drawings and text over image can be so confusing, sometimes it is useful to add a little transparency to the annotations. For this reason, you can choose an opacity ranging from 0.0 to 1.0.

The line below specifies a Image Markup instance with red pen, thickness of 4 points and opacity of 50%:

var markup = $('.img-container img').imageMarkup({color: 'red', width: 10, opacity: .5});

It's All About Layers

Annotations should never modify the underlying image. Not because we want to preserve the original image, but rather because we want to keep annotations as an entity and later save it to the browser's local storage, download or transmit it via web services, for example.

The separate annotation layer also has other advantages: since it encapsulate an element set in a JavaScript object structure, you can easily serialize it as JSON format, and persist annotations to, let's say, a string column in a database table. Conversely, you could later retrieve that same JSON string and restore the annotations.

Figure 2. Image layer (A) under a transparent canvas with annotations (B) produces a new composite image (C).

Once initialized, the Image Markup plugin creates a new HTML5 canvas element that covers the entire image. This canvas is appended as a sibling of the image element, therefore they share the same ancestor.

Drawing Over Image

The plugin features freehand drawing: dragging the mouse over the canvas leaves a trace of straight segments, that becomes a smoothened scribble once the user releases the mouse button. Users can draw an unlimited number of points and independent lines.

And it begins when the user clicks the mouse button. At this moment, a new instance of Path object is created and the default settings are applied to the path (color, width and opacity):

tool.onMouseDown = function (event) {

	switch (event.event.button) {
		// leftclick
		case 0:
			// If we produced a path before, deselect it:
			if (path) {
				path.selected = false;
			}

			path = new paper.Path();
			path.data.id = generateUUID();
			path.strokeColor = settings.color;
			path.strokeWidth = settings.width;
			path.opacity = settings.opacity;
			break;
			// rightclick
		case 2:
			break;
	}
}	

Below, the dragging effect is detected and handled by an implementation of the onMouseDrag event of Paper JS library's tool object: New points are added to the current path object while the user drags the mouse.

tool.onMouseDrag = function (event) {
	switch (event.event.button) {
		// leftclick
		case 0:
			// Every drag event, add a point to the path at the current
			// position of the mouse:

			if (selectedItem) {
				.
				[DRAG (MOVE) THE SELECTED ITEM]
				.
			}
			else if (path)
				path.add(event.point);
			break;
			// rightclick
		case 2:
			break;
	}
}

Now the onMouseUp event is subscribed, and when the user releases the mouse button the path is simplified (that is, lines between segment points are smoothened).

tool.onMouseUp = function (event) {
	switch (event.event.button) {
		// leftclick
		case 0:
			if (selectedItem) {
				.
				[STOP DRAGGING THE SELECTED ITEM]
				.
			}
			else {
				// When the mouse is released, simplify it:
				path.simplify();
				.
				[SAVE THE PATH COMMAND IN COMMAND MANAGER]
				.
			}
			break;
			// rightclick
			.
			.
			.
	}
}	

Figure 3. Each "path" or "scribble" is made up by only a few points, but instead of appearing as straight lines, the segments between them are smoothened to resemble handwriting.

Users can change pen colors, but choosing from a limited set: Black, Red, Green and Yellow.

this.setPenColor = function (color) {
	self.setOptions({ color: color });
	$('.image-markup-canvas').css('cursor', "url(img/" + color + "-pen.png) 14 50, auto");
}

As seen earlier, the transparency and thickness properties are defined in JavaScript initializing code. Users cannot change them.

Text Over Image

This tool works in a "create first, modify later" manner. Users can add text annotations by clicking the Text tool in Context Menu, and a new text element with default message will be dropped on the canvas. The text can be edited by double-clicking the element and typing in the new text in the browser's input dialog. The text color will be the same as the drawing pen.

Notice how the text tool is set up in ContextMenu through the setText function:

$.contextMenu({
	selector: '.image-markup-canvas',
	callback: function (key, options) {
		switch (key) {
		.
		.
		.
			case 'text':
				self.setText();
				break;
		.
		.
		.
		}
	},
	items: {
	.
	.
	.
		"text": { name: "Text", icon: "text" },
	.
	.
	.
	}
});

The default settings are applied to the new instance of the PointText object, and the onDoubleClick function prepares the web browser's default input dialog to ask users for the new text:

this.setText = function () {
	var uid = generateUUID();
	var pos = contextPoint;
	CommandManager.execute({
		execute: function () {
			var TXT_DBL_CLICK = "<<double click to edit>>";
			var txt = TXT_DBL_CLICK;
			var text = new paper.PointText(pos);
			text.content = txt;
			text.fillColor = settings.color;
			text.fontSize = 18;
			text.fontFamily = 'Verdana';
			text.data.uid = uid;
			text.opacity = settings.opacity;

			text.onDoubleClick = function (event) {
				if (this.className == 'PointText') {
					var txt = prompt("Type in your text", this.content.replace(TXT_DBL_CLICK, ''));
					if (txt.length > 0)
						this.content = txt;
				}
			}
		},
		unexecute: function () {
			$(paper.project.activeLayer.children).each(function (index, item) {
				if (item.data && item.data.uid) {
					if (item.data.uid == uid) {
						item.remove();
					}
				}
			});
		}
	});
}

Figure 4. By selecting the text menu item and then double-clicking the text, users can place text annotations over the image.

Selecting Items

You select an item by moving the mouse over it: a series of segment handlers will then indicate which item has been selected.

Figure 5. Path segments showing the selected element.

As the user moves the mouse (without dragging) over an element (path or text), the onMouseMove function of Paper JS library's tool object turns the element into selected state, while deselecting all other elements on canvas:

tool.onMouseMove = function (event) {
	if (!$('.context-menu-list').is(':visible')) {
		position = event.point;
		paper.project.activeLayer.selected = false;
		self.setPenColor(settings.color);
		if (event.item) {
			event.item.selected = true;
			selectedItem = event.item;
			self.setCursorHandOpen();
		}
		else {
			selectedItem = null;
		}
	}
}

Currently, the Image Markup plugin does not support multiple selection. However, I intend to implement this feature in future versions.

Deleting Items

Annotation elements easily be deleted by menu. The Erase menu will delete the selected item, if there is one. Otherwise, it will delete all canvas elements.

The selected element is searched for in the collection of canvas elements, and then removed when found. Notice how the operation is done through the CommandManager object, so that it can be undone later at user's request.

this.erase = function () {
	var strPathArray = new Array();
	$(paper.project.activeLayer.children).each(function (index, item) {
		if (contextSelectedItemId) {
			if (contextSelectedItemId.length == 0 || item.data.id == contextSelectedItemId) {
				var strPath = item.exportJSON({ asString: true });
				strPathArray.push(strPath);
			}
		}
	});

	CommandManager.execute({
		execute: function () {
			$(paper.project.activeLayer.children).each(function (index, item) {
				if (contextSelectedItemId) {
					if (contextSelectedItemId.length == 0 || item.data.id == contextSelectedItemId) {
						item.remove();
					}
				}
			});
		},
		unexecute: function () {
			$(strPathArray).each(function (index, strPath) {
				path = new paper.Path();
				path.importJSON(strPath);
			});
		}
	});
}

Figure 6. Erasing the selected annotation item.

Moving Items

You can move a single element (drawing or text) by drag-and-drop over the whole canvas. However, the current version does not allow multiple elements to be moved at once.

As the user releases the mouse button, the selected element is placed in its final location, the CommandManager is told to enlist the operation, so that it can be undone later.

tool.onMouseUp = function (event) {
	switch (event.event.button) {
		// leftclick
		case 0:
			if (selectedItem) {
				if (mouseDownPoint) {
					var selectedItemId = selectedItem.id;
					var draggingStartPoint = { x: mouseDownPoint.x, y: mouseDownPoint.y };
					CommandManager.execute({
						execute: function () {
							//item was already moved, so do nothing
						},
						unexecute: function () {
							$(paper.project.activeLayer.children).each(function (index, item) {
								if (item.id == selectedItemId) {
									if (item.segments) {
										var middlePoint = new paper.Point(
												((item.segments[item.segments.length - 1].point.x) - item.segments[0].point.x) / 2,
												((item.segments[item.segments.length - 1].point.y) - item.segments[0].point.y) / 2
											);
										item.position =
											new paper.Point(draggingStartPoint.x, draggingStartPoint.y);
									}
									else {
										item.position = draggingStartPoint;
									}
									return false;
								}
							});
						}
					});
					mouseDownPoint = null;
					.
					.
					.

Downloading Merged Images

You can download composed images (that is, the source image plus the drawings and text annotations) as a single image, by clicking the Download menu item in the context menu. The image will go right to your Downloads folder (or whatever folder you have assigned as your browser's default).

$.contextMenu({
	selector: '.image-markup-canvas',
	callback: function (key, options) {
		switch (key) {
			//COMMANDS
			.
			.
			.
			case 'download':
				self.download();
				break;
			.
			.
			.						
		}
	},
	items: {
	.
	.
	.
	"download": { name: "Download", icon: "download" },
	.
	.
	.
	}
});

In the code below, the mergedContext is created as a new instance of Canvas html element (though not appended to the HTML page, and thus not visible). Then, the drawImage method is called twice: one time to draw the underlying image and once more to draw the canvas containing the image annotations created before with the help of the Paper JS library.

this.download = function () {
	var canvas = paper.project.activeLayer.view.element;
	var img = $(canvas).parent().find('img')[0];
	var mergeCanvas = $('<canvas>')
	.attr({
		width: $(img).width(),
		height: $(img).height()
	});

	var mergedContext = mergeCanvas[0].getContext('2d');
	mergedContext.clearRect(0, 0, $(img).width(), $(img).height());
	mergedContext.drawImage(img, 0, 0);
	mergedContext.drawImage(canvas, 0, 0);
	self.downloadCanvas(mergeCanvas[0], "image-markup.png");
}

The actual download code is implemented in an ingenious solution provided by Ken Fyrstenberg, that emulates the click event on an anchor (<a>) element:

this.downloadCanvas = function (canvas, filename) {
	/// create an "off-screen" anchor tag
	var lnk = document.createElement('a'),
		e;

	/// the key here is to set the download attribute of the a tag
	lnk.download = filename;

	/// convert canvas content to data-uri for link. When download
	/// attribute is set the content pointed to by link will be
	/// pushed as "download" in HTML5 capable browsers
	lnk.href = canvas.toDataURL();

	/// create a "fake" click-event to trigger the download
	if (document.createEvent) {

		e = document.createEvent("MouseEvents");
		e.initMouseEvent("click", true, true, window,
						 0, 0, 0, 0, 0, false, false, false,
						 false, 0, null);

		lnk.dispatchEvent(e);

	} else if (lnk.fireEvent) {

		lnk.fireEvent("onclick");
	}
}

The next version of the plugin will allow downloading the annotation image (a .png image with drawings plus text over a transparent background), so that programmers can place it over the original image and obtain the composed image.

Conclusion

As you can see, there is a lot of room for enhancement in the application. The Paper.js framework proved itself up to the task of handling complex graphic scripting and event handling, while providing a clean and simplified set of classes and events.

Also, the application at this point is maybe too much for general purpose, but I think the biggest potential lies on how you adapt it to the specific needs of your project. For example, it can be a strictly educational website, where teachers assess work done online by students. Or it can be an entertainment game where children spot the differences between pairs of images. Or even a collaborative application (something like Google Hangouts) for remote users in online meetings.

If you have any comments, complaints or suggestions, please leave a comment below. I'd like to hear from you and I'm willing to improve the app as new ideas arrive.

History

2014-07-26: First version.