Oil Paint Effect: Implementation of Oil Painting Effect on an Image

• Download source - 529 KB 
• Download demo - 665 KB 

Contents 

• Introduction
• Oil Painting Effect Details
• Parameters of Oil Painting Effect
• Radius 
• Intensity
• Details of PaintEffect Application
• Drawing Image to Screen
• Why OpenGL Display ?
• Load Image
• Save Image
• Resizing the Dialog
• Issues and Limitations
• Points of Interest
• References
• Revision History

Introduction

This article demonstrates an application to create an oil painting effect of an image. Basically this effect is achieved by examining the nearest pixels for all pixels. For every pixel, it finds the maximum repeated color and that color will be considered as the output. In effect, we will get a blocky image with less information, and it will be similar to the painting effect of the image. 

Details of the oil painting algorithm are explained below.

Oil Painting Effect Details

Analysis of nearest pixels is explained with a real example. A pixel from the above image is considered to analyze the oil painting algorithm. This example considers Radius as 2 and Intensity as 10.

For every pixel, the surrounding pixels are analysed. Find the maximum repeated pixels within the area, and place it as output. Processing of a pixel [X,Y] analyzes pixels from [X-Radius,Y-Radius] to [X+Radius,Y+Radius].

/*
/// Function to process Oil Painting Effect.
// Parameters:
pbyDataIn_i: Input RGB buffer of input image.
nRadius:     Radius of processing. This values is used to consider nearest pixels.
             If this value is high, processing cost will increase. 2 ~5 are good values.
fIntensityLevels: Applied to r,g,b values intensity.
                  Increasing this values will create blocky output image.
nWidth: Width of image.
nHeight: Height of image.
pbyDataOut_o : Output RGB buffer.
*/
void PaintEffect::Process( const BYTE* pbyDataIn_i,
                           const int nRadius_i,
                           const float fIntensityLevels_i,
                           const int nWidth_i,
                           const int nHeight_i,
                           BYTE* pbyDataOut_o )
{
    // nRadius pixels are avoided from left, right top, and bottom edges.
    for( int nY = nRadius_i; nY < nHeight_i - nRadius_i; nY++)
    {
        for( int nX = nRadius_i; nX < nWidth_i - nRadius_i; nX++)
        {
            // Find intensities of nearest nRadius pixels in four direction.
            for( int nY_O = -nRadius_i; nY_O <= nRadius_i; nY_O++ )
            {
                for( int nX_O = -nRadius_i; nX_O <= nRadius_i; nX_O++ )
                {
                    int nR = pbyDataIn_i[( nX+nX_O) * 3  + ( nY + nY_O ) * nBytesInARow ];
                    int nG = pbyDataIn_i[( nX+nX_O) * 3  + ( nY + nY_O ) * nBytesInARow + 1];
                    int nB = pbyDataIn_i[( nX+nX_O) * 3  + ( nY + nY_O ) * nBytesInARow + 2];

                    // Find intensity of RGB value and apply intensity level.
                    int nCurIntensity =  ( ( ( nR + nG + nB ) / 3.0 ) * fIntensityLevels_i ) / 255;
                    if( nCurIntensity > 255 )
                        nCurIntensity = 255;
                    int i = nCurIntensity;
                    nIntensityCount[i]++;

                    nSumR[i] = nSumR[i] + nR;
                    nSumG[i] = nSumG[i] + nG;
                    nSumB[i] = nSumB[i] + nB;
                }
            }

            int nCurMax = 0;
            int nMaxIndex = 0;
            for( int nI = 0; nI < 256; nI++ )
            {
                if( nIntensityCount[nI] > nCurMax )
                {
                    nCurMax = nIntensityCount[nI];
                    nMaxIndex = nI;
                }
            }

            pbyDataOut_o[( nX) * 3 + ( nY ) * nBytesInARow ] = nSumR[nMaxIndex] / nCurMax;
            pbyDataOut_o[( nX) * 3 + ( nY ) * nBytesInARow + 1] = nSumG[nMaxIndex] / nCurMax;
            pbyDataOut_o[( nX) * 3 + ( nY ) * nBytesInARow + 2] = nSumB[nMaxIndex] / nCurMax;
        }
    }
}

Here analyzing the nearest pixels of a pixel (X,Y), for this example, X,Y is at the center of the blue rectangle. The nearest pixels of the pixel X,Y is displayed in the right top corner. We will analyse the intensity of these nearest pixels and find out the final R, G, B for pixel (X,Y) which will help create an oil painting effect.

The RGB values of the nearest pixels are shown in the below image.

Intensity of the above pixels are calculated with the following logic. The average of R ,G, and B is multiplied with Intensity and the final intensity value is prepared.

// Find intensity of RGB value and apply intensity level.
int nCurIntensity =  ( ( ( nR + nG + nB ) / 3.0 ) * fIntensityLevels_i ) / 255; 

The calculated intensity of the nearest pixels are shown in the below image.

The intensity values of the nearest pixels range from 6 to 10. The final pixel value depends on the occurrence of the intensity values. The most repeating intensity value is considered for the output. From the above image, the most repeating intensity is 8. Seven nearest pixels have the intensity value 8. On calculating the intensity of each pixel, the sum of R, G, B corresponding to each intensity is also calculated.

// Sum of each pixels intensity is calculated.
nSumR[nCurIntensity] = nSumR[nCurIntensity] + nR;
nSumG[nCurIntensity] = nSumG[nCurIntensity] + nG;
nSumB[nCurIntensity] = nSumB[nCurIntensity] + nB;

The sum of R, G, B component for each intensity value is as shown below.

From the above table, the intensity value 8 is the most repeating intensity. Seven nearest pixels have the intensity value 8, and therefore the output pixel is prepared from the sum of R, G, B components corresponding to intensity 8.

Final R = 949 / 7 = 135
Final G = 902 / 7 = 128
Final B = 458 / 7 = 65

And at last, RGB (131,151,82) at X,Y is changed to RGB (135,128,65) after analyzing RGB values of the nearest pixels.

The output pixel is changed according to the intensity of the nearest pixels. The output image is created by applying the same algorithm for all pixels. The maximum repeated pixels are considered for output, and it removes small (smooth) changes in the image. In effect we will get a rough/blocky image. It will be similar to that of an oil painting effect.

Parameters of Painting Effect

Radius

This parameter decides the nearest pixels to analyze. For every pixel (X,Y) the nearest pixels from (X-Radius, Y-Radius) to ( X+Radius, Y+Radius) are analyzed to prepare the output image.

If the Radius is high, then the processing cost will be high. The iterations inside PaintEffectImpl is as follows: Width * Height * (2 * Radius + 1 ) * (2 * Radius + 1 ).

If Radius is increased, the computation cost of the algorithm will also increase and we will get a much blocky image. Suitable values of Radius are from 3 to 7, which will produce a good oil painting effect.

Intensity

This parameter is used to find the intensity of a pixel. The final intensity of a pixel is calculated by the following logic:

// Calculating intensity with R,G,B values and Intensity parameter.
int nActualIntensity =  ( ( ( nR + nG + nB ) / 3.0 ) * IntensityParameter ) / 255;  

This parameter helps to change the intensity, in-effect the output image becomes blocky.

Details of PaintEffect Application

On loading a new image, the effect will be applied with the help of the PaintEffectImpl class. PaintEffectImpl::Process is used to prepare the oil paint effect applied image. The prototype of PaintEffectImpl::Process is as follows.

/*
Function to prepare Oil Painting Effect.
// Parameters:
pbyDataIn_i: Input RGB buffer of input image.
nRadius:     Radius of processing. This values is used to consider nearest pixels.
             If this value is high, processing cost will increase. 2 ~5 are good values.
fIntensityLevels: Applied to r,g,b values intensity.
                  Increasing this values will create blocky output image.
nWidth: Width of image.
nHeight: Height of image.
pbyDataOut_o : Output RGB buffer.
 */
void PaintEffect::Process( const BYTE* pbyDataIn_i,
                           const int nRadius_i,
                           const float fIntensityLevels_i,
                           const int nWidth_i,
                           const int nHeight_i,
                           BYTE* pbyDataOut_o );

The output image (painting effect applied image) will be available in m_pbyEffectAppliedData. This image is copied to a bitmap object. This bitmap will be blitted into the static window on each repainting of the dialog.

The following code is used to copy the effect applied image to the CBitmap object.

// Code to copy effect applied data to Bitmap object.
HBITMAP hbmDest = ::CreateCompatibleBitmap(hdc, m_nImageWidth, m_nImageHeight);
 
if (hbmDest)
{
    if (SetDIBits(hdc, hbmDest, 0, m_nImageHeight, m_pbyEffectAppliedData, &stBitmapInfo, DIB_RGB_COLORS))
    {
        m_BitmapObject.DeleteObject();
        m_BitmapObject.Attach( hbmDest );
    }
}

Drawing Image to Screen

Output bitmap is displayed to screen with GDI. StretchBlit is used to display the re-sized bitmap to the output window.

void CPaintEffectDlg::DrawWithGDI()
{
    // Use GDI method to draw the bitmap to screen.
    CDC* pDC = GetDlgItem( IDC_STATIC_IMAGE )->GetDC();
 
    if (m_BitmapObject.GetSafeHandle())
    {
        CDC dcMem;
 
        if (dcMem.CreateCompatibleDC(pDC))
        {
            CBitmap* pOldBM = dcMem.SelectObject(&m_BitmapObject);
            BITMAP BM;
 
            m_BitmapObject.GetBitmap(&BM);
 
            RECT ClientRegion;
            GetDlgItem( IDC_STATIC_IMAGE )->GetClientRect( &ClientRegion );
 
            // StretchBlt is used to resize the actual image to region of static window.
            pDC->StretchBlt( 0, 0, ClientRegion.right, ClientRegion.bottom,
                             &dcMem, 0, 0, m_nImageWidth, m_nImageHeight, SRCCOPY );
 
            dcMem.SelectObject(pOldBM);
            
        }
        GetDlgItem( IDC_STATIC_IMAGE )->ReleaseDC( pDC );
    }
} 

Why OpenGL Display? 

Initially this application created without OpenGL support. GDI was used to display the output image into static window. But loading image in different size creates image with improper aspect ratio, and it create bad quality image. The static window[which displays the output image] in the application can display an image of dimension [400,300]. When user loads a new image with different size, a new Bitmap object is created in the actual size of the image, and algorithm is applied on the actual size image. But we have to display the new image in to a static window of dimension 400X300. Therefore CDC::StretchBlit API is used to display the Bitmap of actual size to 400X300. 

// StretchBlt is used to resize the actual image to region of static window.
pDC->StretchBlt( 0, 0, ClientRegion.right, ClientRegion.bottom,
                             &dcMem, 0, 0, m_nImageWidth, m_nImageHeight, SRCCOPY ); 

Sometimes resizing of the image creates bad quality image as seen “OpenGL Display OFF” screen shot in the below Figure. If the resizing is performed with OpenGL texture mapping, the output quality will be good. OpenGL texture is created with GL_LINEAR interpolation type for MIN(minimizing) and MAG(Maximizing) interpolation type. Therefore resizing the texture uses Bi-Linear interpolation to create a good quality image. 

The following image shows the difference between StretchBlit and OpenGL display. 

Load Image

A new image can be loaded with this button. The GDI+ library is used to get the image buffer of all types of image files. After loading a new image, Oil Paint Effect is applied to the image and redrawn to the screen.

void CPaintEffectDlg::OnButtonLoadBitmap() 
{
    // Create a file open Dialog for opening .bmp file.
    CFileDialog* pFileOpenDlg = new CFileDialog( TRUE,L"image", NULL, OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT,
                                                 L"Input Image (*.bmp;*.jpg;*.png;*.tga)|*.bmp;*.jpg;*.png;*.tga||");
    if( IDOK == pFileOpenDlg->DoModal())
    {
        CString csFileName = pFileOpenDlg->GetPathName();
        int nWidth = 0;
        int nHeight = 0;
        BYTE* pbyData = 0;
        BMPLoader BMPLoaderObj;
        if( !BMPLoaderObj.LoadBMP( csFileName.GetBuffer( 0 ), nWidth, nHeight, pbyData ))
        {
            AfxMessageBox( L"BMP Loading failed" );
            return;
        }

        ProcessEffect();

        Invalidate( false );
    }
}

Save Image

Effect applied image can be saved to a file. The size of the input image is used as the size of output image. The PaintEffect dialog will display the resized image and it is not good to save the resized image. The file name is created with the Radius and Intensity parameters.

// To save the processed image.
void CPaintEffectDlg::OnBnClickedButtonSave()
{
    // Construct file name. Radius and Intensity parameters are added in the deafult file name.
    CString csFileName;
    csFileName.Format( L"OilPaintEffect_radius%d_Intensity%d.bmp",m_nRadius, m_nIntensity );
    CFileDialog SaveDlg( false, L"*.bmp", csFileName );
    if( IDOK == SaveDlg.DoModal())
    {
        CString csFileName = SaveDlg.GetPathName();
        BMPLoader SaveBmp;
        SaveBmp.SaveBMP( csFileName, m_nImageWidth, m_nImageHeight, m_pbyEffectAppliedData );
    }
}

Resizing the Dialog

The output image is drawn to screen by resizing the output image to fit into the static window in the dialog. Therefore loading a new image with a different size may stretch or skew the image. If the dialog can resize, the user can view the image in the desired size. The WM_SIZE message is handled and the static window which displays the image is resized according to the new size of the dialog. All controls except the static window which displays images are moved in the X direction based on the size of new window.

// All windows other than image display window is moved in X direction.
MoveWindowInXDirection( IDC_EDIT_RADIUS, nParamWindowMoveX );
MoveWindowInXDirection( IDC_STATIC_RADIUS, nParamWindowMoveX );
MoveWindowInXDirection( IDC_EDIT_INTENSITY, nParamWindowMoveX );
MoveWindowInXDirection( IDC_STATIC_INTENSITY, nParamWindowMoveX );
MoveWindowInXDirection( IDC_STATIC_PARAM, nParamWindowMoveX );
MoveWindowInXDirection( IDC_BUTTON_LOAD_BITMAP, nParamWindowMoveX );
MoveWindowInXDirection( IDC_BUTTON_SAVE, nParamWindowMoveX );
MoveWindowInXDirection( IDC_BUTTON_ABOUT, nParamWindowMoveX );
MoveWindowInXDirection( IDC_CHECK_PAINTING_EFFECT, nParamWindowMoveX );
MoveWindowInXDirection( IDC_STATIC_STATUS, nParamWindowMoveX );
MoveWindowInXDirection( IDC_CHECK_OPENGL_DISPLAY, nParamWindowMoveX );

Issues and Limitations

• Time for processing a big image is too high. Currently the painting effect is applied in a GUI thread, therefore processing a big sized image may take time for displaying the image.
• If the Radius is a high value, then the processing time is too high. No parallel processing method was tried. When I tried to prepare a shader, cg does not support array access without a loop variable. Trying to prepare a GPU implementation of this effect. 

Points of Interest 

• Preparing a real example was a little complex. I prepared another application to draw the nearest pixels in large size and prepared the images used in the Details of Algorithm section.
• I can easily prepare the output of the algorithm. But copying the output properly to screen took a long time. Preparing a BITMAP object and copying this image to a memory DC. In the WM_PAINT message, this memory DC is copied to the output window.
• Resizing of the dialog is handled in a special way. The MoveWindowInXDirection function is created to move a control in the X direction. On each resize, the required movement in X direction is identified and MoveWindowInXDirection is called for all controls. MoveWindowInXDirection gets the previous position of the control and calculates the new position and calls SetWindowPos to apply the new position.
• On changing a parameter related to Oil Paint Effect or loading a new image, the dialog stops responding, and the user won't get any notification. Therefore a static button is added to indicate the algorithm processing is going on. The "Processing Effect..." message indicates the processing of the algorithm is going on, and we need to wait a few seconds to display the new image.
• In first version, there was some painting issues when switching from Desktop and PaintEffect application. After long time of analysis, I observed that the WM_PAINT is called, and the entire scene was drawn. But sometimes it was not proper. I used the DC of static window by calling GetDlgItem() for drawing, and it caused the painting issue. When I prepared a CPaintDC from static window, the paining issue was solved.  

References 

• http://angeljohnsy.blogspot.jp/2011/06/oil-painting-in-matlab.html
• http://supercomputingblog.com/graphics/oil-painting-algorithm/

Revision History 

• 08-October-2012: Initial version. 
• 13-October-2012: Solved painting issue, Added slider control to change Radius and Intensity.