Click here to Skip to main content
15,867,704 members
Articles / Programming Languages / C++11

Bring Your Animations to H264/HEVC Video

Rate me:
Please Sign up or sign in to vote.
4.99/5 (46 votes)
25 Apr 2023MIT9 min read 61.2K   2.8K   77   28
Bring your animations to H264/HEVC video using C++ and C# with h/w acceleration
In this article, we go through four examples. In the first example, we render a red video, in the second example, we load a JPEG image with GDI+ and render once, that is when frame_cnt is zero, in the third example, we display first image and slowly alphablend with the second image until it appears, and in our last example, we show the two pre-rendered text images appearing from the middle of the video.

Table of Contents

Introduction

I just completed my (closed-source) photo slideshow application, Mandy Frenzy, based on the video encoder featured in this article. Below is the video encoded from it. You can download my slideshow application from here or visit its home page.

Last year, I introduced a single header windows-based video encoder for OpenGL that works on Windows 7 and above. See the above video demo! I have decoupled it from the OpenGL thread and make it simpler to encode a 2D frame. All you need is to fill in the frame buffer, frame by frame to create your animations. In this article, I use GDI+ since I am most familiar with it but you are welcome to use your favourite graphics library; The video encoder is not coupled with GDI+. HEVC codec used to come bundled with Windows 10 but now Microsoft has removed it and put it on sale in the Microsoft Store. That HEVC codec has a quality issue where higher bitrate has to be given to maintain the same quality as H264 encoded video. Make sure the video file is not opened or locked by video player before you begin to write to it. The new H264Writer constructor is as follows:

C#
H264Writer(string mp3_file, string dest_file, VideoCodec codec, 
    int width, int height, int fps, int duration /*in milliseconds*/, 
    FrameRenderer* frameRenderer, uint bitrate);
C++
H264Writer(const wchar_t* mp3_file, const wchar_t* dest_file, VideoCodec codec, 
    int width, int height, int fps, int duration /*in milliseconds*/, 
    FrameRenderer* frameRenderer, UINT32 bitrate = 4000000);

The mp3_file parameter is a MP3 file path (which can be empty if you do not want any audio) and dest_file parameter is the resultant video file. codec parameter can be either H264 or HEVC. The width and height parameters refer to the video width and height. fps parameter is the frames per second of the video which I usually specified as 30 or 60. duration parameter refers to the video duration in milliseconds which can be set as -1 to indicate the video duration to be the same as the MP3. bitrate parameters refers to the video bitrate of bytes per second. Remember to set the bitrate higher for high resolution video and HEVC. The Render function signature of pure virtual FrameRenderer is below. The width and height is the video dimension. fps is the frames per second while frame_cnt is the frame count which auto-increments itself on every frame. pixels parameter is the single dimensional array to be filled up with your bitmap data. The return value should be false for catastrophic error which encoding shall be stopped. FrameRenderer is a class whose Render method is called on every frame, it should be implemented by the user.

C#
interface FrameRenderer
{
    public bool Render(int width, int height, int fps, int frame_cnt, uint[] pixels);
};
C++
class FrameRenderer
{
public:
    virtual bool Render(int width, int height, int fps, 
                        int frame_cnt, UINT32* pixels) = 0;
};

Red Video

For our first example, I keep it simple. We just render a red video.

Red Video

This is the main function whereby H264Writer.h is included and H264Writer is instantiated and Process() is called to encode the video. Process() calls the given Render() of RenderRedImage.

C#
using H264WriterDLL;

static void Main(string[] args)
{
    string musicFile = "";
    string videoFile = "C:\\temp\\RedVideoCSharp.mp4";

    RenderRedImage frameRenderer = new RenderRedImage();

    H264Writer writer = new H264Writer(musicFile, videoFile, VideoCodec.H264, 
                            640, 480, 30, 5000, frameRenderer, 40000000);
    if (writer.IsValid())
    {
        if (writer.Process())
        {
            Console.WriteLine("Video written successfully!\n");
            return;
        }
    }
    Console.WriteLine("Video write failed!\n");
}
C++
#include "../Common/H264Writer.h"

int main()
{
    std::wstring musicFile(L"");
    std::wstring videoFile(L"C:\\temp\\RedVideo.mp4");

    RenderRedImage frameRenderer;

    H264Writer writer(musicFile.c_str(), videoFile.c_str(), VideoCodec::H264, 
                      640, 480, 30, 5000, &frameRenderer);
    if (writer.IsValid())
    {
        if (writer.Process())
        {
            printf("Video written successfully!\n");
            return 0;
        }
    }
    printf("Video write failed!\n");
}

Below is the C++ RenderRedImage class. It only renders when frame_cnt is zero, meaning on the first frame because since pixels remains unchanged, there is no need to fill it up again on every frame.

C#
// render a red image once!
public class RenderRedImage : FrameRenderer
{
    public bool Render(int width, int height, int fps, int frame_cnt, UInt32[] pixels)
    {
        if (frame_cnt == 0)
        {
            for (int col = 0; col < width; ++col)
            {
                for (int row = 0; row < height; ++row)
                {
                    int index = row * width + col;
                    pixels[index] = 0xffff0000;
                }
            }
        }
        return true;
    }
}
C++
// render a red image once!
class RenderRedImage : public FrameRenderer
{
public:
    bool Render(int width, int height, int fps, int frame_cnt, UINT32* pixels) override
    {
        if (frame_cnt == 0)
        {
            for (int col = 0; col < width; ++col)
            {
                for (int row = 0; row < height; ++row)
                {
                    int index = row * width + col;
                    pixels[index] = 0xffff0000;
                }
            }
        }
        return true;
    }
};

Pixel format is in Alpha,Red,Green,Blue (ARGB) format. For example, you want a blue video, just change to pixels[index] = 0xff0000ff;

One JPEG Video

For our second example, we load a JPEG image with GDI+ and render once, that is when frame_cnt is zero.

Video with 1 image

Because we are using GDI+ now, we have to include the Gdiplus.h header and its Gdiplus.lib. frameRenderer type is set to RenderJPG class now.

C#
using H264WriterDLL;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Drawing.Imaging;

static void Main(string[] args)
{
    string musicFile = "";
    string videoFile = "C:\\temp\\JpgVideoCSharp.mp4";

    RenderJPG frameRenderer = new RenderJPG();

    H264Writer writer = new H264Writer(musicFile, videoFile, VideoCodec.H264, 
                            640, 480, 30, 10000, frameRenderer, 40000000);
    if (writer.IsValid())
    {
        if (writer.Process())
        {
            Console.WriteLine("Video written successfully!\n");
            return;
        }
    }
    Console.WriteLine("Video write failed!\n");
}
C++
#include "../Common/H264Writer.h"
#include <Gdiplus.h>
#pragma comment(lib, "gdiplus.lib")

int main()
{
    std::wstring musicFile(L"");
    std::wstring videoFile(L"C:\\temp\\JpgVideo.mp4");

    RenderJPG frameRenderer;
    H264Writer writer(musicFile.c_str(), videoFile.c_str(), VideoCodec::H264, 
                      640, 480, 30, 10000, &frameRenderer);
    if (writer.IsValid())
    {
        if (writer.Process())
        {
            printf("Video written successfully!\n");
            return 0;
        }
    }
    printf("Video write failed!\n");
}

RenderJPG is straightforward to those developers familiar with GDI+. It initialize and destroy GDI+ with GdiplusStartup() and GdiplusShutdown respectively. It loads the "yes.jpg" with the Bitmap class. bmp is the Bitmap with the same dimension as the video. We fill bmp with black color using FillRectangle(). Then we calculate the aspect ratio of the jpeg file and video frame. If w_ratio_jpg is greater than w_ratio_bmp, it means image is wider than video so you will see 2 horizontal black bars at the top and bottom of the video, otherwise you shall see 2 vertical black bars on the 2 sides of the video. In other words, we try to render the image as much as to cover the video while maintaining its original aspect ratio. To get bmp pixel pointer, we must call LockBits() and UnlockBits() afterwards after use. You notice in the double for loop, the image is rendered vertically upside down, so that it appears correctly in the video output. In the C# version of RenderJPG class. You see there is no need to initialize and deinitialize GDI+ in the constructor and destructor as with the C++ version because it is being taken care of for developer when System.Drawing library is referenced.

C#
// render a jpeg once!
public class RenderJPG : FrameRenderer
{
    public bool Render(int width, int height, int fps, int frame_cnt, UInt32[] pixels)
    {
        if (frame_cnt == 0)
        {
            Bitmap bmp = new Bitmap(width, height, PixelFormat.Format32bppArgb);
            Bitmap jpg = new Bitmap("image\\yes.jpg", true);
            Graphics g = Graphics.FromImage(bmp);
            float w_ratio_bmp = bmp.Width / (float)bmp.Height;
            float w_ratio_jpg = jpg.Width / (float)jpg.Height;

            SolidBrush brush = new SolidBrush(Color.Black);
            g.FillRectangle(brush, 0, 0, bmp.Width, bmp.Height);

            if (w_ratio_jpg >= w_ratio_bmp)
            {
                int width2 = bmp.Width;
                int height2 = (int)((bmp.Width / (float)jpg.Width) * jpg.Height);
                g.DrawImage(jpg, 0, (bmp.Height - height2) / 2, width2, height2);
            }
            else
            {
                int width2 = (int)((bmp.Height / (float)jpg.Height) * jpg.Width);
                int height2 = bmp.Height;
                g.DrawImage(jpg, (bmp.Width - width2) / 2, 0, width2, height2);
            }

            BitmapData bitmapData = new BitmapData();
            Rectangle rect = new Rectangle(0, 0, bmp.Width, bmp.Height);

            bmp.LockBits(
                rect,
                ImageLockMode.ReadOnly,
                PixelFormat.Format32bppArgb,
                bitmapData);

            unsafe
            {
                uint* pixelsSrc = (uint*)bitmapData.Scan0;

                if (pixelsSrc==null)
                    return false;

                int stride = bitmapData.Stride >> 2;

                for (int col = 0; col < width; ++col)
                {
                    for (int row = 0; row < height; ++row)
                    {
                        int indexSrc = (height - 1 - row) * stride + col;
                        int index = row * width + col;
                        pixels[index] = pixelsSrc[indexSrc];
                    }
                }
            }
            bmp.UnlockBits(bitmapData);

            bmp.Dispose();
            jpg.Dispose();
            brush.Dispose();
            g.Dispose();
        }
        return true;
    }
}
C++
// render a jpg once!
class RenderJPG : public FrameRenderer
{
public:
    RenderJPG()
    {
        // Initialize GDI+ so that we can load the JPG
        Gdiplus::GdiplusStartup(&m_gdiplusToken, &m_gdiplusStartupInput, NULL);
    }
    ~RenderJPG()
    {
        Gdiplus::GdiplusShutdown(m_gdiplusToken);
    }

    // render a jpg once!
    bool Render(int width, int height, int fps, int frame_cnt, UINT32* pixels) override
    {
        using namespace Gdiplus;

        if (frame_cnt == 0)
        {
            Bitmap bmp(width, height, PixelFormat32bppARGB);
            Bitmap jpg(L"image\\yes.jpg", TRUE);
            Graphics g(&bmp);
            float w_ratio_bmp = bmp.GetWidth() / (float)bmp.GetHeight();
            float w_ratio_jpg = jpg.GetWidth() / (float)jpg.GetHeight();

            SolidBrush brush(Color::Black);
            g.FillRectangle(&brush, 0, 0, bmp.GetWidth(), bmp.GetHeight());

            if (w_ratio_jpg >= w_ratio_bmp)
            {
                int width2 = bmp.GetWidth();
                int height2 = (int)((bmp.GetWidth() / 
                                    (float)jpg.GetWidth()) * jpg.GetHeight());
                g.DrawImage(&jpg, 0, (bmp.GetHeight() - height2) / 2, width2, height2);
            }
            else
            {
                int width2 = (int)((bmp.GetHeight() / 
                                   (float)jpg.GetHeight()) * jpg.GetWidth());
                int height2 = bmp.GetHeight();
                g.DrawImage(&jpg, (bmp.GetWidth() - width2) / 2, 0, width2, height2);
            }

            BitmapData bitmapData;
            Rect rect(0, 0, bmp.GetWidth(), bmp.GetHeight());

            bmp.LockBits(
                &rect,
                ImageLockModeRead,
                PixelFormat32bppARGB,
                &bitmapData);

            UINT* pixelsSrc = (UINT*)bitmapData.Scan0;

            if (!pixelsSrc)
                return false;

            int stride = bitmapData.Stride >> 2;

            for (int col = 0; col < width; ++col)
            {
                for (int row = 0; row < height; ++row)
                {
                    int indexSrc = (height - 1 - row) * stride + col;
                    int index = row * width + col;
                    pixels[index] = pixelsSrc[indexSrc];
                }
            }

            bmp.UnlockBits(&bitmapData);

        }
        return true;
    }

private:
    Gdiplus::GdiplusStartupInput m_gdiplusStartupInput;
    ULONG_PTR m_gdiplusToken;
};

Two JPEG Video

For the third example, we display first image and slowly alphablend with the second image until it appears. You can see the effect by looking at the video.

Since main function is exactly the same as previous except frameRenderer is set to Render2JPG, we skip showing it.

Render2JPG is almost similar to RenderJPG, except it loads 2 jpeg with the Bitmap class. The transparency stored in alpha variable is zero(total transparent) and 255(total opaque) when the duration is less or equal to 1000 milliseconds and is more or equal to 2000 milliseconds respectively. Between duration of 1000 and 2000 milliseconds, the alpha is calculated. A little note about the frame_duration = 1000 / fps: it is imprecise because it is in integer. For example, when the fps is 30: 1000/30 gives 33 millseconds but 30 * 33 only yields 990 millseconds, not the original 1000 milliseconds. It is of paramount importance that fps and frame_cnt are used to calculate the timing of appearance of current frame in the video. DO NOT USE a timer to calculate the elapsed time because frame encoding speed can vary depending on the complexity of your rendering logic.

C#
// render 2 jpegs!
public class Render2JPG : FrameRenderer
{
   public Render2JPG(string img1, string img2)
    {
        g = null;
        g2 = null;
        bmp = null;
        bmp2 = null;
        jpg1 = new Bitmap(img1, true);
        jpg2 = new Bitmap(img2, true);
    }
    ~Render2JPG()
    {
        jpg1.Dispose();
        jpg2.Dispose();
        bmp.Dispose();
        bmp2.Dispose();
        g.Dispose();
        g2.Dispose();
    }

    public bool Render(int width, int height, int fps, int frame_cnt, UInt32[] pixels)
    {
        if (bmp == null)
        {
            bmp = new Bitmap(width, height, PixelFormat.Format32bppArgb);
            bmp2 = new Bitmap(width, height, PixelFormat.Format32bppArgb);
            g = Graphics.FromImage(bmp);
            g2 = Graphics.FromImage(bmp2);
        }
        byte alpha = 0;
        float frame_duration = 1000.0f / fps;
        float time = frame_cnt * frame_duration;
        if (time <= 1000.0)
            alpha = 0;
        else if (time >= 2000.0)
            alpha = 255;
        else
            alpha = (byte)(int)(((time) - 1000.0) / 1000.0 * 255);

        float w_ratio_bmp = bmp.Width / (float)bmp.Height;
        float w_ratio_jpg = jpg1.Width / (float)jpg1.Height;

        SolidBrush brush = new SolidBrush(Color.Black);
        g.FillRectangle(brush, 0, 0, bmp.Width, bmp.Height);

        if (w_ratio_jpg >= w_ratio_bmp)
        {
            int width2 = bmp.Width;
            int height2 = (int)((bmp.Width / (float)jpg1.Width) * jpg1.Height);
            g.DrawImage(jpg1, 0, (bmp.Height - height2) / 2, width2, height2);
            g2.DrawImage(jpg2, 0, (bmp2.Height - height2) / 2, width2, height2);
        }
        else
        {
            int width2 = (int)((bmp.Height / (float)jpg1.Height) * jpg1.Width);
            int height2 = bmp.Height;
            g.DrawImage(jpg1, (bmp.Width - width2) / 2, 0, width2, height2);
            g2.DrawImage(jpg2, (bmp2.Width - width2) / 2, 0, width2, height2);
        }

        BitmapData bitmapData = new BitmapData();
        BitmapData bitmapData2 = new BitmapData();
        Rectangle rect = new Rectangle(0, 0, bmp.Width, bmp.Height);

        bmp.LockBits(
            rect,
            ImageLockMode.ReadOnly,
            PixelFormat.Format32bppArgb,
            bitmapData);

        bmp2.LockBits(
            rect,
            ImageLockMode.ReadOnly,
            PixelFormat.Format32bppArgb,
            bitmapData2);

        unsafe
        {
            uint* pixelsSrc = (uint*)bitmapData.Scan0;
            uint* pixelsSrc2 = (uint*)bitmapData2.Scan0;

            if (pixelsSrc == null || pixelsSrc2 == null)
                return false;

            int stride = bitmapData.Stride >> 2;

            for (int col = 0; col < width; ++col)
            {
                for (int row = 0; row < height; ++row)
                {
                    int indexSrc = (height - 1 - row) * stride + col;
                    int index = row * width + col;
                    pixels[index] = Alphablend(pixelsSrc2[indexSrc], 
                                               pixelsSrc[indexSrc], alpha, 0xff);
                }
            }
        }
        bmp.UnlockBits(bitmapData);
        bmp2.UnlockBits(bitmapData2);

        brush.Dispose();

        return true;
    }

    private uint Alphablend(uint dest, uint source, byte nAlpha, byte nAlphaFinal)
    {
        byte nInvAlpha = (byte) ~nAlpha;

        byte nSrcRed = (byte)((source & 0xff0000) >> 16);
        byte nSrcGreen = (byte)((source & 0xff00) >> 8);
        byte nSrcBlue = (byte)(source & 0xff);

        byte nDestRed = (byte)((dest & 0xff0000) >> 16);
        byte nDestGreen = (byte)((dest & 0xff00) >> 8);
        byte nDestBlue = (byte)(dest & 0xff);

        byte nRed = (byte)((nSrcRed * nAlpha + nDestRed * nInvAlpha) >> 8);
        byte nGreen = (byte)((nSrcGreen * nAlpha + nDestGreen * nInvAlpha) >> 8);
        byte nBlue = (byte)((nSrcBlue * nAlpha + nDestBlue * nInvAlpha) >> 8);

        return (uint)(nAlphaFinal << 24 | nRed << 16 | nGreen << 8 | nBlue);
    }

    private Bitmap jpg1;
    private Bitmap jpg2;
    private Graphics g;
    private Graphics g2;
    private Bitmap bmp;
    private Bitmap bmp2;
}
C++
// render 2 jpg
class Render2JPG : public FrameRenderer
{
public:
    Render2JPG(const std::wstring& img1, const std::wstring& img2)
    : jpg1(nullptr), jpg2(nullptr), g(nullptr), g2(nullptr), bmp(nullptr), bmp2(nullptr)
    {
        // Initialize GDI+ so that we can load the JPG
        Gdiplus::GdiplusStartup(&m_gdiplusToken, &m_gdiplusStartupInput, NULL);
        jpg1 = new Gdiplus::Bitmap(img1.c_str(), TRUE);
        jpg2 = new Gdiplus::Bitmap(img2.c_str(), TRUE);
    }
    ~Render2JPG()
    {
        delete jpg1;
        delete jpg2;
        delete bmp;
        delete bmp2;
        delete g;
        delete g2;
        Gdiplus::GdiplusShutdown(m_gdiplusToken);
    }

    // render 2 jpg
    // This function takes a long time.
    bool Render(int width, int height, int fps, int frame_cnt, UINT32* pixels) override
    {
        using namespace Gdiplus;

        if (bmp == nullptr)
        {
            bmp = new Bitmap(width, height, PixelFormat32bppARGB);
            bmp2 = new Bitmap(width, height, PixelFormat32bppARGB);
            g = new Graphics(bmp);
            g2 = new Graphics(bmp2);
        }
        BYTE alpha = 0;
        float frame_duration = 1000.0 / fps;
        float time = frame_cnt * frame_duration;
        if (time <= 1000.0)
            alpha = 0;
        else if (time >= 2000.0)
            alpha = 255;
        else
            alpha = (BYTE)(int)(((time)-1000.0) / 1000.0 * 255);

        float w_ratio_bmp = bmp->GetWidth() / (float)bmp->GetHeight();
        float w_ratio_jpg = jpg1->GetWidth() / (float)jpg1->GetHeight();

        SolidBrush brush(Color::Black);
        g->FillRectangle(&brush, 0, 0, bmp->GetWidth(), bmp->GetHeight());

        if (w_ratio_jpg >= w_ratio_bmp)
        {
            int width2 = bmp->GetWidth();
            int height2 = (int)((bmp->GetWidth() / 
                                (float)jpg1->GetWidth()) * jpg1->GetHeight());
            g->DrawImage(jpg1, 0, (bmp->GetHeight() - height2) / 2, width2, height2);
            g2->DrawImage(jpg2, 0, (bmp2->GetHeight() - height2) / 2, width2, height2);
        }
        else
        {
            int width2 = (int)((bmp->GetHeight() / 
                               (float)jpg1->GetHeight()) * jpg1->GetWidth());
            int height2 = bmp->GetHeight();
            g->DrawImage(jpg1, (bmp->GetWidth() - width2) / 2, 0, width2, height2);
            g2->DrawImage(jpg2, (bmp2->GetWidth() - width2) / 2, 0, width2, height2);
        }

        BitmapData bitmapData;
        BitmapData bitmapData2;
        Rect rect(0, 0, bmp->GetWidth(), bmp->GetHeight());

        bmp->LockBits(
            &rect,
            ImageLockModeRead,
            PixelFormat32bppARGB,
            &bitmapData);

        bmp2->LockBits(
            &rect,
            ImageLockModeRead,
            PixelFormat32bppARGB,
            &bitmapData2);

        UINT* pixelsSrc = (UINT*)bitmapData.Scan0;
        UINT* pixelsSrc2 = (UINT*)bitmapData2.Scan0;

        if (!pixelsSrc || !pixelsSrc2)
            return false;

        int stride = bitmapData.Stride >> 2;

        for (int col = 0; col < width; ++col)
        {
            for (int row = 0; row < height; ++row)
            {
                int indexSrc = (height - 1 - row) * stride + col;
                int index = row * width + col;
                pixels[index] = Alphablend(pixelsSrc2[indexSrc], 
                                           pixelsSrc[indexSrc], alpha, 0xff);
            }
        }

        bmp->UnlockBits(&bitmapData);
        bmp2->UnlockBits(&bitmapData2);

        return true;
    }

private:
    inline UINT Alphablend(UINT dest, UINT source, BYTE nAlpha, BYTE nAlphaFinal)
    {
        BYTE nInvAlpha = ~nAlpha;

        BYTE nSrcRed = (source & 0xff0000) >> 16;
        BYTE nSrcGreen = (source & 0xff00) >> 8;
        BYTE nSrcBlue = (source & 0xff);

        BYTE nDestRed = (dest & 0xff0000) >> 16;
        BYTE nDestGreen = (dest & 0xff00) >> 8;
        BYTE nDestBlue = (dest & 0xff);

        BYTE nRed = (nSrcRed * nAlpha + nDestRed * nInvAlpha) >> 8;
        BYTE nGreen = (nSrcGreen * nAlpha + nDestGreen * nInvAlpha) >> 8;
        BYTE nBlue = (nSrcBlue * nAlpha + nDestBlue * nInvAlpha) >> 8;

        return nAlphaFinal << 24 | nRed << 16 | nGreen << 8 | nBlue;
    }
    private:
    Gdiplus::GdiplusStartupInput m_gdiplusStartupInput;
    ULONG_PTR m_gdiplusToken;
    Gdiplus::Bitmap* jpg1;
    Gdiplus::Bitmap* jpg2;
    Gdiplus::Graphics* g;
    Gdiplus::Graphics* g2;
    Gdiplus::Bitmap* bmp;
    Gdiplus::Bitmap* bmp2;
};

pixels[index] is determined by the Alphablend() function.

Text Animation Video

For our last example, we show the two prerendered text images appearing from the middle of the video. See the video below for example.

There is a thin white rectangle expanding progressively. renderbmp variable is the one where top half of bmp and bottom half of bmp2 are shown. bmp is rendered with jpg1 progressively moving up while bmp2 is rendered with jpg2 progressively moving down. The jpg1 and jpg2 are misnomers since the image loaded are actually PNGs. Bitmap class can load both JPEG and PNG. JPEG is best for storing photographs while PNG is for storing illustrations.

C#
// text animation!
public class RenderText : FrameRenderer
{
    public RenderText(string img1, string img2)
    {
        g = null;
        g2 = null;
        bmp = null;
        bmp2 = null;
        jpg1 = new Bitmap(img1, true);
        jpg2 = new Bitmap(img2, true);
    }
    ~RenderText()
    {
        jpg1.Dispose();
        jpg2.Dispose();
        bmp.Dispose();
        bmp2.Dispose();
        g.Dispose();
        g2.Dispose();
    }

    public bool Render(int width, int height, int fps, int frame_cnt, UInt32[] pixels)
    {
        if (bmp == null)
        {
            bmp = new Bitmap(width, height, PixelFormat.Format32bppArgb);
            bmp2 = new Bitmap(width, height, PixelFormat.Format32bppArgb);
            g = Graphics.FromImage(bmp);
            g2 = Graphics.FromImage(bmp2);
        }
        Bitmap renderbmp = new Bitmap(width, height, PixelFormat.Format32bppArgb);
        Graphics render_g = Graphics.FromImage(renderbmp);

        float rectProgress = 0.0f;
        float textProgress = 0.0f;
        float frame_duration = 1000.0f / fps;
        float time = frame_cnt * frame_duration;

        SolidBrush brush = new SolidBrush(Color.Black);
        render_g.FillRectangle(brush, 0, 0, width, height);
        g.FillRectangle(brush, 0, 0, width, height);

        int rectHeight = 4;

        int rectWidth = (int)(width * 0.8f);
        if (time >= 1000.0f)
            rectProgress = 1.0f;
        else
            rectProgress = time / 1000.0f;


        if (time >= 2000.0f)
            textProgress = 1.0f;
        else if (time <= 1000.0f)
            textProgress = 0.0f;
        else
            textProgress = (time - 1000.0f) / 1000.0f;

        g.DrawImage(jpg1, (width - jpg1.Width) / 2, 
                          (height / 2) - (int)(jpg1.Height * textProgress), 
                          jpg1.Width, jpg1.Height);
        g.FillRectangle(brush, 0, height / 2 - 4, width, height / 2 + 4);
        render_g.DrawImage(bmp, 0, 0, width, height);

        g2.DrawImage(jpg2, (width - jpg2.Width) / 2, 
		                   (int)((height / 2 - jpg2.Height) + 
                           (int)(jpg2.Height * textProgress)), 
                           jpg2.Width, jpg2.Height);
        g2.FillRectangle(brush, 0, 0, width, height / 2 + 4);
        render_g.DrawImage(bmp2, 0, height / 2 + 4, 
                           new Rectangle(0, height / 2 + 4, width, height / 2 - 4), 
                           GraphicsUnit.Pixel);

        SolidBrush whitebrush = new SolidBrush(Color.White);
        int start_x = (width - (int)(rectWidth * rectProgress)) / 2;
        int pwidth = (int)(rectWidth * rectProgress);
        render_g.FillRectangle(whitebrush, start_x, 
                               (height - rectHeight) / 2, pwidth, 
                               rectHeight);

        BitmapData bitmapData = new BitmapData();
        Rectangle rect = new Rectangle(0, 0, width, height);

        renderbmp.LockBits(
            rect,
            ImageLockMode.ReadOnly,
            PixelFormat.Format32bppArgb,
            bitmapData);

        unsafe
        {
            uint* pixelsSrc = (uint*)bitmapData.Scan0;

            if (pixelsSrc == null)
                return false;

            int stride = bitmapData.Stride >> 2;

            for (int col = 0; col < width; ++col)
            {
                for (int row = 0; row < height; ++row)
                {
                    int indexSrc = (height - 1 - row) * stride + col;
                    int index = row * width + col;
                    pixels[index] = pixelsSrc[indexSrc];
                }
            }
        }
        renderbmp.UnlockBits(bitmapData);

        renderbmp.Dispose();
        brush.Dispose();
        whitebrush.Dispose();
        render_g.Dispose();

        return true;
    }

    private Bitmap jpg1;
    private Bitmap jpg2;
    private Graphics g;
    private Graphics g2;
    private Bitmap bmp;
    private Bitmap bmp2;
}
C++
class RenderText : public FrameRenderer
{
public:
    RenderText(const std::wstring& img1, const std::wstring& img2) 
        : jpg1(nullptr), jpg2(nullptr), g(nullptr), 
          g2(nullptr), bmp(nullptr), bmp2(nullptr)
    {
        // Initialize GDI+ so that we can load the JPG
        Gdiplus::GdiplusStartup(&m_gdiplusToken, &m_gdiplusStartupInput, NULL);
        jpg1 = new Gdiplus::Bitmap(img1.c_str(), TRUE);
        jpg2 = new Gdiplus::Bitmap(img2.c_str(), TRUE);
    }
    ~RenderText()
    {
        delete jpg1;
        delete jpg2;
        delete bmp;
        delete bmp2;
        delete g;
        delete g2;
        Gdiplus::GdiplusShutdown(m_gdiplusToken);
    }


    // render text
    bool Render(int width, int height, int fps, int frame_cnt, UINT32* pixels) override
    {
        using namespace Gdiplus;

        if (bmp == nullptr)
        {
            bmp = new Bitmap(width, height, PixelFormat32bppARGB);
            bmp2 = new Bitmap(width, height, PixelFormat32bppARGB);
            g = new Graphics(bmp);
            g2 = new Graphics(bmp2);
        }

        Bitmap renderbmp(width, height, PixelFormat32bppARGB);
        Graphics render_g(&renderbmp);

        float rectProgress = 0.0f;
        float textProgress = 0.0f;
        float frame_duration = 1000.0f / fps;
        float time = frame_cnt * frame_duration;

        SolidBrush brush(Color::Black);
        render_g.FillRectangle(&brush, 0, 0, width, height);
        g->FillRectangle(&brush, 0, 0, width, height);

        int rectHeight = 4;

        int rectWidth = (int)(width * 0.8f);
        if (time >= 1000.0f)
            rectProgress = 1.0f;
        else
            rectProgress = time / 1000.0f;

        if (time >= 2000.0f)
            textProgress = 1.0f;
        else if (time <= 1000.0f)
            textProgress = 0.0f;
        else
            textProgress = (time - 1000.0f) / 1000.0f;

        g->DrawImage(jpg1, (width - jpg1->GetWidth()) / 2, 
                        (height / 2) - (int)(jpg1->GetHeight() * textProgress), 
                        jpg1->GetWidth(), jpg1->GetHeight());
        g->FillRectangle(&brush, 0, height / 2 - 4, width, height / 2 + 4);
        render_g.DrawImage(bmp, 0, 0, width, height);

        g2->DrawImage(jpg2, (width - jpg2->GetWidth()) / 2, 
                               (int)((height / 2 - jpg2->GetHeight()) + 
                               (int)(jpg2->GetHeight() * textProgress)), 
                               jpg2->GetWidth(), jpg2->GetHeight());
        g2->FillRectangle(&brush, 0, 0, width, height / 2 + 4);
        render_g.DrawImage(bmp2, 0, height / 2 + 4, 0, height / 2 + 4, width, 
                           height / 2 - 4, Gdiplus::UnitPixel);

        SolidBrush whitebrush(Color::White);
        int start_x = (width - (int)(rectWidth * rectProgress)) / 2;
        int pwidth = (int)(rectWidth * rectProgress);
        render_g.FillRectangle(&whitebrush, start_x, (height - rectHeight) / 2, 
                               pwidth, rectHeight);

        BitmapData bitmapData;
        Rect rect(0, 0, width, height);

        renderbmp.LockBits(
            &rect,
            ImageLockModeRead,
            PixelFormat32bppARGB,
            &bitmapData);

        UINT* pixelsSrc = (UINT*)bitmapData.Scan0;

        if (!pixelsSrc)
            return false;

        int stride = bitmapData.Stride >> 2;

        for (int col = 0; col < width; ++col)
        {
            for (int row = 0; row < height; ++row)
            {
                int indexSrc = (height - 1 - row) * stride + col;
                int index = row * width + col;
                pixels[index] = pixelsSrc[indexSrc];
            }
        }

        renderbmp.UnlockBits(&bitmapData);

        return true;
    }
private:
    Gdiplus::GdiplusStartupInput m_gdiplusStartupInput;
    ULONG_PTR m_gdiplusToken;
    Gdiplus::Bitmap* jpg1;
    Gdiplus::Bitmap* jpg2;
    Gdiplus::Graphics* g;
    Gdiplus::Graphics* g2;
    Gdiplus::Bitmap* bmp;
    Gdiplus::Bitmap* bmp2;
};

Can I Use this H264Writer for OpenGL?

The answer is definitely yes if you do not wish to use the more complicated OpenGL H264Writer that requires win32 thread synchronization. OpenGL and that H264Writer renders/encodes in tandem as shown below. Number in the blue box is the frame number in OpenGL whereas number in the magenta box is corresponding frame in encoding process. In that graph, it is assumed OpenGl and encoding frame take the same amount of processing time. After OpenGL fill in the frame buffer and signals the win32 event for the video encoding thread to take over.

2 threads

Whereas in this H264Writer featured in this article, if you call the renderFunction in which OpenGL fill in the frame buffer, you get single threaded performance as OpenGL and encoding take place in the same thread. Encoding can utilitize more than 1 thread under the hood. For the sake of simplicity, we assume encoding uses 1 thread.

Single threaded

It is better to use the more complicated OpenGL H264Writer solely for performance reason. The code is hosted at GitHub. Remember to copy the image folder to Debug or Release folder before running the executable. Have fun with converting your cool animations to H264/HEVC video to share with others and keepsake for posterity!

Hardware Acceleration

Hardware acceleration is available now in the library by setting MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS in attributes and passing them to MFCreateSinkWriterFromURL

C++
CComPtr<IMFAttributes> attrs;
MFCreateAttributes(&attrs, 1);
attrs->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, TRUE);

hr = MFCreateSinkWriterFromURL(m_DestFilename.c_str(), nullptr, attrs, &m_pSinkWriter);

When you run a video encoder without OpenGL, you should see "GPU 1 - Video Encode" on Windows 10 task manager. With OpenGL, you see "GPU 1 - Copy".

Image 5

Quality Parameters in Constructor in v0.4.2

  • int numWorkerThreads: 0 leaves to default
  • int qualityVsSpeed: [0:100] 0 for speed, 100 for quality
  • RateControlMode mode: 3 modes to choose from UnconstrainedVBR, Quality, CBR (VBR is variable bitrate and CBR is constant bitrate)
  • int quality: Only valid when mode is Quality. [0:100] 0 for smaller file size and lower quality, 100 for bigger file size and higher quality

How to Record Real-Time Animation?

  • Create a Win32 event and let Render() wait for it to be signalled.
  • Set a timer to run periodically to signal the Win32 event.
  • Updates to the pixels have to be synchronized, you do not want the pixels are being updated and read by the video encoder at the same time which results in tearing effect
  • Note: Do not be lazy and use Sleep() because we do not know how long to sleep between each interval as the time needed to encode each frame might be different.

Software that use this library

History

  • 25th April, 2023: Added a section on Software that use this library. Please let me know if your software is using this video encoding library so that I can add it to this list.
  • 27th October, 2022: Added Mandy Frenzy video and links in the introduction
  • 8th March, 2022: Added a section on How to record real-time animation?
  • 6th March, 2022: Added HasH264() and HasHEVC() to check for presence of hardware accelerated or software encoder on the system. Remember to call CoInitialize() before calling HasH264() and HasHEVC() in C++. C# application calls CoInitialize() by default. Put back option of software encoding for testing in VMs.
    Note: Quality settings are ignored when software encoding is selected because they do not work.
  • 17th February, 2022: Added HasHEVC() to check for presence of hardware accelerated HEVC encoder on the system.
  • 9th August, 2020: Uploaded precompiled C# DLLs for those who have difficulty in building the C++/CLI DLL. Your C# application mode must be either x86 or x64 to use the DLLs. Strictly, no AnyCPU mode which can then default to either x86 or x64.
  • 26th July, 2020: Added 0.4.2 version to adjust quality and encoding speed parameters (See quality section for more information)
  • 25th July, 2020: Added 0.4.0 version for hardware acceleration (See hardware acceleration section for more information)
  • 20th July, 2019: Added C++/CLI version for C# use
  • 19th July, 2019: Added OpenGL section
  • 2nd July, 2019: Initial version

Other Articles in the Bring Your... Series

License

This article, along with any associated source code and files, is licensed under The MIT License


Written By
Software Developer (Senior)
Singapore Singapore
Shao Voon is from Singapore. His interest lies primarily in computer graphics, software optimization, concurrency, security, and Agile methodologies.

In recent years, he shifted focus to software safety research. His hobby is writing a free C++ DirectX photo slideshow application which can be viewed here.

Comments and Discussions

 
QuestionContact Pin
zapsolution30-Apr-23 22:29
zapsolution30-Apr-23 22:29 
AnswerRe: Contact Pin
Shao Voon Wong2-May-23 14:05
mvaShao Voon Wong2-May-23 14:05 
GeneralRe: Contact Pin
zapsolution3-May-23 1:39
zapsolution3-May-23 1:39 
SuggestionSome suggestions and questions Pin
++i10-Nov-22 3:04
++i10-Nov-22 3:04 
GeneralRe: Some suggestions and questions Pin
Shao Voon Wong10-Nov-22 18:18
mvaShao Voon Wong10-Nov-22 18:18 
PraiseOpenGL H264Writer insight = my vote of 5 Pin
Member 1507871629-Jun-22 16:16
Member 1507871629-Jun-22 16:16 
QuestionHEVC for an app's animations? Pin
dandy7217-Feb-22 6:07
dandy7217-Feb-22 6:07 
AnswerRe: HEVC for an app's animations? Pin
Shao Voon Wong19-Feb-22 13:31
mvaShao Voon Wong19-Feb-22 13:31 
GeneralMy vote of 5 Pin
tugrulGtx17-Feb-22 1:44
tugrulGtx17-Feb-22 1:44 
GeneralRe: My vote of 5 Pin
Shao Voon Wong17-Feb-22 2:51
mvaShao Voon Wong17-Feb-22 2:51 
BugSystem.BadImageFormatException Pin
Liza Yemini6-Feb-22 22:50
Liza Yemini6-Feb-22 22:50 
GeneralRe: System.BadImageFormatException Pin
Shao Voon Wong7-Feb-22 1:51
mvaShao Voon Wong7-Feb-22 1:51 
Questioncool Pin
Christ Kennedy28-Nov-20 1:35
mvaChrist Kennedy28-Nov-20 1:35 
its great at editing a single frame but I'm producing a cartoon animator and have 20fps images to put on the screen. I tried using your single jpg example with a different image source

Bitmap jpg = formLIS_SequencePreview.instance.getImage(formLIS_SequencePreview.instance.Frame_Start + frame_cnt);


but it only draws the first image and never refreshes to the remaining sequence of images.
stepping through with debugger only sees this code once ?
is there a way to source a sequence of multiple frame?
my code is perfect until i don't find a bug...

AnswerRe: cool Pin
Shao Voon Wong28-Nov-20 2:40
mvaShao Voon Wong28-Nov-20 2:40 
QuestionPrecompiled dll? Pin
alxxl6-Aug-20 23:24
alxxl6-Aug-20 23:24 
AnswerRe: Precompiled dll? Pin
Shao Voon Wong8-Aug-20 21:57
mvaShao Voon Wong8-Aug-20 21:57 
GeneralRe: Precompiled dll? Pin
alxxl13-Aug-20 1:16
alxxl13-Aug-20 1:16 
GeneralRe: Precompiled dll? Pin
Shao Voon Wong13-Aug-20 1:20
mvaShao Voon Wong13-Aug-20 1:20 
GeneralRe: Precompiled dll? Pin
alxxl13-Aug-20 9:54
alxxl13-Aug-20 9:54 
QuestionPlease help me with using H264WriterDLL; Pin
Piotr Dusiński29-Jul-20 1:27
Piotr Dusiński29-Jul-20 1:27 
AnswerRe: Please help me with using H264WriterDLL; Pin
Shao Voon Wong8-Aug-20 21:57
mvaShao Voon Wong8-Aug-20 21:57 
Questioncan be used for live animation Pin
Filip De Brabander27-Jul-20 4:39
Filip De Brabander27-Jul-20 4:39 
AnswerRe: can be used for live animation Pin
Shao Voon Wong8-Aug-20 22:01
mvaShao Voon Wong8-Aug-20 22:01 
AnswerRe: can be used for live animation Pin
robschoenaker10-Aug-20 0:22
robschoenaker10-Aug-20 0:22 
QuestionWrong use of Dispose/Finalizer Pin
spi22-Jul-19 1:20
professionalspi22-Jul-19 1:20 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.