Writing Custom Information in Linux Core-dump, using GoogleCoreDumper

• Download source - 82.9 KB

Introduction

I am quite proud to say that I am a Debug Enthusiast. People like me believe in the below quote...

Debugging is an art...

Debugging can sometimes be hell like roaming around a maze in final challenge of Triwizard tournament or like searching for a needle of unknown size and shape in pacific ocean.

Being an debug enthusiast, I spend my time exploring ways to better my debugging skills and also to reach out to vent my learning to folks like you [who stop by @ my blog... ;) ].

One such new learning which I had in recent days is GoogleCoreDumper.

It is a wonderful tool, which aids you in creating a GDB readable ELF corefile. The most important aspect is, Coredump can be dumped, without stopping your application.

This is normal to me... as I am used to gcore [however googCoreDumper might be helpful in systems where ulimit setting for writing a core is an issue. For more details, look here.]

So, what's special about googCore according to me....????? ;) :P

Writing customized notes in the corefile...

Background

Before understanding customized notes, let us first understand what are notes in corefiles...

ELF notes are special information, which will get stored in the ELF header. ELF as the name suggests, is a popular file format, opted by most OS vendors to convey the information in a uniform manner.

An Executable file [exe], coredump, etc. are ELF files. ELF note appear in ELF header [metadata of the ELF file] which conveys vendor & platform specific information like:

• OS platform
• Signal name which caused the core
• Executable name which the core belongs to
• Endianess hardware, etc...

I am not going to bore you with the theory anymore. This link has got much information in case you are interested in learning more about ELF.

Predominantly, these notes are vendor specific and cannot be customized unless you keep your hands in kernel and tweak something. Though there were some ELF writers like elfIO, elfutils, etc., the simplicity and usability doesn't match with GoogleCoreDumper (according to me :P :) ).

In this post, let me take you through how a typical debugging problem could be solved by using GoogleCoreDumper especially in the custom notes...

Using the Code

Let us take a scenario / problem to understand the code.

I have hosted the source in github which can be found here.

Problem: How to get pmap [process memory map] from a linux coredump, where kernel < 3.6

I work in a server product, which is a C++ app and crashdumps from deployment site are the primary source of information for me to debug. One of the pains in debugging is to rule-out theories based on some proof of concept. This product employs both proprietary plugins and 3^rd party / opensource plugins, all hosted in the same process space.

If a crash / core occurs in deployment site with the release mode binaries, all I have is just the function addresses in hex format. As I had explained earlier, my first job is to understand whether the crashed instruction is part of proprietary plugin [ ha... I know where to look @ ;) ] or opensource [ well... some what lucky... :)] or 3^rd party [phew!!! this is not in my court.... :) ].

So how do I get to know.. what region of memory space is occupied by every shared library in the process space... the answer is pmap.

However pmap utility will work only with the running process and not with core.

Well.. wouldn't the crash capture the memory map as part of core ???

I can read your mind.... ;)

The answer is, until this check-in the memory map of the running process, was not captured in crashdump. In new kernel, this has been resolved, but what about old kernel... [like my situation... :( ].

Now read the blog title... that's it.. you have the solution...

Implementation Details

GoogleCoreDumper can be used to both read & write coredumps. It also provides facility to add custom notes in the corefile through SetCoreDumpNotes() API. Surprisingly, these functions aren't explained in the project wiki page. Nevertheless the main pages as part of package was still useful.

SetCoreDumpNotes() API function sets the attributes in params to add additional notes to core dumps when called with GetCoreDumpWith() or WriteCoreDumpWith(). The notes are specified as an array of the CoredumperNote structs. The CoredumperNote is being defined as follows:

struct CoredumperNote {
const char *name; // The vendor name
unsigned int type; // A vendor specific type
unsigned int description_size; // The size of the description field
const void *description; // The note data
};

The name, including a terminating null character, will be 4 byte aligned. The type is a user chosen value. The description_size specifies the amount of bytes to write from the description pointer in memory to the core dump note. The description will be padded to be 4 byte aligned.

You can dump as much information you want, provided the total corefile size doesn't exceed the ulimit setting of the system.

Code

The intent of this exercise is to capture the memory map of the process while writing the coredump file and to extract the same from the resultant core file.

Part 1 - Saving information in the corefile

First, a signal handler is registered to capture the signal and being the core writing algorithm:

int main()
{
 signal(SIGSEGV, signal_handler);
 cout << "Starting the program with pid" << getpid() << endl;
 int c;
 cout << "OK am now waiting for the signal" << endl;
 while(1);
 return 0;
}

and now we will see, what the signal_handler is actually doing.

/*
 * googCore.cpp
 *
 * Author: viswa
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <google/coredumper.h> // Header for googCore API
#include <signal.h> // for handling signal
#include <iostream>
#include <fstream>
#include <sys/types.h>
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sstream>
#include <limits>
#include <errno.h>

using namespace std;

#define NT_VISWA 0x46494c46 // NT_FILE + 1

void signal_handler(int SIGNO)
{

:

// Jumping to relevant details...

:

// now lets proceed to dump the core
 struct CoreDumpParameters theParam;
 struct CoredumperNote theNote;

theNote.name = "kspviswa-special-notes-mmap";
 theNote.type = NT_VISWA;
 theNote.description_size = fsize;
 theNote.description = pInit; // pInit is the ptr to pmap of the running process

// Clear theParam so that default values get initialized
 ClearCoreDumpParameters(&theParam);

// Save theParam with the custom notes
 SetCoreDumpNotes(&theParam, &theNote, 1);

// Write the notes to core

status = WriteCoreDumpWith(&theParam, "viswa-core");

The NT_VISWA is a special flag which helps to uniquely identify the special notes in the elfReader.

Part 2 - Reading special notes from the corefile

In this exercise, I had used readelf to customize / tweak the reader so that special notes can be read [ I love opensource extremely for these customization ;) ] .

static const char *
get_note_type (unsigned e_type)
{

:

/**
 * ########## VISWA #############
 */
 case NT_VISWA:
 return _("NT_VISWA (kspviswa-special-notes-mmap)");

}

/**
 * ########## VISWA #############
 */
static int print_core_note_special(Elf_Internal_Note *pnote)
{

printf("\n Printing the special notes for kspviswa... :-)\n");
 printf("%s",(char*) pnote->descdata);

return 1;
}

static int
process_note (Elf_Internal_Note * pnote)
{

:

/**
 * ########## VISWA #############
 */
 else if (const_strneq (pnote->namedata, "kspviswa-special-notes-mmap"))
 return print_core_note_special (pnote);
 else
 return 1;
}

Outputs

Starting the main program under "sudo" mode:

Pmap of the process, before core..

Customized readelf [readelfviswa] reads the custom notes:

Points of Interest

GoogleCoreDumper was extremely cool and fun to work with. Solving a realworld problem with my hobby was lot more fantastic. Not only just pmap, any interesting information like memory profiling, number of active connections to a shared resource, total count of shared libraries, state of global caches / buffers, total number of heap space contributed by a particular module, etc. can be logged as notes to the corefile and all you need to do is to tweak readelf to read those for you... I enjoy debugging. :) Hope you enjoyed my blog too...

Once again, thanks for taking the time to go through this post. Feel free to leave your comments / feedback and I will be more than willing to respond at the earliest.

Happy coding [& debugging :) ] !!!!!

History

• Sep 7^th 2014. Initial version

I can provide a library version if anyone plans to use it in their project. Please let me know in the comments section.