IgProf history

While the current version of IgProf is entirely the work of Lassi Tuura and Giulio Eulisse, it borrows most of the ideas from several other projects. Our main contribution has been to provide the existing concepts in a package that actually works out of the box and is fast and reliable enough to made significant use of.

Like everyone else, we started with gprof. It’s terribly limited by modern standards, and having to recompile all the source is just not going to happen – but we still like it, and a lot of the IgProf output borrows from gprof. We then hoped $LD_PROFILE to be an improvement, but that quickly turned out not to go anywhere.

At some point we learnt about the Mozilla jprof tool and got very excited. It actually worked for programs using dynamically loaded shared libraries! We were delighted to have in our hands finally a profiler capable of sampling IGUANA and other CMS software at near-interactive rates – a profiler that actually worked. We noticed we could easily fix in the source any problems jprof had with CMS software. As time went on, patches piled up and we had implemented a memory profiling jprof look-a-like, MemProfLib. IgProf was born when we eventually took it unto ourselves to rewrite the whole lot from ground up. Giulio made sure we handled threads correctly, and made some very significant performance optimisations.

Anyone well versed with GLIBC internals and shared library symbol versioning knows the terror it is to try to provide binary-compatible substitute across several operating system generations. We started thinking that installing hooks into the code by dynamically modifying the machine code couldn’t be harder than what we were experiencing already. In concept it’s similar to other dynamic instrumentation tools such as Jeffrey Richter’s INJLIB (see MSJ May 1994 - Vol 9, Issue 5), Jonathan Rentzsch’s mach_override (now mach_star), Secure Reality’s injectso, or Dyninst, and of the most recent renown, Sun’s DTrace. Of course it would be silly to claim both Giulio and Lassi weren’t excited about dynamically rewriting the machine the code to vector our stuff into it, so it’s also fair to say we did it because we could and thought it an interesting learning exercise! At any rate the dynamic tapping functionality, now provided in separate library in IgHook, has been a great deal more robust than depending on the $LD_PRELOAD symbol override ever was. It also works when symbol overriding can’t work, for example if for one reason or the other symbols cannot easily be re-defined or simply are not going to be looked up in the $LD_PRELOAD library by the dynamic linker.

Since the switch to the dynamic instrumentation architecture the profiler has been rewritten in its entirety at least twice. One of the big changes was to completely separate all the profiling modules and to make it easier to write arbitrary new modules that “tap” into functions of interest. We’ve also worked hard to provide an infrastructure for multiple profiling backends to coexist peacefully. This is a lot more complex than it sounds what with multi-threaded applications receiving asynchronous SIGPROF signals and the memory profiler trapping calls from all manner of operating system bowels while all kinds of interesting pthread mutexes are being held!

Lately, initial support for generic instrumentation and precise profiling of applications has been introduced, thanks to the work done by Mikko Kurkela.

In 2014 IgProf was accepted as GSoC project within CERN SFT organization. In such a context Filip Nybäck implemented initial support for ARM64 (aarch64) and PAPI based energy profiling.