Performance Evaluation of Vortex-Compiled Applications

2 Benchmarking Applications

In its default compilation mode, when compiling Cecil applications Vortex generates code that supports fairly good source level debugging. However, we have not yet spent much time optimizing our debugging support and the simple scheme currently used incurs a fairly hefty runtime cost, often on the order of 30% or 40% in optimized code. Therefore, when benchmarking Cecil applications we disable debugging support by setting the compiler options debug_support and interrupt_checking to false (e.g., Cecil> no_debug_support; no_interrupt_checking). Note that code compiled with and without debugging support cannot be mixed; toggling the debug_support option will automatically invalidate all compiled code.

Another thing to be aware of is that, to reduce the costs of gathering profiling data, Vortex does not fully instrument calls that have been statically-bound purely by means of some static analysis (for example, intraprocedural class analysis, class hierarchy analysis or static class prediction). In most cases this does not matter, but if different levels of static analysis are going to be used (for example to measure the effectiveness of various flavors of static class analysis), then it is critical that the profile data to be used in the experiments be gathered from executables compiled with the same level of static analysis. For example, to measure the impact of class hierarchy analysis, one might want to compare the following four combinations of optimizations:

intra: intraprocedural class analysis
intra+profile: intraprocedural class analysis + profile-guided class prediction
intra+CHA: intraprocedural class analysis + class hierarchy analysis
intra+CHA+profile: intraprocedural class analysis + class hierarchy analysis + profile-guided class prediction.

To do this requires two sets of profile information: a profile of the intra version of the program to build the intra+profile version and a profile of the intra+CHA version to build the intra+CHA+profile version. Future versions of Vortex may support an additional, more expensive level of profiling that instruments all statically-bound calls, but this is not currently implemented. Similarly, as mentioned above, Vortex does not perform enough instrumentation of specialized programs, and the missing information can degrade the quality of the profile data substantially.

If you are going to be doing a non-trivial amount of benchmarking and/or performance evaluation using Vortex, you should invest some time and become familiar with the config family of scripts that we have developed to support these tasks (found in $VORTEX_HOME/bin/shell). The file configs.perl is included in the rest of the scripts; it defines configuration information and handles command line arguments. buildConfig uses Vortex to build executable, runProgs measures the resulting exectuables, and showData displays experimental results.

Performance Evaluation of Vortex-Compiled Applications - 25 MARCH 1997

[Next] [Previous] [Top]

Generated with Harlequin WebMaker