SourceForge.net Logo
News
Downloads
Mailing Lists
CVS
NOTE: There will be a new release soon. Until that, please use the CVS version! -S., 2004-08-09
Latest Release as of 2002-08-25: oobench-0.5.0.tar.gz

OO Bench - A Brief Description

Sven C. Köhler

There are lies, damned lies, and benchmarks.


Contents

History
Date Author Description
2002-01-07 Sven C. Köhler Initialization
2002-08-25 Sven C. Köhler Preparation for 0.5.0

Introduction

What is it?

OO-Bench compares the speed of the same object-oriented tasks in several object-oriented languages. It also has a statistics tool (written in Java), which can be used to easily compare the speed of the several versions of a given benchmark.

Supported Languages so far

So far supported languages are Java, C++, and Objective-C. Also take a look at the {Java,C++,Objective-C}/README files.

The directories for CSharp, Eiffel, Smalltalk, and CLOS are already there, but the benchmarks are currently in development.

Aims

  • Simple, easy to understand, and easy to port.
  • Follows the idioms and best practices advised by each language as much as possible
  • Designed to make it easy to look up how a particular problem is best solved in another language

Remarks

  • As the motto suggests, the focus is on source code rather than on benchmark results. This is even more true, since intensive reviews of the benchmarks are yet due.

About Measuring Time

To be compatible with Java and Windows, only the actual execution time is measured (Not the user time of the process). So try to keep your system as Idle as possible, and avoid swapping (On UNIX: Watch the pi and po columns in vmstat during execution of a benchmark).

About Measuring Memory Usage

The benchmarks for C++ and Objective-C use sbrk() to monitor heap usage, that is not supported on all platforms (So you might see no or inappropriate results for memory allocation). Also, heaps don't shrink on most systems, so you won't see memory freed. Therefore, memory usage does currently not influence the results produced by the statistics tool.

Supported Compilers/Libraries

OO Bench should work at least with the following compilers, and libraries :

  • C++: GCC-2.95.2 (GCC-3 should work too), to build GNU-Make is needed;
  • ObjC: GCC-2.95.2 with libFoundation, to build GNU-Make is needed;
  • Java: JDK-1.3, to build GNU-Make, ant, and Perl is needed.

User Guide

Usually, one does the following:

    [oobench-X.Y.Z] $ gmake           # Configure, and compile all files.
    [oobench-X.Y.Z] $ gmake bench     # Run all benchmarks.  Results are
                                      # logged to benchResult.txt.

If you just want to test, whether the benchmarks were compiled correctly, you can also use the test target:

[oobench-X.Y.Z] $ gmake test          # Run all benchmarks with minimal    
                                      # values (much faster than the bench
                                      # target).  Results are logged to   
                                      # testResult.txt.

The bench and test targets can be used in any subdirectory of oobench, hence you can also do this:

[oobench-X.Y.Z/Java] $ gmake bench  # Run all benchmarks below Java 
                                    # directory.  Results are not
                                    # logged.

If you want only to execute a certain benchmark, you can do it that way:

[oobench-X.Y.Z/C++/Patterns/MVC] $ gmake bench # Run MVC benchmark only.
                                               # Results are not logged.

Running the Statistics Program

USAGE: ./tools/Statistics/statistics <ID file> <log file[,tag]> [<log file[,tag]> ...]

IDs file

The IDs file, which is located in the the oobench-X.Y.Z directory, contains a short description of each benchmark topic, which is then used by the statistics program when generating the report .

Tag

This can be used, if you want to compare several bench results, that use i. e. the same language, or cannot be distinguished for some other reason.

Example: You have one benchResult.txt that used JDK-1.3, and one that used JDK-1.4. In this case, you can use the ",tag" notation, so the statistics program can distinguish the two set of results.

Example Run

[oobench-X.Y.Z] $ ./tools/Statistics/statistics IDs benchResult.txt

Output looks like this:

#
# Output generated by OO Bench's statistics program.
# See http://oobench.sourceforge.net/ for more information
# about OO Bench.
#
# Command was: ./tools/Statistics/statistics IDs benchResult.linux.txt
#
# Note: Performance is measured only by the speed of execution.
#       Estimated times are not included in statistics as well.
=========================================================================
Total average performance (if comparison was available):
C++:  XX%
Java: YY%
ObjC: ZZ%
=========================================================================
[1    ] Data structures benchmarks
Average performance for this section (if comparison was available):
-------------------------------------------------------------------
C++:  XX%
Java: YY%
ObjC: ZZ%
[...]



Sven C. Koehler 2002-08-29