OO Bench - A Brief Description
Sven C. Köhler
There are lies, damned lies, and benchmarks.
History |
Date |
Author |
Description |
2002-01-07 |
Sven C. Köhler |
Initialization |
2002-08-25 |
Sven C. Köhler |
Preparation for 0.5.0 |
OO-Bench compares the speed of the same object-oriented tasks in several
object-oriented languages. It also has a statistics tool (written
in Java), which can be used to easily compare the speed of the several
versions of a given benchmark.
So far supported languages are Java, C++, and Objective-C.
Also take a look at the {Java,C++,Objective-C}/README files.
The directories for CSharp, Eiffel, Smalltalk, and CLOS are already
there, but the benchmarks are currently in development.
- Simple, easy to understand, and easy to port.
- Follows the idioms and best practices advised by each language as
much as possible
- Designed to make it easy to look up how a particular problem is
best solved in another language
- As the motto suggests, the focus is on source code rather than on
benchmark results. This is even more true, since intensive reviews of the
benchmarks are yet due.
To be compatible with Java and Windows, only the actual execution time is
measured (Not the user time of the process). So try to keep your system
as Idle as possible, and avoid swapping (On UNIX: Watch the pi and po
columns in vmstat during execution of a benchmark).
The benchmarks for C++ and Objective-C use sbrk() to monitor heap usage,
that is not supported on all platforms (So you might see no or
inappropriate results for memory allocation). Also, heaps don't shrink
on most systems, so you won't see memory freed. Therefore, memory usage
does currently not influence the results produced by the statistics tool.
OO Bench should work at least with the following compilers, and libraries :
- C++: GCC-2.95.2 (GCC-3 should work too), to build GNU-Make is needed;
- ObjC: GCC-2.95.2 with libFoundation, to build GNU-Make is needed;
- Java: JDK-1.3, to build GNU-Make, ant, and Perl is needed.
Usually, one does the following:
[oobench-X.Y.Z] $ gmake # Configure, and compile all files.
[oobench-X.Y.Z] $ gmake bench # Run all benchmarks. Results are
# logged to benchResult.txt.
If you just want to test, whether the benchmarks were compiled correctly,
you can also use the test target:
[oobench-X.Y.Z] $ gmake test # Run all benchmarks with minimal
# values (much faster than the bench
# target). Results are logged to
# testResult.txt.
The bench and test targets can be used in any subdirectory of oobench,
hence you can also do this:
[oobench-X.Y.Z/Java] $ gmake bench # Run all benchmarks below Java
# directory. Results are not
# logged.
If you want only to execute a certain benchmark, you can do it that way:
[oobench-X.Y.Z/C++/Patterns/MVC] $ gmake bench # Run MVC benchmark only.
# Results are not logged.
USAGE: ./tools/Statistics/statistics <ID file> <log file[,tag]>
[<log file[,tag]> ...]
The IDs file, which is located in the the oobench-X.Y.Z directory,
contains a short description of each benchmark topic, which is then used by
the statistics program when generating the report .
This can be used, if you want to compare several bench results, that use
i. e. the same language, or cannot be distinguished for some other reason.
Example: You have one benchResult.txt that used JDK-1.3, and one
that used JDK-1.4. In this case, you can use the ",tag" notation, so the
statistics program can distinguish the two set of results.
[oobench-X.Y.Z] $ ./tools/Statistics/statistics IDs benchResult.txt
Output looks like this:
#
# Output generated by OO Bench's statistics program.
# See http://oobench.sourceforge.net/ for more information
# about OO Bench.
#
# Command was: ./tools/Statistics/statistics IDs benchResult.linux.txt
#
# Note: Performance is measured only by the speed of execution.
# Estimated times are not included in statistics as well.
=========================================================================
Total average performance (if comparison was available):
C++: XX%
Java: YY%
ObjC: ZZ%
=========================================================================
[1 ] Data structures benchmarks
Average performance for this section (if comparison was available):
-------------------------------------------------------------------
C++: XX%
Java: YY%
ObjC: ZZ%
[...]
Sven C. Koehler
2002-08-29
|