Compiler Design Lab

CGO 2015 Artifact Evaluation

A Graph-Based Higher-Order Intermediate Representation

Artifact

The CGO 2015 paper A Graph-Based Higher-Order Intermediate Representation was successfully evaluated during Artifact Evaluation. This website contains detailed instructions how to reproduce the numbers presented in the paper. The artifact can either be tested via

VirtualBox Image

VirtualBox is a virtualization software package for x86 and AMD64/Intel64-based computers. Our pre-configured image contains the source code and the required compilers in the proper version for the described benchmarks from the paper (see Section 6: Implementation and Evaluation):

    • Thorin (branch cgo): https://github.com/AnyDSL/thorin
    • Impala (branch cgo): https://github.com/AnyDSL/impala
  • Download the VirtualBox Appliance
  • Import the Appliance via File->Import Appliance
  • Login to the machine via user impala and password impala (root password is also impala)
  • Execute the evaluation script evaluate.sh by typing
    ./evaluate.sh
    The evaluation is started upon execution of the evaluation script.
    Note: Running the benchmark script may take up to two hours or longer.

Build Instructions for a Manual Build

Disclaimer: These instructions worked around 2015. You will need an old system to compile the old sources. On new systems you might run into problems. We highly recommend our pre-configured VirtualBox image for modern systems. 

mkdir cgo
cd cgo
export CUR=`pwd`
wget http://llvm.org/releases/3.4.2/llvm-3.4.2.src.tar.gz
tar xf llvm-3.4.2.src.tar.gz
mv llvm-3.4.2.src llvm
cd llvm/tools
wget http://llvm.org/releases/3.4.2/cfe-3.4.2.src.tar.gz
tar xf cfe-3.4.2.src.tar.gz
mv cfe-3.4.2.src clang
cd ${CUR}
git clone https://github.com/richardmembarth/c3ms.git -b cgo15
git clone https://github.com/AnyDSL/thorin.git -b cgo15
git clone https://github.com/AnyDSL/impala.git -b cgo15
git clone https://github.com/AnyDSL/benchmarks-impala.git -b cgo15
mkdir llvm_build
mkdir llvm_install
mkdir thorin/build
mkdir impala/build
cd llvm_build
cmake ../llvm -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=${CUR}/llvm_install
make install -j4 # use appropriate number of threads here
cd ${CUR}/c3ms
make
cd ${CUR}/thorin/build
cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_DIR=${CUR}/llvm_install/share/llvm/cmake
make -j4 # use appropriate number of threads here
cd ${CUR}/impala/build
cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_DIR=${CUR}/llvm_install/share/llvm/cmake -DTHORIN_DIR=${CUR}/thorin
make -j4 # use appropriate number of threads here
export PATH=${CUR}/llvm_install/bin:${CUR}/impala/build/bin:$PATH

Testing the Compiler

Using the instructions above the impala compiler should now be in your path. Invoking impala --help shows available options. As impala currently does not ship with a standard library, impala emits LLVM files. We link via clang small C-wrappers to communicate with the outside world and to build the final executable. So let's compile hello_world.impala: 

cd ${CUR}/impala/test
clang infrastructure/call_impala_main.c -O2 -c      # compile wrappers
impala --emit-llvm -O2 codegen/hello_world.impala   # produce hello_world.bc
clang hello_world.bc call_impala_main.o             # link wrapper and hello_world.bc to executable
./a.out

Running the benchmark script

Please, double-check that the impala compiler (see above) as well as your freshly built clang binaries are in your path: 

which impala
which clang
This should point to ${CUR}/impala/build/bin/impala and ${CUR}/llvm_install/bin/clang, respectively. Also, you need a recent version of ghc and rust version 0.11 installed. If you don't want to install these compilers you can just benchmark vs C. Simply comment the relevant part in the ${CUR}/benchmarks-impala/run.sh script (lines 28-32 for rust and lines 34-40 for ghc). Finally, let's run the actual benchmark script: 
cd ${CUR}/benchmarks-impala
./evaluate.sh

Note: Running the benchmark script may take up to two hours or longer.

Information about the Benchmarks

The benchmarks are located in the benchmarks_impala directory.
Different files extensions are used for the different languages.
The script evaluates the languages of every benchmark in the following order:

  • C
  • Haskell
  • Impala (our implementation)
  • Rust

Note:

  • The printed performance numbers measure the time in kiloseconds (1000 seconds)
  • The benchmarks can be found in the repository (branch cgo): git@github.com:AnyDSL/benchmarks-impala.git

Information about the Script

  • Initially, the executables of the different benchmarks are built using the compilers described in the paper
  • After compilation, the median execution times of the different benchmarks are computed and printed on the command line (median times are printed in green)
  • In the second step, the Halstead numbers are computed and printed on the command line
For streaming the results into a file use: 
./evaluate.sh > out

Information about the computed and published Numbers

The computed performance numbers might by different form the numbers in the paper. This is typically caused by your hardware setup which might be different from the one used in the paper.

Publication

Conferences

  • A Graph-Based Higher-Order Intermediate Representation - CGO 2015
    (2nd place: Artifact Evaluation for CGO/PPoPP'15)
    Leißa, R., Köster, M. and Hack, S.
    International Symposium on Code Generation and Optimization, 2015. [doi] [url] [pdf] [bib]

People