Adding buildroot packages (would like to eventually add SPEC 2017)

[andrewzigerelli]

Steps so far:
made directory in buildroot_packages/spec2017

In there, made Config.in:

config BR2_PACKAGE_SPEC2017_BENCHMARK
	bool "SPEC2017_BENCHMARK"
	help
	  SPEC CPU 2017 Benchmark suite.

	  https://www.spec.org/cpu2017

if BR2_PACKAGE_SPEC2017_BENCHMARK

config BR2_PACKAGE_SPEC2017_BENCHMARK_BUILD_LIST
	string "build_list"
	default 500.perlbench_r
	help
	  Space separated list of parsec packages to build.

endif

external.desc, which is just a description, and

and external.mk

################################################################################
#
# SPEC2017_BENCHMARK
#
################################################################################

SPEC2017_BENCHMARK_VERSION = 1.0
SPEC2017_BENCHMARK_SITE = $(LKMC_SPEC2017_BENCHMARK_SRCDIR)
SPEC2017_BENCHMARK_SITE_METHOD = local

define SPEC2017_BENCHMARK_BUILD_CMDS
  # TODO make this nicer, only untar when extract step is done.
  # EXTRACT_CMDS and EXTRA_DOWNLOADS would be good candidates,
  # but they don't run with OVERRIDE_SRCDIR.
  # '$(@D)/get-inputs' $(if $(filter $(V),1),-v,) '$(DL_DIR)' '$(@D)/'
  # We run the benchmarks with the wrong arch here to generate the inputs on the host.
  # This is because on gem5 this takes too long to do.
  export GNU_TARGET_NAME='$(GNU_TARGET_NAME)' && \
  export GNU_HOST_NAME='$(GNU_HOST_NAME)' && \
  export HOSTCC='$(HOSTCC)' && \
  export M4='$(HOST_DIR)/usr/bin/m4' && \
  export MAKE='$(MAKE)' && \
  export OSTYPE=linux && \
  export TARGET_CROSS='$(TARGET_CROSS)' && \
  export CC="$(TARGET_CC)" && \
  export CPP="$(TARGET_CPP)" && \
  export CXX="$(TARGET_CXX)" && \
  export HOSTTYPE=$(BR2_ARCH) && \
  cd $(@D) && \
  . shrc && \
  for pkg in $(BR2_PACKAGE_SPEC2017_BENCHMARK_BUILD_LIST); do \
    runcpu --config=myconf.cfg --action=buildsetup $$pkg && \
    cd $(@D)/benchspec/CPU/$$pkg/build/build_base_myconf-m64.000 & \
    make -j 20 & \
    runcpu --config=myconf.cfg --action=runsetup --size ref --threads 1 $$pkg \
  done
endef

define SPEC2017_BENCHMARK_INSTALL_TARGET_CMDS
  mkdir -p '$(TARGET_DIR)/spec2017/'
  for pkg in $(BR2_PACKAGE_SPEC2017_BENCHMARK_BUILD_LIST); do \
    $(INSTALL) -D -m 0755 '$(@D)/benchspec/CPU/$$pkg/run' '$(TARGET_DIR)/spec2017/'
  done
endef

$(eval $(generic-package))

Since the src is not in a submodule, I did the following:

added the spec_src dir to common.py:

diff --git a/common.py b/common.py
index db9f702..be3c381 100644
--- a/common.py
+++ b/common.py
@@ -52,6 +52,8 @@ linux_config_dir = os.path.join(common.root_dir, 'linux_config')
 rootfs_overlay_dir = os.path.join(common.root_dir, 'rootfs_overlay')
 qemu_src_dir = os.path.join(submodules_dir, 'qemu')
 parsec_benchmark_src_dir = os.path.join(submodules_dir, 'parsec-benchmark')
+home_path = os.environ['HOME']
+spec2017_benchmark_src_dir = os.path.join(home_path, 'spec2017')
 ccache_dir = os.path.join('/usr', 'lib', 'ccache')
 default_build_id = 'default'
 arch_short_to_long_dict = collections.OrderedDict([

changed build-buildroot script:

diff --git a/build-buildroot b/build-buildroot
index 0d0e9ea..9e4acd3 100755
--- a/build-buildroot
+++ b/build-buildroot
@@ -146,6 +146,7 @@ usually extra Buildroot targets.
                 'make', common.Newline,
                 'LKMC_GEM5_SRCDIR="{}"'.format(common.gem5_src_dir), common.Newline,
                 'LKMC_PARSEC_BENCHMARK_SRCDIR="{}"'.format(common.parsec_benchmark_src_dir), common.Newline,
+                'LKMC_SPEC2017_BENCHMARK_SRCDIR="{}"'.format(common.spec2017_benchmark_src_dir), common.Newline,
                 'O={}'.format(common.buildroot_build_dir), common.Newline,
                 'V={}'.format(int(args.verbose)), common.Newline,
             ] +

I rebuild with:

./build-buildroot   --arch x86_64 --config 'BR2_OPTIMIZE_3=y'   --config 'BR2_PACKAGE_SPEC2017_PACKAGE=y'   --
  sample_package-reconfigure ;

I don't see my package being compiled (and nothing copied to out/buildroot/build/default/x86_64/target'.).

What am I missing?

Also, I have just hardcoded some options for now, but once I get it working, I need to generate spec configs and such. What is the appropriate way to add this option? As a submodule and setup the config in external.mk?

I currently have a python script which generates the readfiles as well, which I can add (or change to bash).

[cirosantilli]

Edit: created a dedicated section documenting this: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/657c59249e56d861bb0a437a1b1c757797281910#add-new-files-to-the-buildroot-image

Hi Andrew,

I can only spot on thing wrong, the last command says sample_package-reconfigure, but you likely want spec2017-reconfigure?

For further configs, they will be treated exactly like other Buildroot configs and enabled with --config on lkmc. Have a look at the Buildroot manual to see how to add them, e.g. see what BR2_PACKAGE_UTIL_LINUX does: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/550897ce1766e8df0b4ffcfdc17206f788d9f67f#gdb-step-debug-multicore-userland Your example already has two configs: BR2_PACKAGE_SPEC2017_BENCHMARK and BR2_PACKAGE_SPEC2017_BENCHMARK_LIST, others will look exactly the same.