rust: generate target specification files on the fly #694

ojeda · 2022-03-03T22:51:46Z

Now that we have host programs support, add a "Rust script"
to generate the rustc target spec file.

The script has been written so that its output matches exactly
the current arch/*/rust/*.json static target files we have
at the moment, to have a baseline and reduce the changes done
by this commit. Thus the static files are now gone.

The build system is set up in a way that we do not need to rebuild
any Rust code as long as the target.json contents do not change,
even if the kernel configuration changes for something else.
Furthermore, the script does not need to be rebuilt even if
the kernel configuration changes.

With this, KBUILD_RUST_TARGET is also gone -- one should modify
the script instead of trying to override the target file.

After this, the script should be cleaned (e.g. remove unneeded
keys) and fixed to configure the architectures properly (e.g.
enable CPU features as needed, etc.).

There were several design choices:

The language: this could have been done in a shell script,
C or Perl. But as soon as Rust is an option, it is more
convenient than any of those. And it is an ideal showcase for
the Rust host programs support.

Python could have also been a natural option for the script,
though it cannot be used as part as the kernel compilation
because it is not required (as far as I can see in the docs).
Even if it were, Rust offers some advantages anyway (though
the json module would have been fairly useful here).

Compiling the Rust host program takes a lot of time compared
to other languages, at least currently (e.g. 300ms), but there
is no need to recompile it on config changes, so it is fine.
And, of course, running it is very fast.
Whether or not to use conditional compilation.

Initially I took advantage of the rustc_cfg flags, to make
the script look closer to the actual Rust kernel code.

However, that means the script depends on the config, which
means it is not an independent tool anymore (though there is
at least one other host program that gets passed the flags,
but it is not common), plus requires recompiling it every
time the configuration changes just to generate the actual
file. It also made things more tricky on the build system.
Splitting the logic into arch/* or not.

In order to have arch maintainers work independently, we could
move part of the logic into their trees.

However, given how small the script is and that some logic
is shared anyway, it looks a little bit odd. So I decided
to avoid it. We can revisit this if needed.
Whether to use a proper JSON generator or not, whether to
use a HashMap, whether to set up a type that knows about
all the keys that can be generated (or maybe even import
part of the rustc code to be in sync), etc.

The script has been written to be as simple as possible
while making it easy to change the actual business logic.
We should reevaluate as it gets developed.

Signed-off-by: Miguel Ojeda [email protected]

Now that we have host programs support, add a "Rust script" to generate the `rustc` target spec file. The script has been written so that its output matches exactly the current `arch/*/rust/*.json` static target files we have at the moment, to have a baseline and reduce the changes done by this commit. Thus the static files are now gone. The build system is set up in a way that we do not need to rebuild any Rust code as long as the `target.json` contents do not change, even if the kernel configuration changes for something else. Furthermore, the script does not need to be rebuilt even if the kernel configuration changes. With this, `KBUILD_RUST_TARGET` is also gone -- one should modify the script instead of trying to override the target file. After this, the script should be cleaned (e.g. remove unneeded keys) and fixed to configure the architectures properly (e.g. enable CPU features as needed, etc.). There were several design choices: - The language: this could have been done in a shell script, C or Perl. But as soon as Rust is an option, it is more convenient than any of those. And it is an ideal showcase for the Rust host programs support. Python could have also been a natural option for the script, though it cannot be used as part as the kernel compilation because it is not required (as far as I can see in the docs). Even if it were, Rust offers some advantages anyway (though the `json` module would have been fairly useful here). Compiling the Rust host program takes a lot of time compared to other languages, at least currently (e.g. 300ms), but there is no need to recompile it on config changes, so it is fine. And, of course, running it is very fast. - Whether or not to use conditional compilation. Initially I took advantage of the `rustc_cfg` flags, to make the script look closer to the actual Rust kernel code. However, that means the script depends on the config, which means it is not an independent tool anymore (though there is at least one other host program that gets passed the flags, but it is not common), plus requires recompiling it every time the configuration changes just to generate the actual file. It also made things more tricky on the build system. - Splitting the logic into `arch/*` or not. In order to have arch maintainers work independently, we could move part of the logic into their trees. However, given how small the script is and that some logic is shared anyway, it looks a little bit odd. So I decided to avoid it. We can revisit this if needed. - Whether to use a proper JSON generator or not, whether to use a `HashMap`, whether to set up a type that knows about all the keys that can be generated (or maybe even import part of the `rustc` code to be in sync), etc. The script has been written to be as simple as possible while making it easy to change the actual business logic. We should reevaluate as it gets developed. Signed-off-by: Miguel Ojeda <[email protected]>

bjorn3 · 2022-03-03T23:23:38Z

I will take a look tomorrow.

ojeda · 2022-03-04T00:07:57Z

Thanks a lot! No rush.

bjorn3 · 2022-03-04T08:48:47Z