Add support for C sources and headers to dub

[cschlote]

Add changes to support C source and header files with Dub

• This is a work-in-project PR as basis for discussions on scope changes and impacts involved with adding C source and header file support to dub.
• The goal is to be able to use C files (.c, .h and .i) in Dub projects the same way as usual D files (.d and .di). Dub shall be able to pick up these files automatically and pass them to the D compiler.
• We need some switch to enable the support in JSON/SDL files (backward compatibility, test staging). For this 'cSourcePaths' and 'cImportPaths' were added. If specified, dub will pick up C source and pass extra paths to the compiler.
• We need some way to detect existing ImportC support in D compilers (separate issue, WIP)
• We must generalize some code by using isSourceFile(), isHeaderFile() and isImportCFile() instead of just checking for ".d" extension. (separate issue, WIP)

[rikkimax]

This is going to need tests before going in.

However, I have no idea how that can happen without a build of at least one compiler with the preprocessor support working in the CI.

[cschlote]

Yes for sure it needs testing in the CI. But nobody is hindering us to work out things in advance. In the meantime we can use and test the patched dub with our own projects.

I will continue with this patch stack, rebase it from time to time.

@rikkimax Is there already some code to determine the compiler version / D support level in dub? if we implement that first, we can test the patched dub in the CI - it should disable ImportC support in this case as some kind of negative test. And when the next DMD/LDC/GDC is released with ImportC support, it should switch to ImportC support.

[rikkimax]

@rikkimax Is there already some code to determine the compiler version / D support level in dub?

Yes.

https://github.com/dlang/dub/blob/master/source/dub/compilers/compiler.d#L150

https://github.com/dlang/dub/blob/master/source/dub/platform.d#L252

(note ignore the determine platform stuff in that module, as that only applies to the dub executable).

There is code to compare compiler versions, however, you'll need to dig a bit into it I think.

dub/source/dub/recipe/packagerecipe.d

Line 607 in a284d97

package(dub) void checkPlatform(const scope ref ToolchainRequirements tr, BuildPlatform platform, string package_name)

[nordlow]

Replace with

if (!buildsettings.sourceFiles.any!(f => f.endsWith(".d", ".c")))

. startsWith and endsWith can take any (N >= 1) number of explicit needle parameters, where N is known at compile-time.

[WebFreak001]

can we move the list of hardcoded file endings somewhere else entirely? It's repeated many times in the code.

[cschlote]

I tested the current state of the PR with many of my projects and dub packages. I found other bugs, but nothing related to this PR.
I created some test code for ImportC here: https://gitlab.vahanus.net/dlang/examples/importc-tests.git

Although I agree, that there is much more potential to refactor the dub sources, I'm also a friend of agile development moving in small steps to a final goal. As this PR adds initial support for Import C with the next upcoming DMD release, we should focus on this current scope and proceed with other refactoring and improvements in other PRs. I would be nice to have working support for ImportC in dub before the next DMD release.

[ibuclaw]

The goal is to be able to use C file (.c, .h and .i) in Dub projects the same way as usual D files (.d and .di). Dub shall be able to pick up these files automatically and pass them to the D compiler.

FYI the GDC compiler has been able to compile C/C++/Go/Fortran/Rust... sources since 2004. It's all delegated by the D compiler driver when you invoke it normally: gdc file1.d file2.cpp file3.go file4.rs -o combined.exe.

Calling the gate haveImportCSupport is could be misleading, as don't really need importC in order for the D compiler to compile foreign language sources.

[redthing1]

I tried the PR and seems to work quite well! Hope to see merge soon.

[cschlote]

The goal is to be able to use C file (.c, .h and .i) in Dub projects the same way as usual D files (.d and .di). Dub shall be able to pick up these files automatically and pass them to the D compiler.

FYI the GDC compiler has been able to compile C/C++/Go/Fortran/Rust... sources since 2004. It's all delegated by the D compiler driver when you invoke it normally: gdc file1.d file2.cpp file3.go file4.rs -o combined.exe.

Calling the gate haveImportCSupport is could be misleading, as don't really need importC in order for the D compiler to compile foreign language sources.

Interesting. I have never tried to feed other language sources to GDC, yet.

But the goal of this PR is to teach 'dub' to pass C files to the DMD, LDC2 and GDC compilers. Otherwise no code is generated and the linker will emit errors on the missing stuff. So although GDC might be able to compile these files without any addition effort, without this PR the 'dub' project won't build as expected.

[WebFreak001]

maybe instead of trying to apply this to sourcePaths/sourceFiles, should we add a new key to the dub.json/sdl like "cPaths" or "sourceCPaths" or "cSourcePaths"?

Or a more generic "foreignSourcePaths"?

For C paths we could maybe also consider implementing ctod or similar in DUB, so compilers without importC will work as well, or if you want to manually enable it because of importC restrictions.

Advantages: we don't suddenly start compiling C source files inside D projects, which might have been there incrementally while porting to D. (a case Walter brings up often)

Additionally tooling can recognize when people want to explicitly enable ImportC.

[cschlote]

maybe instead of trying to apply this to sourcePaths/sourceFiles, should we add a new key to the dub.json/sdl like "cPaths" or "sourceCPaths" or "cSourcePaths"?
Or a more generic "foreignSourcePaths"?

I kept my C/foreign in separate directories and used pre-build/gen Hooks to convert them to object files. The object ist then added to the linker flags.

Starting with ImportC I tend to add my C sources inside of the usuall D directories. Simply because C is then no longer foreign. For more complex C projects, which can't be compiled with ImportC, I would use external hooks and linklibs as before.

For C paths we could maybe also consider implementing ctod or similar in DUB, so compilers without importC will work as well, or if you want to manually enable it because of importC restrictions.

I wouldn't mix up things here. We have pre Hooks to handle the complicated stuff, or Import for the simpler code. CtoD never worked for me, because most old C code uses the PrePro for funny things.

Advantages: we don't suddenly start compiling C source files inside D projects, which might have been there incrementally while porting to D. (a case Walter brings up often)

Ok, that would be a point. A switch like 'enable-importc' in the dub file would be a good workaround. if enabled, dub would pass C files to the compiler silently. Otherwise I will gibe a warning instead that C files are found, but not yet sent to the compiler until the 'enable-importc' property ist set. Later this could be removed when most Dub projects are adapted.

Additionally tooling can recognize when people want to explicitly enable ImportC.

How?

[cschlote]

One of my own old dub projects from 2019 uses C files, which are compiled to object files. This PR passes the C file now directly to the D compiler. Unfortunatelly this doesn't work. So we need a configuration switch/option as proposed by @WebFreak001 .

[cschlote]

I added new keyword to dub.json/sdl: 'cSourcePaths' and 'cImportPaths'
They allow to pickup C sources and headers in the given paths.
Tested with https://gitlab.vahanus.net/dlang/examples/importc-tests.git (for 'cSourcePaths').
For 'cImportPaths' more testing and likely some changes are needed.
Please review and comment.

[WebFreak001]

also add your test project to the tests folder

[WebFreak001]

please don't force push while still reviewing, it makes it hard to understand what has changed, especially on a change with this many changed lines of code.

I think for this we will need to squash it all at the end anyway, so we can do this after the reviews pass I think.

[cschlote]

please don't force push while still reviewing, it makes it hard to understand what has changed, especially on a change with this many changed lines of code.

I think for this we will need to squash it all at the end anyway, so we can do this after the reviews pass I think.

Ok, I tend to rebase my changes to the latest master, especially when other changes happen on master, which conflict with the PR. So I can resolve them now. I added the requested changes as commits to the current patch stack.

And yes, I will rework the patch stack at the end and split up the changes into reasonable commits. Some functions in utils are currently unused - probably we want them in a separate PR and with active use in the source.

[cschlote]

Cleanup up the patch stack. I re-submitted the PR as #2544.