Formatting code is in conflict with unnamed_addr

[RalfJung]

The code for passing arguments into format strings relies on equality of function pointers. This is in conflict with the fact that we set unnamed_addr on all functions.

Global variables can be marked with unnamed_addr which indicates that the address is not significant, only the content. Constants marked like this can be merged with other constants if they have the same initializer.

rust/src/libcore/fmt/mod.rs

Lines 295 to 301 in 618f5a0

	fn as_usize(&self) -> Option<usize> {
	if self.formatter as usize == ArgumentV1::show_usize as usize {
	Some(unsafe { *(self.value as *const _ as *const usize) })
	} else {
	None
	}
	}

It might be possible for a user to write a function that gets merged with ArgumentV1::show_usize, and then cause bugs in the above code.

@oli-obk @eddyb

[oli-obk]

These uses are impossible to abuse for users (this is an Argument internal API that is only used if it's a usize anyway). The invariants that make this possible are very subtle though, and we should totally document this better

[RalfJung]

The invariants that make this possible are very subtle though, and we should totally document this better

I wonder which invariant you had in mind here?

[oli-obk]

We are only using the as_usize method in getcount, which is only used for the width and the precision fields. These must be usize anyway, so the as_usize function essentially double checks whether we are doing this for usize.

[RalfJung]

So you are saying format_args! actually ensures that width/precision fields are always created with from_usize?

[oli-obk]

That's my reading of the code, but I might have missed some stuff. The formatting code is very... frameworky

[eddyb]

Yeah, from_usize is used by format_args! in all cases as_usize will be called, the behavior of the latter for non-usize formatting arguments is impossible to observe without going through the unstable API.

But maybe we should benchmark (cc @anp) the cost of using an enum that relies on the null pointer optimization, i.e. Option where Some(fn_ptr) indicates formatting-only, while None indicates usize, which can be formatted (or not?), but also extracted.

[eddyb]

What if... we actually called the function?! And there was a Formatter field that the fmt::Display impl for usize would fill in if a flag was set, and return without printing anything?