Don’t parse the pipeline as text when it is directed from an EXE to another EXE or file. Keep the bytes as-is.

[be5invis]

Currently PowerShell parses STDOUT as string when piping from an EXE, while in some cases it should be preserved as a byte stream, like this scenario:

curl.exe http://whatever/a.png > a.png

or

node a.js | gzip -c > out.gz

Affected patterns include: native | native, native > file and (maybe) cat file | native.

[be5invis]

@vors @lzybkr
The current NativeCommandProcessor breaks:

• LF line endings.
• Non-ASCII text within UTF-8 without BOM header.
• Binary file redirects (like curl.exe’s output).
• > layouts text into 80 columns by default.

[ForNeVeR]

Maybe add a cmdlet/operator to call native command and get its raw output (as a byte array / stream?), something like this:

# Consider ^& operator is an alias for Get-CommandRawOutputStream; this is just an example syntax
$output = ^& curl.exe http://whatever/a.png # $output now is a byte array or stream
$output > C:\Temp\file.png # file.png now is a valid image file

# This should be valid, too:
^& curl.exe http://whatever/a.png > C:\Temp\file.png

This opens an opportunity for some additional usage patterns (you can put this raw content into variables, and pipe raw content from native commands to managed cmdlets).

[ForNeVeR]

But maybe we could add a special kind of redirection operator (like 2>&1, 3>&1, *>&1 we already have), something like this (where %>&1 is a new redirection operator that redirects command "raw output" without processing it as a string):

$output = curl.exe http://whatever/a.png %>&1
$output > C:\Temp\file.png

# Or even this:
curl.exe http://whatever/a.png %> C:\Temp\file.png # which is just awesome

Overall: I don't think that this kind of redirection should be tied to only native commands or some limited list of usage patterns (e.g. native | native).

[be5invis]

@ForNeVeR My proposal is that:

1. For native | native, keep the bytes as-is. This is already purposed by @vors.
2. For ps | native, add a set of cmdlets which encodes PS objects into bytes, perhaps ps | encode-text utf-8 | native.
3. For native | ps, we can use the type system to identify whether a cmdlet accepts “raw input”. For cmdlets like out-file or maybe decode-text, it will keep the bytes from native, and other cmdlets will use the parsed string as its input.

[ForNeVeR]

@be5invis okay, it seems like this proposal also supports all the relevant use cases I can imagine.

[GeeLaw]

Shouldn't this open up an RFC since this is a breaking change (changes the observed behaviour)?

A workaround for this is to provide a cmdlet that stores the content in a temporary file. A working example is Use-RawPipeline in PowerShell Gallery. The current implementation is to store the file, but it could also be streamlined so that the file doesn't have to be stored.

[jhclark]

See also #559, where this appears to be actively discussed and worked on by @vors on the PowerShell team.