Proposal - module build & publish via the task runner and psake

[rkeithhill]

There is interest in being able to publish a module to the PowerShell Gallery from VSCode using this extension. I have been doing a fair amount of work with ASP.NET Core projects. Those projects use gulp as a task runner to perform during the "publish" of a web site. VSCode has a task runner that can be configured to run various tools including gulp. However for PowerShell module publish, I think we should use something more "PowerShell oriented". So I've been playing with PSake to build and publish modules within VSCode.

Here is the psake.ps1 file (equivalent to gulpfile.js) that would be in the workspace folder:

Properties {    
    $NugetApiKeyEnc = @'XXXX'@
    $ReleaseNotesPath = "$PSScriptRoot\ReleaseNotes.md"

    $PublishDir = "$PSScriptRoot\.publish"
    $Exclude = @(
        'psake.ps1',
        '*Tests.ps1',
        '.git*',
        '.publish',
        '.vscode'
    )
}

Task default -depends Build

Task Publish -depends Test {
    $secureString = $NugetApiKeyEnc | ConvertTo-SecureString
    $nugetApiKeyBstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secureString)
    $nugetApiKey = [Runtime.InteropServices.Marshal]::PtrToStringAuto($nugetApiKeyBstr)
    [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($nugetApiKeyBstr)

    $ReleaseNotes = Get-Content $ReleaseNotesPath -Raw
    Publish-Module -Path $PublishDir -NuGetApiKey $nugetApiKey `
                   -ReleaseNotes $ReleaseNotes -WhatIf
}

Task Test -depends Build {
    Import-Module Pester
    Invoke-Pester $PSScriptRoot
}

Task Build -depends Clean {
    Copy-Item $PSScriptRoot\* -Destination $PublishDir -Exclude $Exclude -Recurse -Container 
}

Task Clean -depends Init {
    Remove-Item $PublishDir\* -Recurse -Force
}

Task Init {
   if (!(Test-Path $PublishDir)) {
       New-Item -ItemType Directory $PublishDir
   }
}

Note that most of this file is boilerplate except for the Properties section. That is what the user will spend the most time configuring for their module. The NuGetApi key is something that should be protected and not checked in. What you see above is the output from:

$ss = Read-Host 'Enter NuGet Api Key' -AsSecureString
$ss | ConvertFrom-SecureString | Out-Clipboard

Now to get this to work from VSCode, I've modified the tasks.json file to:

// A task runner that invokes Pester to run all Pester tests under the
// current workspace folder.

// NOTE: This Pester task runner requires an updated version of Pester (>=3.4.0)
// in order for the problemMatcher to find failed test information (message, line, file).
// If you don't have that version, you can update Pester from the PSGallery like so:
//
// PS C:\> Update-Module Pester
//
// If that gives an error like:
// "Module 'Pester' was not installed by using Install-Module, so it cannot be updated."
// then execute:
//
// PS C:\> Install-Module Pester -Scope CurrentUser -Force
//

// Available variables which can be used inside of strings.
// ${workspaceRoot}: the root folder of the team
// ${file}: the current opened file
// ${fileBasename}: the current opened file's basename
// ${fileDirname}: the current opened file's dirname
// ${fileExtname}: the current opened file's extension
// ${cwd}: the current working directory of the spawned process
{
    "version": "0.1.0",

    // Start PowerShell
    "command": "${env.windir}\\sysnative\\windowspowershell\\v1.0\\PowerShell.exe",

    // The command is a shell script
    "isShellCommand": true,

    // Show the output window always
    "showOutput": "always",

    "args": [
        "-NoProfile", "-ExecutionPolicy", "Bypass"        
    ],

    // Associate with test task runner
    "tasks": [
        {
            "taskName": "Build",
            "suppressTaskName": true,
            "isBuildCommand": true,
            "args": [
                "Write-Host 'Invoking PSake...'; Invoke-PSake psake.ps1;",
                "Invoke-Command { Write-Host \"Completed Build task in task runner\" }"                
            ]
        },
        {
            "taskName": "Publish",
            "suppressTaskName": true,
            "args": [
                "Write-Host 'Invoking PSake...'; Invoke-PSake psake.ps1 Publish;",
                "Invoke-Command { Write-Host \"Completed Publish task in task runner\" }"                
            ]
        },
        {
            "taskName": "Pester",
            "suppressTaskName": true,
            "isTestCommand": true,
            "args": [
                "Write-Host 'Invoking Pester...'; Invoke-Pester -PesterOption @{IncludeVSCodeMarker=$true};",
                "Invoke-Command { Write-Host \"Completed all tests tasks in task runner\" }"
            ],
            "problemMatcher": [
                {
                    "owner": "powershell",
                    "fileLocation": ["absolute"],
                    "severity": "error",
                    "pattern": [
                        {
                            "regexp": "^\\s*(\\[-\\]\\s*.*?)(\\d+)ms\\s*$",
                            "message": 1
                        },
                        {
                            "regexp": "^\\s+at\\s+[^,]+,\\s*(.*?):\\s+line\\s+(\\d+)$",
                            "file": 1,
                            "line": 2
                        }
                    ]
                }
            ]
        }
    ]
}

The result looks like this:

Still a ways to go. The Build's copy task isn't copying the en-US dir correctly. And I have no problemMatcher for the psake output yet. Still this shows some promise.

What we still need though is a command for scaffolding a new module manifest file from the VSCode UI. And there should be a .gitignore file with .publish in it.

[daviwil]

This is really cool! I'd really prefer it if PowerShellEditorServices had built-in module publishing support but the reality is that we'd end up needing something like PSake anyway because each module has its own unique build steps. I think we could require that PSake be installed to use that command, though. Doing the initial work using VS Code tasks is helpful to learn how this should work.

I really like the idea of a .publish file though, or at least some standard naming convention for module build files. If we have well-established task names then we'll easily be able to run anyone's clean/build/publish tasks.

Now I'm curious to use PSake for Editor Services and ISE Preview ;)

Related: https://github.com/psake/psake-vscode

[KirkMunro]

I don't agree that each module has its own unique build steps. Most modules don't have build steps at all. Most of them are just unsigned script modules. Some modules have unique build steps, but the vast majority of them don't, other than perhaps signing the release.

I like the idea of simple publishing support (by simple I mean sign and publish) in PowerShellEditorServices, along with extension points where that process can do other required work if necessary to cover the unique build steps. For example, similar to $ExecutionContext.InvokeCommand.PreCommandLookupAction and $ExecutionContext.InvokeCommand.PostCommandLookupAction, it would be great if module authors could specify pre-/post- publish steps that are just PowerShell scripts that do whatever work is required to publish the module.

[daviwil]

Good point! I suppose if we had simple hooks around a general publish action then the user could use whatever they wanted (including PSake) to inject the build behavior they need for their module project.

[rkeithhill]

@KirkMunro I started out with the opinion that folks would only need to tweak these properties for their module:

Properties {    
    $NugetApiKeyEnc = @'XXXX'@
    $ProjectUri = 'http://pscx.codeplex.com/'
    $LicenseUri = 'http://pscx.codeplex.com/license'
    $IconUri = 'http://download-codeplex.sec.s-msft.com/Download?ProjectName=pscx&DownloadId=8046&Build=20941'
    $Tags = 'Utilities','Xml','Zip','Clipboard','Base64'    
    $ReleaseNotesPath = "$PSScriptRoot\ReleaseNotes.md"

    $PublishDir = "$PSScriptRoot\.publish"
    $Exclude = @(
        'psake.ps1',
        '*Tests.ps1',
        '.git*',
        '.publish',
        '.vscode'
    )
}

But then I realized that sometimes you want to publish to an internal repo. No biggie. We can use the -Repository parameter and add another property that defaults to PSGallery (if I understand how that param works - never used it before). But then what if someone wants to publish to an internal file share? Then you no longer want to use Publish-Module, you just need to use Copy-Item and most of the properties don't apply. In addition, I've seen some cases where folks extract module version numbers from other files or assemblies. So while it is mostly a standard build process it can vary and I think whatever we do it should anticipate that.

@daviwil Note that .publish is a folder in which copy the artifacts to be published. You don't want to publish directly from the workspace folder because A) it likely contains more artifacts than should be published (ie .vscode, your test files, .gitignore, etc). B) I could be wrong here but I believe the name of the folder you publish needs to be your module name and your workspace folder may or may not have that name and C) ideally you want to run your tests in your publish folder (to ensure you didn't forget a file) and then remove those tests before publish. So whatever approach we take I believe it requires copying parts of the workspace's content to a temp dir of some sort from which the publish will happen. BTW I'm not stuck on .publish perhaps the dir should be called out?

WRT how the PowerShell supports module publishing, I believe "this" is the way. I encourage folks to check out the Tasks docs here. The first two paragraphs read:

Lots of tools exist to automate tasks like building, packaging, testing or deploying software systems. Examples include Make, Ant, Gulp, Jake, Rake and MSBuild.

These tools are mostly run from the command line and automate jobs outside the inner software development loop (edit, compile, test and debug). Given their importance in the development life-cycle, it is very helpful to be able run them and analyze their results from within VS Code. Please note that task support is only available when working on a workspace folder. It is not available when editing single files.

As I mentioned web sites are using gulpfile.js to do this sort of stuff (preparing, copying, minifying, etc). I think the VSCode way is to use tasks. I just object to using gulp. :-) I think something like psake would make more sense for a PowerShell user. Unfortunately PSake is currently not on the PSGallery but @dlwyatt is on it.

BTW I'm not sure if the file should be called psake.ps1. Perhaps it should be called build.ps1 or publish.ps1?

I believe I'm going to write this up in a blog post as folks can use the existing infrastructure to do this today with the current VSCode and PowerShell extension. I'm still hoping we get better support for pre-configuring the tasks.json file. Until then, we could look into contributing a task snippet tsc - build & publish PowerShell module that folks could use. And for the build/publish ps1 file, that could be a PowerShell snippet. BTW we could also do a snippet that would be the equivalent of New-ModuleManifest. Although I would love it if VSCode supported adding template files defined by an extension i.e. right click in navigator and select Add File template -> PowerShell -> Module manifest file.

[rkeithhill]

Submitted a feature request to VSCode related to this proposal.

[rkeithhill]

Also, what do folks recommend in terms of protecting the NugetApiKey? The example above uses secure strings encrypted with DPAPI. This means that the user who created the encrypted key can put it in a file. Check it in, push to the public repo safely. However, on a multi-user project, only that user can execute the publish and only on the same machine the user created the encrypted api key on.

Is there a GitHub feature for storing secrets that can be safely accessed only by project team members and not others?

[dlwyatt]

For something built into an editor, I'd prefer to see it prompt me to enter the API key, with a checkbox to save it. How the saved value gets encrypted is entirely up to you; could just put it in $env:APPDATA\VSCode\Secrets or something, and in that case, your current DPAPI code would be fine (since $env:APPDATA is in the user's profile anyway.) The only difference is the bit where if the encrypted / saved copy isn't found, it prompts the user.

[daviwil]

I could be wrong here but I believe the name of the folder you publish needs to be your module name and your workspace folder may or may not have that name

Yep, that is true.

Although I would love it if VSCode supported adding template files defined by an extension

VS Code uses Yeoman for creating things from templates but I'm not sure if there's an option for creating a single file (as opposed to a whole "project"). I had started a generator-powershell project on the PowerShell GitHub org to get a basic PowerShell module project generator going but haven't made progress or worked on it in a while.

I like Dave's idea for storing the API key after first request. I don't think there's a GitHub feature for shared secrets but it'd be handy if they added something like that.

[rkeithhill]

I'd prefer to see it prompt me to enter the API key

@daviwil Can a VSCode extension pop a dialog box for text entry? Or are we limited to the Debug Console and Read-Host prompting at runtime?

[daviwil]

Yep, check out showInputBox on the API page. I've actually wired it up to Read-Host via a protocol message in the language service as you can see here.

[KirkMunro]

But then I realized that sometimes you want to publish to an internal repo. No biggie. We can use the -Repository parameter and add another property that defaults to PSGallery (if I understand how that param works - never used it before). But then what if someone wants to publish to an internal file share? Then you no longer want to use Publish-Module, you just need to use Copy-Item and most of the properties don't apply. In addition, I've seen some cases where folks extract module version numbers from other files or assemblies. So while it is mostly a standard build process it can vary and I whatever we do should anticipate that.

@rkeithhill I agree that the build/publish process can vary. For publication to a file share though, Publish-Module still works just fine (PowerShellGet supports using a file share as the repository -- it's incredibly easy to set up and works very well for this scenario). So the publication portion (pushing the collection of files to the repository) should remain largely the same. The being/end work may change (updating version numbers before publishing, automated tweeting after publishing, etc.), and that's where PrePublish and PostPublish actions would come in.

[KirkMunro]

+1 for David's idea for prompting for the key and storing it for each user who needs to publish.

[rkeithhill]

Yep, check out showInputBox on the API page. I've actually wired it up to Read-Host via a protocol message in the language service as you can see here.

That's not in 0.4.1 though, right? Read-Host prompts via the Debug Console (and also doesn't support -AsSecureString). :-) It's also not as obvious where you are supposed to type in the response as a dialog box would be.

So a build script could look for the api key in a set location, and if it is not found, use Read-Host to allow the user to enter the key. We could then encrypt that for storage on the user's file system so that the next time, we wouldn't need to prompt them. Of course, if they ever wanted to use a new key they would have to know how to either A) delete the encrypted key file or B) force the script to prompt for a new key.

FWIW I'd like for there to be not too much magic in how this works. That is why I like using a PSake file. While I love me some VSCode w/PowerShell there are times I do stuff from the PowerShell prompt. I'd love to be able to wander into my module workspace directory and type Invoke-PSake build.ps1 to allow me to manually test my module before publishing and Invoke-PSake build.ps1 Publish to publish it.

[rkeithhill]

WRT to prompting for the NuGet API key, what about the rest of this information:

    $ProjectUri = 'http://pscx.codeplex.com/'
    $LicenseUri = 'http://pscx.codeplex.com/license'
    $IconUri = 'http://download-codeplex.sec.s-msft.com/Download?ProjectName=pscx&DownloadId=8046&Build=20941'
    $Tags = 'Utilities','Xml','Zip','Clipboard','Base64'    
    $ReleaseNotesPath = "$PSScriptRoot\ReleaseNotes.md"

I don't think VSCode supports anything as fancy as a form, does it? If not, then would you be willing to be peppered with input dialogs for each of the above properties?