feat: add gevals action to github workflows #505
Conversation
Signed-off-by: Calum Murray <[email protected]>
|
@manusa this also includes all the eval scenarios we ported from kubectl-ai. I realize that makes this PR quite large, so if you prefer I can split this one up |
|
@manusa @nader-ziada a sample run (which failed, due to not enough tasks succeeding to pass the threshold) can be seen here: https://github.com/Cali0707/kubernetes-mcp-server/actions/runs/19628912129 |
|
Note that this will need a few repository secrets to be set before this can be merged:
As we move to this running a matrix of models and/or having maintainers trigger this with specific models in comments, we will need more secrets and need to figure out naming conventions for those |
| Both examples should produce: | ||
| - ✅ Task passed - pod created successfully | ||
| - ✅ Assertions passed - appropriate tools were called | ||
| - ✅ Verification passed - pod exists and is running |
There was a problem hiding this comment.
are there more results captured?
There was a problem hiding this comment.
Yes, overtime we are working to expand the top level info exported. For now, this is what we show (along with a full trace of all the MCP messages)
For now, this is all the info shared at a top level
manusa
left a comment
manusa
left a comment
There was a problem hiding this comment.
LGTM, thx! (just a couple of minor observations)
I've added the required secrets to the repository.
I haven't checked the evals/**/* files since I think these have been directly ported from the gevals repo which were ported from kubectl-ai (#505 (comment))
| @@ -0,0 +1,35 @@ | |||
| # Gevals evaluation support | |||
There was a problem hiding this comment.
The targets in this file seem more generic than just for the gevals scenario.
Maybe we want to rename the mk file to something else.
| # Check if commenter is a maintainer (has write access) | ||
| PERMISSION=$(curl -s -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \ | ||
| "https://api.github.com/repos/${{ github.repository }}/collaborators/${{ github.event.comment.user.login }}/permission" \ |
There was a problem hiding this comment.
@manusa here is where we check if the person has write access to help address the concern
3 participants
This PR adds the gevals action to run in github workflows. There are a few ways to trigger this:
/run-gevals)I will do future follow ups to allow us to: