Blog

Customer needs to automate code statical analysis into the integration workflow using a SAST tool.

The detailed task specification comes from ChatGPT, suggesting semgrep ran from a docker image semgrep/semgrep.

I thought it wasn’t a bad idea. ChatGPT suggested to integrate the tool as a pre-commit git hook, that is fair for local development, but not for automated continuous integration. Ok, a pretty useless suggestion, but something to start from.

Ask to LLM Agent

The tools adopted for CI/CD is Jenkins, so I start interacting with LLM to suggest a groovy pipeline to integrate semgrep/semgrep for code analysis.

I forgot to state to LLM engine (Claude) that Jenkins was ran as container, managed by a docker swarm instance, running into a VM separated from production (since docker swarm does not have namespace, I fully agree with this arrangement).

Code generated rely on separate script involving the execution of

docker run -v $jenkinsworkspace:/code semgrep/semgrep ….

Thats easy and cool. But wait, Jenkins ran in a docker, the agent used has the docker command, but it connect to the host /var/docker.socket

This means that there is only a docker daemon on the host. When that docker daemon receive a command from docker CLI, it take argument as it is: mounting a volume from a path means just mounting the host’s path, because it is the only path known by docker daemon.

The script generated by the LLM try to mount a container’s path to be read by a docker daemon. I feel lazy, and I want to let Claude agent fix the code for me.

system architecture and pipeline design

In details, the system architecture and pipeline for backend code (with or without SAST) was designed to do this steps:

1. build a docker image of the service
2. run unit tests and integration tests on the newly created docker image (using docker-compose for side services)
3. (deploy by) push the docker image into a private repository
4. Update the service into production or dev environment

The first prompt I gave to the agent was to read the existing Jenkinsfile and integrate a SAST step.

Please Claude, fix it

I stated the problem, and suggested to use a volume.

The agent suggested code:

1. creates a volume
2. extracting code from image into the new volume
3. extract code from volume into the container filesystem
4. remove the volume
5. create a new volume for analysing the code
6. copy code from container filesystem into a new volume
7. analyse the code
8. remove the new volume

Isn’t here some repeated staff? The agent say no.

This story go on for half an hour…

The point was that the code was arranged in 2 function (bash function), and the agent treat those as silos.

I didn’t want to waste my time explaining things to the agent, so I refactor the code my myself.

I started with the idea to do staff in lazy mode, I ended with fighting the rigidity of AI Agent way of solving staff by adding operations.

1. define unique volume name (using jenkins job number)
2. create the volume
3. extract code from image into the volume
4. analyse the code contained into the volume
5. remove the volume

But I have to use my hand to arrange code this way.

I also must say that code generated by agents are full of checks, some of those are clever and nice to have, some are paranoid driven. So the job is to remove extra-code.

The integrated SAST step in the pipeline

At the end the Jenkins pipeline for the job build new artifacts: SAST reports.

Newly generated artifacts can be inspected to fix the code and release a more secure code.

The step defined as:

        stage('SAST Security Scan') {
            steps {
                script {
                    echo "🔒 Starting SAST scan for JavaScript/Node.js application"
                    
                    // Extract and scan from built Docker image (code only exists in image)
                    echo "🐳 Extracting and scanning code from Docker image: ${env.LOCTAG}"
                    def sastExitCode = sh(
                        script: "./scripts/sast-scan-image.sh '${env.LOCTAG}'",
                        returnStatus: true
                    )
                    
                    // Read results summary - check both locations
                    def summaryContent = ""
                    if (fileExists('sast-summary.txt')) {
                        summaryContent = readFile('sast-summary.txt').trim()
                    } else if (fileExists('/tmp/sast-summary.txt')) {
                        summaryContent = readFile('/tmp/sast-summary.txt').trim()
                    }
                    
                    if (summaryContent) {
                        env.SAST_RESULTS = summaryContent
                        echo "SAST Results: ${env.SAST_RESULTS}"
                        
                        // Parse results for detailed logging
                        def results = env.SAST_RESULTS.split(',')
                        def highIssues = results[0].split(':')[1] as Integer
                        def mediumIssues = results[1].split(':')[1] as Integer
                        def lowIssues = results[2].split(':')[1] as Integer
                        
                        echo """
🔍 SAST Scan Summary:
   🔴 High Severity: ${highIssues}
   🟡 Medium Severity: ${mediumIssues}  
   🟢 Low Severity: ${lowIssues}
                        """
                    } else {
                        echo "⚠️ No SAST summary found - assuming no issues"
                        env.SAST_RESULTS = "HIGH:0,MEDIUM:0,LOW:0"
                    }
                    
                    // Set build status based on SAST results
                    if (sastExitCode == 2) {
                        currentBuild.result = 'FAILURE'
                        error("❌ SAST scan failed due to high severity security issues")
                    } else if (sastExitCode == 1) {
                        currentBuild.result = 'UNSTABLE'
                        echo "⚠️ SAST scan marked build as unstable due to medium severity issues"
                    } else {
                        echo "✅ SAST scan passed successfully"
                    }
                }
            }
            post {
                always {
                    // Copy SAST results from /tmp if they exist there
                    sh '''
                        # Copy results from /tmp to workspace for archiving
                        cp /tmp/semgrep-*.json . 2>/dev/null || true
                        cp /tmp/semgrep-*.txt . 2>/dev/null || true  
                        cp /tmp/sast-*.txt . 2>/dev/null || true
                    '''
                    
                    // Archive all SAST results
                    archiveArtifacts artifacts: 'semgrep-*.json, semgrep-*.txt, sast-*.txt', 
                                   fingerprint: true, 
                                   allowEmptyArchive: true
                    
                    // Display scan results in build description
                    script {
                        if (env.SAST_RESULTS) {
                            def results = env.SAST_RESULTS.split(',')
                            def highIssues = results[0].split(':')[1]
                            def mediumIssues = results[1].split(':')[1]
                            def lowIssues = results[2].split(':')[1]
                            
                            currentBuild.description = """
SAST: H:${highIssues} M:${mediumIssues} L:${lowIssues}
                            """.trim()
                        }
                    }
                }
                failure {
                    echo '❌ SAST scan failed - check security findings before proceeding'
                }
                unstable {
                    echo '⚠️ SAST scan found medium severity issues - review before deployment'
                }
                success {
                    echo '✅ SAST scan completed successfully'
                }
            }
        }

here archiveArtifacts artifacts is the way to index the artifacts listed on top of Jenkins interface

Is agent mode good or bad idea on coding?

I am still reluctant to adopt agent mode. Someway it suggest good idea, but someway it use those idea in awful way.

And worst of all, it keeps saying “you are perfectly right”, while it finds another idiot way of producing unnecessary code.

I think that behind LLM usage there is an unsaid interest conflict:

The more the agent interact and create crufty code, the more token are consumed, the more fee are charged.

This is not about making things done, it is about give away your money thinking you find the cheaper developer in the market: the LLM Agent.

But someway it helps to know staff. My orientation is to use a mix of agent mode, and internal prompt, fixing staff by hand when it is a matter of refactoring, or clearly crufty code. I prefer to think to LLM as an useful stochastic parrot.

My services