[FIXED] Why does this Jenkins build script assign the same node ID twice?

Issue

I'm using Knapsack Pro to split up unit tests and run them in parallel, and I've recently noticed that sometimes, it assigns the same node index to multiple nodes.

The build script looks like this (trimmed for brevity):

def num_nodes = 10;
def nodes = [:]

env.total_nodes = num_nodes

for (i in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) {
    int index = i
    nodes["ci_node_${i}"] = {
        node('ec2-slave') {
            ...
            env.new_index = index
            stage('Build & run rspec') {
                dir("$WORKSPACE") {
                    sh '''#!/bin/bash
                        cd $WORKSPACE
                        export NODE_ENV=production
                        export KNAPSACK_PRO_CI_NODE_TOTAL=${total_nodes}
                        export KNAPSACK_PRO_CI_NODE_BUILD_ID=${BUILD_TAG}
                        export KNAPSACK_PRO_CI_NODE_INDEX=${new_index}
                        export KNAPSACK_PRO_COMMIT_HASH=${commithash}
                        export KNAPSACK_PRO_BRANCH=${BRANCH_NAME}
                        export KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/smokes/*_spec.rb"
                        env
                        ...
                    '''
                }
            }
        }
    }
}
try {
    parallel nodes // run CI nodes in parallel
} catch (err) {
    echo "Build Failed - ${err.getMessage()}"
} finally {
    ...
}

But when I grep the logs for all the values of KNAPSACK_PRO_CI_NODE_INDEX that get printed by env, I sometimes (but not always) see something like this:

KNAPSACK_PRO_CI_NODE_INDEX=0
KNAPSACK_PRO_CI_NODE_INDEX=2
KNAPSACK_PRO_CI_NODE_INDEX=1
KNAPSACK_PRO_CI_NODE_INDEX=5
KNAPSACK_PRO_CI_NODE_INDEX=3
KNAPSACK_PRO_CI_NODE_INDEX=8
KNAPSACK_PRO_CI_NODE_INDEX=6
KNAPSACK_PRO_CI_NODE_INDEX=6
KNAPSACK_PRO_CI_NODE_INDEX=7
KNAPSACK_PRO_CI_NODE_INDEX=9

Node 4 never gets assigned, and node 6 gets assigned twice. How is this happening?

note: That horrible hard-coded for-loop is an attempt to work around this, which may be relevant:

• http://blog.freeside.co/2013/03/29/groovy-gotcha-for-loops-and-closure-scope
• Passing loop variable to sh in Jenkinsfile doesn't scope correctly
• Parallel jenkins job using a for loop

Solution

env.new_index = index

env is a global object that is part of the main script. env.new_index is a global property.

Example:

node {
    env.VAR1 = "value"
}

println(env.VAR1)

will print

[Pipeline] Start of Pipeline
[Pipeline] node
[Pipeline] {
[Pipeline] }
[Pipeline] // node
[Pipeline] echo
value
[Pipeline] End of Pipeline
Finished: SUCCESS

If you want to define env vars per node then it's better to use withEnv

            withEnv(["new_index=$index"]) {
                stage('Build & run rspec') {
                    dir("$WORKSPACE") {
                        sh '''#!/bin/bash
                            cd $WORKSPACE
                            export NODE_ENV=production
                            export KNAPSACK_PRO_CI_NODE_TOTAL=${total_nodes}
                            export KNAPSACK_PRO_CI_NODE_BUILD_ID=${BUILD_TAG}
                            export KNAPSACK_PRO_CI_NODE_INDEX=${new_index}
                            export KNAPSACK_PRO_COMMIT_HASH=${commithash}
                            export KNAPSACK_PRO_BRANCH=${BRANCH_NAME}
                            export KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN="spec/smokes/*_spec.rb"
                            env
                            ...
                        '''
                    }
                }
            }

Answered By - Alex K.