Add support for WebHook Templates based on Velocity template framework

[netwolfuk]

There are a number of limitations with the current templateJson Template format, which could be solved by using a smarter templating framework.

Conditionals
if, else statements for conditions like:

• if buildStatus == successful, then set the colour to green
• if changes.size > 0, then output a list of changes

Loops
for and while loops for handling list/array/set type objects. Items like changes, vcsroots, etc could benefit from some sort of iteration support

Dot notation for access objects/method/fields
I have been hesitant to add too many complex objects to the WebHookPayloadContent class, because accessing them from within a template is impossible. I wondered about adding some sort dot notation macros, so that one could do ${changes[0].version} or ${changes[0].change.username} but I've not found any useful libraries to do it in a generic way.

Putting it all together
From previous experience with the Velocity Template framework provides a way to accomplish all of the above, I think the following code should be possible.

#if ( $changes.size > 0 )
  <b>Changes:</b><br>
  #foreach ( $change in $changes )
    Change by: $change.username <br>
    Message: $change.comment <br>
    Changed Files: <br>
    #foreach ( $file in $change.files )
      $file<br>
    #end
  #end
#else
  <b>No Changes found.</b><br>
#end

Technical Notes

1. Adding a new template type templateVelocityJson or similar name would be used to differentiate the current templates from velocity ones.
2. Velocity 2.0 was just released. It should be considered unless it causes issues.
3. Does ACE editor support velocity 2.0? Is there really that much difference in syntax?
4. Does ACE editor support velocity with a JSON hint? It seems unlikely.
5. Should not require any change in REST API. The template format is just a String.

[netwolfuk]

Mostly working. Just needs example template created.

[netwolfuk]

1.2.0 alpha.1 released which includes this fix.

[taizoon]

Thanks for the 1.2.0 alpha.1 release. An example template to extract the commit message (change comment) for a slack notification would really help. Thanks again.

[netwolfuk]

Sorry I won't get to look at this for a few days, but you could try this as a place to start...

#macro( showchanges $mychanges)
#if ( $mychanges.size() > 0 ) 
#foreach( $change in $mychanges )
#substr($change.version,0,7,32) :: $change.change.username :: $change.change.comment
#end
#else No Changes found #end
#end
{  
    "username": "TeamCity",
    "icon_url" : "https://raw.githubusercontent.com/tcplugins/tcWebHooks/master/docs/icons/teamcity-logo-48x48.png",
    "attachments": [ 
        { 
            "title": "#capitalise(${buildStateDescription}) : ${buildName} <${buildStatusUrl}|build #${buildNumber}>", 
            "fallback": "#capitalise(${buildStateDescription}) : ${buildName} build #${buildNumber}", 
            "color": "good",
            "fields" : [
                { "title" : "Status", "value" : "${buildStatus}" },
                { "title" : "Project Name", "value" : "<${rootUrl}/project.html?projectId=${projectExternalId}|${projectName}>", "short": true },
                { "title" : "Build Name", "value" : "<${rootUrl}/viewType.html?buildTypeId=${buildExternalTypeId}|${buildName}>", "short": true },
                { "title" : "Commit", "value" : "<${buildStatusUrl}&tab=buildChangesDiv|#substr($build_vcs_number,0,7,32)>", "short": true },
                { "title" : "Changes", "value" : "#showchanges( $changes )", "short": true },
                { "title" : "Triggered By", "value" : "${triggeredBy}", "short" : true },
                { "title" : "Agent", "value" : "${agentName}", "short" : true }
            ]
        }
    ]
}

[taizoon]

Thank you very much for a quick response. I tried the above template and didn't have much luck. When I tried sending a test notification, I got a "Error: WebHook endpoint returned non-2xx response (Bad Request) (400)". Thanks again for your input.

[netwolfuk]

Ah. Ok. I'll take a look this week hopefully

[netwolfuk]

Is there anything useful in the TeamCity log?

[netwolfuk]

I suspect the payload is not valid JSON. You could try a function called #to_json($var) or something similar. I'm not in front of a computer at the moment so can't remember the syntax.

It's likely the change comment has a quote or something in it.

[netwolfuk]

Sorry it's escapejson not to_json

[taizoon]

Thanks for the suggestion. Unfortunately neither helped. This is the error from teamcity-server.log

[2019-04-22 15:25:47,291]   INFO -   jetbrains.buildServer.SERVER - AbstractWebHookExecutor ::  :: WebHook triggered : <slack web hook url> using format jsonTemplate returned 400 Bad Request
[2019-04-22 15:25:47,291]   WARN -   jetbrains.buildServer.SERVER - AbstractWebHookExecutor ::  WebHook (url: <slack web hook url> proxy: null:0) returned HTTP status 400
[2019-04-22 15:25:47,291]  ERROR -   jetbrains.buildServer.SERVER - AbstractWebHookExecutor :: ef8d06c9-b068-4576-8bdb-41de0e2ff65c :: WebHook endpoint returned non-2xx response (Bad Request)

It appears to have trouble invoking macros it seems. I changed the macro to just return a string as follows

#macro( showchanges )
#No Changes found
#end

and invoked it like this

{ "title" : "Changes", "value" : "#showchanges( )", "short": true },

and got a 400 while sending a test notification with the same error as above in the server log. I tried different combinations of returning a string in the macro like with and without quotes but all resulted in that same error.

Thanks again for helping with this. Much appreciated.

[netwolfuk]

Oh. It looks like the template is configured as jsonTemplate, rather than jsonVelocityTemplate.

Can you try editing the template settings (little menu at the top right corner) and setting it to be the velocity version?

[taizoon]

Excellent.. that did the trick. Thanks a million for all your help.

[netwolfuk]

You're welcome. Thanks for the feedback.

I'm not sure if you will need to escape the values as JSON. It seems to work ok for me although I've done very little testing, and tend to have simple changes.

The example above looks pretty ugly when it's shown in slack. Feel free to improve it, and submit your improvements to the templates project.

https://github.com/tcplugins/tcWebHooksTemplates

Exporting is supported from the UI in 1.2, so that process should be pretty easy now.

[netwolfuk]

This process has highlighted two things for me.

1. There should be a way to "include" code in a template. Currently that macro needs to be added to the top of every event template.

2. Internally the template type (jsonTemplate vs jsonVelocityTemplate) is stored in the webhook config. It should only refer to the template id. This causes a problem because if I decide to bundle a new slack template based on velocity, all existing webhooks would need to be reconfigured. 😣

[netwolfuk]

Added #134

[netwolfuk]

This is available for testing in the v1.2.0-alpha.3 release.

[Nachiket26]

Hi,

I am little to this - as I can see. Thanks @netwolfuk for this amazing plugin.

I am facing the exact same problem mentioned here : #130. I have multiple vcs roots, committers and the changes are not seen correctly.

I am getting the same error as @taizoon : #103 (comment)

and the problem is I am using TeamCity 9.1.1 with the plugin version 1.1.318.391. I cannot see the option for Velocity Template. Not sure if I need to install anything additionally or it is pre-built.

Could you please confirm if I can use the Velocity Template for the versions above or I have to look for work-arounds?

Thanks & Regards

[netwolfuk]

Hi @Nachiket26 The support for velocity templates is only available from the 1.2.0 versions (currently in alpha, but still very stable). However, I'm not testing against version 9 of TeamCity anymore.

tcWebHooks 1.2.0 might work, but I've not tested it on anything before TeamCity 10.

I'll do a quick test now to see if it works with TeamCity 9

[netwolfuk]

Hi @Nachiket26 I just downloaded and installed TeamCity 9.1.7 (couldn't find a docker image for 9.1.1) and tcWebHooks 1.2.0 alpha 4 works quite well. I have found one minor bug, where you can't edit project webhooks and template from the Project Configuration screen, but there are other ways to do that anyway.

I'd say give it a go on a test teamcity if you have one.

[Nachiket26]

Thanks a lot @netwolfuk for a quick turn-around. I am trying it now locally and then on official server. I'll let you know how it goes.

[netwolfuk]

oops, sorry. I am testing on tcWebHooks 1.2.0 alpha 5. You'll need to use alpha5, since it fixes a bug for versions of teamcity < 2018.1.5

[netwolfuk]

I'm seeing some issues with the editing of Templates on TeamCity 9.1.7 with tcWeHooks 1.2.0-alpha5. I'll need to do some more testing on it. Let me know if you can successfully edit a WebHook template on 9.1.1

[Nachiket26]

Hi @netwolfuk, I was just giving 1.2.0 alpha 5 a try on my TC 9.1.1 server. It seems as you said, there are issues in editing project webhooks and sending test notifications as well - I keep on getting the error :

Error has occurred during request processing (Internal Server Error).
Error: java.lang.ArrayIndexOutOfBoundsException
Error initializing REST API

[netwolfuk]

ok. that's what I'm seeing too. I will keep looking at it, but I won't be able to get back to you today. I thought it might be related to #112 , but I suspect you would've seen it in 1.1.318.391 since it's in that version too. I'll keep digging, but it might take me a few days. I appreciate your patience.

[Nachiket26]

@netwolfuk Thanks for your help. Surprisingly, I did not see such problems in 1.1.318.391.

[netwolfuk]

Ok. The editing of project webhooks is a bug and will affect all versions of TeamCity if no webhook templates exist for a project (which is all projects initially).

The other two issues are probably both related to the rest API issue you saw. I think I have found a possible cause for that too, but I need to do some more testing.

[Nachiket26]

@netwolfuk That's great! Thanks for looking into it.

I was looking into 1.1.318.391 source yesterday to see if I could make a place for multiple changes - but I didn't get a good solution.

If you want me try something out on TeamCity 9.x - I can help.

[netwolfuk]

FYI, TC9.1 support now fixed. See #156

[netwolfuk]

Ok. The editing of project webhooks is a bug and will affect all versions of TeamCity if no webhook templates exist for a project (which is all projects initially).

The other two issues are probably both related to the rest API issue you saw. I think I have found a possible cause for that too, but I need to do some more testing.

These are all resolved in master. I will do a release this week to resolve these issues.