Block parser to drop events issue

[RyanKershawWhittle]

Hi,

I am having issues implementing a sc4s block parser to drop events:

block parser name_name_vmware-postfilter() {
channel {
#In this case the outcome is drop the event other logic such as adding indexed fields or editing the message is possible
rewrite(r_set_dest_splunk_null_queue);
};
};
application name_name_vmware-postfilter[sc4s-postfilter] {
filter {
"${fields.sc4s_vendor}" eq "vmware" and
"${fields.sc4s_product}" eq "vsphere"
#Note regex will capture and drop events based on the keywords below.
and message('^- - (lookupsvc-localhost_access|vpxd-main|statsmonitor|vmdird|sps|trustmanagement-svcs|vpxd-svcs-perf|sps-gc|osfsd|sso-tomcat|wcpsvc|vmafdd|vpxd-svcs-access|cis-license|ui-threadmonitor|ssoadminserver|analytics|content-library|vapi-endpoint-access|lookupsvc-gc|vmcad|sca-vmon.std|ui-main|vapi-endpoint|trustmanagement-gc|vmon|ui-access|sca-gc|applmgmt-audit|observability-main|vapi-runtime|vapi-gc|vpxd-profiler|ui-gc|ui-runtime|ui-apigw|vapi-vcentershim|vum-vmacore|envoy-access) - -');
};
parser { name_name_vmware-postfilter(); };
};

The regex looks to capture keywords at the start of each event. Regex has been validated and works fine. Its worth mentioning I have tried several different iterations of regex to get this working and none seem to work with the parser..

If i comment # out the 'and message' line it works fine and drop events based on vendor and product.

please can you provide guidance on this.

[mstopa-splunk]

Hi @RyanKershawWhittle, it may be caused by nilvalues - which may not be part of the message macro. Please try regex with literals (lookupsvc-localhost_access|vpxd-main... but without nilvalues - and without matching beginning (or end) of the string. If this doesn't unblock you please send one or two raw log message examples for me to reproduce and check.

[mstopa-splunk]

The reason is that vpxd is not part of the message, it goes to program.

I checked the following to correctly filter out vmware vsphere events different than vpxd:

block parser name_name_vmware-postfilter() {
    channel {
        rewrite(r_set_dest_splunk_null_queue);
    };
};

application name_name_vmware-postfilter[sc4s-postfilter] {
    filter {
        "${fields.sc4s_vendor}" eq "vmware" 
        and "${fields.sc4s_product}" eq "vsphere"
        and not program('vpxd');
    };
    parser { name_name_vmware-postfilter(); };
};

You can find programs here: https://github.com/splunk/splunk-connect-for-syslog/blob/main/package/etc/conf.d/conflib/syslog/app-syslog-vmware_vsphere.conf

Please let me know if that solves the issue or process id (i.e. vpxd 1453, vpxd 56724) should be also included to the filter.

[mstopa-splunk]

Hi @RyanKershawWhittle , I see. Let's turn off regex in the program filter by switching to string glob, no wildcards will result in the exact match:

block parser name_name_vmware-postfilter() {
    channel {
        rewrite(r_set_dest_splunk_null_queue);
    };
};

application name_name_vmware-postfilter[sc4s-postfilter] {
    filter {
        "${fields.sc4s_vendor}" eq "vmware" 
        and "${fields.sc4s_product}" eq "vsphere"
        and not program('vpxd' type("string") flags("global"));
    };
    parser { name_name_vmware-postfilter(); };
};

now it blocks vpxd-main but not vpxd

[RyanKershawWhittle]

Hi @mstopa-splunk

This works!

Thank you for your help on this one. You've been extremely helpful and fast at responding. Really appreciate it.

[RyanKershawWhittle]

@mstopa-splunk

If i want to include more programs to NOT be dropped what is the correct syntax:?

    and not program('vpxd' type("string") flags("global"));
    
    I.e. 'vpxd, vpxa'
          'vpxd vpxa'
          
       ??

[mstopa-splunk]

@RyanKershawWhittle and not program will work. The verbose way is good and self documenting but if you have more of them and they make regular patterns you can use wildcards. When you remove flags("global") you can use regex, which will be less efficient, but it's still successfully used in other filters

[RyanKershawWhittle]

Hi @mstopa-splunk

Apologies let me be more clear. I want to keep 'hostd' and 'vpxa' programs. I have tried this config but it wont work:

block parser piaas_vmware_vcenter-postfilter() {

    channel {

        rewrite(r_set_dest_splunk_null_queue);

    };

};

 


 

application piaas_vmware_vcenter-postfilter[sc4s-postfilter] {

    filter {

        "${fields.sc4s_vendor}" eq "vmware"

        and "${fields.sc4s_product}" eq "vsphere"

        and not program('vpxd' type("string") flags("global"));

        and not program('hostd' type("string") flags("global"));

        and not program('vpxa' type("string") flags("global"));

    };

    parser { piaas_vmware_vcenter-postfilter(); };

};

I presume it needs to be in one line ? i.e. and not program('vpxa hostd vpxa' type("string") flags("global"));

Wanted to confirm what syntax this is within the ' '

https://github.com/splunk/splunk-connect-for-syslog/blob/main/package/etc/conf.d/conflib/syslog/app-syslog-vmware_vsphere.conf

[mstopa-splunk]

Right, please try removing semicolons prior to and. if it doesn't help, please provide a raw example for either of those two for us to investigate

[RyanKershawWhittle]

Hi @mstopa-splunk

Removing the ; didnt work.

block parser piaas_vmware_vcenter-postfilter() {

    channel {

        rewrite(r_set_dest_splunk_null_queue);

    };

};

 


 

application piaas_vmware_vcenter-postfilter[sc4s-postfilter] {

    filter {

        "${fields.sc4s_vendor}" eq "vmware"

        and "${fields.sc4s_product}" eq "vsphere"

        and not program('vpxd' type("string") flags("global"));

        and not program('hostd' type("string") flags("global"))

        and not program('vpxa' type("string") flags("global"))

    };

    parser { piaas_vmware_vcenter-postfilter(); };

};

example logs:

log - - Vpxa - - [Originator@6876 sub="vpxLro" opID="HB-host-31@8946840-6a6e49fd-ec"][VpxLRO] -- FINISH lro-7598181

log - - Hostd - - [Originator@6876 sub="Libs" opID="b2c6a5c4"]memory per disk MB: 81

[mstopa-splunk]

All right, it needs to ignore case:

block parser piaas_vmware_vcenter-postfilter() {
    channel {
        rewrite(r_set_dest_splunk_null_queue);
    };
};

application piaas_vmware_vcenter-postfilter[sc4s-postfilter] {

    filter {
        "${fields.sc4s_vendor}" eq "vmware"
        and "${fields.sc4s_product}" eq "vsphere"
        and not program('vpxd' type("string") flags(ignore-case, global))
        and not program('hostd' type("string") flags(ignore-case, global))
        and not program('vpxa' type("string") flags(ignore-case, global));

    };
    parser { piaas_vmware_vcenter-postfilter(); };
};

[RyanKershawWhittle]

@mstopa-splunk thank you very much that worked!

Will close this.

Thanks once again