enhancement - move object import from IP/Domain to Indicator

[johnfellers]

Just throwing this out there, but I think it would work out better if all the objects were imported into CRITs as indicators first. By importing as an indicator, it should automatically create the corresponding IP/Domain object and create a relationship back to the indicator.

I know this would mean a good portion would need to be rewritten, but I think that it offers a better approach because it allows for more flexibility to set specific parameters on the object during import, such as setting the confidence level on the individual indicator instead of the overarching campaign. Additionally it allows for setting impact levels per indicator. This will also allow for specifying a default confidence level on a per 'feed' basis, which would be helpful for setting levels on feeds that a user 'trusts' more over other feeds.

You also mentioned that you would like to tag objects that you are importing. This can be accomplished by setting the bucket_list during object import. I was playing around with grabbing this information from the 'notes' that you are currently capturing. It works, but there will definitely need to be some 'cleaning up' as there sometimes can be quite a bit of non-tag like text inside the notes.

I read that you prefer pull requests for enhancements, and I will gladly provide one once I have enough time from my $dayjob to dedicate to it, but wanted to put this out there for the time being. You can take a look at what i have done so far on my fork under the crits-update branch.

Here is some example code for the baler.py that i have thus far:

def bale_CRITs_indicator(base_url, data, indicator_que):
    """ One thread of adding indicators to CRITs"""
    # set the confidence level from the config
    if config.has_option('Baler', 'default_indicator_confidence'):
        data['indicator_confidence'] = config.get('Baler', 'default_indicator_confidence')
    else:
        logger.info('Lacking a default indicator confidence, we will default to Low.')
        data['indicator_confidence'] = 'low'
    # move url initialization here because all objects are imported as indicators
    url = base_url + 'indicators/'
    while not indicator_que.empty():
        indicator = indicator_que.get()
        if indicator[1] == 'IPv4':
            # set the indicator object variables
            data['value'] = indicator[0]
            data['type'] = 'Address - ipv4-addr'
            data['reference'] = indicator[3]

            # set the tags associated with the indicator
            # note: this needs to be fixed as each feed has different forms of data
            # data['bucket_list'] = indicator[4]

            # getting the source automatically:
            source = re.findall(r'\/\/(.*?)\/', data['reference'])
            if source:
                data['source'] = source[0]
            res = requests.post(url, data=data, verify=False)
            if not res.status_code in [201, 200, 400]:
                logger.info("Issues with adding: %s" % data['ip'])