Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DomainTools Expert Bot (initial version) #1004

Draft
wants to merge 5 commits into
base: develop
Choose a base branch
from

Conversation

aaronkaplan
Copy link
Member

demo of how to use the domaintools expert to fetch scoring for a domain

@codecov-io
Copy link

codecov-io commented Jun 11, 2017

Codecov Report

Merging #1004 into develop will increase coverage by 2.26%.
The diff coverage is 85.93%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1004      +/-   ##
===========================================
+ Coverage     75.8%   78.06%   +2.26%     
===========================================
  Files          216      223       +7     
  Lines         9108     9136      +28     
  Branches      1215        0    -1215     
===========================================
+ Hits          6904     7132     +228     
- Misses        1920     2004      +84     
+ Partials       284        0     -284
Impacted Files Coverage Δ
...elmq/tests/bots/experts/domaintools/test_expert.py 100% <100%> (ø)
intelmq/bots/experts/domaintools/expert.py 79.54% <79.54%> (ø)
...ntelmq/bots/collectors/alienvault_otx/collector.py 34.78% <0%> (-17.85%) ⬇️
...q/bots/collectors/blueliv/collector_crimeserver.py 32.14% <0%> (-13.7%) ⬇️
...telmq/bots/experts/ripencc_abuse_contact/expert.py 88.88% <0%> (-9.48%) ⬇️
intelmq/bots/collectors/misp/collector.py 34.48% <0%> (-6.9%) ⬇️
...telmq/tests/bots/outputs/postgresql/test_output.py 93.54% <0%> (-6.46%) ⬇️
...elmq/bots/collectors/mail/collector_mail_attach.py 25.71% <0%> (-5.72%) ⬇️
intelmq/bots/outputs/stomp/output.py 30.55% <0%> (-5.56%) ⬇️
intelmq/bots/experts/maxmind_geoip/expert.py 18.42% <0%> (-5.27%) ⬇️
... and 108 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4efedf4...3a400cf. Read the comment docs.

@aaronkaplan aaronkaplan force-pushed the domaintools-expert branch 11 times, most recently from e848623 to 5972061 Compare June 11, 2017 22:25
@aaronkaplan aaronkaplan requested a review from a user June 12, 2017 02:47
@aaronkaplan aaronkaplan added this to the v1.0 Stable Release milestone Jun 12, 2017
@aaronkaplan aaronkaplan assigned aaronkaplan and ghost Jun 12, 2017
@aaronkaplan
Copy link
Member Author

pls review

score = self.domaintools_get_score(event.get(key_fqdn))
if score is not None:
extra["domaintools_score"] = score
event.add("extra", extra)
Copy link

@ghost ghost Jun 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails if extra is already present, with force it would overwrite existing data


try:
score = resp['risk_score']
except exceptions.NotFoundException:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use except (exceptions.NotFoundException, exceptions.BadRequestException)


def domaintools_get_score(self, fqdn):
score = None
if fqdn:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's already checked in the process() method

except exceptions.NotFoundException:
score = None
except exceptions.BadRequestException:
score = None
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If nothing is returned explicitly, the return value is None anyway

"extra": '{"domaintools_score": 0}',
"time.observation": "2015-01-01T00:00:00+00:00"
}
NONEXISTING_INPUT = {"__type": "Event",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the lib raises the BadRequestException for this test which is not equivalent to 'not existing', but the result is similar.

@classmethod
def set_bot(self):
self.bot_reference = DomaintoolsExpertBot
self.sysconfig = {'user': 'mkendrick_first2017', 'password': 'c0e4e-e2527-dc6af-824a4-229d5'}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this here intentionally?

@ghost ghost modified the milestones: v1.1 Feature release, v1.0 Stable Release Jun 12, 2017
@ghost ghost added the feature Indicates new feature requests or new features label Jun 14, 2017
@ghost ghost changed the base branch from master to develop July 5, 2017 15:38
self.logger.info("Loading Domaintools expert.")
if (not API):
self.logger.exception("need to have the domaintools API installed. See https://github.com/domaintools/python_api")
self.stop()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong usage of logger.exception. It prints the traceback of a fromerly catched exception.

Replace both lines with raise ValueError("your error message"). The bot class performs the stop itself.

self.logger.exception("need to have the domaintools API installed. See https://github.com/domaintools/python_api")
self.stop()
if (not self.parameters.user):
self.logger.exception("need to specify user for domaintools expert in runtime.conf. Exiting")
Copy link

@ghost ghost Jul 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing dot at end of line. And you can remove the "Exiting", that's done by the bot class.

score = resp['risk_score']
except exceptions.NotFoundException:
score = None
except exceptions.BadRequestException:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a bad request really the same as no result? Just raise the error, so the bot can retry.

@ghost ghost removed their assignment Jul 27, 2017
@ghost
Copy link

ghost commented Jul 27, 2017

pls review

Done

@ghost
Copy link

ghost commented Aug 14, 2017

Do you want to work on these issues or should I revise this bot?

@ghost ghost added the needs: feedback label Oct 11, 2017
@SYNchroACK
Copy link
Contributor

Do you want to work on these issues or should I revise this bot?

I did some work on this one. Waiting confirmation from @aaronkaplan
aaronkaplan#1

@SYNchroACK SYNchroACK changed the title initial version of the domain tools expert DomainTools Expert Bot (initial version) Jan 17, 2018
@ghost ghost removed the nice to have label Jan 31, 2018
Applied improvements on DomainTools Expert
@ghost ghost self-assigned this May 15, 2018
@ghost ghost unassigned aaronkaplan May 25, 2018
@ghost
Copy link

ghost commented Jun 19, 2018

I don't have an API key, so can't test this at the moment

@ghost ghost modified the milestones: 1.1.0, More bots and feeds Jun 19, 2018
@ghost
Copy link

ghost commented Feb 5, 2019

Closing because of inactivity. Please reopen if needed.

@ghost ghost closed this Feb 5, 2019
@aaronkaplan
Copy link
Member Author

aaronkaplan commented Nov 4, 2019

definitely needed. Reopening.
What we can do is to re-write this PR completely but , yes , domaintools is needed.

@aaronkaplan aaronkaplan reopened this Nov 4, 2019
@ghost ghost marked this pull request as draft May 14, 2020 11:42
@ghost
Copy link

ghost commented Apr 8, 2021

Is this PR still being worked on?

@ghost ghost removed their assignment Aug 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: bots feature Indicates new feature requests or new features needs: feedback
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants