HTTP CT logs features

[kidmose]

@kdhageman has done some work on the area of HTTPS Certificate Transparency logs, and also has a script that extract some features for a domain.

We want those reimplemented in richkit (as a first iteration) such that richkit has a function for each feature, that given a domain name will return the value for the feature.

The script is likely based on a API/data source at Censys, which is very batch oriented (Along the lines that a batch, whether for 1 or for 1000 domains, has a fixed price). It seems likely that https://crt.sh/?q=example.com is a better candidate for richkit for now.

Not knowing the state or nature of the script, it might be necessary to analyse it to understand each feature and reimplement it from scratch here, but I'm sure Kaspar can provide some advice.

This is done when richkit has a method for each of the features, with the documentation and testing to with it.

@kdhageman : If you don't get arround to push the script to a repo, then perhaps you can share the current version here?

[kdhageman]

I have transferred the repo to the aau-network-security group, it's called bulldozer.
You can find the specific code here. It's written in Golang, so it cannot be transferred to richkit trivially. I sent a description of the features to @gianmarcomennecozzi

[gianmarcomennecozzi]

Just to let you know guys, this is how i structured the data retrieved from crt.sh. It is a list of the following obj:

{   
   "ID":987119772,
   "Issuer":"C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA",
   "Algorithm":"sha256WithRSAEncryption",
   "ValidationL":[
      "2.16.840.1.114412.1.1",
      "2.23.140.1.2.2"
   ],
   "NotBefore":"2018-11-28T00:00:00",
   "NotAfter":"2020-12-02T12:00:00",
   "Validity":735,
   "SANFeatures":{
      "DomainCount":8,
      "UniqueApexCount":"None",
      "UniqueSLDCount":"None",
      "ShortestSAN":11,
      "LongestSAN":15,
      "SANsMean":13,
      "MinSublabels":"None",
      "MaxSublabels":"None",
      "MeanSublabels":"None",
      "UniqueTLDsCount":"None",
      "UniqueTLDsDomainCount":"None",
      "ApexLCS":"None",
      "LenApexLCS":"None",
      "LenApexLCSnorm":"None"
   }
}

The None fields are still in progress