Add script to update generated SDS to SCAP 1.3 #4302

yuumasato · 2019-04-26T16:16:12Z

Description:

Add script that updates generated DataStream to be compliant with SCAP 1.3.

How to test

Use scapval-1.3.2
- Use openjdk-1.8.0 on a RHEL environment. (for some reason it doesn't work on Fedora)
- Increase allocated memory: -Xmx1024m
Workaround bugs in scap-schematron-rules-1.3.2 (just to test that scapval passes):
- group_id and user_id cannot be zero, tracked here Multiple schematron false-positives OVAL-Community/OVAL#23
  - workaround it :
    - sed -i -E "s/(group_id .*\">)/\1-1/" ./ssg-rhel7-ds.xml
    - sed -i -E "s/(user_id .*\">)/\1-1/" ./ssg-rhel7-ds.xml
- OCIL checks for security_patches_up_to_date is causing fail of SRC-377, when requirement is about OVAL checks. I have e-mailed NIST about this to try to sort out what is the issue.
  - workaround it:
    - awk '/check-content-ref.*up_to_date_ocil/{for(x=NR-1;x<=NR+1;x++)d[x];}{a[NR]=$0}END{for(i=1;i<=NR;i++)if(!(i in d))print a[i]}' ./ssg-rhel7-ds.xml > ./ssg-rhel7-ds-1.3.xml
Validate it: java -Xmx1024m -jar scapval-1.3.2.jar -file ./ssg-rhel7-ds-1.3.xml -scapversion 1.3 > scap-1.3.log
Report of scapval with workarounds: https://fedorapeople.org/~wsato/scap-1.3/validation-report.html

Rationale:

SCAP 1.3 should supersede SCAP 1.2
In the near future OpenSCAP should be update to handle generation of SCAP 1.3 DS.
- More specifically the command oscap ds sds-compose

pep8speaks · 2019-04-26T16:16:16Z

Hello @yuumasato! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-04-29 14:41:25 UTC

jan-cerny

Please also fix PEP8 issues.

jan-cerny · 2019-04-29T08:14:03Z

build-scripts/update_sds_to_scap_1_3.py

+        component_id = component_id[1:]
+
+        # Locate the <xccdf:check> of the <xccdf:Rule> with id security_patches_up_to_date
+        oval_check = datastreamtree.find(".//ds:component[@id='%s']//xccdf:Rule[@id='xccdf_org.ssgproject.content_rule_security_patches_up_to_date']/xccdf:check[@system='%s']" % (component_id, oval_ns), ns )


This way of using namespaces us not supported in Python 2.6. You need to use eg.
find(".//{%s}Benchmark" % (ssg.constants.XCCDF12_NS))

I have stopped using the namespace dictionary in favor of suggested namespace approach.

jan-cerny · 2019-04-29T08:19:28Z

build-scripts/update_sds_to_scap_1_3.py

+        component_id = component_id[1:]
+
+        # Locate the <xccdf:check> of the <xccdf:Rule> with id security_patches_up_to_date
+        oval_check = datastreamtree.find(".//ds:component[@id='%s']//xccdf:Rule[@id='xccdf_org.ssgproject.content_rule_security_patches_up_to_date']/xccdf:check[@system='%s']" % (component_id, oval_ns), ns )


Pyhton 2.6 doesn't support XPath attributes, you have to loop and filter manually:

for rule in rules: id_ = rule.get("id") if id_ == "xccdf_org.ssgproject.content_rule_security_patches_up_to_date": checks = .... ...

That is unfortunate.
I have implemented the filtering manually.

jan-cerny · 2019-04-29T08:22:56Z

build-scripts/update_sds_to_scap_1_3.py

+
+        # Add the component-ref to list of datastreams' checks
+        ds_checks = datastreamtree.find(".//ds:checks", ns)
+        check_component_ref = ds_checks.findall("ds:component-ref[@id='%s']" % component_ref_uri[1:], ns)


It would be nice to have a comment explaining this [1:].

I have made it explicit, and also added a comment.

Great, but there are still 2 other occurrences of this [1:] which might not be clear.

Sure, thank you for asking for these clarifications.
I have added some comments.

It is not supported in python 2.6

Use of 'attrib' parameter is not supported in python 2.7.

Python 2.6 doesn't support attribute filtering in XPath expressions. For every findall, run through the elements lookcing for the one which interests us.

jan-cerny · 2019-04-29T13:51:55Z

build-scripts/update_sds_to_scap_1_3.py

+
+        # The component-ref ID is the catalog uri without leading '#'
+        component_ref_feed_id = component_ref_uri[1:]
+


When I build RHEL 7 data stream I get in scap_org.open-scap_cref_ssg-rhel7-pcidss-xccdf-1.2.xml component-ref this element:
<ns2:uri name="ecurity-data-oval-com.redhat.rhsa-RHEL7.xml.bz2" uri="#scap_org.open-scap_cref_ecurity-data-oval-com.redhat.rhsa-RHEL7.xml.bz2"/>. There is missing s 2 times.

Nice catch, I ended up introducing a bug during manual filtering. Fixed in ce7ec59.

Commits 15c70d0 and 304270f rename variables to make them clear about what components they are about.

jan-cerny · 2019-04-29T13:52:26Z

build-scripts/update_sds_to_scap_1_3.py

+oval_ns = ssg.constants.oval_namespace
+xccdf_ns = ssg.constants.XCCDF12_NS
+ds_ns = ssg.constants.datastream_namespace
+xlink_ns = ssg.constants.xlink_namespace


This is just renaming variables. Please use them directly.

I have updated the import definition and changed code to use them.

scrutinizer-notifier · 2019-04-29T14:50:51Z

The inspection completed: 4 updated code elements

yuumasato · 2019-04-29T15:22:05Z

@jan-cerny Thank you for all the reviews.
The content is finally building on RHEL6, and the PEP8 issues are fixed as well.

yuumasato · 2019-04-29T15:38:04Z

Here is validation report from rhel8 content, with sed and awk workarounds applied.
https://fedorapeople.org/~wsato/scap-1.3/rhel8-validation-report.html

yuumasato · 2019-04-29T16:01:05Z

With these changes, OpenSCAP 1.3.0 will fail to scan if --fetch-remote-resources is not provided.

$ oscap-ssh rhel7.6-kicked 22 xccdf eval --profile pci-dss ./ssg-rhel7-ds.xml 
Connecting to 'rhel7.6-kicked' on port '22'...
Connected!
Copying input file './ssg-rhel7-ds.xml' to remote working directory '/tmp/tmp.bMdBnKwcR6'...
ssg-rhel7-ds.xml                                                                                                                                                                 100%   26MB 163.4MB/s   00:00    
Starting the evaluation...
WARNING: Datastream component 'scap_org.open-scap_cref_security-data-oval-com.redhat.rhsa-RHEL7.xml.bz2' points out to the remote 'https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL7.xml.bz2'. Use '--fetch-remote-resources' option to download it.
WARNING: Skipping 'https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL7.xml.bz2' file which is referenced from datastream
OpenSCAP Error: Could not extract scap_org.open-scap_cref_ssg-rhel7-xccdf-1.2.xml with all dependencies from datastream. [ds_sds_session.c:211]
oscap exit code: 1
Copying back requested files...
Removing remote temporary directory...
Disconnecting ssh and removing master ssh socket directory...
Exit request sent.

This should be fixed in OpenSCAP.

jan-cerny · 2019-04-30T09:14:00Z

@yuumasato Moreover,
oscap xccdf eval --fetch-remote-resources --profile ospp build/ssg-rhel8-ds.xml fails because https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL8.xml.bz2 doesn't exist yet, so that probably needs to be checked as well.

jan-cerny · 2019-04-30T09:14:17Z

Great job!

yuumasato added this to the 0.1.44 milestone Apr 26, 2019

yuumasato mentioned this pull request Apr 26, 2019

[WIP] Compose SCAP 1.3 DataStreams OpenSCAP/openscap#1323

Closed

Add script to update SDS to SCAP 1.3

2d3f3e2

yuumasato force-pushed the generate_scap_1.3_content branch from 64ef48a to 2d3f3e2 Compare April 26, 2019 16:33

jan-cerny requested changes Apr 29, 2019

View reviewed changes

yuumasato added 2 commits April 29, 2019 11:08

Fix import of urlparse for python2

73d0fb5

Handle tilde during mangling

8a97f8f

jan-cerny self-assigned this Apr 29, 2019

yuumasato added 4 commits April 29, 2019 13:26

Drop use of namespace dictionary

f0b7494

It is not supported in python 2.6

Separate element creation and attribute setting

2f0e4bd

Use of 'attrib' parameter is not supported in python 2.7.

Explicitly define feed component-ref ID

a519fa0

Filter the component and rules manually

bbcd271

Python 2.6 doesn't support attribute filtering in XPath expressions. For every findall, run through the elements lookcing for the one which interests us.

yuumasato force-pushed the generate_scap_1.3_content branch from 705809d to bbcd271 Compare April 29, 2019 13:07

yuumasato added 2 commits April 29, 2019 15:36

Filter existence of uri and component-ref manually

6786ff5

Comment why we strip some leading chars

57dadd3

jan-cerny requested changes Apr 29, 2019

View reviewed changes

yuumasato added 4 commits April 29, 2019 16:19

Fix bug in component filtering

ce7ec59

Add ds_ prefix to component variables related to DS

15c70d0

Add checklists_ prefix to components related to checklist components

304270f

Reuse namespace definitions from ssg.constants

3af98cc

jan-cerny merged commit 279d0e3 into ComplianceAsCode:master Apr 30, 2019

yuumasato deleted the generate_scap_1.3_content branch April 30, 2019 13:57

yuumasato mentioned this pull request Apr 30, 2019

Build multiple scap versions #4319

Merged

This was referenced Apr 30, 2019

Allow DS session to continue without remote resources OpenSCAP/openscap#1324

Merged

SCAP 1.3 Validation #3144

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add script to update generated SDS to SCAP 1.3 #4302

Add script to update generated SDS to SCAP 1.3 #4302

yuumasato commented Apr 26, 2019 •

edited

Loading

pep8speaks commented Apr 26, 2019 •

edited

Loading

jan-cerny left a comment

jan-cerny Apr 29, 2019

yuumasato Apr 29, 2019

jan-cerny Apr 29, 2019

yuumasato Apr 29, 2019

jan-cerny Apr 29, 2019

yuumasato Apr 29, 2019

jan-cerny Apr 29, 2019

yuumasato Apr 29, 2019

jan-cerny Apr 29, 2019

yuumasato Apr 29, 2019 •

edited

Loading

yuumasato Apr 29, 2019

jan-cerny Apr 29, 2019

yuumasato Apr 29, 2019

scrutinizer-notifier commented Apr 29, 2019

yuumasato commented Apr 29, 2019

yuumasato commented Apr 29, 2019 •

edited

Loading

yuumasato commented Apr 29, 2019

jan-cerny commented Apr 30, 2019

jan-cerny commented Apr 30, 2019


		# The component-ref ID is the catalog uri without leading '#'
		component_ref_feed_id = component_ref_uri[1:]

Add script to update generated SDS to SCAP 1.3 #4302

Add script to update generated SDS to SCAP 1.3 #4302

Conversation

yuumasato commented Apr 26, 2019 • edited Loading

Description:

How to test

Rationale:

pep8speaks commented Apr 26, 2019 • edited Loading

Comment last updated at 2019-04-29 14:41:25 UTC

jan-cerny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuumasato Apr 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scrutinizer-notifier commented Apr 29, 2019

yuumasato commented Apr 29, 2019

yuumasato commented Apr 29, 2019 • edited Loading

yuumasato commented Apr 29, 2019

jan-cerny commented Apr 30, 2019

jan-cerny commented Apr 30, 2019

yuumasato commented Apr 26, 2019 •

edited

Loading

pep8speaks commented Apr 26, 2019 •

edited

Loading

yuumasato Apr 29, 2019 •

edited

Loading

yuumasato commented Apr 29, 2019 •

edited

Loading