Create a tool for stubs in cstruct v4 #72

Miauwkeru · 2024-02-27T10:12:52Z

The only thing that does not work as intended yet is the dereference of pointer. As in, it doesn't return the desired type information but just T

codecov · 2024-02-27T10:14:27Z

Codecov Report

Attention: Patch coverage is 58.38509% with 67 lines in your changes missing coverage. Please review.

Project coverage is 89.65%. Comparing base (7f6a584) to head (20a00bd).

Files	Patch %	Lines
dissect/cstruct/tools/stubify.py	38.31%	66 Missing ⚠️
dissect/cstruct/types/base.py	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #72      +/-   ##
==========================================
- Coverage   92.04%   89.65%   -2.39%     
==========================================
  Files          20       21       +1     
  Lines        2136     2292     +156     
==========================================
+ Hits         1966     2055      +89     
- Misses        170      237      +67

Flag	Coverage Δ
unittests	`89.65% <58.38%> (-2.39%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dissect/cstruct/tools/stubify.py

dissect/cstruct/types/structure.py

dissect/cstruct/cstruct.py

Schamper · 2024-08-09T16:39:43Z

dissect/cstruct/types/base.py

+    def _type_stub(cls, name: str = "") -> str:
+        return f"{name}: {cls.__name__}"
+
+    def to_stub(cls, name: str) -> str:


Maybe name this differently? Also I think this might as well be a private method. Maybe _to_type_stub? to_stub might be confusing because it also makes me think of the compiler.

i renamed it to to_type_stub as it is a function that gets called externally.

I don't agree on the "externally" part, I'd classify this usage as internal and would never expect anyone to ever call this externally. This is not part of any expected usage API.

Schamper · 2024-08-19T09:13:15Z

dissect/cstruct/tools/stubify.py

+    description = """
+        Create stub files for cstruct definitions.
+
+        These stub files are in a `.pyi` format and provides `type` information to cstruct definitions.


Why is type monospaced? Also this would need to be double back ticks anyway, if you're going to include this on the dissect.cstruct doc page.

Schamper · 2024-08-19T09:13:40Z

dissect/cstruct/tools/stubify.py

+        This in turn gives a developer insight into the elements inside the definition and 
+        parameter completion when dealing with cstructs.


Suggested change

This in turn gives a developer insight into the elements inside the definition and

parameter completion when dealing with cstructs.

Schamper · 2024-08-19T09:16:00Z

dissect/cstruct/tools/stubify.py

+
+    prev_offset = buffer.tell()
+    for const, value in c_structure.consts.items():
+        buffer.write(indent(f"{const}: {type(value).__name__}=...\n", prefix=indentation))


Suggested change

buffer.write(indent(f"{const}: {type(value).__name__}=...\n", prefix=indentation))

buffer.write(indent(f"{const}: {type(value).__name__} = ...\n", prefix=indentation))

Schamper · 2024-08-19T09:17:13Z

dissect/cstruct/tools/stubify.py

Can you add some unit tests for this file? All the coverage warnings make it difficult to review.

Schamper · 2024-08-19T09:25:34Z

dissect/cstruct/types/enum.py

+        output = ""
+        with io.StringIO() as buf:
+            buf.write(cls._class_stub())
+            for key in cls.__members__.keys():
+                buf.write(f"    {key} = ...\n")
+            output = buf.getvalue()
+
+        return output


Suggested change

output = ""

with io.StringIO() as buf:

buf.write(cls._class_stub())

for key in cls.__members__.keys():

buf.write(f" {key} = ...\n")

output = buf.getvalue()

return output

result = [cls._class_stub()]

for key in cls.__members__.keys():

result.append(f" {key} = ...")

return "\n".join(result)

Should also get rid of the newline in _class_stub. Might be confusing for custom type implementations that you're responsible for ending with a newline.

Schamper · 2024-08-19T10:30:29Z

pyproject.toml

@@ -31,6 +31,9 @@ homepage = "https://dissect.tools"
 documentation = "https://docs.dissect.tools/en/latest/projects/dissect.cstruct"
 repository = "https://github.com/fox-it/dissect.cstruct"

+[project.scripts]
+stubify = "dissect.cstruct.tools.stubify:main"


Suggested change

stubify = "dissect.cstruct.tools.stubify:main"

cstruct-stubify = "dissect.cstruct.tools.stubify:main"

Schamper · 2024-08-19T10:30:59Z

tests/test_stub.py

+                class __anonymous_0__(Union):
+                    a: WcharArray
+                    b: CharArray
+                    def __init__(self, a: WcharArray=..., b: CharArray=...): ...
+                a: WcharArray
+                b: CharArray
+                def __init__(self, __anonymous_0__: __anonymous_0__=...): ...


Anonymous structures should get folded in, I don't think there's any use in them being typed?

Schamper · 2024-08-19T10:50:03Z

dissect/cstruct/types/base.py

@@ -181,6 +181,15 @@ def _write_0(cls, stream: BinaryIO, array: list[BaseType]) -> int:
        """
        return cls._write_array(stream, array + [cls()])

+    def _class_stub(cls) -> str:
+        return f"class {cls.__name__}({cls.__base__.__name__}):\n"


Suggested change

return f"class {cls.__name__}({cls.__base__.__name__}):\n"

return f"class {cls.__name__}({cls.__base__.__name__}):"

Schamper · 2024-08-19T10:50:16Z

dissect/cstruct/types/enum.py

@@ -84,6 +85,19 @@ def _write_0(cls, stream: BinaryIO, array: list[BaseType]) -> int:
        data = [entry.value if isinstance(entry, Enum) else entry for entry in array]
        return cls._write_array(stream, data + [cls.type()])

+    def _class_stub(cls) -> str:
+        return f"class {cls.__name__}({cls.__base__.__name__}, {cls.type.__name__}):\n"


Suggested change

return f"class {cls.__name__}({cls.__base__.__name__}, {cls.type.__name__}):\n"

return f"class {cls.__name__}({cls.__base__.__name__}, {cls.type.__name__}):"

Schamper · 2024-08-19T10:52:02Z

dissect/cstruct/types/structure.py

+        buffer = io.StringIO()
+        buffer.write(f"class {cls.__name__}({cls.__base__.__name__}):\n")
+        call_args = ["self"]
+        for key, field in cls.lookup.items():
+            if isinstance(field.type, StructureMetaType):
+                class_info = field.type.to_type_stub()
+                buffer.write(indent(class_info, prefix=" " * 4))
+            call_args.append(f"{field.type_stub()}=...")
+
+        for field in cls.fields.values():
+            type_info = field.type_stub()
+            buffer.write(indent(f"{type_info}\n", prefix=" " * 4))
+
+        call = ", ".join(call_args)
+        buffer.write(indent(f"def __init__({call}): ...\n", prefix=" " * 4))
+
+        output = buffer.getvalue()
+        buffer.close()
+        return output


Suggested change

buffer = io.StringIO()

buffer.write(f"class {cls.__name__}({cls.__base__.__name__}):\n")

call_args = ["self"]

for key, field in cls.lookup.items():

if isinstance(field.type, StructureMetaType):

class_info = field.type.to_type_stub()

buffer.write(indent(class_info, prefix=" " * 4))

call_args.append(f"{field.type_stub()}=...")

for field in cls.fields.values():

type_info = field.type_stub()

buffer.write(indent(f"{type_info}\n", prefix=" " * 4))

call = ", ".join(call_args)

buffer.write(indent(f"def __init__({call}): ...\n", prefix=" " * 4))

output = buffer.getvalue()

buffer.close()

return output

result = [f"class {cls.__name__}({cls.__base__.__name__}):"]

args = ["self"]

for field in cls.fields.values():

if isinstance(field.type, StructureMetaType) and not field.type.__anonymous__:

result.append(indent(field.type.to_type_stub(), prefix=" " * 4))

result.append(f" {field._type_stub()}")

args.append(f"{field._type_stub()} = ...")

result.append(indent(f"def __init__({', '.join(args)}): ...", prefix=" " * 4))

return "\n".join(result)

Miauwkeru changed the base branch from main to refactor-v4 February 27, 2024 10:13

Miauwkeru force-pushed the stub branch from 3a2a6b0 to 0ca4810 Compare February 27, 2024 10:15

Schamper self-requested a review March 4, 2024 00:38

Miauwkeru marked this pull request as ready for review March 11, 2024 12:50

Miauwkeru marked this pull request as draft March 11, 2024 12:50

Schamper force-pushed the refactor-v4 branch from 0ef089a to 2ce458b Compare March 22, 2024 12:15

Schamper force-pushed the refactor-v4 branch from 2ce458b to 7f406ab Compare April 4, 2024 08:51

Base automatically changed from refactor-v4 to main May 17, 2024 13:15

Miauwkeru force-pushed the stub branch from 30b91e9 to 31f582c Compare August 9, 2024 13:45

Miauwkeru marked this pull request as ready for review August 9, 2024 14:56

Miauwkeru added 6 commits August 9, 2024 14:57

Adding stub creation tooling

13e068e

Add stub information on types

f4f16c1

Fix linting issues

55e40e1

remove unused argument

c089250

Not using a wildcard for the types

6a043ad

Cleanup and move typedefs at the start of the file

314982c

Miauwkeru force-pushed the stub branch from 4bc48e3 to 314982c Compare August 9, 2024 14:57

Schamper requested changes Aug 9, 2024

View reviewed changes

Miauwkeru added 5 commits August 12, 2024 09:51

Add suggestion for stubify tool

a2bf25a

Move stub functions from cstruct to stubify.py

c6544c3

Improve the description

b2a270d

rename to_stub to to_type_stub

d9f992d

Add suggestions

20a00bd

Miauwkeru requested a review from Schamper August 12, 2024 10:26

Schamper requested changes Aug 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a tool for stubs in cstruct v4 #72

Create a tool for stubs in cstruct v4 #72

Miauwkeru commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading

Schamper Aug 9, 2024

Miauwkeru Aug 12, 2024

Schamper Aug 16, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

Schamper Aug 19, 2024

		This in turn gives a developer insight into the elements inside the definition and
		parameter completion when dealing with cstructs.

	buffer.write(indent(f"{const}: {type(value).__name__}=...\n", prefix=indentation))
	buffer.write(indent(f"{const}: {type(value).__name__} = ...\n", prefix=indentation))

	stubify = "dissect.cstruct.tools.stubify:main"
	cstruct-stubify = "dissect.cstruct.tools.stubify:main"

	return f"class {cls.__name__}({cls.__base__.__name__}):\n"
	return f"class {cls.__name__}({cls.__base__.__name__}):"

Create a tool for stubs in cstruct v4 #72

Are you sure you want to change the base?

Create a tool for stubs in cstruct v4 #72

Conversation

Miauwkeru commented Feb 27, 2024 • edited Loading

codecov bot commented Feb 27, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Miauwkeru commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading