Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite type system and compiler (v4) #35

Merged
merged 54 commits into from
May 17, 2024
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
27c0b83
Initial draft of v4
Schamper May 7, 2023
46ca391
Initial draft of v4 compiler
Schamper May 27, 2023
e591918
Fix enum for older Python versions
Schamper May 27, 2023
66d68e3
Fix zero length arrays in the compiler
Schamper Jun 24, 2023
1377c48
Add debug log to compiler
Schamper Jun 24, 2023
2c0a9c0
Allow dynamically sized unions
Schamper Jun 24, 2023
703a3b1
Small performance optimizations
Schamper Jun 30, 2023
7407e5f
Remove magic operator support
Schamper Jul 10, 2023
e11c04a
Track fields by ID for type injection in compiler
Schamper Jul 21, 2023
58b09f7
Allow self-referencing structures
Schamper Jul 21, 2023
655b537
Remove commented code
Schamper Jul 21, 2023
b5d5f80
Add missing type hint
Schamper Jul 28, 2023
f6e8e63
Add support for EOF length identifier
Schamper Aug 2, 2023
1f7bd10
Fix tests
Schamper Aug 31, 2023
fd12f69
Address review comments
Schamper Sep 4, 2023
c7eecf7
Address comments
Schamper Sep 4, 2023
e6ef6e5
Fix references
Schamper Sep 4, 2023
d61549c
Test array class name
Schamper Nov 24, 2023
f1f623f
Fix alignment and size of new types
Schamper Nov 24, 2023
ddbf96b
Add data-backed unary operator selection (#58)
xloem Dec 11, 2023
64083f4
Add support for real-time updating of union (v4) (#49)
Schamper Dec 21, 2023
418a337
Fix tail alignment for unions
Schamper Dec 24, 2023
c918054
Fix dynamic size check of sub structures
Schamper Dec 24, 2023
2338882
Fix setting dynamic field for enums
Schamper Dec 24, 2023
3009a7c
Modern type hint
Schamper Jan 6, 2024
134c660
Add the `default()` method back in (#55)
Schamper Jan 8, 2024
4550b34
Add overloads for dumps and write for BC (v4) (#54)
Schamper Jan 8, 2024
af12ad0
Add missing return type hint to dumpstruct
Schamper Jan 9, 2024
4c30958
Fix type hints
Schamper Jan 14, 2024
f2e0056
Add LEB128 variable-lengh integer support (#69)
joost-j Feb 16, 2024
85d99b5
Add LEB128 to root __init__
Schamper Feb 19, 2024
3c4c316
Add ArrayType class attribute to provide for user-created array types.
Dec 9, 2023
77e4cf2
Refine custom array type and add unit test
Schamper Feb 20, 2024
e03be00
Add support for None alignment
Schamper Feb 20, 2024
00345ae
Add additional tests for re-packing result from read
Schamper Feb 20, 2024
adfc3b9
Use new Union type-hint syntax
Schamper Feb 20, 2024
eaf8b68
Add additional tests
Schamper Feb 20, 2024
585d13f
Remove redundant tests
Schamper Feb 20, 2024
768735f
Improve pointer repr
Schamper Feb 20, 2024
4327537
Remove experimental code for dynamic module name
Schamper Feb 20, 2024
7f406ab
Add support for pointers of pointers
Schamper Feb 21, 2024
8686f53
Update tests/test_align.py
Schamper Apr 26, 2024
c9b8aba
Update tests/test_align.py
Schamper Apr 26, 2024
25f12a4
Fix test type hints
Schamper May 6, 2024
0a11c9d
Update structure.py
Schamper May 6, 2024
877110b
Update test
Schamper May 6, 2024
9bbc3ba
Update tests/utils.py
Schamper May 6, 2024
210f0a8
Update enum.py
Schamper May 6, 2024
4937662
Update compiler.py
Schamper May 6, 2024
e854073
Small changes
Schamper May 6, 2024
d96d301
Use new style optional type hints
Schamper May 6, 2024
2fd8aa3
Fix enum and flag str behaviour
Schamper May 13, 2024
d655171
Merge branch 'main' into refactor-v4
Schamper May 13, 2024
0ab1683
Add examples and unit tests for custom types in cstruct v4 (#83)
Miauwkeru May 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ assert a.dumps() == d
The API to access enum members and their values is similar to that of the native Enum type in Python 3. Functionally, it's best comparable to the IntEnum type.

### Custom types
You can implement your own types by subclassing `BaseType` or `RawType`, and adding them to your cstruct instance with `addtype(name, type)`
You can implement your own types by subclassing `BaseType` or `RawType`, and adding them to your cstruct instance with `add_type(name, type)`

### Custom definition parsers
Don't like the C-like definition syntax? Write your own syntax parser!
Expand Down
69 changes: 36 additions & 33 deletions dissect/cstruct/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
from dissect.cstruct.bitbuffer import BitBuffer
from dissect.cstruct.compiler import Compiler
from dissect.cstruct.cstruct import cstruct, ctypes, ctypes_type
from dissect.cstruct.exceptions import (
Error,
Expand All @@ -8,18 +7,25 @@
ResolveError,
)
from dissect.cstruct.expression import Expression
from dissect.cstruct.types.base import Array, BaseType, RawType
from dissect.cstruct.types.bytesinteger import BytesInteger
from dissect.cstruct.types.chartype import CharType
from dissect.cstruct.types.enum import Enum, EnumInstance
from dissect.cstruct.types.flag import Flag, FlagInstance
from dissect.cstruct.types.instance import Instance
from dissect.cstruct.types.leb128 import LEB128
from dissect.cstruct.types.packedtype import PackedType
from dissect.cstruct.types.pointer import Pointer, PointerInstance
from dissect.cstruct.types.structure import Field, Structure, Union
from dissect.cstruct.types.voidtype import VoidType
from dissect.cstruct.types.wchartype import WcharType
from dissect.cstruct.types import (
LEB128,
Array,
BaseType,
Char,
CharArray,
Enum,
Field,
Flag,
Int,
MetaType,
Packed,
Pointer,
Structure,
Union,
Void,
Wchar,
WcharArray,
)
from dissect.cstruct.utils import (
dumpstruct,
hexdump,
Expand All @@ -40,31 +46,28 @@
)

__all__ = [
"Compiler",
"Array",
"Union",
"Field",
"Instance",
"cstruct",
"ctypes",
"ctypes_type",
"LEB128",
"Structure",
"Expression",
"PackedType",
"Pointer",
"PointerInstance",
"VoidType",
"WcharType",
"RawType",
"Array",
"BaseType",
"CharType",
"Char",
"CharArray",
"Enum",
"EnumInstance",
"Expression",
"Field",
"Flag",
"FlagInstance",
"BytesInteger",
"Int",
"MetaType",
"Packed",
"Pointer",
"Structure",
"Union",
"Void",
"Wchar",
"WcharArray",
"BitBuffer",
"cstruct",
"ctypes",
"ctypes_type",
"dumpstruct",
"hexdump",
"pack",
Expand Down
8 changes: 4 additions & 4 deletions dissect/cstruct/bitbuffer.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
from __future__ import annotations

from typing import TYPE_CHECKING, BinaryIO, Union
from typing import TYPE_CHECKING, BinaryIO

if TYPE_CHECKING:
from dissect.cstruct.types import RawType
from dissect.cstruct.types import BaseType


class BitBuffer:
Expand All @@ -17,7 +17,7 @@ def __init__(self, stream: BinaryIO, endian: str):
self._buffer = 0
self._remaining = 0

def read(self, field_type: RawType, bits: Union[int, bytes]) -> int:
def read(self, field_type: BaseType, bits: int) -> int:
if self._remaining == 0 or self._type != field_type:
self._type = field_type
self._remaining = field_type.size * 8
Expand All @@ -43,7 +43,7 @@ def read(self, field_type: RawType, bits: Union[int, bytes]) -> int:

return v

def write(self, field_type: RawType, data: int, bits: int) -> None:
def write(self, field_type: BaseType, data: int, bits: int) -> None:
if self._remaining == 0 or self._type != field_type:
if self._type:
self.flush()
Expand Down
Loading