Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update signature section with correct commands and add info on sigdb. #119

Merged
merged 2 commits into from
Mar 12, 2024
Merged

Update signature section with correct commands and add info on sigdb. #119

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
05d0c13
Update signature section with correct commands and add info on sigdb.
wargio Mar 11, 2024
0c79691
Fixed comments
wargio Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
344 changes: 228 additions & 116 deletions src/analysis/signatures.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,120 +1,35 @@
# Signatures

Rizin supports the [HexRays FLIRT](https://hex-rays.com/products/ida/tech/flirt/) signature format which allows you to create,
parse or apply the signatures on the fly. Rizin is also capable of applying
the signatures automatically, when the [sigdb](https://github.com/rizinorg/sigdb) is defined by the user via the
variable `flirt.sigdb.path`. All the applied signatures are stored in the `flirt`
flag space.
Rizin supports the [HexRays FLIRT](https://hex-rays.com/products/ida/tech/flirt/) signature format, enabling users to effortlessly create, parse, or apply signatures on the go. Additionally, Rizin can automatically apply these signatures when the user defines the [sigdb](https://github.com/rizinorg/sigdb) via the `flirt.sigdb.path` variable. All applied signatures are then stored within the `flirt` flag space.

The HexRays FLIRT format is made by two formats:
The HexRays FLIRT format has two formats:

- `.pat` text string format
- `.sig` compressed format which includes the architecture type used to generate the signature
- `.pat` A human-readable text string format
- `.sig` A compressed format, which includes a name, version and the architecture type utilized for generating the signature.

Signature commands are available under the `zf` command namespace:
```
[0x00000000]> zf?
Usage: zf<cds> # Manage FLIRT signatures
| zfc <filename> # Create a FLIRT file (.pac or .sig)
| zfd <filename> # Open a FLIRT file (.pac or .sig) and dumps its contents
| zfs <filename> # Open a FLIRT file (.pac or .sig) and tries to apply the signatures to the loaded binary
```

To apply a signature file manually or a folder of signatures, you need to call `zfs`
```
$ rizin /path/to/binary/with/stripped.elf
[0x000051c0]> aaa
[0x000051c0]> zfs signature.sig # relative or absolutes paths are accepted
Found 206 FLIRT signatures via signature.sig
[0x000051c0]>
```

To create signature you need to analyze the binary first, then you can create it by calling `zfc`.
You can modify the behaviour of the signature creation via the following options:

- `flirt.node.optimize` - FLIRT optimization option when creating a signature file (none: 0, normal: 1, smallest: 2), `default: 2`
- `flirt.sig.deflate` - enables/disables FLIRT zlib compression when creating a signature file (available only for .sig files), `default: true`
- `flirt.sig.file` - FLIRT file list (comma separated) for sig format (msdos, win, os2, netware, unix, other, all, none), `default: all`
- `flirt.sig.library` - FLIRT library name for sig format, `default: Built with rizin x.y.z`
- `flirt.sig.os` - FLIRT operating system list (comma separated) for sig format (aixar, aout, ar, bin, coff, dos:com, dos:com:old, dos:exe, dos:exe:old, dosdrv, elf, intelhex, le, loader, lx, moshex, ne, nlm, omf, omflib, pe, pilot, srec, w32run, zip, all, none), `default: all`
- `flirt.sig.version` - FLIRT version for sig format, `default: 10`

Example of signature creation with `.sig` format:
```
$ rizin /path/to/binary/with/symbols.elf
[0x000051c0]> aa
[0x000051c0]> # setting library name
[0x000051c0]> e flirt.sig.library="My Awesome Signature"
[0x000051c0]> # creating signature
[0x000051c0]> zfc signature.sig # relative or absolutes paths are accepted
707 FLIRT signatures were written in 'signature.sig'
[0x000051c0]>
Signature commands are available under the `F` command namespace:
```bash
[0x00000000]> F?
Usage: F<cdsfal> # FLIRT signature management
| Fc <filename> # Create a FLIRT file (.pat or .sig)
| Fd <filename> # Open a FLIRT file (.pat or .sig) and dumps its contents
| Fs <filename> # Open a FLIRT file (.pat or .sig) and tries to apply the signatures to the loaded binary
| Ff # Outputs the flirt function signature info
| Fa [<filter>] # Apply signatures from sigdb
| Fl[t] # Lists all available signatures in sigdb
```

Example of signature creation with `.pat` format:
```
$ rizin /path/to/binary/with/symbols.elf
[0x000051c0]> aa
[0x000051c0]> # setting internal node optimization to none
[0x000051c0]> e flirt.node.optimize=0
[0x000051c0]> # creating signature
[0x000051c0]> zfc signature.pat # relative or absolutes paths are accepted
707 FLIRT signatures were written in 'signature.pat'
[0x000051c0]>
```
## Signature matching

You can print the contents of a FLIRT signature file via `zfd`
```
[0x000051c0]> zfd signature.sig
SIG format
Signature: Built with rizin 0.4.0-git, 8 modules
Version: 1
Architecture: 0 (x86)
F30F:
1EFA:
31ED4989D15E4889E24883E4F050544C............48..........:
0. 09 CDDC 002E 0000:entry0 (REF )
41574C............41564989D641554989F541544189FC5548....:
0. 45 7595 0065 0000:__libc_csu_init (REF )
4883EC0848:
83C408C3......................................:
0. 00 0000 000D 0000:_fini (REF )
............4885C074..FFD04883C408C3..........:
0. 00 0000 001B 0000:_init (REF )
80............75..5548..............4889E574..48........:
0. 21 300E 0041 0000:__do_global_dtors_aux (REF )
C3......................................................:
0. 00 0000 0005 0000:__libc_csu_fini (REF )
E9......................................................:
0. 00 0000 0009 0000:entry.init0 (REF )
6F866E0100004889F84189C80F1187F8130000F30F6F8E7E0100000F118F:
0. A7 A662 00C8 0000:Curl_persistconninfo (REF )
[0x000051c0]>
```
Before applying a signature, **you must first analyze the binary** (see [code analysis](code_analysis.md) chapter).

It is also possible to use `rz-sign` to create (automatically) or to convert or to dump FLIRT signatures:
```
$ rz-sign -h
Usage: rz-sign [options] [file]
-h this help message
-a [-a] add extra 'a' to analysis command (available only with -o option)
-e [k=v] set an evaluable config variable (available only with -o option)
-c [output.pat] [input.sig] parses a FLIRT signature and converts it to its other format
-o [output.sig] [input.bin] performs an analysis on the binary and generates the FLIRT signature.
-d [flirt.sig] parses a FLIRT signature and dump its content
-q quiet mode
-v show version information
Examples:
rz-sign -d signature.sig
rz-sign -c new_signature.pat old_signature.sig
rz-sign -o libc.sig libc.so.6
In Rizin, there are two methods to apply signatures to a file: using sigdb or manually specifying the signature file.

$ rz-sign -e "flirt.sig.library=My Awesome Signature" -o signature.sig /path/to/binary/with/symbols.elf
```
If `e flirt.sigdb.path` is configured before executing the analysis (`aaa` command), then `sigdb` files are automatically applied to the binary.
However, if a manual approach is preferred, it is possible to select the FLIRT signature file from `sigdb` using the `Fa` command. To list all available signature files in sigdb, use the `Fl` command.

If instead you want to manually select the FLIRT signature file from `sigdb`
you have to use `aaF` commands:
```
An example of selecting sigdb signatures via `Fa` could be:
```bash
[0x00409c70]> pdf
; CALL XREF from main @ 0x4017c1
/ fcn.00409c70 (int64_t arg1, int64_t arg2, int64_t arg3, int64_t arg4, int64_t arg5, int64_t arg6, int64_t arg7, int64_t arg8, int64_t arg9, int64_t arg10, int64_t arg11, int64_t arg_e0h);
Expand All @@ -127,20 +42,27 @@ you have to use `aaF` commands:
| ; var int64_t var_38h_2 @ rsp+0x38
| ; var int64_t var_40h_2 @ rsp+0x40
[0x00409c70]> e flirt.sigdb.path=/path/to/sigdb
[0x00409c70]> aaF?
Usage: aaF [l] # applies signatures from sigdb automatically
| aaF <filter> applies signatures from sigdb automatically
| aaFl lists all the signatures available in sigdb
[0x00409c70]> aaFl~gcc
elf/x86/64/ubuntu-libgcc-10.sig
elf/x86/64/ubuntu-libgcc-11.sig
elf/x86/64/ubuntu-libgcc-7.sig
elf/x86/64/ubuntu-libgcc-8.sig
[0x00409c70]> aaF ubuntu-libgcc
[0x00409c70]> Fa?
Usage: Fa [<filter>] # Apply signatures from sigdb
[0x00409c70]> Fl?
Usage: Fl[t] # Lists all available signatures in sigdb (table mode)
| Flt # Lists all available signatures in sigdb (table mode)
[0x00409c70]> Flt:name/str/gcc
bin arch bits name modules details
--------------------------------------------------------------------------------------------
elf x86 64 ubuntu-libgcc-10.sig 487 Ubuntu GCC support library Version 10 (rizin.re)
elf x86 64 ubuntu-libgcc-11.sig 298 Ubuntu GCC support library Version 11 (rizin.re)
elf x86 64 ubuntu-libgcc-12.sig 307 Ubuntu GCC support library Version 12 (rizin.re)
elf x86 64 ubuntu-libgcc-7.sig 247 Ubuntu GCC support library Version 7 (rizin.re)
elf x86 64 ubuntu-libgcc-8.sig 520 Ubuntu GCC support library Version 8 (rizin.re)
elf x86 64 ubuntu-libgcc-9.sig 535 Ubuntu GCC support library Version 9 (rizin.re)
[0x00409c70]> Fa gcc
Applying elf/x86/64/ubuntu-libgcc-10.sig signature file
Applying elf/x86/64/ubuntu-libgcc-11.sig signature file
Applying elf/x86/64/ubuntu-libgcc-12.sig signature file
Applying elf/x86/64/ubuntu-libgcc-7.sig signature file
Applying elf/x86/64/ubuntu-libgcc-8.sig signature file
Applying elf/x86/64/ubuntu-libgcc-9.sig signature file
[0x00409c70]> pdf
; CALL XREF from main @ 0x4017c1
;-- fcn.00409c70:
Expand All @@ -151,3 +73,193 @@ Applying elf/x86/64/ubuntu-libgcc-8.sig signature file
| ; var int64_t var_18h @ rsp+0xf0
|
```

To manually apply a signature file, you need to use the `Fs` command.
```bash
$ rizin /path/to/binary/with/stripped.elf
[0x00009690]> aaa
[0x00009690]> Fs signature.sig # relative or absolutes paths are accepted
Found 536 FLIRT signatures via signature.sig
[0x00009690]>
```

## Signature creation

Before creating a signature, **you must first analyze the binary** (see [code analysis](code_analysis.md) chapter).

You can generate the signature by invoking the `Fc <filename>` command. You can customize the behavior of signature creation using the following options:

- Signature creation options:
- `flirt.ignore.unknown`: When enabled, during FLIRT creation, it will disregard any function starting with `fcn.`, `default: true`.
- `flirt.node.optimize`: FLIRT optimization option when creating a signature file (none: 0, normal: 1, smallest: 2), `default: 2`.
- `.sig` (compressed format) specific options:
- `flirt.sig.deflate`: Enables or disables the FLIRT zlib compression when creating a signature file (available only for .sig files), `default: true`.
- `flirt.sig.file`: FLIRT file list (comma separated) for `.sig` format (msdos, win, os2, netware, unix, other, all, none), `default: all`.
- `flirt.sig.library`: FLIRT library name for `.sig` format, `default: Built with rizin x.y.z`.
- `flirt.sig.os`: FLIRT operating system list (comma separated) for `.sig` format (aixar, aout, ar, bin, coff, dos:com, dos:com:old, dos:exe, dos:exe:old, dosdrv, elf, intelhex, le, loader, lx, moshex, ne, nlm, omf, omflib, pe, pilot, srec, w32run, zip, all, none), `default: all`.
- `flirt.sig.version`: FLIRT version for `.sig` format `default: 10`.
- [sigdb](https://github.com/rizinorg/sigdb) specific options:
- `analysis.apply.signature`: Enables or disables the automatic application of signatures to the loaded binary, `default: true`.
- `flirt.sigdb.load.extra`: Load signatures from the extra path, `default: true` and its path can be found via `rizin -H RZ_EXTRA_SIGDB`.
- `flirt.sigdb.load.home`: Load signatures from the home path, `default: true`.
- `flirt.sigdb.load.system`: Load signatures from the system path, `default: true` and its path can be found via `rizin -H RZ_SIGDB`.
- `flirt.sigdb.path`: Additional user defined rizin sigdb location to load on the filesystem `default: (empty)`.

Example of signature creation in `.sig` (compressed) format:
```bash
$ rizin /path/to/binary/with/symbols.elf
[0x00009690]> aa
[0x00009690]> # setting library name
[0x00009690]> e flirt.sig.library="My Awesome Library Name"
[0x00009690]> # creating signature
[0x00009690]> Fc signature.sig # relative or absolutes paths are accepted
704 FLIRT signatures were written in 'signature.sig'
[0x00009690]>
```

Example of signature creation in `.pat` (human readable) format:
```bash
$ rizin /path/to/binary/with/symbols.elf
[0x00009690]> aa
[0x00009690]> # disable internal node optimization
[0x00009690]> e flirt.node.optimize=0
[0x00009690]> # creating signature
[0x00009690]> Fc signature.pat # relative or absolutes paths are accepted
704 FLIRT signatures were written in 'signature.pat'
[0x00009690]>
```

You can view the contents of a FLIRT signature file via `Fd <filename>`.
```bash
[0x00009690]> Fd signature.sig
SIG format
Signature: Built with rizin x.y.z, 704 modules
Version: 10
Architecture: 0 (x86)
09CA74..C30F1F00E9:
0. 00 0000 000D 0000:sock_state_cb
0F:
B6:
07:
3C..74..84C074..4C8D86FF000000EB..0F1F40003C..74..4939F074:
0. 00 0000 0103 0000:Curl_auth_digest_get_pair
84C0:
0F84........0FB60E84C90F84........4885D20F84........53:
0. 01 A5A9 0105 0000:Curl_strncasecompare
75..E9........0F1F400041......448D50E0448D429F4189C345:
0. 07 6545 009B 0000:Curl_strcasecompare
4F080FB7570883....66C1C20880....74..83....41B8010000006681..:
0. 00 0000 008A 0000:Curl_ipv6_scope.part.0
87910B00008B9748010000F7D0C0E80781..........0F9EC108C874..31:
0. 11 F2A8 00AF 0000:http_should_fail
B707:
66C1C008C3:
0. 00 0000 0008 0000:Curl_read16_be
C3:
0. 00 0000 0004 0000:Curl_read16_le
```

You can also utilize `rz-sign` to automatically create, convert, or dump FLIRT signatures:
```bash
$ rz-sign -h
Usage: rz-sign [options] [file]
-h Show this help
-a [-a] Add extra 'a' to analysis command (available only with -o option)
-e [k=v] Set an evaluable config variable (available only with -o option)
-c [output.pat] [input.sig] Parse a FLIRT signature and convert it to its other format
-o [output.sig] [input.bin] Perform an analysis on the binary and generate the FLIRT signature
-d [flirt.sig] Parse a FLIRT signature and dump its content
-q Quiet mode
-v Show version information
Examples:
rz-sign -d signature.sig
rz-sign -c new_signature.pat old_signature.sig
rz-sign -o libc.sig libc.so.6

$ rz-sign -e "flirt.sig.library=My Awesome Library Name" -o signature.sig /path/to/binary/with/symbols.elf
```

For library files that use the `.a` or `.la` formats, it is highly recommended to unpack them using `ar` and utilize the `.o` files as sources for the signature files.

## Signature database creation

It is possible to create personal `sigdb` using the [sigdb tools](https://github.com/rizinorg/sigdb-tools) and the scripts in [sigdb source](https://github.com/rizinorg/sigdb-source/tree/main/.scripts) available in the RizinOrg repository.

- [sigdb tools](https://github.com/rizinorg/sigdb-tools)
- `generate-pat-from-obj.py` Generates unoptimized `.pat` signatures from `.o`, `.lo`, or `.obj` object files. The tool creates a new file with the same name as the original object file but with the extension `.pat`.
- `generate-obj-from-lib.py` Unpacks Windows `.lib` files into `.o` (object files). This tool accepts a folder containing all the `.lib` files to unpack as input and generates multiple directories (named after the respective libraries) along with their contents (beware that this tool requires linux utilities).
- `launchpad-deb-scraper.py` This script downloads `deb` packages from launchpad leveraging high-performance asynchronous i/o. It is possible to filter the `deb` packages by architecture.

- [sigdb source](https://github.com/rizinorg/sigdb-source/tree/main/.scripts)
- `generate-pat.py` is a tool that generates a `.pat` file from one or multiple `.pat` files. It offers various options to automatically resolve conflicts. Additionally, it is possible to run it in **test mode** (also known as a dry run) to preview its actions without making any actual changes.
- `generate-sig.py` is a tool that is only used to generate `.sig` files for the release version of a signature database. this compress and automatically names the libraries following the [sigdb source format](https://github.com/rizinorg/sigdb-source/blob/main/README.md#mandatory-folder-structure).

Example of sigdb generation using the tools mentioned above:

```bash
# create output folders
$ mkdir mysigdb mysigdb-extract mysigdb-source

# check if there are lib files
$ find libs/ -type f -name '*.lib'
libs/libcrypto.lib
libs/libssl.lib
# unpack all lib files
$ python ~/sigdb-tools/generate-obj-from-lib.py -v --input libs/
input dir: libs/
found 2 file to ingest

[1|2] parsing libssl.lib
[2|2] parsing libcrypto.lib

done.

# create a sigdb source folder from the extracted files.
$ python ~/sigdb-tools/generate-pat-from-obj.py -i libs/ --output mysigdb-extract
input dir: libs/
found 580 file to ingest

done.
$ find mysigdb-extract/ -type f -name '*.pat' | head
mysigdb-extract/coff/x86/64/libcrypto.lib.ext/bf_ofb64.c.obj.pat
mysigdb-extract/coff/x86/64/libcrypto.lib.ext/aes_cfb.c.obj.pat
mysigdb-extract/coff/x86/64/libcrypto.lib.ext/t_req.c.obj.pat
mysigdb-extract/coff/x86/64/libcrypto.lib.ext/md5-mingw64-x86_64.S.obj.pat
mysigdb-extract/coff/x86/64/libcrypto.lib.ext/tasn_utl.c.obj.pat
mysigdb-extract/coff/x86/64/libcrypto.lib.ext/asn1_par.c.obj.pat
...

# create sigdb source folder structure.
$ mkdir -p "mysigdb-source/coff/x86/64/openssl_3_0_0"

# create sigdb library description files.
# see https://github.com/rizinorg/sigdb-source/blob/main/README.md#mandatory-folder-structure
$ echo "OpenSSL 3.0.0 for Windows" > "mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.description"
$ sha1sum openssl.3.0.0.zip > "mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.src.sha1"

# create sigdb source .pat from extracted files in mysigdb-extract
$ python ~/sigdb-source/.scripts/generate-pat.py --auto --recursive -d "mysigdb-extract/" -o "mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.pat"
output: mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.pat
input: 577 pat files
parsed a total of 6163 signatures and dropped 2082 (~34%) signatures.
There were 452 duplicates out of 4081 signatures (~11%).
mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.pat has been created

# sigdb source directory is now complete.
$ find mysigdb-source/ -type f
mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.description
mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.pat
mysigdb-source/coff/x86/64/openssl_3_0_0/openssl_3_0_0.src.sha1

# create release compressed files in mysigdb/ using mysigdb-source/.
$ python generate-sig.py -s mysigdb-source/ -o mysigdb --test
source: mysigdb-source
output: mysigdb
Generating OpenSSL 3.0.0 for Windows signature (as openssl_3_0_0.sig) from openssl_3_0_0.pat

# Use the generated sigdb with rizin.
$ rizin -e "flirt.sigdb.path=mysigdb/" -qc 'Fl'
bin arch bits name modules details
------------------------------------------------------------------------------
coff x86 64 openssl_3_0_0.sig 3357 OpenSSL 3.0.0 for Windows (rizin.re)
```