Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.0.0: Improve object encoding and decoding #862

Merged
merged 33 commits into from
Jun 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
d003221
Improved encoding & decoding work, still work in progress
jasonpaulos Mar 25, 2024
6bdc9ba
Fix up transactions unit test
jasonpaulos Mar 27, 2024
6e80bd6
Fixing type errors
jasonpaulos Mar 28, 2024
71999a6
Update cucumber test branch
jasonpaulos Mar 28, 2024
6f58347
Fix more type errors
jasonpaulos Mar 29, 2024
312fd65
Untyped model
jasonpaulos Apr 2, 2024
04af841
Fixed untyped value conversions
jasonpaulos Apr 2, 2024
564c889
Temporarily disable block parsing unit tests
jasonpaulos Apr 3, 2024
6637ead
Fix integration test errors
jasonpaulos Apr 4, 2024
6235396
Update examples
jasonpaulos Apr 4, 2024
1ecb5e2
Initial work on unifying msgp and json paths with a schema object
jasonpaulos Apr 5, 2024
abcbd9a
Make everything Encodable, including generated models
jasonpaulos Apr 11, 2024
d6c4913
Fix most unit tests
jasonpaulos Apr 30, 2024
54aeb78
Update json decoding references
jasonpaulos May 1, 2024
3b99c59
Passing unit tests
jasonpaulos May 2, 2024
3d911fc
Type issue
jasonpaulos May 2, 2024
65e0724
Fixing integration test failures
jasonpaulos May 2, 2024
381347d
fix mistake
jasonpaulos May 2, 2024
7e6ca55
Simplifying code
jasonpaulos May 10, 2024
0a493ae
Remove BaseModel, introduce tests for Schemas
jasonpaulos May 10, 2024
272df51
Fix issues
jasonpaulos May 10, 2024
b11917b
Clean up box translation
jasonpaulos May 10, 2024
3016e9a
Regenerate types, simplify toEncodingData
jasonpaulos May 23, 2024
ce0d857
Clean up generation of toEncodingData and fromEncodingData functions
jasonpaulos May 28, 2024
d72e724
Mark types.ts as generated following https://github.com/github-lingui…
ohill May 29, 2024
b864c24
set linguist-generated=true https://docs.github.com/en/repositories/w…
ohill May 29, 2024
f79528e
Remove `required` from NamedMapEntry & introduce OptionalSchema in it…
jasonpaulos May 30, 2024
6c90132
Fix integration test error
jasonpaulos May 31, 2024
8a29264
Expose and test encode/decodeJSON
jasonpaulos Jun 4, 2024
f67acb6
Migration guide update
jasonpaulos Jun 4, 2024
d11db00
typo
jasonpaulos Jun 4, 2024
c82a856
docstrings
jasonpaulos Jun 4, 2024
86a6e11
Update .test-env
jasonpaulos Jun 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make everything Encodable, including generated models
jasonpaulos committed Apr 11, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit abcbd9a0b3f441a07b3fa189b1b7fc8c7a85bbad
6,342 changes: 3,439 additions & 2,903 deletions src/client/v2/algod/models/types.ts
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a reminder, this file is generated, and its changes are a result of algorand/generator#73

Large diffs are not rendered by default.

5,925 changes: 3,135 additions & 2,790 deletions src/client/v2/indexer/models/types.ts
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a reminder, this file is generated, and its changes are a result of algorand/generator#73

Large diffs are not rendered by default.

31 changes: 12 additions & 19 deletions src/client/v2/untypedmodel.ts
Original file line number Diff line number Diff line change
@@ -1,32 +1,25 @@
import {
MsgpackEncodable,
MsgpackEncodingData,
JSONEncodable,
JSONEncodingData,
msgpackEncodingDataToJSONEncodingData,
jsonEncodingDataToMsgpackEncodingData,
} from '../../encoding/encoding.js';
import { Encodable, MsgpackEncodingData } from '../../encoding/encoding.js';
import { UntypedSchema } from '../../encoding/schema/index.js';

export class UntypedValue implements Encodable {
static encodingSchema = new UntypedSchema();

export class UntypedValue implements MsgpackEncodable, JSONEncodable {
public readonly data: MsgpackEncodingData;

constructor(data: MsgpackEncodingData) {
this.data = data;
}

public msgpackPrepare(): MsgpackEncodingData {
return this.data;
// eslint-disable-next-line class-methods-use-this
public getEncodingSchema(): UntypedSchema {
return UntypedValue.encodingSchema;
}

public jsonPrepare(): JSONEncodingData {
return msgpackEncodingDataToJSONEncodingData(this.data);
}

public static fromDecodedMsgpack(data: MsgpackEncodingData): UntypedValue {
return new UntypedValue(data);
public toEncodingData(): MsgpackEncodingData {
return this.data;
}

public static fromDecodedJSON(data: JSONEncodingData): UntypedValue {
return new UntypedValue(jsonEncodingDataToMsgpackEncodingData(data));
public static fromEncodingData(data: unknown): UntypedValue {
return new UntypedValue(data as MsgpackEncodingData);
}
}
2 changes: 2 additions & 0 deletions src/encoding/schema/index.ts
Original file line number Diff line number Diff line change
@@ -7,3 +7,5 @@ export { ByteArraySchema, FixedLengthByteArraySchema } from './bytearray.js';

export { ArraySchema } from './array.js';
export { NamedMapSchema, NamedMapEntry, allOmitEmpty } from './map.js';

export { UntypedSchema } from './untyped.js';
38 changes: 34 additions & 4 deletions src/encoding/schema/map.ts
Original file line number Diff line number Diff line change
@@ -53,11 +53,18 @@ export class NamedMapSchema extends Schema {
const map = new Map<string, MsgpackEncodingData>();
for (const entry of this.entries) {
if (data.has(entry.key)) {
// TODO: option to omit empty values
if (
entry.omitEmpty &&
entry.valueSchema.isDefaultValue(data.get(entry.key))
) {
continue;
}
map.set(
entry.key,
entry.valueSchema.prepareMsgpack(data.get(entry.key))
);
} else if (entry.required) {
throw new Error(`Missing required key: ${entry.key}`);
}
}
return map;
@@ -66,13 +73,21 @@ export class NamedMapSchema extends Schema {
public fromPreparedMsgpack(
encoded: MsgpackEncodingData
): Map<string, unknown> {
if (!(encoded instanceof Map)) {
throw new Error('NamedMapSchema data must be a Map');
}
const map = new Map<string, unknown>();
for (const entry of this.entries) {
if (encoded instanceof Map && encoded.has(entry.key)) {
if (encoded.has(entry.key)) {
map.set(
entry.key,
entry.valueSchema.fromPreparedMsgpack(encoded.get(entry.key))
);
} else if (entry.required) {
if (entry.omitEmpty) {
map.set(entry.key, entry.valueSchema.defaultValue());
}
throw new Error(`Missing required key: ${entry.key}`);
}
}
return map;
@@ -85,21 +100,36 @@ export class NamedMapSchema extends Schema {
const obj: { [key: string]: JSONEncodingData } = {};
for (const entry of this.entries) {
if (data.has(entry.key)) {
// TODO: option to omit empty values
if (
entry.omitEmpty &&
entry.valueSchema.isDefaultValue(data.get(entry.key))
) {
continue;
}
obj[entry.key] = entry.valueSchema.prepareJSON(data.get(entry.key));
} else if (entry.required) {
throw new Error(`Missing required key: ${entry.key}`);
}
}
return obj;
}

public fromPreparedJSON(encoded: JSONEncodingData): Map<string, unknown> {
if (!(encoded instanceof Map)) {
throw new Error('NamedMapSchema data must be a Map');
}
const map = new Map<string, unknown>();
for (const entry of this.entries) {
if (encoded instanceof Map && encoded.has(entry.key)) {
if (encoded.has(entry.key)) {
map.set(
entry.key,
entry.valueSchema.fromPreparedJSON(encoded.get(entry.key))
);
} else if (entry.required) {
if (entry.omitEmpty) {
map.set(entry.key, entry.valueSchema.defaultValue());
}
throw new Error(`Missing required key: ${entry.key}`);
}
}
return map;
39 changes: 39 additions & 0 deletions src/encoding/schema/untyped.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import {
Schema,
MsgpackEncodingData,
JSONEncodingData,
msgpackEncodingDataToJSONEncodingData,
jsonEncodingDataToMsgpackEncodingData,
} from '../encoding.js';

/* eslint-disable class-methods-use-this */

export class UntypedSchema extends Schema {
public defaultValue(): undefined {
return undefined;
}

public isDefaultValue(data: unknown): boolean {
return data === undefined;
}

public prepareMsgpack(data: unknown): MsgpackEncodingData {
// Value is already MsgpackEncodingData, since it is returned as such from
// fromPreparedMsgpack and fromPreparedJSON
return data as MsgpackEncodingData;
}

public fromPreparedMsgpack(
encoded: MsgpackEncodingData
): MsgpackEncodingData {
return encoded;
}

public prepareJSON(data: unknown): JSONEncodingData {
return msgpackEncodingDataToJSONEncodingData(data as MsgpackEncodingData);
}

public fromPreparedJSON(encoded: JSONEncodingData): MsgpackEncodingData {
return jsonEncodingDataToMsgpackEncodingData(encoded);
}
}
94 changes: 56 additions & 38 deletions src/signedTransaction.ts
Original file line number Diff line number Diff line change
@@ -1,24 +1,57 @@
import {
MsgpackEncodable,
MsgpackEncodingData,
JSONEncodable,
JSONEncodingData,
encodeMsgpack,
decodeMsgpack,
msgpackEncodingDataToJSONEncodingData,
Encodable,
encodeMsgpack2 as encodeMsgpack,
decodeMsgpack2 as decodeMsgpack,
} from './encoding/encoding.js';
import { Address } from './encoding/address.js';
import { base64ToBytes } from './encoding/binarydata.js';
import { Transaction } from './transaction.js';
import { LogicSig } from './logicsig.js';
import {
EncodedMultisig,
encodedMultiSigMsgpackPrepare,
encodedMultiSigFromDecodedMsgpack,
encodedMultiSigFromDecodedJSON,
ENCODED_MULTISIG_SCHEMA,
} from './types/transactions/index.js';
import {
AddressSchema,
FixedLengthByteArraySchema,
NamedMapSchema,
} from './encoding/schema/index.js';

export class SignedTransaction implements Encodable {
static encodingSchema = new NamedMapSchema([
{
key: 'txn',
valueSchema: Transaction.encodingSchema,
required: true,
omitEmpty: true,
},
{
key: 'sig',
valueSchema: new FixedLengthByteArraySchema(64),
required: false,
gmalouf marked this conversation as resolved.
Show resolved Hide resolved
omitEmpty: true,
},
{
key: 'msig',
valueSchema: ENCODED_MULTISIG_SCHEMA,
required: false,
omitEmpty: true,
},
{
key: 'lsig',
valueSchema: LogicSig.encodingSchema,
required: false,
omitEmpty: true,
},
{
key: 'sgnr',
valueSchema: new AddressSchema(),
required: false,
omitEmpty: true,
},
]);

export class SignedTransaction implements MsgpackEncodable, JSONEncodable {
/**
* The transaction that was signed
*/
@@ -74,57 +107,42 @@ export class SignedTransaction implements MsgpackEncodable, JSONEncodable {
}
}

public msgpackPrepare(): MsgpackEncodingData {
const data: Map<string, MsgpackEncodingData> = new Map([
['txn', this.txn.msgpackPrepare()],
]);
// eslint-disable-next-line class-methods-use-this
public getEncodingSchema() {
return SignedTransaction.encodingSchema;
}

public toEncodingData(): Map<string, unknown> {
const data = new Map<string, unknown>([['txn', this.txn.toEncodingData()]]);
if (this.sig) {
data.set('sig', this.sig);
}
if (this.msig) {
data.set('msig', encodedMultiSigMsgpackPrepare(this.msig));
}
if (this.lsig) {
data.set('lsig', this.lsig.msgpackPrepare());
data.set('lsig', this.lsig.toEncodingData());
}
if (this.sgnr) {
data.set('sgnr', this.sgnr.publicKey);
data.set('sgnr', this.sgnr);
}
return data;
}

public static fromDecodedMsgpack(data: unknown): SignedTransaction {
public static fromEncodingData(data: unknown): SignedTransaction {
if (!(data instanceof Map)) {
throw new Error(`Invalid decoded SignedTransaction: ${data}`);
}
return new SignedTransaction({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should there be a check for more than one signature on decode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not so sure about this. If you try to create a signed txn with more than one signature, then yes we should error and stop you because you're doing something wrong and it's best to fail early.

But if you're just trying to read such a transaction that someone else created, I don't see a clear benefit to erroring in that case. There are almost an unlimited number of ways you can invalidate a transaction, and I'm not sure it's worth the effort to try to validate every field completely upon decoding (we do validate that field types are correct, but field values/consistency is another thing entirely). That will be done when you send the transaction to a node for processing anyway.

txn: Transaction.fromDecodedMsgpack(data.get('txn')),
txn: Transaction.fromEncodingData(data.get('txn')),
sig: data.get('sig'),
msig: data.get('msig')
? encodedMultiSigFromDecodedMsgpack(data.get('msig'))
: undefined,
lsig: data.get('lsig')
? LogicSig.fromDecodedMsgpack(data.get('lsig'))
? LogicSig.fromEncodingData(data.get('lsig'))
: undefined,
sgnr: data.get('sgnr') ? new Address(data.get('sgnr')) : undefined,
});
}

public jsonPrepare(): JSONEncodingData {
return msgpackEncodingDataToJSONEncodingData(this.msgpackPrepare());
}

public static fromDecodedJSON(data: unknown): SignedTransaction {
if (data === null || typeof data !== 'object') {
throw new Error(`Invalid decoded SignedTransaction: ${data}`);
}
const obj = data as Record<string, any>;
return new SignedTransaction({
txn: Transaction.fromDecodedJSON(obj.txn),
sig: obj.sig ? base64ToBytes(obj.sig) : undefined,
msig: obj.msig ? encodedMultiSigFromDecodedJSON(obj.msig) : undefined,
lsig: obj.lsig ? LogicSig.fromDecodedJSON(obj.lsig) : undefined,
sgnr: obj.sgnr ? Address.fromString(obj.sgnr) : undefined,
sgnr: data.get('sgnr'),
});
}
}
Loading