Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose Encoder and Decoder in TiktokenTokenizer #7314

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Expose Encoder and Decoder in TiktokenTokenizer #7314

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
af97d02
Expose Encoder and Decoder properties in TiktokenTokenizer
razshare Nov 15, 2024
140aa42
Fixing TiktokenTests to use the public properties Encoder and Decoder…
razshare Nov 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -998,12 +998,12 @@ public override OperationStatus Decode(IEnumerable<int> ids, Span<char> destinat
/// <summary>
/// Gets the dictionary mapping token bytes to Ids.
/// </summary>
internal IReadOnlyDictionary<ReadOnlyMemory<byte>, int> Encoder => _encoder;
public IReadOnlyDictionary<ReadOnlyMemory<byte>, int> Encoder => _encoder;

/// <summary>
/// Gets the dictionary mapping Ids to token utf-8 bytes.
/// </summary>
internal IReadOnlyDictionary<int, ReadOnlyMemory<byte>> Decoder => _decoder;
public IReadOnlyDictionary<int, ReadOnlyMemory<byte>> Decoder => _decoder;

private const string EndOfText = "<|endoftext|>";
private const string FimPrefix = "<|fim_prefix|>";
Expand Down
6 changes: 3 additions & 3 deletions test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -729,13 +729,13 @@ public void TestPreciseTokenLimits(string text, string[] expectedTokens, (int In
}
}

// We are not exposing the Encoder, Decoder, or Vocabulary so far. For now, use reflection to test it.
private static IReadOnlyDictionary<ReadOnlyMemory<byte>, int>? GetEncoder(TiktokenTokenizer tiktoken)
=> typeof(TiktokenTokenizer).GetProperty("Encoder", BindingFlags.Instance | BindingFlags.NonPublic)?.GetValue(tiktoken) as IReadOnlyDictionary<ReadOnlyMemory<byte>, int>;
=> tiktoken.Encoder;

private static IReadOnlyDictionary<int, ReadOnlyMemory<byte>>? GetDecoder(TiktokenTokenizer tiktoken)
=> typeof(TiktokenTokenizer).GetProperty("Decoder", BindingFlags.Instance | BindingFlags.NonPublic)?.GetValue(tiktoken) as IReadOnlyDictionary<int, ReadOnlyMemory<byte>>;
=> tiktoken.Decoder;

// We are not exposing the Vocabulary so far. For now, use reflection to test it.
private static IReadOnlyDictionary<string, int>? GetVocabulary(TiktokenTokenizer tiktoken)
=> typeof(TiktokenTokenizer).GetProperty("Vocabulary", BindingFlags.Instance | BindingFlags.NonPublic)?.GetValue(tiktoken) as IReadOnlyDictionary<string, int>;
}
Expand Down
Loading