Skip to content

Commit

Permalink
Structural improvements to SpeziLLM (#45)
Browse files Browse the repository at this point in the history
# Structural improvements to SpeziLLM

## ♻️ Current situation & Problem
As of now, when adjusting LLMonFHIR and SpeziFHIR, we noticed some
structural limitations of the SpeziLLM library.


## ⚙️ Release Notes 
- Structural improvements to SpeziLLM, providing a different abstraction
methodology for working with LLMs.


## 📚 Documentation
Proper documentation is written for all components.


## ✅ Testing
Proper manual testing as well as UI tests


## 📝 Code of Conduct & Contributing Guidelines 

By submitting creating this pull request, you agree to follow our [Code
of
Conduct](https://github.com/StanfordSpezi/.github/blob/main/CODE_OF_CONDUCT.md)
and [Contributing
Guidelines](https://github.com/StanfordSpezi/.github/blob/main/CONTRIBUTING.md):
- [x] I agree to follow the [Code of
Conduct](https://github.com/StanfordSpezi/.github/blob/main/CODE_OF_CONDUCT.md)
and [Contributing
Guidelines](https://github.com/StanfordSpezi/.github/blob/main/CONTRIBUTING.md).
  • Loading branch information
philippzagar authored Feb 23, 2024
1 parent 94c1f3b commit 6892c5d
Show file tree
Hide file tree
Showing 92 changed files with 2,855 additions and 1,556 deletions.
9 changes: 6 additions & 3 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,15 @@ let package = Package(
.library(name: "SpeziLLMOpenAI", targets: ["SpeziLLMOpenAI"])
],
dependencies: [
.package(url: "https://github.com/MacPaw/OpenAI", .upToNextMinor(from: "0.2.5")),
.package(url: "https://github.com/MacPaw/OpenAI", .upToNextMinor(from: "0.2.6")),
.package(url: "https://github.com/StanfordBDHG/llama.cpp", .upToNextMinor(from: "0.1.8")),
.package(url: "https://github.com/StanfordSpezi/Spezi", from: "1.1.0"),
.package(url: "https://github.com/StanfordSpezi/SpeziStorage", from: "1.0.0"),
.package(url: "https://github.com/StanfordSpezi/SpeziOnboarding", from: "1.0.0"),
.package(url: "https://github.com/StanfordSpezi/SpeziSpeech", from: "1.0.0"),
.package(url: "https://github.com/StanfordSpezi/SpeziChat", .upToNextMinor(from: "0.1.4")),
.package(url: "https://github.com/StanfordSpezi/SpeziViews", from: "1.0.0")
.package(url: "https://github.com/StanfordSpezi/SpeziChat", .upToNextMinor(from: "0.1.8")),
.package(url: "https://github.com/StanfordSpezi/SpeziViews", from: "1.0.0"),
.package(url: "https://github.com/groue/Semaphore.git", exact: "0.0.8")
],
targets: [
.target(
Expand All @@ -47,6 +48,7 @@ let package = Package(
dependencies: [
.target(name: "SpeziLLM"),
.product(name: "llama", package: "llama.cpp"),
.product(name: "Semaphore", package: "Semaphore"),
.product(name: "Spezi", package: "Spezi")
],
swiftSettings: [
Expand All @@ -65,6 +67,7 @@ let package = Package(
dependencies: [
.target(name: "SpeziLLM"),
.product(name: "OpenAI", package: "OpenAI"),
.product(name: "Semaphore", package: "Semaphore"),
.product(name: "Spezi", package: "Spezi"),
.product(name: "SpeziChat", package: "SpeziChat"),
.product(name: "SpeziSecureStorage", package: "SpeziStorage"),
Expand Down
97 changes: 52 additions & 45 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,14 +91,14 @@ The target enables developers to easily execute medium-size Language Models (LLM
#### Setup
You can configure the Spezi Local LLM execution within the typical `SpeziAppDelegate`.
In the example below, the `LLMRunner` from the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) target which is responsible for providing LLM functionality within the Spezi ecosystem is configured with the `LLMLocalRunnerSetupTask` from the [SpeziLLMLocal](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillmlocal) target. This prepares the `LLMRunner` to locally execute Language Models.
In the example below, the `LLMRunner` from the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) target which is responsible for providing LLM functionality within the Spezi ecosystem is configured with the `LLMLocalPlatform` from the [SpeziLLMLocal](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillmlocal) target. This prepares the `LLMRunner` to locally execute Language Models.
```
```swift
class TestAppDelegate: SpeziAppDelegate {
override var configuration: Configuration {
Configuration {
LLMRunner {
LLMLocalRunnerSetupTask()
LLMLocalPlatform()
}
}
}
Expand All @@ -107,27 +107,30 @@ class TestAppDelegate: SpeziAppDelegate {
#### Usage
The code example below showcases the interaction with the `LLMLocal` through the the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner), which is injected into the SwiftUI `Environment` via the `Configuration` shown above..
Based on a `String` prompt, the `LLMGenerationTask/generate(prompt:)` method returns an `AsyncThrowingStream` which yields the inferred characters until the generation has completed.
The code example below showcases the interaction with local LLMs through the the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner), which is injected into the SwiftUI `Environment` via the `Configuration` shown above.
The `LLMLocalSchema` defines the type and configurations of the to-be-executed `LLMLocalSession`. This transformation is done via the [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner) that uses the `LLMLocalPlatform`. The inference via `LLMLocalSession/generate()` returns an `AsyncThrowingStream` that yields all generated `String` pieces.
```swift
struct LocalLLMChatView: View {
@Environment(LLMRunner.self) var runner: LLMRunner
// The locally executed LLM
@State var model: LLMLocal = .init(
modelPath: ...
)
@State var responseText: String
func executePrompt(prompt: String) {
// Execute the query on the runner, returning a stream of outputs
let stream = try await runner(with: model).generate(prompt: "Hello LLM!")
for try await token in stream {
responseText.append(token)
}
}
struct LLMLocalDemoView: View {
@Environment(LLMRunner.self) var runner
@State var responseText = ""
var body: some View {
Text(responseText)
.task {
// Instantiate the `LLMLocalSchema` to an `LLMLocalSession` via the `LLMRunner`.
let llmSession: LLMLocalSession = runner(
with: LLMLocalSchema(
modelPath: URL(string: "URL to the local model file")!
)
)
for try await token in try await llmSession.generate() {
responseText.append(token)
}
}
}
}
```
Expand All @@ -142,15 +145,15 @@ In addition, `SpeziLLMOpenAI` provides developers with a declarative Domain Spec
#### Setup
In order to use `LLMOpenAI`, the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner) needs to be initialized in the Spezi `Configuration`. Only after, the `LLMRunner` can be used to execute the ``LLMOpenAI``.
In order to use OpenAI LLMs within the Spezi ecosystem, the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner) needs to be initialized in the Spezi `Configuration` with the `LLMOpenAIPlatform`. Only after, the `LLMRunner` can be used for inference of OpenAI LLMs.
See the [SpeziLLM documentation](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) for more details.
```swift
class LLMOpenAIAppDelegate: SpeziAppDelegate {
override var configuration: Configuration {
Configuration {
LLMRunner {
LLMOpenAIRunnerSetupTask()
LLMOpenAIPlatform()
}
}
}
Expand All @@ -159,29 +162,33 @@ class LLMOpenAIAppDelegate: SpeziAppDelegate {
#### Usage
The code example below showcases the interaction with the `LLMOpenAI` through the the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner), which is injected into the SwiftUI `Environment` via the `Configuration` shown above.
Based on a `String` prompt, the `LLMGenerationTask/generate(prompt:)` method returns an `AsyncThrowingStream` which yields the inferred characters until the generation has completed.
The code example below showcases the interaction with an OpenAI LLM through the the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner), which is injected into the SwiftUI `Environment` via the `Configuration` shown above.
The `LLMOpenAISchema` defines the type and configurations of the to-be-executed `LLMOpenAISession`. This transformation is done via the [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner) that uses the `LLMOpenAIPlatform`. The inference via `LLMOpenAISession/generate()` returns an `AsyncThrowingStream` that yields all generated `String` pieces.
```swift
struct LLMOpenAIChatView: View {
@Environment(LLMRunner.self) var runner: LLMRunner
@State var model: LLMOpenAI = .init(
parameters: .init(
modelType: .gpt3_5Turbo,
systemPrompt: "You're a helpful assistant that answers questions from users.",
overwritingToken: "abc123"
)
)
@State var responseText: String
func executePrompt(prompt: String) {
// Execute the query on the runner, returning a stream of outputs
let stream = try await runner(with: model).generate(prompt: "Hello LLM!")
for try await token in stream {
responseText.append(token)
}
struct LLMOpenAIDemoView: View {
@Environment(LLMRunner.self) var runner
@State var responseText = ""
var body: some View {
Text(responseText)
.task {
// Instantiate the `LLMOpenAISchema` to an `LLMOpenAISession` via the `LLMRunner`.
let llmSession: LLMOpenAISession = runner(
with: LLMOpenAISchema(
parameters: .init(
modelType: .gpt3_5Turbo,
systemPrompt: "You're a helpful assistant that answers questions from users.",
overwritingToken: "abc123"
)
)
)
for try await token in try await llmSession.generate() {
responseText.append(token)
}
}
}
}
```
Expand Down
54 changes: 39 additions & 15 deletions Sources/SpeziLLM/Helpers/Chat+Append.swift
Original file line number Diff line number Diff line change
Expand Up @@ -16,24 +16,30 @@ extension Chat {
/// If the `overwrite` parameter is `true`, the existing message is overwritten.
///
/// - Parameters:
/// - output: The `ChatEntity/Role/assistant` output `String` (part) that should be appended.
/// - overwrite: Indicates if the already present content of the assistant message should be overwritten.
/// - output: The `ChatEntity/Role/assistant` output `String` (part) that should be appended. Can contain Markdown-formatted text.
/// - complete: Indicates if the `ChatEntity` is complete after appending to it one last time via the ``append(assistantOutput:complete:overwrite:)`` function.
/// - overwrite: Indicates if the already present content of the assistant message should be overwritten.
@MainActor
public mutating func append(assistantOutput output: String, overwrite: Bool = false) {
if self.last?.role == .assistant {
self[self.count - 1] = .init(
role: .assistant,
content: overwrite ? output : ((self.last?.content ?? "") + output)
)
} else {
self.append(.init(role: .assistant, content: output))
public mutating func append(assistantOutput output: String, complete: Bool = false, overwrite: Bool = false) {
guard let lastChatEntity = self.last,
lastChatEntity.role == .assistant else {
self.append(.init(role: .assistant, content: output, complete: complete))
return
}

self[self.count - 1] = .init(
role: .assistant,
content: overwrite ? output : (lastChatEntity.content + output),
complete: complete,
id: lastChatEntity.id,
date: lastChatEntity.date
)
}

/// Append an `ChatEntity/Role/user` input to the `Chat`.
///
/// - Parameters:
/// - input: The `ChatEntity/Role/user` input that should be appended.
/// - input: The `ChatEntity/Role/user` input that should be appended. Can contain Markdown-formatted text.
@MainActor
public mutating func append(userInput input: String) {
self.append(.init(role: .user, content: input))
Expand All @@ -42,8 +48,8 @@ extension Chat {
/// Append an `ChatEntity/Role/system` prompt to the `Chat`.
///
/// - Parameters:
/// - systemPrompt: The `ChatEntity/Role/system` prompt of the `Chat`, inserted at the very beginning.
/// - insertAtStart: Defines if the system prompt should be inserted at the start of the conversational context, defaults to `true`.
/// - systemPrompt: The `ChatEntity/Role/system` prompt of the `Chat`, inserted at the very beginning. Can contain Markdown-formatted text.
/// - insertAtStart: Defines if the system prompt should be inserted at the start of the conversational context, defaults to `true`.
@MainActor
public mutating func append(systemMessage systemPrompt: String, insertAtStart: Bool = true) {
if insertAtStart {
Expand All @@ -62,10 +68,28 @@ extension Chat {
/// Append a `ChatEntity/Role/function` response from a function call to the `Chat.
///
/// - Parameters:
/// - functionName: The name of the `ChatEntity/Role/function` that is called by the LLM.
/// - functionResponse: The response `String` of the `ChatEntity/Role/function` that is called by the LLM.
/// - functionName: The name of the `ChatEntity/Role/function` that is called by the LLM.
/// - functionResponse: The response `String` of the `ChatEntity/Role/function` that is called by the LLM.
@MainActor
public mutating func append(forFunction functionName: String, response functionResponse: String) {
self.append(.init(role: .function(name: functionName), content: functionResponse))
}


/// Marks the latest chat entry as `ChatEntity/completed`, if the role of the chat is `ChatEntity/Role/assistant`.
@MainActor
public mutating func completeAssistantStreaming() {
guard let lastChatEntity = self.last,
lastChatEntity.role == .assistant else {
return
}

self[self.count - 1] = .init(
role: .assistant,
content: lastChatEntity.content,
complete: true,
id: lastChatEntity.id,
date: lastChatEntity.date
)
}
}
29 changes: 29 additions & 0 deletions Sources/SpeziLLM/Helpers/Chat+Init.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
//
// This source file is part of the Stanford Spezi open-source project
//
// SPDX-FileCopyrightText: 2022 Stanford University and the project authors (see CONTRIBUTORS.md)
//
// SPDX-License-Identifier: MIT
//

import SpeziChat


extension Chat {
/// Creates a new `Chat` array with an arbitrary number of system messages.
///
/// - Parameters:
/// - systemMessages: `String`s that should be used as system messages.
public init(systemMessages: [String]) {
self = systemMessages.map { systemMessage in
.init(role: .system, content: systemMessage)
}
}


/// Resets the `Chat` array, deleting all persisted content.
@MainActor
public mutating func reset() {
self = []
}
}
78 changes: 0 additions & 78 deletions Sources/SpeziLLM/LLM.swift

This file was deleted.

Loading

0 comments on commit 6892c5d

Please sign in to comment.