Load-balancing / auto-scaling for LLM serving on Google Cloud

glossary.md

@@ -50,3 +50,21 @@ It makes neural network models more memory-efficient and computationally faster
     3. The model's tendency to generate creative but incorrect information.
 
 Researchers and developers continually work to mitigate and reduce hallucination in LLMs to make them more reliable and trustworthy in their outputs.
+
+## References
+- ["Large Languge model"] (https://www.analyticsvidhya.com/blog/2023/03 an-introduction-to-large-language-models-llms/)
+
+- ["Transformer architecture"] (https://machinelearningmastery.com/the-transformer-model/)
+
+- ["Tokenization"] (https://www.analyticsvidhya.com/blog/2020/05/what-is-tokenization-nlp/)
+
+- ["Embedding Layer"] (https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/)
+
+- ["Prompt:"](https://machinelearningmastery.com/a-gentle-introduction-to-prompt-engineering/)
+
+- ["Transfer learning"] (https://machinelearningmastery.com/transfer-learning-for-deep-learning/)
+
+["Distributed learning"](https://cs.brown.edu/people/acrotty/pubs/Galakatos2017_ReferenceWorkEntry_DistributedMachineLearning.pdf)
+
+- ["Fine-tuning"](https://intellipaat.com/blog/fine-tuning/)
+- 

my-llm-deployment.yaml

@@ -0,0 +1,17 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: my-llm-deployment
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: my-llm-service
+  template:
+    metadata:
+      labels:
+        app: my-llm-service
+    spec:
+      containers:
+      - name: my-llm-container
+        image: gcr.io/your-project/llm-image:tag

my-llm-hpa.yaml

@@ -0,0 +1,16 @@
+apiVersion: autoscaling/v2beta2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: my-llm-hpa
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: my-llm-deployment
+  minReplicas: 1
+  maxReplicas: 10
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      targetAverageUtilization: 50

my-llm-service.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-llm-service
+spec:
+  selector:
+    app: my-llm-service
+  ports:
+  - protocol: TCP
+    port: 80
+    targetPort: 80
+  type: LoadBalancer

src/llm_vm/Vertex_AI_Google_API_based_fine_tuning.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+
+from typing import Optional
+
+
+from google.auth import default
+from google.cloud import aiplatform
+import pandas as pd
+import vertexai
+from vertexai.language_models import TextGenerationModel
+from vertexai.preview.language_models import TuningEvaluationSpec
+
+
+credentials, _ = default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
+
+
+def tuning(
+    project_id: str,
+    location: str,
+    model_display_name: str,
+    training_data: pd.DataFrame | str,
+    train_steps: int = 10,
+    evaluation_dataset: Optional[str] = None,
+    tensorboard_instance_name: Optional[str] = None,
+) -> TextGenerationModel:
+    """Tune a new model, based on a prompt-response data.
+
+    "training_data" can be either the GCS URI of a file formatted in JSONL format
+    (for example: training_data=f'gs://{bucket}/{filename}.jsonl'), or a pandas
+    DataFrame. Each training example should be JSONL record with two keys, for
+    example:
+      {
+        "input_text": <input prompt>,
+        "output_text": <associated output>
+      },
+    or the pandas DataFame should contain two columns:
+      ['input_text', 'output_text']
+    with rows for each training example.
+
+    Args:
+      project_id: GCP Project ID, used to initialize vertexai
+      location: GCP Region, used to initialize vertexai
+      model_display_name: Customized Tuned LLM model name.
+      training_data: GCS URI of jsonl file or pandas dataframe of training data.
+      train_steps: Number of training steps to use when tuning the model.
+      evaluation_dataset: GCS URI of jsonl file of evaluation data.
+      tensorboard_instance_name: The full name of the existing Vertex AI TensorBoard instance:
+        projects/PROJECT_ID/locations/LOCATION_ID/tensorboards/TENSORBOARD_INSTANCE_ID
+        Note that this instance must be in the same region as your tuning job.
+    """
+    vertexai.init(project=project_id, location=location, credentials=credentials)
+    eval_spec = TuningEvaluationSpec(evaluation_data=evaluation_dataset)
+    eval_spec.tensorboard = aiplatform.Tensorboard(
+        tensorboard_name=tensorboard_instance_name
+    )
+    model = TextGenerationModel.from_pretrained("text-bison@001")
+
+    model.tune_model(
+        training_data=training_data,
+        # Optional:
+        model_display_name=model_display_name,
+        train_steps=train_steps,
+        tuning_job_location="europe-west4",
+        tuned_model_location=location,
+        tuning_evaluation_spec=eval_spec,
+    )
+
+    print(model._job.status)
+
+    return model
+
+
+if __name__ == "__main__":
+    tuning()