#Install Llama Stack

This document describes how to install and deploy Llama Stack Server on Kubernetes using the Llama Stack Operator.

#Upload Operator

Download the Llama Stack Operator installation file (e.g., llama-stack-operator.alpha.ALL.xxxx.tgz).

Use the violet command to publish to the platform repository:

violet push --platform-address=platform-access-address --platform-username=platform-admin --platform-password=platform-admin-password llama-stack-operator.alpha.ALL.xxxx.tgz

#Install Operator

1. Go to the Administrator view in the Alauda Container Platform.

2. In the left navigation, select Marketplace / Operator Hub.

3. In the right panel, find Alauda build of Llama Stack and click Install.

4. Keep all parameters as default and complete the installation.

#Deploy Llama Stack Server

After the operator is installed, deploy Llama Stack Server by creating a LlamaStackDistribution custom resource:

Note: Prepare the following in advance; otherwise the distribution may not become ready:

• Inference URL: VLLM_URL must point at a vLLM OpenAI-compatible HTTP base URL (for example an in-cluster vLLM or KServe InferenceService) that serves the target model.
• Secret (optional): VLLM_API_TOKEN is only needed when the vLLM endpoint requires authentication. If vLLM has no auth, do not set it. When required, create a Secret in the same namespace and reference it from containerSpec.env (see the commented example in the manifest below).
• Storage Class: Ensure the default Storage Class exists in the cluster; otherwise the PVC cannot be bound and the resource will not become ready.

apiVersion: llamastack.io/v1alpha1
kind: LlamaStackDistribution
metadata:
  annotations:
    cpaas.io/display-name: ""
  name: demo
  namespace: default
spec:
  network:
    exposeRoute: false                             # Whether to expose the route externally
  replicas: 1                                      # Number of server replicas
  server:
    containerSpec:
      name: llama-stack
      port: 8321
      env:
        - name: VLLM_URL
          value: "http://vllm-predictor.default.svc.cluster.local/v1"   # vLLM OpenAI-compatible base URL
        - name: VLLM_MAX_TOKENS
          value: "8192"                            # Maximum output tokens

        # Optional: VLLM_API_TOKEN — add only when the vLLM endpoint requires authentication.
        # If vLLM is deployed without auth, omit the entire block below (do not set VLLM_API_TOKEN).
        # Example after creating: kubectl create secret generic vllm-api-token -n default --from-literal=token=<TOKEN>
        # - name: VLLM_API_TOKEN
        #   valueFrom:
        #     secretKeyRef:
        #       key: token
        #       name: vllm-api-token

    distribution:
      name: starter                                # Distribution name (options: starter, postgres-demo, meta-reference-gpu)
    storage:
      mountPath: /home/lls/.lls
      size: 1Gi                                    # Requires the "default" Storage Class to be configured beforehand

After deployment, the Llama Stack Server will be available within the cluster. The access URL is displayed in status.serviceURL, for example:

status:
  phase: Ready
  serviceURL: http://demo-service.default.svc.cluster.local:8321

#Tool calling with vLLM on KServe

The following applies to the vLLM predictor on KServe, not to the LlamaStackDistribution manifest. For agent flows that use tools (client-side tools or MCP), the vLLM process must expose tool-call support. Add predictor container args as required by upstream vLLM, for example:

args:
  - --enable-auto-tool-choice
  - --tool-call-parser
  - hermes

Choose --tool-call-parser (and any related flags) according to the served model and the vLLM documentation for that model family.