Skip to content

metrics.md

Latest commit

 

History

History
126 lines (109 loc) · 8.52 KB

metrics.md

File metadata and controls

126 lines (109 loc) · 8.52 KB

简体中文

Monitoring Metrics

After FastDeploy is launched, it supports continuous monitoring of the FastDeploy service status through Metrics. When starting FastDeploy, you can specify the port for the Metrics service by configuring the metrics-port parameter.

Category Metric Name Type Description Unit
Request fastdeploy:requests_number Counter Total number of received requests count
Request fastdeploy:request_success_total Counter Number of successfully processed requests count
Request fastdeploy:num_requests_running Gauge Number of requests currently running count
Request fastdeploy:num_requests_waiting Gauge Number of requests currently waiting count
Latency fastdeploy:time_to_first_token_seconds Histogram Time to generate the first token (TTFT) s
Latency fastdeploy:time_per_output_token_seconds Histogram Time interval between generated tokens (TPOT) s
Latency fastdeploy:e2e_request_latency_seconds Histogram End-to-end request latency distribution s
Latency fastdeploy:request_inference_time_seconds Histogram Time spent in the RUNNING phase s
Latency fastdeploy:request_queue_time_seconds Histogram Time spent in the WAITING phase s
Latency fastdeploy:request_prefill_time_seconds Histogram Time spent in the Prefill phase s
Latency fastdeploy:request_decode_time_seconds Histogram Time spent in the Decode phase s
Token fastdeploy:prompt_tokens_total Counter Total number of processed prompt tokens count
Token fastdeploy:generation_tokens_total Counter Total number of generated tokens count
Token fastdeploy:request_prompt_tokens Histogram Prompt token count per request count
Token fastdeploy:request_token_ratio Histogram Token generation rate per Request count
Token fastdeploy:request_generation_tokens Histogram Generation token count per request count
Token fastdeploy:request_params_max_tokens Histogram Distribution of max_tokens per request count
Batch fastdeploy:available_batch_size Gauge Number of additional requests that can be inserted during Decode count
Batch fastdeploy:batch_size Gauge Actual batch size during inference count
Batch fastdeploy:max_batch_size Gauge Maximum batch size configured at service startup count
KV Cache fastdeploy:cache_config_info Gauge Cache configuration info of the inference engine count
KV Cache fastdeploy:hit_req_rate Gauge Prefix cache hit rate at the request level %
KV Cache fastdeploy:hit_token_rate Gauge Prefix cache hit rate at the token level %
KV Cache fastdeploy:cpu_hit_token_rate Gauge CPU-side token-level prefix cache hit rate %
KV Cache fastdeploy:gpu_hit_token_rate Gauge GPU-side token-level prefix cache hit rate %
KV Cache fastdeploy:prefix_cache_token_num Counter Total number of tokens in prefix cache count
KV Cache fastdeploy:prefix_gpu_cache_token_num Counter Total number of prefix cache tokens on GPU count
KV Cache fastdeploy:prefix_cpu_cache_token_num Counter Total number of prefix cache tokens on CPU count
KV Cache fastdeploy:available_gpu_block_num Gauge Available GPU blocks in cache (including unreleased prefix blocks) count
KV Cache fastdeploy:free_gpu_block_num Gauge Number of free GPU blocks in cache count
KV Cache fastdeploy:max_gpu_block_num Gauge Total number of GPU blocks initialized at startup count
KV Cache fastdeploy:max_cpu_block_num Gauge Total number of CPU blocks initialized at startup count
KV Cache fastdeploy:available_gpu_resource Gauge Ratio of available GPU blocks to total GPU blocks %
KV Cache fastdeploy:gpu_cache_usage_perc Gauge GPU KV cache utilization %
KV Cache fastdeploy:send_cache_failed_num Counter Total number of cache send failures count

Accessing Metrics

  • Access URL: http://localhost:8000/metrics
  • Metric Type: Prometheus format

Trace Events

FastDeploy outputs structured trace events to trace.log at key stages of request processing, useful for diagnosing per-request latency bottlenecks. Each trace log entry contains fields such as timestamp (milliseconds), request_id, event, and stage.

Common Events (Mixed / All Instances)

Stage Event Description
PREPROCESSING PREPROCESSING_START API Server begins preprocessing the request
PREPROCESSING PREPROCESSING_END Engine receives the request, preprocessing complete
SCHEDULE REQUEST_SCHEDULE_START Request enters the scheduling flow
SCHEDULE REQUEST_QUEUE_START Request enters the scheduling queue
SCHEDULE REQUEST_QUEUE_END Request dequeued from the scheduling queue
SCHEDULE RESOURCE_ALLOCATE_START Resource allocation begins for the request
SCHEDULE PREPARE_PREFIX_CACHE_START Prefix cache block matching begins
SCHEDULE PREPARE_PREFIX_CACHE_END Prefix cache block matching complete
SCHEDULE RESOURCE_ALLOCATE_END Resource allocation complete
SCHEDULE REQUEST_SCHEDULE_END Scheduling flow complete
PREFILL INFERENCE_START Request sent to GPU for inference
PREFILL FIRST_TOKEN_GENERATED First token generated
DECODE DECODE_START Enters Decode phase
DECODE INFERENCE_END Inference complete (all tokens generated)
DECODE PREEMPTED Request preempted
DECODE RESCHEDULED_INFERENCE_START Preempted request rescheduled for execution
POSTPROCESSING WRITE_CACHE_TO_STORAGE_START Begins writing KV Cache to external storage
POSTPROCESSING WRITE_CACHE_TO_STORAGE_END KV Cache written to external storage
POSTPROCESSING POSTPROCESSING_START Post-processing begins
POSTPROCESSING POSTPROCESSING_END Post-processing complete, response sent

PD Disaggregation — Prefill (P) Instance Events

Stage Event Description
SCHEDULE ASK_DECODE_RESOURCE_START P begins requesting resources from D (sends ZMQ request)
SCHEDULE ASK_DECODE_RESOURCE_END P receives resource allocation confirmation from D (with dest_block_ids)
PREFILL PREFILL_INFERENCE_END P instance Prefill inference complete
POSTPROCESSING CHECK_CACHE_TRANSFER_START P begins waiting for KV Cache transfer to complete
POSTPROCESSING CHECK_CACHE_TRANSFER_END KV Cache transfer confirmed, ready to send first token to D

PD Disaggregation — Decode (D) Instance Events

Stage Event Description
DECODE DECODE_PROCESS_PREALLOCATE_REQUEST_START D begins processing resource allocation request from P
DECODE DECODE_PROCESS_PREALLOCATE_REQUEST_END D completes resource allocation and returns dest_block_ids to P
DECODE DECODE_PROCESS_PREFILLED_REQUEST_START D receives first token from P, begins processing Prefilled request
DECODE DECODE_PROCESS_PREFILLED_REQUEST_END D adds Prefilled request to running queue
DECODE DECODE_INFERENCE_END D instance Decode inference complete

Request Lifecycle Sequence

Mixed mode (single instance, full inference):

PREPROCESSING_START → PREPROCESSING_END → REQUEST_QUEUE_START → REQUEST_QUEUE_END
→ RESOURCE_ALLOCATE_START → RESOURCE_ALLOCATE_END → INFERENCE_START
→ FIRST_TOKEN_GENERATED → DECODE_START → INFERENCE_END
→ POSTPROCESSING_START → POSTPROCESSING_END

PD Disaggregation — Prefill (P) Instance:

PREPROCESSING_START → PREPROCESSING_END → REQUEST_QUEUE_START → REQUEST_QUEUE_END
→ ASK_DECODE_RESOURCE_START → ASK_DECODE_RESOURCE_END
→ RESOURCE_ALLOCATE_START → RESOURCE_ALLOCATE_END
→ INFERENCE_START → PREFILL_INFERENCE_END
→ CHECK_CACHE_TRANSFER_START → CHECK_CACHE_TRANSFER_END → [send first token to D]

PD Disaggregation — Decode (D) Instance:

PREPROCESSING_START → PREPROCESSING_END → REQUEST_QUEUE_START → REQUEST_QUEUE_END
→ DECODE_PROCESS_PREALLOCATE_REQUEST_START → DECODE_PROCESS_PREALLOCATE_REQUEST_END
→ [wait for P to complete prefill and transfer KV Cache]
→ DECODE_PROCESS_PREFILLED_REQUEST_START → DECODE_PROCESS_PREFILLED_REQUEST_END
→ INFERENCE_START → DECODE_INFERENCE_END
→ POSTPROCESSING_START → POSTPROCESSING_END