SK-2813: resolve pr comments and update claude.md file

Author: saileshwar-skyflow

.claude/CLAUDE.md

@@ -1 +1,84 @@
-See @../AGENTS.md for documentation on how to configure and use Claude agents in this workspace.
+# CLAUDE.md
+
+## Commands
+
+**Install dependencies:**
+```bash
+pip install -r requirements.txt
+pip install ".[dev]"   # includes ruff and codespell
+```
+
+**Run all tests:**
+```bash
+python -m coverage run --source=skyflow \
+  --omit=skyflow/generated/*,skyflow/utils/validations/*,skyflow/vault/data/*,skyflow/vault/detect/*,skyflow/vault/tokens/*,skyflow/vault/connection/*,skyflow/error/*,skyflow/utils/enums/*,skyflow/vault/controller/_audit.py,skyflow/vault/controller/_bin_look_up.py \
+  -m unittest discover
+```
+
+**Run a single test:**
+```bash
+python -m unittest tests.vault.controller.test__vault.TestVault.test_insert
+```
+
+**Lint:**
+```bash
+ruff check . --output-format=github
+```
+
+**Spell check:**
+```bash
+codespell
+```
+
+**Build package:**
+```bash
+python setup.py sdist bdist_wheel
+```
+
+**Commit message format:** All commits must include a Jira ticket ID, e.g. `SDK-123: description`. CI enforces this on PRs.
+
+## Architecture
+
+### Entry point and builder
+
+`skyflow/client/skyflow.py` — `Skyflow` is a facade built via `Skyflow.builder()`. The inner `Builder` class accumulates vault configs, connection configs, and optional Skyflow-level credentials. Calling `.build()` validates all configs and instantiates `VaultClient` + controller objects per config. Each `add_vault_config()` call creates a `Vault` controller and a `Detect` controller backed by the same `VaultClient`.
+
+### Three controllers
+
+- `skyflow/vault/controller/_vault.py` — `Vault`: insert, get, update, delete, query, detokenize, tokenize, upload_file. All methods follow: validate → `__initialize()` → call generated API → parse response.
+- `skyflow/vault/controller/_connections.py` — `Connection`: single `invoke()` method that builds an HTTP request via `requests.PreparedRequest` and sends it directly (bypasses the generated client).
+- `skyflow/vault/controller/_detect.py` — `Detect`: deidentify/reidentify text and files. File operations use long-polling (`__poll_for_processed_file`) with exponential back-off up to a configurable `wait_time` (max 64 s).
+
+### VaultClient (auth and token caching)
+
+`skyflow/vault/client/client.py` — Holds the generated REST client (`skyflow/generated/rest/`) and manages bearer token lifecycle. `initialize_client_configuration()` is called before every API request; it skips re-initialization if a non-expired token exists. Credential resolution order: config-level credentials → Skyflow-level credentials → `SKYFLOW_CREDENTIALS` env var.
+
+### Service account / JWT layer
+
+`skyflow/service_account/_utils.py` — Standalone functions for bearer token generation (`generate_bearer_token`, `generate_bearer_token_from_creds`) and signed data tokens (`generate_signed_data_tokens`). Signs a JWT with RS256 then exchanges it at the `tokenURI` via `AuthClient`. `is_expired()` decodes JWT without verification to check `exp`.
+
+### Generated REST client
+
+`skyflow/generated/rest/` — **Do not edit.** Auto-generated by Fern from the Skyflow API spec. All vault/token/query/detect/file API calls go through this layer. The `Skyflow` class in `generated/rest/client.py` is the raw HTTP client (separate from `skyflow/client/skyflow.py`).
+
+### Validation
+
+`skyflow/utils/validations/_validations.py` — All request and config validation lives here. Every public SDK method calls a corresponding `validate_*` function before touching the API. Config-level validation (vault, connection, credentials) runs at `build()` / `add_*_config()` time.
+
+### Error handling
+
+`skyflow/error/_skyflow_error.py` — `SkyflowError(message, http_code, request_id, grpc_code, http_status, details)`. `handle_exception()` in `skyflow/utils/_utils.py` converts raw generated-client exceptions into `SkyflowError` by inspecting `content-type` of the error response.
+
+### Messages and constants
+
+- `skyflow/utils/_skyflow_messages.py` — All user-facing error strings, log strings, and HTTP status enums. Strings are grouped into `Error`, `ErrorLogs`, `Info`, `ErrorCodes`, `HttpStatus` inner enums.
+- `skyflow/utils/constants.py` — Named constant classes (`CredentialField`, `JWT`, `SdkMetricsKey`, etc.) and top-level constants (`PROTOCOL`, `SKY_META_DATA_HEADER`).
+- `skyflow/utils/enums/` — `LogLevel`, `Env`, `RedactionType`, `TokenMode`, `ContentType`, `DetectEntities`, etc.
+
+### Request/Response objects
+
+Plain dataclasses in `skyflow/vault/data/`, `skyflow/vault/tokens/`, `skyflow/vault/detect/`, and `skyflow/vault/connection/`. Response objects always have an `errors` field (list or `None`).
+
+### SDK metadata header
+
+Every API call appends `sky-metadata: <json>` via `__get_headers()`. The metrics dict (`skyflow/utils/_utils.py:get_metrics()`) is computed once per process and cached in `_CACHED_METRICS`.

skyflow/error/_skyflow_error.py

@@ -14,4 +14,4 @@ def __init__(self,
         self.http_status = http_status if http_status else SkyflowMessages.HttpStatus.BAD_REQUEST.value
         self.details = details
         self.request_id = request_id
-        super().__init__()
+        super().__init__(message)

skyflow/service_account/_utils.py

@@ -240,8 +240,7 @@ def generate_signed_data_tokens_from_creds(credentials, options):
     return get_signed_tokens(json_credentials, options)
 
 def get_signed_data_token_response_object(signed_token, actual_token):
-    response_object = {
+    return {
         ResponseField.TOKEN: actual_token,
-        ResponseField.SIGNED_TOKEN: signed_token
+        ResponseField.SIGNED_TOKEN: signed_token,
     }
-    return response_object.get(ResponseField.TOKEN), response_object.get(ResponseField.SIGNED_TOKEN)

skyflow/utils/_utils.py

@@ -421,22 +421,30 @@ def parse_invoke_connection_response(api_response: requests.Response):
             error_response = json.loads(content)
             error_from_client = api_response.headers.get(HttpHeader.ERROR_FROM_CLIENT)
 
-            status_code = error_response.get(ResponseField.ERROR, {}).get(ResponseField.HTTP_CODE, status_code)
-            http_status = error_response.get(ResponseField.ERROR, {}).get(ResponseField.HTTP_STATUS)
-            grpc_code = error_response.get(ResponseField.ERROR, {}).get(ResponseField.GRPC_CODE)
-            details = error_response.get(ResponseField.ERROR, {}).get(ResponseField.DETAILS)
-            message = error_response.get(ResponseField.ERROR, {}).get(ResponseField.MESSAGE, SkyflowMessages.Error.UNKNOWN_ERROR_DEFAULT_MESSAGE.value)
-            
+            http_status = None
+            grpc_code = None
+            details = None
+
+            error_obj = error_response.get(ResponseField.ERROR) if isinstance(error_response, dict) else None
+            if isinstance(error_obj, dict):
+                status_code = error_obj.get(ResponseField.HTTP_CODE, status_code)
+                http_status = error_obj.get(ResponseField.HTTP_STATUS)
+                grpc_code = error_obj.get(ResponseField.GRPC_CODE)
+                details = error_obj.get(ResponseField.DETAILS)
+                message = error_obj.get(ResponseField.MESSAGE, message)
+            elif isinstance(error_obj, str) and error_obj:
+                message = error_obj
+
             if error_from_client is not None:
-                if details is None: 
+                if details is None:
                     details = []
                 error_from_client_bool = error_from_client.lower() == BooleanString.TRUE
                 details.append({ResponseField.ERROR_FROM_CLIENT: error_from_client_bool})
 
             raise SkyflowError(message, status_code, request_id, grpc_code, http_status, details)
-            
+
         except json.JSONDecodeError:
-            raise SkyflowError(content if content else message, status_code, request_id)
+            raise SkyflowError(message, status_code, request_id)
 
 def parse_deidentify_text_response(api_response: DeidentifyStringResponse):
     entities = [convert_detected_entity_to_entity_info(entity) for entity in api_response.entities]
@@ -480,21 +488,46 @@ def handle_exception(error, logger):
 
 def handle_json_error(err, data, request_id, logger):
     try:
-        if isinstance(data, dict):  # If data is already a dict
+        if isinstance(data, dict):
             description = data
         elif isinstance(data, ErrorResponse):
             description = data.dict()
         else:
             description = json.loads(data)
-        status_code = description.get(ResponseField.ERROR, {}).get(ResponseField.HTTP_CODE, HttpStatusCode.INTERNAL_SERVER_ERROR)
-        http_status = description.get(ResponseField.ERROR, {}).get(ResponseField.HTTP_STATUS)
-        grpc_code = description.get(ResponseField.ERROR, {}).get(ResponseField.GRPC_CODE)
-        details = description.get(ResponseField.ERROR, {}).get(ResponseField.DETAILS, [])
 
-        description_message = description.get(ResponseField.ERROR, {}).get(ResponseField.MESSAGE, SkyflowMessages.Error.UNKNOWN_ERROR_DEFAULT_MESSAGE.value)
-        log_and_reject_error(description_message, status_code, request_id, http_status, grpc_code, details, logger = logger)
+        if ResponseField.ERROR in description:
+            error_obj = description.get(ResponseField.ERROR, {})
+            status_code = error_obj.get(ResponseField.HTTP_CODE, HttpStatusCode.INTERNAL_SERVER_ERROR)
+            http_status = error_obj.get(ResponseField.HTTP_STATUS)
+            grpc_code = error_obj.get(ResponseField.GRPC_CODE)
+            details = error_obj.get(ResponseField.DETAILS, [])
+            description_message = error_obj.get(ResponseField.MESSAGE, SkyflowMessages.Error.UNKNOWN_ERROR_DEFAULT_MESSAGE.value)
+        elif ResponseField.RESPONSES in description:
+            responses = description.get(ResponseField.RESPONSES, [])
+            messages = []
+            status_code = HttpStatusCode.INTERNAL_SERVER_ERROR
+            for resp in responses:
+                resp_status = resp.get(ResponseField.STATUS, HttpStatusCode.INTERNAL_SERVER_ERROR)
+                resp_body = resp.get(ResponseField.BODY, {})
+                if isinstance(resp_status, int) and resp_status >= 400:
+                    status_code = resp_status
+                    error_msg = resp_body.get(ResponseField.ERROR)
+                    if error_msg:
+                        messages.append(str(error_msg))
+            description_message = '; '.join(messages) if messages else SkyflowMessages.Error.UNKNOWN_ERROR_DEFAULT_MESSAGE.value
+            http_status = None
+            grpc_code = None
+            details = []
+        else:
+            status_code = HttpStatusCode.INTERNAL_SERVER_ERROR
+            http_status = None
+            grpc_code = None
+            details = []
+            description_message = SkyflowMessages.Error.UNKNOWN_ERROR_DEFAULT_MESSAGE.value
+
+        log_and_reject_error(description_message, status_code, request_id, http_status, grpc_code, details, logger=logger)
     except json.JSONDecodeError:
-        log_and_reject_error(SkyflowMessages.Error.INVALID_JSON_RESPONSE.value, err, request_id, logger = logger)
+        log_and_reject_error(SkyflowMessages.Error.INVALID_JSON_RESPONSE.value, err, request_id, logger=logger)
 
 def handle_text_error(err, data, request_id, logger):
     log_and_reject_error(data, err.status, request_id, logger =  logger)

skyflow/utils/constants.py

@@ -117,6 +117,7 @@ class ResponseField:
     TYPE = 'type'
     TOKENIZED_DATA = 'tokenized_data'
     SIGNED_TOKEN = 'signed_token'
+    RESPONSES = 'responses'
 
 
 class CredentialField:
@@ -195,6 +196,7 @@ class DeidentifyFileRequestField:
     OUTPUT_OCR_TEXT = 'output_ocr_text'
     MASKING_METHOD = 'masking_method'
     PIXEL_DENSITY = 'pixel_density'
+    DENSITY = 'density'
     MAX_RESOLUTION = 'max_resolution'
     OUTPUT_PROCESSED_AUDIO = 'output_processed_audio'
     OUTPUT_TRANSCRIPTION = 'output_transcription'
@@ -230,6 +232,7 @@ class DeidentifyField:
     ENTITY_UNQ_COUNTER = 'entity_unq_counter'
     ENTITY_UNIQUE_COUNTER = 'entity_unique_counter'
     ENTITY_ONLY = 'entity_only'
+    VAULT_TOKEN = 'vault_token'
     ENTITIES = 'entities'
     MAX_DAYS = 'max_days'
     MIN_DAYS = 'min_days'

skyflow/utils/enums/detect_output_transcriptions.py

@@ -4,4 +4,5 @@ class DetectOutputTranscriptions(Enum):
     DIARIZED_TRANSCRIPTION = "diarized_transcription"
     MEDICAL_DIARIZED_TRANSCRIPTION = "medical_diarized_transcription"
     MEDICAL_TRANSCRIPTION = "medical_transcription"
-    TRANSCRIPTION = "transcription"
+    TRANSCRIPTION = "transcription"
+    PLAINTEXT_TRANSCRIPTION = "plaintext_transcription"

skyflow/utils/validations/_validations.py

@@ -232,8 +232,10 @@ def validate_update_vault_config(logger, config):
     if ConfigField.ENV in config and config.get(ConfigField.ENV) not in Env:
         raise SkyflowError(SkyflowMessages.Error.INVALID_ENV.value.format(vault_id), invalid_input_error_code)
 
-    if ConfigField.CREDENTIALS in config and config.get(ConfigField.CREDENTIALS):
-        validate_credentials(logger, config.get(ConfigField.CREDENTIALS), ConfigType.VAULT, vault_id)
+    if ConfigField.CREDENTIALS not in config:
+        raise SkyflowError(SkyflowMessages.Error.EMPTY_CREDENTIALS.value.format(ConfigType.VAULT, vault_id), invalid_input_error_code)
+
+    validate_credentials(logger, config.get(ConfigField.CREDENTIALS), ConfigType.VAULT, vault_id)
 
     return True
 
@@ -255,8 +257,10 @@ def validate_connection_config(logger, config):
         SkyflowMessages.Error.INVALID_CONNECTION_URL.value.format(connection_id)
     )
 
-    if ConfigField.CREDENTIALS in config:
-        validate_credentials(logger, config.get(ConfigField.CREDENTIALS), ConfigType.CONNECTION, connection_id)
+    if ConfigField.CREDENTIALS not in config:
+        raise SkyflowError(SkyflowMessages.Error.EMPTY_CREDENTIALS.value.format(ConfigType.CONNECTION, connection_id), invalid_input_error_code)
+
+    validate_credentials(logger, config.get(ConfigField.CREDENTIALS), ConfigType.CONNECTION, connection_id)
 
     return True
 
@@ -298,7 +302,7 @@ def validate_file_from_request(file_input: FileInput):
     if has_file:
         file = file_input.file
         # Validate file object has required attributes
-        if not hasattr(file, FileUploadField.FILE_NAME) or not isinstance(file.name, str) or not file.name.strip():
+        if not hasattr(file, FileUploadField.NAME) or not isinstance(file.name, str) or not file.name.strip():
             raise SkyflowError(SkyflowMessages.Error.INVALID_FILE_TYPE.value, invalid_input_error_code)
 
         # Validate file name

skyflow/vault/controller/_detect.py

@@ -187,7 +187,7 @@ def output_to_dict_list(output):
         return DeidentifyFileResponse(
             file_base64=base64_string,
             file=file_obj,
-            type=first_output.get(DeidentifyField.TYPE, DetectStatus.UNKNOWN),
+            type=first_output.get(DeidentifyField.TYPE, None),
             extension=extension,
             word_count=word_count,
             char_count=char_count,
@@ -207,6 +207,7 @@ def __get_token_format(self, request):
             DeidentifyField.DEFAULT: getattr(request.token_format, DeidentifyField.DEFAULT, None),
             DeidentifyField.ENTITY_UNQ_COUNTER: getattr(request.token_format, DeidentifyField.ENTITY_UNIQUE_COUNTER, None),
             DeidentifyField.ENTITY_ONLY: getattr(request.token_format, DeidentifyField.ENTITY_ONLY, None),
+            DeidentifyField.VAULT_TOKEN: getattr(request.token_format, DeidentifyField.VAULT_TOKEN, None)
         }
 
     def __get_transformations(self, request):
@@ -346,7 +347,7 @@ def deidentify_file(self, request: DeidentifyFileRequest):
                     DeidentifyField.ALLOW_REGEX: request.allow_regex_list,
                     DeidentifyField.RESTRICT_REGEX: request.restrict_regex_list,
                     DeidentifyFileRequestField.MAX_RESOLUTION: getattr(request, DeidentifyFileRequestField.MAX_RESOLUTION, None),
-                    DeidentifyFileRequestField.PIXEL_DENSITY: getattr(request, DeidentifyFileRequestField.PIXEL_DENSITY, None),
+                    DeidentifyFileRequestField.DENSITY: getattr(request, DeidentifyFileRequestField.PIXEL_DENSITY, None),
                     DeidentifyField.REQUEST_OPTIONS: {'additional_headers': self.__get_headers()}
                 }
 

skyflow/vault/detect/_deidentify_file_response.py

@@ -1,4 +1,5 @@
 import io
+from typing import Optional
 from skyflow.vault.detect._file import File
 
 class DeidentifyFileResponse:
@@ -17,6 +18,7 @@ def __init__(
         entities: list = None,  # list of dicts with keys 'file' and 'extension'
         run_id: str = None,
         status: str = None,
+        errors: Optional[list] = None,
     ):
         self.file_base64 = file_base64
         self.file = File(file) if file else None
@@ -31,6 +33,7 @@ def __init__(
         self.entities = entities if entities is not None else []
         self.run_id = run_id
         self.status = status
+        self.errors = errors
 
     def __repr__(self):
         return (
@@ -40,7 +43,7 @@ def __repr__(self):
             f"char_count={self.char_count!r}, size_in_kb={self.size_in_kb!r}, "
             f"duration_in_seconds={self.duration_in_seconds!r}, page_count={self.page_count!r}, "
             f"slide_count={self.slide_count!r}, entities={self.entities!r}, "
-            f"run_id={self.run_id!r}, status={self.status!r})"
+            f"run_id={self.run_id!r}, status={self.status!r}, errors={self.errors!r})"
         )
 
     def __str__(self):

skyflow/vault/detect/_deidentify_text_response.py

@@ -1,19 +1,21 @@
-from typing import List
+from typing import List, Optional
 from ._entity_info import EntityInfo
 
 class DeidentifyTextResponse:
-    def __init__(self, 
+    def __init__(self,
                 processed_text: str,
-                entities: List[EntityInfo], 
+                entities: List[EntityInfo],
                 word_count: int,
-                char_count: int):
+                char_count: int,
+                errors: Optional[list] = None):
         self.processed_text = processed_text
         self.entities = entities
         self.word_count = word_count
         self.char_count = char_count
+        self.errors = errors
 
     def __repr__(self):
-        return f"DeidentifyTextResponse(processed_text='{self.processed_text}', entities={self.entities}, word_count={self.word_count}, char_count={self.char_count})"
+        return f"DeidentifyTextResponse(processed_text='{self.processed_text}', entities={self.entities}, word_count={self.word_count}, char_count={self.char_count}, errors={self.errors})"
 
     def __str__(self):
         return self.__repr__()