feat: new populate initial data from prod#927
Open
Winzen wants to merge 37 commits intobackup-before-chatbot-stagingfrom
Open
feat: new populate initial data from prod#927Winzen wants to merge 37 commits intobackup-before-chatbot-stagingfrom
Winzen wants to merge 37 commits intobackup-before-chatbot-stagingfrom
Conversation
Winzen
changed the title
Winzen
added
the
staging
…and_more #944 fix: migrations account 0025_role_description_en_role_description_es_and_more NOTE: This migration was created automatically by the modeltranslation extension on 2025-02-04 but was not committed. Because our start-server.sh runs the makemigrations command, the migration file was created directly in the production pod and applied as 0021. This created divergences between our migration files in the repository and the actual state of the production database. We fixed this on December 17, 2025 by: - Deleting the original 0021 migration entry from the django_migrations table - Creating this migration file with the correct sequence number (0025) - Running a fake migration to register 0025 in django_migrations without re-executing the SQL
chore: update resources - main, staging and dev #945 - fix if expression (it was always evaluating to truthy) - update pod resources for dev, staging and production environments - add 10 minutes timeout to deployments
renamed `ChatbotThreadListView` to `ThreadListView` and created `CheckpointView` for deleting checkpoints refactored views added chatbot env variables temp: added a script for running django in dev mode and updated the Dockerfile for installing the `chatbot` package updated migrations fixed django admin added `djangorestframework` and `djangorestframework-simplejwt` packages as dependencies comment could be empty/null added jwt tokens urls changed `chatbot` package installation path created serializers and refactored views, added jwt auth, error handling, etc. reminder for custom authentication rules updated dockerfile and compose file for chromadb fixing a TypeError (the assistant expects `thread_id` as a string) and using the `get_sync_sql_assistant` helper method Adding chatbot as submodule Download submodules in ci. Also update README to alert on cloning submodules minor stuff adjusting indentation updated chatbot package and using validated data updated chatbot package version to `v0.3.0` validated data is serialized to a UUID but `UserMessage` expects a string added healthcheck to vector database service created a request validation function using pydantic just renaming views and endpoints added authentication to the checkpoint deletion endpoint added docstrings update `chatbot` package to version `v0.4.0` update .gitignore updated volume mount path to match the expected path on chroma v0.6.3 updated `chatbot` package to `v0.4.1` and using Serializers for request validation updated chatbot package version to v0.4.2 created `ChatbotDatabase` to read metadata from the local database and query data on BigQuery created chatbot authentication field on user account model sending feedbacks to langsmith fixed authentication rule (when a user is not found in db, django simple jwt returns `None`, so it has no `has_chatbot_access` attribute checking if the thread exists before trying to delete its checkpoints and returning 201 on thread creation added unit tests for the endpoints added `chatbot` dependency update env variables file optimize docker image the `chatbot` package was added to the api dependencies, so it doesn't need to be installed it directly. add chatbot migrations removed unused type edited server starting scripts update compose file updated chatbot admin models update account admin model to show the `has_chatbot_access` flag remove unused pydantic import cloning submodules in ci workflow add missing `Account` migrations we don't need to checkout submodules in the deploy step because the docker image is already built and we're only using the helm templates we need to use a deploy key to checkout the `chatbot` submodule because it's private we don't need to checkout submodules in this action because it only watches for new chart versions building `db_url` on runtime add non-sensitive env variables apps.py so chatbot shows in admin remove chatbot related env variables update `chatbot` package version to `0.4.4` enable `chatbot` package logging update poetry lock file fix imports and formatting to pass linting CI workflow fix imports and formatting to pass linting CI workflow increase dev ram memory limit to 1.5Gi remove chroma related env variables stopped installing packages needed for chromadb update dependencies update chatbot views and tests update `chatbot` package to version `v0.4.5` add `__init__.py` file to `apps` so python sees it as a package and pytest can import its chatbot module correctly. discard changes in Dockerfile setting model name directly in plain text use pgvector for similarity search use special valua `__all__` to include all model fields remove chroma from compose files add `PGVECTOR_COLLECTION` env variable minor fixes added `EMBEDDING_MODEL` env variable created `populate_pgvector` command improved logging running `populate_pgvector` script in the foreground run `populate_pgvector` command when starting the server rewrite query when invoking the assistant update `chatbot` package to version `0.5.1` set top-k to 5, i.e., retrieve the metadata of the 5 most relevant datasets remove deploy key setup as the submodule is now public update log message improved comments remove `model_provider` arg support multiple chats using full table ids and spaces instead of tabs for tables metadata remove default value, as `None` is the default already. add optional ordering for `ThreadListView` and `MessageListView` update `chatbot` app test cases add a single migration file return only non-deleted threads refactor thread deletion endpoint make `title` a required field in the `Thread` model and populate old threads `title` field update `chatbot` package to version `0.5.2` prepare `MessagePair` model for streaming add streaming support use `Response` and `status` from DRF update `chatbot` package to version `0.5.2` fix chatbot migration comments fix: correctly parse parallel tool calls Add token bridge endpoint for chatbot authentication fix: use poetry to install and run the project fix: make poetry venv accessible to all users add comments to `Dockerfile` increase helm timeout period to 10 minute revert `Dockerfile` changes to debug deploy timeout restore `Dockerfile` updates to use Poetry chore: update chatbot package (#857) chore: update chatbot package to `v0.6.1` (#858) fix: update chatbot v0.6.1 (#859) feat: react agent (#873) feat: query billing limit (#875) feat: custom react agent (#876) chore: system prompt tuning + other stuff (#884) fix: empty ai message (#887) refactor: truncate tool output (#895) chore: update agent prompt (#897) chore: add usage guides (#903) style: run ruff linter chore: add `contains=tables` parameter (#905) chore: chatbot service account (#911) feat: show sql query (#912) chore: remove submodule `chatbot` (#896) feat: add user ID to BigQuery query job (#926) chore: improve metadata usage (#928) perf: add async support (#932) perf: use rest transport (#933) chore: adjust chatbot env (#937) docs: update readme (#929) chore: prepare for staging and prod (#938) chore: update poetry lock file chore: update `.env.example` file refactor: using already existing `BACKEND_URL` variable from django settings (#940) fix: base backend url (#941) remove unused variable `BACKEND_BASE_URL` from `chatbot.agent.tools` module chore: fix migrations order chore: set `pipefail` option before curl chore: set shell with pipefail chore: fix locustfile exceptions messages chore: update ruff chore: fix ruff lint errors chore: fix migrations order chore: update ruff and sqlfmt pre-commit config chore: sort dependencies in alphabetical order chore: preserve quotes in yamlfix chore: migrate to official ruff action with latest version fix: template response processing in `LoggerMidleware` chore: revert changes in backend.custom.environment chore: standardized images versions and api ports fix: remove duplicated timeout
feat: chatbot to staging
fix: fixing missing static files in local development #948
feat: chatbot to main
fix: optimize search view
chore: use gthread
# Descrição do Problema: Atualmente, estamos enfrentando dificuldades ao tentar abrir alguns datasets devido a problemas de otimização. O uso da CPU atinge 100%, o que impede a conclusão do processo de carregamento, resultando em um erro de timeout. ### Ação: Estou implementando a limpeza de alguns campos para garantir que os datasets possam ser abertos, ao menos temporariamente, enquanto aguardamos uma solução definitiva de otimização.
feat: catalog download
Feat/catolog download
Este PR implementa o fluxo completo de notificação por e-mail para usuários inscritos em atualizações de tabelas, incluindo verificação de mudanças, envio de e-mails e execução automática via tarefa agendada.
Este PR adiciona o comando `disable_unhealthy_flow_schedules`, responsável por **desativar automaticamente schedules de flows não saudáveis no Prefect**, com base no histórico recente de execuções. Estamos adicionando o comando `disable_unhealthy_flow_schedules`, que: * Identifica flows com execuções recentes problemáticas * Avalia a saúde do flow a partir das **duas últimas execuções concluídas** * Desativa automaticamente o schedule de flows considerados não saudáveis
Adiciona filtro na query GraphQL `LastTwoCompletedRunsWithTasks` para garantir que apenas `flow_runs` com `start_time` diferente de `null` sejam retornadas.
- Add `DJANGO_JWT_ALGORITHM` env variable. - Create `db_network` (external network) in the compose file.
- Add `account.uuid` to JWT token payload. - Add unique constraint to `account.uuid`.
# fix: disable_unhealthy_flow_schedules and _check_for_updates
## disable_unhealthy_flow_schedules
No Prefect foi identificado um comportamento intermitente na desativação de schedules.
Devido a uma **race condition entre o scheduler e a API**, quando o scheduler ainda está processando ticks no momento da mutation, o schedule pode não ser efetivamente desativado na primeira chamada — mesmo retornando `success`.
Isso fazia com que flows considerados não saudáveis continuassem gerando novas execuções, exigindo nova tentativa manual para efetivar a desativação.
---
Foi implementada uma segunda tentativa imediata de desativação do schedule para cada flow validado como unhealthy:
```python
for _ in range(2): # Existe um bug onde o Flow não desativa com apenas uma query
self.set_flow_schedule(flow_id=flow.id, active=False)
```
Essa abordagem mitiga a race condition do scheduler e garante que o schedule seja efetivamente desativado, tornando o processo determinístico e evitando a necessidade de reexecução manual.
## check_for_updates
Foi identificado um erro na função check_for_updates. Algumas tables não possuem o atributo last_updated_at, o que gerava exceção durante a comparação e interrompia o fluxo normal da execução.
Também foi ajustada a função check_for_updates para tratar casos onde table.last_updated_at não está presente, evitando que a ausência desse atributo gere exceção e quebre a execução. Agora, nesses casos, a função trata o erro de forma segura e retorna False, preservando a estabilidade do processo.
Winzen
force-pushed
the
feat/local_populate
branch
from
035f5c7 to
49feb72
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.