Great project! How to test?

[philon123]

Hi, I just found your project while googling for sqlc and timescaledb.. it was a slightly unrelated search result but spot on for what I'm looking for - I found several of the tools you use but missed some critical ones like xo and rtk-query. So, awesome! And thanks!

I cloned your repo and ran a go mod tidy. But now I'm lost. I see the screenshot of the CLI on this repo, but can't seem to run it becaue I'm missing some .env file:
./bin/project [2023-04-16T14:25:02+0200]: File .env.dev does not exist Terminated

Trying to run ./cmd/cli/main.go in vscode doesn't work out of the box since it's defaulting to the "attach to process" launch config.

I didn't mess with it further now, first wanted to check if I'm on the right path. Any help would be apppreciated.

[danicc097]

Quite surprising that this even turns up in search results (and that someone finds it useful).

I'm not sure what part of this repo you're interested in, since I use timescaledb for opentelemetry only, the rest is Postgres.
I don't use rtk-query anymore, I recommend https://orval.dev/ instead which you can find in frontend.
I also use a custom xo fork tightly coupled to postgres and the pgx driver only (see project's install-tools function) with heavily modified templates in internal.

If you want to run everything, first run the install-???.sh scripts if you're missing something from there, project bootstrap to install deps, ensure project check-build-deps is alright, then project gen to ensure it works and you get the same output.
If you copy .env.ci and modify its contents replacing ci for dev it should work as is.

To run backend and frontend in dev mode: project run-dev.
To run backend tests, the first time run project test.backend-setup then project test directly.
Frontend is just like any other web app via package.json, nothing fancy there

You can run opentelemetry services with bin/deploy-instrumentation, which needs a .env file (see .env.ci in that folder)

And thank you for your comments, hopefully you can extract something useful out of this

[philon123]

Wow, this is more complex than I thought initially. I just found the 1500 lines of bash in bin/project ;) I'll document my setup experience here, hopefully for your benefit. A section in the README with instructions would be great!

• Error executing install-pg_format.sh on line 10. Because of the preceding sudo, the files are owned by root, so the non-sudo rm fails.
• Needed to copy docker/.env.ci to docker/.env
• protoc and column got installed to ~/bin, which is unusual. I needed to add that to my PATH manually, and will need to update my .bashrc to persist that change. Why not install it in a more usual path, or locally in the bin folder?
• I'm on Ubuntu-22.10, which doesn't have docker-compose installed any more. Instead, it's docker compose. I modified the deps-check.sh line 91 accordingly and I replaced all occurances in the code.
• missing from project bootstrap: sudo apt install direnv
• missing from project bootstrap (use /usr/bin/ for all the custom binaries?): sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
• yq version check failed. My version string is yq (https://github.com/mikefarah/yq/) version v4.33.3, so it should be ok right? I don't understand the regex enough to fix it unfortunately, so I'll comment out the check
• bin/project gen hangs indefinitely with Waiting for the postgres database to be ready. I don't see network activity, so it's not downloading any containers.
• Running bin/project run-dev to bring up the DB and any other stuff - but I get an error: network traefik-net declared as external, but could not be found. docker network create traefik-net fixes that.
• trying bin/project gen again, this time with error: github.com/jackc/pgtype@v1.11.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod - a go mod vendor fixes it by modifying vendor/modules.txt
• Now bin/project run-dev throws error gen.proto > Could not make proto path relative: internal/python-ml-app-protos/tfidf/v1/service.proto: No such file or directory. A git submodule init and git submodule update fixes that.
• missing pnpm. sudo npm install -g @pnpm/exe
• missing sponge. sudo apt install moreutils
• bin/project run-dev again, but error from gen.frontend about missing dependencies. So cd frontend && pnpm install. But that gives error:  ERR_PNPM_PREPARE_PACKAGE  Failed to prepare git-hosted package fetched from "https://codeload.github.com/danicc097/openapi-typescript-validator/tar.gz/cd83d434bd9b924f157f16172f1ffacb36d53f8a": openapi-typescript-validator@4.0.0 postpublish: git push && git push --tags Exit status 128

At this point I don't know how to continue. I feel like I got quite close though? :) I hope my comments are helpful and I'd be over the moon if you could help me finish the setup!

The reason I'm interested in your project is that the amount of tooling required to bring up a DB centric project with backend and frontend is just so high. And it require a lot of LOC spent on basic operations like infra setup, CRUD, and form validation (both on backend and frontend). A stack like this would be hugely beneficial to new projects, especially for small team sizes.

[danicc097]

• Error executing install-pg_format.sh on line 10. Because of the preceding sudo, the files are owned by root, so the non-sudo rm fails.
  • I just realized pgformatter is available through standard package managers. I'll replace with a dep list and include sudo apt install pgformatter
• Needed to copy docker/.env.ci to docker/.env
  • Ill add the list of .env.templates that need to be changed, as well as available envs since theyre not obvious (dev,ci,prod)
• protoc and column got installed to ~/bin, which is unusual. I needed to add that to my PATH manually, and will need to update my .bashrc to persist that change. Why not install it in a more usual path, or locally in the bin folder?
  • I agree these should definitely go in the project's tools folder
    somewhere and add its path to .envrc.
• I'm on Ubuntu-22.10, which doesn't have docker-compose installed any more. Instead, it's docker compose. I modified the deps-check.sh line 91 accordingly and I - replaced all occurances in the code.
  • I should actually replace everything with the new docker compose instead
    of docker-compose (https://docs.docker.com/compose/compose-v2/). On a
    tangent, this got me thinking of adding
    a gitignored .envrc.local and an alias
    workaround.
    This way, you can have your .envrc.local with export_alias docker-compose 'docker compose $@' for example.
• missing from project bootstrap: sudo apt install direnv
  • Ill add pgformatter to this as well
• missing from project bootstrap (use /usr/bin/ for all the custom binaries?): sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/- bin/yq && sudo chmod +x /usr/bin/yq
  • Maybe yq should also be project specific and a install-yq.sh dumps it to tools/..., since we have kislyuk/yq and mikefarah's
• yq version check failed. My version string is yq (https://github.com/mikefarah/yq/) version v4.33.3, so it should be ok right? I don't understand the regex enough to - fix it unfortunately, so I'll comment out the check
  • I see they added the v prefix recently, I fixed it now
• bin/project gen hangs indefinitely with Waiting for the postgres database to be ready. I don't see network activity, so it's not downloading any containers.
  • Not quite sure where this could come from honestly. Was gen_db created
    in your database? That's whats the whole gen pipeline connects to exclusively
• Running bin/project run-dev to bring up the DB and any other stuff - but I get an error: network traefik-net declared as external, but could not be found. docker - network create traefik-net fixes that.
  • Yep, was only being created on run-prod, good catch. I moved it to
    run_shared_services which makes much more sense
• trying bin/project gen again, this time with error: github.com/jackc/pgtype@v1.11.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod - a go mod vendor fixes it by modifying vendor/modules.txt
  • I still haven't dedicated time to figure out how to use vendor properly so its always
    synced and used, should investigate
• Now bin/project run-dev throws error gen.proto > Could not make proto path relative: internal/python-ml-app-protos/tfidf/v1/service.proto: No such file or directory. - A git submodule init and git submodule update fixes that.
  • A git submodule update --init --recursive on project bootstrap
    should help
• missing pnpm. sudo npm install -g @pnpm/exe
  • To bootstrap it goes
• missing sponge. sudo apt install moreutils
  • To bootstrap it goes, was only mentioned in github actions
• bin/project run-dev again, but error from gen.frontend about missing dependencies. So cd frontend && pnpm install. But that gives error:  ERR_PNPM_PREPARE_PACKAGE  Failed to prepare git-hosted package fetched from "https://codeload.github.com/danicc097/openapi-typescript-validator/tar.gz/cd83d434bd9b924f157f16172f1ffacb36d53f8a": openapi-typescript-validator@4.0.0 postpublish: git push && git push --tags Exit status 128
  • Not sure about this one either. --ignore-scripts
    is definitely not what we want from our side. Since that is a fork I won't publish
    anywhere I just removed postpublish. Try it again in a while, if it still
    doesn't work then try with pnpm 7.6.0

I'll attempt to craft meaningful setup instructions at the very least
(shamelessly stolen from this whole issue for the most part :)).

The actual app is just an incomprehensible broken demo at this point, but indeed the
barebones could be useful, as you mention, for small teams. There are many rough
edges especially around code generation now, leading to that unwieldly bash mess in
project, which needs to be adapted to every app and its openapi spec, database, etc. since it's three way generation between internal structs (via openapi-go), openapi spec and database models/enums
(not sure how to decouple any of that, probably not even possible, but I haven't yet given it a thought since I don't plan
for this repo to become a template, just an example app).

Indeed you are quite close to running it, maybe also check the github
action and then
https://github.com/danicc097/openapi-go-gin-postgres-sqlc/blob/main/.github/workflows/tests.yaml
as it should work out of the box

[philon123]

• Alright, now cd frontend && pnpm install worked! Thanks for the fix. Also bin/project gen worked.
• bin/project test.backend-setup almost worked, it just failed right at the end running prettier. The error is not vbery descriptive: gen > gen.frontend > 🛑 Failed to run afterAllFilesWrite hook: Error: Command failed with exit code 1: prettier --write [...]
• bin/project test spits out a bunch of form validation errors, but looks like that's expected (trying to pass 500 to a value that may be max 100, things like that). However, then it starts spitting errors that look real. api_events_test.go:76: Couldn't run test server: New: error creating provider Get "https://authserver.dev.localhost/oidc/.well-known/openid-configuration": dial tcp 127.0.0.1:443: connect: connection refused.. The attached screenshot is the last output that the test print before exiting.
  
• I figured the frontend container is just missing, so bin/project run-dev: It prints with the same prettier error from earlier. And it fails to start Vite dev server, because of permission denied listening on 127.0.0.1:80 I don't want to run it as root, so I modify .env.dev to use port 5143 (I see it defaults to that port if none is provided)
• again bin/project run-dev, with error [Air] 2023/04/18 13:20:30 Couldn't run: NewServer: error creating provider Get "https://authserver.dev.localhost/oidc/.well-known/openid-configuration": dial tcp 127.0.0.1:443: connect: connection refused Looking through the code, I find it's trying to connect to simple-oidc-server but that's listening on port 10001 and not sure how the SSL part is supposed to work. This is where I'm stuck now.
• The frontend actually loads! But with error [plugin:vite:import-analysis] Failed to resolve import "react-redux" from "src/hooks/auth/useAuthenticatedUser.tsx". Does the file exist?

[danicc097]

• bin/project test.backend-setup almost worked, it just failed right at the end running prettier. The error is not vbery descriptive: gen > gen.frontend > 🛑 Failed to run afterAllFilesWrite hook: Error: Command failed with exit code 1: prettier --write [...]
  - that's the last step in orval generation, shouldn't give any trouble. Do you have a ./node_modules/.bin/prettier? I doubt it's permission related but you never know.
  bin/project test spits out a bunch of form validation errors, but looks like that's expected (trying to pass 500 to a value that may be max 100, things like that). However, then it starts spitting errors that look real. api_events_test.go:76: Couldn't run test server: New: error creating provider Get "https://authserver.dev.localhost/oidc/.well-known/openid-configuration": dial tcp 127.0.0.1:443: connect: connection refused.. The attached screenshot is the last output that the test print before exiting.
  • it looks like you need a running traefik reverse proxy as in the action. Also make sure your containers are actually running:
    

I figured the frontend container is just missing, so bin/project run-dev: It prints with the same prettier error from earlier. And it fails to start Vite dev server, because of permission denied listening on 127.0.0.1:80 I don't want to run it as root, so I modify .env.dev to use port 5143 (I see it defaults to that port if none is provided)
- hmm it should definitely not use 80 by default, port: Number(Config.FRONTEND_PORT) || 5143. I barely touched the frontend, but it's missing tests in CI with a config.ci.json. You need to fill in config.template.json to config.json. An example is:

{
  "APP_ENV": "dev",
  "DOMAIN": "localhost",
  "API_VERSION": "/v2",
  "API_PREFIX": "",
  "API_PORT": "8090",
  "FRONTEND_PORT": "3020",
  "AUTH_SERVER_UI_LOGIN": "https://authserver.dev.localhost/oidc/login/username",
  "AUTH_SERVER_UI_PROFILE": "https://authserver.dev.localhost/oidc/profile"
}

(not all fields used at the moment)
again bin/project run-dev, with error [Air] 2023/04/18 13:20:30 Couldn't run: NewServer: error creating provider Get "https://authserver.dev.localhost/oidc/.well-known/openid-configuration": dial tcp 127.0.0.1:443: connect: connection refused Looking through the code, I find it's trying to connect to simple-oidc-server but that's listening on port 10001 and not sure how the SSL part is supposed to work. This is where I'm stuck now.
- Again this looks like traefik + mkcert setup missing; see the github action
The frontend actually loads! But with error [plugin:vite:import-analysis] Failed to resolve import "react-redux" from "src/hooks/auth/useAuthenticatedUser.tsx". Does the file exist?
- You will find many TS errors at this point since Im not focusing on frontend at this time and I migrated redux -> zustand + backend types were changed a lot as well.

[danicc097]

By the way, main branch is currently broken, I need to fix project swallowing exit codes which causes CI to pass. I'll just merge what I have now

Edit: actually lets wait... https://www.githubstatus.com/

[philon123]

• Do you have a ./node_modules/.bin/prettier? I doubt it's permission related but you never know.
  --> Yep, it's there

• You need to fill in config.template.json to config.json
  --> I already have that

• it looks like you need a running traefik reverse proxy as in the action. Also make sure your containers are actually running
  --> Cloning https://github.com/danicc097/traefik-bootstrap.git and inside there ./mkcert-certificates and ./compose-up, fixed the issue.

• Error from bin/project run-dev: [Air] 2023/04/18 17:22:12 Post "http://localhost:14268/api/traces": dial tcp 127.0.0.1:14268: connect: connection refused. That's the jaeger port, do I need to bring that up manually as well?

• I can reach the Swagger docs now. Nice! However, executing any function fails because there are no authorization headers set. How to get those set up?

• The react-redux import failing was indeed because it was missing from package.json. After adding that, I can load the frontend!

• The logs give one warning/error, I'll post it here just in case, even though it doesn't seem to be causing an issue for me:
  

So, I'm up and running! Thanks for all the help. Now I need to understand what's actually going on ;) It's parsing the DB schema and generating OpenAPI specs, backend API stubs and frontend validations from it right?

[danicc097]

• opentelemetry is optional, you can manually bring it up with bin/deploy-instrumentation. Then you can see jaeger traces, etc. if you make a request to some working endpoint like /user/me, see https://github.com/danicc097/openapi-go-gin-postgres-sqlc/blob/9aa2d41b9c803179eb9993379f33e8b42c9b4b8e/internal/rest/api_user.go

• To execute queries manually via the docs page you need to paste any user's api key in the database (user_api_keys) to x-api-key (its added to the swagger ui via securitySchemes) I think the initial data script already creates some

• Unless you plan to use redux, you can safely delete any "react-redux" imports

• Ive been getting that warning for a while, you can probably ignore it

The db schema itself is independent from the spec. However, in your spec you can reference database enums, or internal structs from https://github.com/danicc097/openapi-go-gin-postgres-sqlc/blob/8b8075f27b8172723b6dc75f270a59e862603e44/internal/postgen/structs.gen.go
which get generated by project. All db structs are generated with a heavily modified xo, the rest are regular structs. You can add struct tags that the openapi-go library interprets, so that gen-schema generates a temp openapi spec that I read with yq and merge with the current spec, in a nutshell. Validation is essentially done with regular openapi libraries for go and javascript

[philon123]

I rebased my changes on top of current main branch, if you want I can push a merge? It has

• docker compose instead of docker compose
• uses newer pnpm, which results in a differently formatted package.lock
• adds react-redux to package.json
• additional cleanup after installing protoc
• remove pgtype (it was complaining that it's unused)
• generated files look different for unknown reasons

[danicc097]

• docker compose instead of docker compose
  • absolutely
• uses newer pnpm, which results in a differently formatted package.lock
  - pnpm is also a good candidate to have a check.bin.pnpm and install.bin.pnpm which is the kind of flow Im using on lastest dev branch. As soon as I finish fixing project you can give it a try if you want, pnpm --version is easy to check
• adds react-redux to package.json
  - Actually I want to get rid of it for this app :) now it's zustand which is much simpler to use. I need to remove all references from frontend
• additional cleanup after installing protoc
  - I moved all installation scripts and checks to project check-build-deps. now it will both check and prompt to install. THe installation is either via apt or with bins moved to "./bin/tools" so it doesnt mess up your other projects. I refactored protoc which left some files there I believe
• remove pgtype (it was complaining that it's unused)
  • not sure where this is coming from, I guess some old import somewhere
• generated files look different for unknown reasons
  • I messed up the latest commit yet again, sorry about that. I need to fix the project script so that I handle subshell errors properly and in turn CI works as it should, as well as adding some checks like git diff so this never happens again

Edit: I think you can even try dev branch's project check-build-deps as it is now, and it should output all green in your case as well. if you end up using it let me know if thats case

[philon123]

Ok, sounds like you have most changes covered yourself, except for docker compose (which is a simple search and replace). So there is no need for my hack job to see the wild ;) Ah and I found the issue of why prettier was failing. It just can't find the executable (`frontend/node_modules/.bin/prettier`). Manually executing the command with a relative path worked fine. I didn't want to add that folder to PATH and `run-dev` builds a container anyway where I don't know the paths. So I'm not sure how to fix it. Am Mi., 19. Apr. 2023 um 08:55 Uhr schrieb danicc097 < ***@***.***>:

…

- docker compose instead of docker compose - absolutely - uses newer pnpm, which results in a differently formatted package.lock - pnpm is also a good candidate to have a check.bin.pnpm and install.bin.pnpm which is the kind of flow Im using on lastest dev branch. As soon as I finish fixing project you can give it a try if you want, pnpm --version is easy to check - adds react-redux to package.json - Actually I want to get rid of it for this app :) now it's zustand which is much simpler to use. I need to remove all references from frontend - additional cleanup after installing protoc - I moved all installation scripts and checks to project check-build-deps. now it will both check and prompt to install. THe installation is either via apt or with bins moved to "./bin/tools" so it doesnt mess up your other projects. I refactored protoc which left some files there I believe - remove pgtype (it was complaining that it's unused) - not sure where this is coming from, I guess some old import somewhere - generated files look different for unknown reasons - I messed up the latest commit yet again, sorry about that. I need to fix the project script so that I handle subshell errors properly and in turn CI works as it should, as well as adding some checks like git diff so this never happens again — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAXYVAJL2RN2IVQOK5OPVLXB6D7XANCNFSM6AAAAAAXAC2LRY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[danicc097]

Its weird that its not finding prettier if you're running the scripts as is, ill check later when i get into the frontend. running manually maybe there's path issues. Maybe adding node modules' bins to .envrc path simplifies some things, Ill think about it. You can already do that locally if you create a .envrc.local in your root and add export PATH=$PWD/frontend/node_modules/.bin:$PATH, which is what I'll add to envrc really.

By the way, I pushed fixes to the dependency checks and tests to main, I think all is good now for those 2, dont mind the failed CI