Consider using Python Docker SDK instead of CLI?

[C-Loftus]

Thank you for the great work the team has done on this script.

I was wondering if it would be possible to use the Docker SDK directly in the Python script instead of requiring the docker cli. In a containerized environment it is easy to add an extra Python dependency and mount the docker socket, but it is a bit cumbersome to add the docker binary itself since:

1. the docker binary is compiled and handling different architectures requires different downloads
2. the docker binary is harder to debug and version compared to the python sdk which matches other python tooling

I think this was intended to be added in #19 but it may have fallen through during the other updates to the script.

[C-Loftus]

And just for context, there's a good chance that I could help with this by submitting a PR. But I wanted to create this as an issue first to see if this is something that would be in scope to change or if the maintainers would prefer to leave it as is

[Qup42]

My thoughts for now though @hannahbast will probably also want to weigh in on this.

• One big advantage of the current approach is that you can just print the shell commands that you execute. This makes it transparent what the program is doing and enables the user to run (and modify) the command themselves.
• By invoking commands via the cli it is also extremely easy to use another runtime like podman. I think that's really neat, but I don't know how much it is actually used.
• I simply haven't though of the viewpoint that using the socket via the SDK is more flexible. I can totally understand the wish for only wanting to provide the docker socket.

[C-Loftus]

Thank you for your response.

By invoking commands via the cli it is also extremely easy to use another runtime like podman. I think that's really neat, but I don't know how much it is actually used.

My understanding of Podman, although please correct me if I am wrong, is that you can use the Docker Python sdk to connect to Podman given the fact it provides a Docker compatible API (in addition to the CLI). So the same Docker SDK code should work with both.

containers/podman-py#336 (comment)

One big advantage of the current approach is that you can just print the shell commands that you execute. This makes it transparent what the program is doing and enables the user to run (and modify) the command themselves.

I think that is a fair point with regards tracing which commands are run. That is likely easier to do with regular Docker CLI commands.

As one other thought, I suppose the answer to this specific point is also a question for the maintainers of how much of a wrapper vs abstraction layer this program should be. (i.e. is qlever-control intended for users to be modifying and extending regularly? Or moreso intended to be used for abstracting away the arguments of the original qlever management commands like IndexBuilderMain?)

With regards to making changes, from my perspective, I think using the Python SDK is actually still easier to make changes with. You can type check the SDK and get static error checks if a function call is mistyped / invalid. This is harder to verify if building a CLI command string.

[hannahbast]

@C-Loftus Have you noticed that there is now a qlever package for Ubuntu and Debian (and macOS)? See https://docs.qlever.dev/quickstart/

Maybe then a more meaningful setup would be to just install these as part of your Docker image. Then you have qlever-index, qlever-server, and qlever at your disposal (note that the binaries have been renamed a while ago)

[C-Loftus]

Thanks for your response. Yes I have seen those and they are very useful, however, my understanding is that regardless of how they are packaged, whether in pypi or debian, they currently depend upon the docker CLI binary since it uses the CLI commands via subprocesses. Which requires an extra dependency in the container. (i.e. you can't just mount the socket but need to install the docker CLI as well)

Ultimately though this is not a blocker for me, I have since gone ahead and just downloaded the extra dependencies and am version controlling the docker CLI in my container. Moreso just a general comment.

[Qup42]

With the debian package you no longer need docker (though you can still use it if you want to). The debian package qlever contains the control script (also available through pip) and compiled binaries. You can just execute qlever-server and qlever-index like a normal program after installing the pacakge.

[C-Loftus]

I am going to close this since I think using the debian package is generally sufficient. I do think that using the docker sdk directly has some benefits that are worth it instead of using shell script commands, but the majority of this is satisfied by just using debian and doing everything inside that. Also there are some ways I can use testcontainers as well for launching emphemeral containers on macos so the plaform specific nature of debian is less of an issue than I thought.