Use venv for Python development

[hemildesai]

Addresses https://github.com/replicate/replicate/issues/424

[hemildesai]

Not sure why CI / Test (Python) (3.6, ubuntu-20.04) is failing. The error seems to be No matching distribution found for tensorflow-cpu==2.4.0 (from -r requirements-test.txt (line 4)).

[bfirsh]

Yeah, super weird isn't it. I restarted the test a couple of times so it's a real failure.

Maybe it's using a different version of pip because it's inside a virtualenv? I'm trying to figure out what might be different in this installation.

[bfirsh]

Thanks so much for the contribution btw! I'll do some poking around later today if I get time...

[hemildesai]

Cool, thanks for looking into it. Let me know if I can help.

[hemildesai]

@bfirsh it looks like the pip version is the culprit - tensorflow/tensorflow#39130 (comment)

[bfirsh]

Hah, oh god. Nice find. 🙄

[bfirsh]

Several times during this project we've thought "ok we need to burn all Python packaging to the ground and create a new packaging system" but then reminded ourselves we should keep our eyes on the ball, haha. This is also why most of the codebase is in Go -- so we don't have to deal with Python's godawful packaging/dependencies. cc @dscape

[hemildesai]

Yeah, sometimes with issues like this, it can be quite frustrating.

[hemildesai]

Hey @bfirsh, let me know if this is good to go or if there are any changes required.

[bfirsh]

Thanks so much @hemildesai! We're just in the process of shipping a release right now so I don't want to change up the development environment while we do that. But, as soon as we've shipped it, this is first thing I'm taking a look at. :)

[bfirsh]

Sorry for being slow here -- I had a quick look at this the other day. I think it's a bit confusing that the Python library is installed locally in a virtualenv, but the binary is still installed globally. That'll make it easy to get into a situation where different versions installed in two different places. I wonder if we can install the binary inside the venv in development?

[hemildesai]

@bfirsh Sure. Can you point me to where the binary is installed so that I can change it to be installed in the venv?