Error loading dolly model locally

[ethanenguyen]

Hello,

I'm trying to load a fine tuned dolly model locally behind a firewall and deepseepd-mii keeps looking up huggingface for the model. I tried to pass the model path to model but that still didn't work. Is there an example of mii.deploy for a local model? I looked through both the example folder and issues but couldn't find a working example.

Thanks,

mii_configs = {"tensor_parallel": 1,
               "dtype": "fp16",
               "port_number": 50950,
               "meta_tensor": True,
               "skip_model_check": True,
               # "trust_remote_code": True,
               }
name = "fine-tuned"

mii.deploy(task='text-generation',
           model='databricks/dolly-v2-7b',
           model_path="/local/dolly/trained_model/fine-tuned-dolly-v2-7b/",
           deployment_name=name + "_deployment",
           mii_config=mii_configs)

[mrwyattii]

@ethanenguyen This is not possible with the latest release of MII, but it will be once we finish/merge #218. Eventually you will be able to pass pipeline kwargs in the mii_configs like so:

mii_configs = {"pipeline_kwargs": 
                  {"model_kwargs": 
                      {"local_files_only":True}
                  }
              }

In the meantime, I've created a branch that you can use to achieve this: https://github.com/microsoft/DeepSpeed-MII/tree/mrwyattii/local-files-only

Install with pip install git+https://github.com/microsoft/DeepSpeed-MII@mrwyattii/local-files-only. Then add "local_files_only": True to your mii_configs.

[ethanenguyen]

thank you @mrwyattii . It works