Update voyageai docs

[fzowl]

Updating the VoyageAI documentation in relation to the Unstructured-IO/unstructured#4109

[Paul-Cornell]

@fzowl @cragwolfe

I'm not able to merge this PR as currently presented. I think you're working with a very old version of the tech docs sources:

• api-reference/how-to/embedding.mdx no longer exists in the current tech docs.
• The contents of open-source/core-functionality/embedding.mdx are very different here than what exists in the current tech docs.
• platform/workflows-automation.mdx no longer exists in the current tech docs.

Please let me know how you'd like to proceed.

[fzowl]

@Paul-Cornell @cragwolfe Ah, sorry, it was my mistake. Can you please check now?!

[Paul-Cornell]

@fzowl @cragwolfe Something still isn't looking right to me here. Specifically:

• For api-reference/workflow/workflows.mdx and snippets/general-shared-text/chunk-limits-embedding-models.mdx, you list 17 available Voyage AI embedding models. However, if I look at our SaaS product today, I count only 8 available Voyage AI embedding models. Why the difference?
• For open-source/how-to/embedding.mdx, the tech docs already currently link out to https://docs.voyageai.com/docs/embeddings, where Voyage AI's list of supported models is (supposedly) being maintained. We don't list this kind of information for any of the other embedding model providers listed in our open source tech docs. As such, I'm hesitant about making this exception here--especially as this list will likely become outdated fairly quick, especially in a portion of the tech docs that is not as frequently maintained.

Please let me know how you'd like to proceed.

[fzowl]

@Paul-Cornell @cragwolfe

• The lists you mentioned (api-reference/workflow/workflows.mdx and snippets/general-shared-text/chunk-limits-embedding-models.mdx) contain old models, which we support. Still, we recommend our users to migrate to our recent models for better quality and efficiency. I checked the code in https://github.com/Unstructured-IO/unstructured/ and i see all 17 models are listed there. First of all, I'm not sure what the best approach would be, but I feel we should keep the older models as well. Maybe i can list older models in these files in a different (Older models) list? Can you please help me where do you find only 8 available models?
• I see your point, i removed the VoyageAI model lists from open-source/how-to/embedding.mdx. Shall i remove the VoyageAI Advanced Features section also? Not sure where to highlight these features, or whether to highlight them at all.

[Paul-Cornell]

@fzowl @cragwolfe Our SaaS product currently lists only 8 of those 17 models. To see these, sign in to your Unstructured account. In the UI, on the sidebar, click Workflows, and then click New Workflow. With Build it Myself already selected, click Continue. Add a Transform > Embedder node to the workflow. Notice in the node's settings pane, only 8 embedding models are listed under Voyage AI.

I would not add anything special to open-source/how-to/embedding.mdx. Just make sure that the page we point to (https://docs.voyageai.com/docs/embeddings) is the correct one and has all the information worth knowing about here.

[fzowl]

@Paul-Cornell @cragwolfe I just tried the SaaS and i see the old models, my recent change (Unstructured-IO/unstructured#4109) and the recent models (i added with the previous PR) are not listed. voyage-3.5, voyage-3.5-lite and voyage-context-3 models are new. Other than these, the list in the PR and in the documentation contains models that are old, but we still support: voyage-2, voyage-02, voyage-large-2-instruct, voyage-large-2-instruct, voyage-large-2 and voyage-3-m-exp. These are Gen 2 models.
I'm not sure how the PR i raised (and was merged) gets into the SaaS or from where the list in SaaS comes from...probably that's a different repo? Please let me know so that i can adjust.

Regarding the open-source/how-to/embedding.mdx i already removed the VoyageAI model lists. I can confirm that the page is correct.