Vext v1.6: Enhanced LLM Project Endpoint, Better Logging, New Model, and More

Our development cycle is around 2 weeks per sprint, and we really packed as much improvement into this one as possible!

We're thrilled to announce the release of Vext v1.6, a significant leap forward in enabling developers and businesses to max out the potential of LLMs with greater ease, efficiency, and control.

Enhanced Endpoint Capabilities

Long Polling Support

One of the standout features in this update is the introduction of long polling for endpoints. If you're working with extensive LLM pipelines, you're likely familiar with the frustration of timeouts.

With long polling support, this ensures that you can retrieve the result without having to wait for the result upon fetching, giving your requests the time they need to process fully. This feature is a game-changer for those aiming to build more reliable and robust LLM-powered applications.

How it works

1. Include a new parameter, long_polling, with a boolean value in your request body alongside the existing "payload"

2. Upon submission, the system will respond with a unique request_id

3. To check the status of your request, fetch the same endpoint again, this time providing only the request_id previously received

4. The system will then return the current progress of your request

5. Repeat the status check process as necessary until you receive the final generated response

You can learn more about it on our help center article, or check out the API reference.

More Information Per API Call

The LLM pipeline endpoint now delivers an expanded set of data with each API call. This includes citations, file paths (with optional public access toggling), and referred chunks for those utilizing RAG. These enhancements open the door to creating more nuanced, credible, and feature-rich LLM applications.

Check out the API reference to learn more.

Enhanced Logging for Unmatched Transparency

Query Consumption Visibility

For projects involving multiple LLMs or smart functions within an LLM pipeline, understanding resource utilization is crucial. The detailed logging that shows how many queries were consumed during each run, providing clear insights into your project's operational footprint.

Execution Time and Source Tracking

Moreover, the new logging capabilities extend to execution time visibility for each pipeline step and the sources used for RAG, including citations. This transparency ensures you can optimize performance and verify the credibility of generated content.

Log Export Options

Understanding the need for flexible data analysis, Vext 1.6 allows users to download logs in both .xlsx and .csv formats, catering to a wide range of analytical preferences.

Enhanced Data Source Management

Original and Processed Data Views

To offer users unprecedented control over their data, Vext 1.6 enables access to both the original files and the processed data views. This dual-view capability ensures that users can understand and leverage their data fully.

Public Access Toggling

Further enhancing data control, users can now toggle the public access setting for original files. This feature is essential for managing data privacy and sharing preferences seamlessly.

Expanded Model Options

Mistral 7B and Mixtral 8x7B

We are adding two powerful model options to cater to diverse project requirements. The Mistral 7B, a 7B dense Transformer model, offers fast deployment and customization for various use cases, supporting English and code. For projects demanding more robust capabilities, the Mixtral 8x7B leverages a 7B sparse Mixture-of-Experts model, with 12B active parameters out of 45B total, supporting multiple languages and code.

With Vext, you now have a very powerful lineup of models to choose from for various kinds of use cases, from simple classifier, basic chat bot, to complex automation.

AI Project Interface Improvements

Larger Input Field

In response to user feedback, we replaced the list format for system prompts with a larger, single input field. This change grants developers more freedom and flexibility in crafting system prompts, facilitating creativity and efficiency in project development.