NVIDIA unveils ChatGPT-rival ‘Chat with RTX’, an AI chatbot that runs locally on your PC

NVIDIA, an American computer hardware manufacturing company, on Tuesday, February 12, announced that it has launched a feature called “Chat with RTX.”

This tool allows users to personalise a chatbot with their content while offline on their PC. 

Currently, Chat with RTX is available for free to download. However, the system requirements to run Chat with RTX include: 

• Platform: Windows
• GPU: NVIDIA GeForce RTX 30 or 40 Series GPU or NVIDIA RTX Ampere or Ada Generation GPU with at least 8GB of VRAM
• RAM: 16GB or greater
• OS: Windows 11
• Driver: 535.11 or later

Chat with RTX utilises retrieval-augmented generation (RAG), NVIDIA TensorRT-LLM software, and NVIDIA RTX acceleration to bring generative AI capabilities to local GeForce-powered Windows PCs. 

NVIDIA TensorRT-LLM is an open-source library that accelerates and optimises the inference performance of the latest large language models (LLMs). It now supports more pre-optimised models for PCs. 

According to the company, developers can use the reference project to develop and deploy their RAG-based applications for RTX, accelerated by TensorRT-LLM.

Here’s how it works:

Easily connects to local files

Users can quickly, and easily connect local files on a PC as a dataset to an open-source large language model like Mistral or Llama 2, enabling queries for quick, contextually relevant answers.

NVIDIA says rather than searching through notes or saved content, users can simply type queries. 

For example, one could ask, “What was the restaurant my partner recommended while in Las Vegas?” Chat with RTX will scan local files the user points it to and provide the answer with context.

The tool supports various file formats, including .txt, .pdf, .doc/.docx, and .xml. Point the application to the folder containing these files, and the tool will load them into its library in just seconds.

Additionally, users can provide the URL of a YouTube playlist, and the app will load the transcriptions of the videos in the playlist, enabling users to query the content they cover.

For example, ask for travel recommendations based on content from favourite influencer videos, or get quick tutorials and how-tos based on top educational resources.

Rather than relying on cloud-based LLM services, Chat with RTX lets users process sensitive data on a local PC without the need to share it with a third party or have an internet connection.

The announcement comes a month after NVIDIA launched GeForce RTX SUPER desktop GPUs for supercharged generative AI performance, new AI laptops, and new NVIDIA RTX, an accelerated AI software and tools for both developers and consumers.