BitTorrent for LLM? Exo software is a distributed LLM solution that can run even on old smartphones and computers

Date:
Wed, 26 Feb 2025 20:39:00 +0000

Description:
Exo enables distributed AI inference by combining computing power from multiple devices, reducing reliance on expensive high-performance hardware.

FULL STORY ======================================================================Exo supports LLaMA, Mistral, LlaVA, Qwen, and DeepSeek Can run on Linux, macOS, Android, and iOS, but not Windows AI models needing 16GB RAM can run on two 8GB laptops

Running large language models ( LLMs ) typically requires expensive, high-performance hardware with substantial memory and GPU power. However, Exo software now looks to offer an alternative by enabling distributed artificial intelligence (AI) inference across a network of devices.

The company allows users to combine the computing power of multiple
computers, smartphones, and even single-board computers (SBCs) like Raspberry Pis to run models that would otherwise be inaccessible.

This decentralized approach shares similarities with the SETI@home project, which distributed computing tasks across volunteer machines. By leveraging a peer-to-peer (P2P) network, Exo eliminates the need for a single, powerful system, making AI inference more accessible to individuals and organizations. How Exo distributes AI workloads

Exo aims to challenge the dominance of large technology companies in AI development. By decentralizing inference, it seeks to give individuals and smaller organizations more control over AI models, similar to initiatives focused on expanding access to GPU resources.

"The fundamental constraint with AI is compute," argues Alex Cheema, co-founder of EXO Labs. "If you dont have the compute, you cant compete. But if you create this distributed network, maybe we can."

The software dynamically partitions LLMs across available devices in a network, assigning model layers based on each machines available memory and processing power. Supported LLMs include LLaMA, Mistral, LlaVA, Qwen, and DeepSeek.

Users can install Exo on Linux, macOS, Android, or iOS, though Windows
support is not currently available. A minimum Python version of 3.12.0 is required, along with additional dependencies for systems running Linux fitted with NVIDIA GPUs.

One of Exos key strengths is that, unlike traditional setups that rely on high-end GPUs, it enables collaboration between different hardware configurations.

For example, an AI model requiring 16GB of RAM can run on two 8GB laptops working together. A more demanding model like DeepSeek R1, requiring approximately 1.3TB of RAM, could theoretically operate on a cluster of 170 Raspberry Pi 5 devices with 8GB RAM each.

Network speed and latency are critical concerns, and Exo's developers acknowledge that adding lower-performance devices may slow inference latency but insists that overall throughput improves with each device added to the network.

Security risks also arise when multiple machines share workloads, requiring safeguards to prevent data leaks and unauthorized access.

Adoption is another hurdle, as developers of AI tools currently rely on large-scale data centers. The low-cost of Exo's approach may appeal. but
Exo's approach simply wont match the speed of those high-end AI clusters.

Via CNX Software You may also like UK creative industries launch Make it Fair campaign against AI content theft We've listed the best AI video editors
right now We've also rounded up the best AI writers available



======================================================================
Link to news story: https://www.techradar.com/computing/bittorrent-for-llm-exo-software-is-a-distr ibuted-llm-solution-that-can-run-even-on-old-smartphones-and-computers


--- Mystic BBS v1.12 A47 (Linux/64)
* Origin: tqwNet Technology News (1337:1/100)