HighScienceHub

Bridging Curiosity to Clarity in Tech, Science & AI

Run DeepSeek-R1 on Your Local Machine: A Step-by-Step Guide to the 671B AI Model

@06 mins

How Unsloth’s Breakthrough Compression Makes It Possible for Everyone

Why This Matters

DeepSeek-R1, a 671-billion-parameter AI model, was previously restricted to cloud APIs due to its colossal size (720GB). Thanks to Unsloth, an open-source project led by ex-NVIDIA engineers, the model is now compressed to 131GB (80% smaller) and optimized for local deployment. Whether you’re a developer, researcher, or hobbyist, you can now experiment with one of the largest AI models ever built—even on consumer-grade hardware.

Key Requirements

Before diving in, ensure your system meets these specs:

  • Minimum (CPU-only):
  • RAM: 20GB
  • Disk Space: 140GB
  • OS: Linux/Windows/macOS (Linux recommended for stability).
  • GPU Acceleration (Recommended):
  • NVIDIA GPU with 24GB+ VRAM (e.g., RTX 4090) for 2-3 tokens/sec.
  • Optimal: 80GB+ combined RAM+VRAM (e.g., dual H100 GPUs for 140 tokens/sec).

How to Run DeepSeek-R1 Locally

Step 1: Install Prerequisites

You’ll need Python and key libraries. Open a terminal and run:

# Create a virtual environment (optional but recommended)  
python -m venv unsloth-env  
source unsloth-env/bin/activate  # Linux/macOS  
.\unsloth-env\Scripts\activate   # Windows  

# Install PyTorch with CUDA support (if using NVIDIA GPU)  
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121  

# Install Unsloth and Hugging Face libraries  
pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"  
pip install transformers accelerate bitsandbytes

Step 2: Download the Quantized Model
Unsloth’s quantized DeepSeek-R1 is hosted on Hugging Face. Use the transformers library to load it:

from unsloth import FastLanguageModel  
model, tokenizer = FastLanguageModel.from_pretrained(  
    model_name = "unsloth/deepseek-r1-131gb-quantized",  
    max_seq_length = 2048,  
    dtype = None,  # Auto-detect (float16 for GPU, float32 for CPU)  
    load_in_4bit = True,  # 4-bit quantization for GPU users  
)

Note**: The 131GB download will take time. Use resume_download=True if interrupted.

Step 3: Run Inference

Use this script to generate text:

inputs = tokenizer("Explain quantum computing in 3 sentences:", return_tensors="pt").to("cuda")  
outputs = model.generate(**inputs, max_new_tokens=100)  
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance Tips:

1. For GPU Users:

• Enable load_in_4bit or load Lin_8bit to reduce VRAM usage.

• Use batch_size=1 for real-time interaction; increase for batch processing.

2. For CPU Users:

Add device_map="cpu" and torch_dtype=torch.float32 in
from_pretrained ().

Expect slower speeds (~0.5 tokens/sec on 20GB RAM).

Step 4: Optimize for Your Hardwares

Unsloth supports advanced optimisations:

  • Flash Attention 2: Speed up inference by 30% on compatible GPUs (RTX 30xx/40xx or A100/H100).
model = FastLanguageModel.get_peft_model(  
    model,  
    r=16,  # LoRA rank  
    use_gradient_checkpointing=True,  
    use_flash_attention_2=True,  
)

Mixed Precision: Use fp16 or bf16 for NVIDIA GPUs.

Use Cases and Limitations

What It’s Great For

  • Batch processing (e.g., summarizing documents, data labeling).
  • Privacy-sensitive tasks (medical/legal text analysis).
  • Experimenting with cutting-edge LLM capabilities offline.

Current Limitations

  • Speed: CPU inference is slow; GPU costs remain high for optimal performance.
  • Quantization Trade-offs: Some niche tasks may lose nuance (e.g., creative writing).
  • Hardware Barriers: Dual H100 GPUs are expensive but required for API-beating speeds.

Unsloth’s work is a game-changer for decentralizing AI. While running a 671B model locally isn’t seamless yet, this opens doors for developers to innovate without relying on cloud providers. For most users, start with a CPU or single GPU setup to test feasibility, then scale as needed.

Ready to try it? Download the model here and join Unsloth’s GitHub community for updates!

Let me know if you’d like help troubleshooting specific setups! 🚀

Related News