llm-visuals

rust
mit

A live terminal dashboard for local LLMs running on your machine.

173
Stars
v0.14.0
Latest release
6 days ago
Last activity
Very Active
Status

llm-visuals is a terminal dashboard for monitoring active local LLM inference and GPU performance.

 

It detects running inference servers, including ollama, and displays prefill and decoding speed measured in tokens per second, and time to first token with context usage and GPU memory and load statistics. The performance view combines throughput histories and it has the Mixture-of-Experts (MoE) balance view showing the GPU load of each expert in active inference.

 

One other useful view in llm-visuals is the bandwidth section that indicates which part of the hardware pipeline is bottlenecked or actively decode / or prefill bound. The active experts can be visualized in 3D to give the perspective of the current GPU inference activity. 

 

For machine learning engineers, researchers, inference engineers and software engineers that running local models, llm-visuals is useful to help with investigating inference slowdowns and watching how requests use available hardware.

Related tools

Get Updates On Terminal Trove.

No spam, just updates on Terminal Trove. See an example update.