> For the complete documentation index, see [llms.txt](https://sully.gitbook.io/sully.ai-docs/FEM1tMOJsSSszbLOgKLA/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sully.gitbook.io/sully.ai-docs/FEM1tMOJsSSszbLOgKLA/eval-based-system/model-performance-and-optimization.md).

# Model Performance & Optimization

### Overview

In the field of compute optimization, custom kernel implementations, memory-mapped model loading, and batch processing optimization are crucial for enhancing performance. For cost optimization, strategies like intelligent model routing, caching, and resource sharing are essential to efficiently manage resources and reduce expenses.

### Model Performance Summary 

<figure><img src="/files/RGQEFJZvlGuFitRZEmeK" alt=""><figcaption></figcaption></figure>

### Technical Implementation

**Memory Management:**

* **Scrolling window approach** for real-time analysis
* **15-20MB in live RAM** per 2-hour stream allocation
* Efficient embedding storage and retrieval

**Processing Optimization:**

* **Custom chunking function** with dynamic sizing
* **ICD codes as ground truth anchors** for medical reasoning
* **Built-in self-calibration** for adaptive weight adjustment

**Performance Metrics**

| LLaMA-4 Optimization                                                                                                                                                                                                                                                                                                                                                                                                                   | DeepSeek Performance                                                                                                                                                             | Nano Model (4.1)                                                                                                                                                                                                                                                                            |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p></p><ul><li>Successfully optimized to reach <strong>theoretical maximum TPS</strong> for 400B parameter model</li><li>Fits alongside DeepSeek in memory (total <strong>1.1T parameters</strong> simultaneously)</li><li>Startup time: <strong>8-12 minutes</strong></li><li>KV cache calculation: <strong>\~30 minutes</strong> (one-time per startup)</li><li>Attention head recalibration: <strong>2-3 minutes</strong></li></ul> | <p></p><ul><li>Running at <strong>30,000 tokens/second</strong> for 700B parameter model on single node</li><li>Production-ready performance for large-scale inference</li></ul> | <p></p><ul><li>Optimized for evaluator tasks with <strong>lower compute requirements</strong></li><li>Showing promising results for <strong>reference-free evaluations</strong></li><li>Current testing indicates potential for <strong>20 notes/second generation speed</strong></li></ul> |

### Performance Optimization Strategies

#### Adaptive Processing

**Dynamic Resource Allocation:**

* Automatic scaling based on reasoning complexity
* **Context-aware compute allocation**
* **Priority-based processing queues**

**Model Selection:**

* **Nano models** for simple evaluator tasks
* **Medium models** (2.8B) for speculative generation
* **Large models** (400B+) for complex reasoning verification

#### Efficiency Optimizations

**Compute Optimization:**

* Custom kernel implementations for specific operations
* **Memory-mapped model loading** for faster startup
* **Batch processing optimization** for concurrent requests

**Cost Optimization:**

* **Intelligent model routing** based on complexity requirements
* **Caching strategies** for repeated reasoning patterns
* **Resource sharing** across concurrent sessions


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://sully.gitbook.io/sully.ai-docs/FEM1tMOJsSSszbLOgKLA/eval-based-system/model-performance-and-optimization.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.