Skip to content

vllm.model_executor.models.step3p5

Inference-only Jurassic model.

Classes:

FP32ReplicatedLinear

Bases: ReplicatedLinear

Use FP32 for higher precision.

Source code in vllm/model_executor/models/step3p5.py 
65
66
67
68
69
70
71
72
73
74
75
class FP32ReplicatedLinear(ReplicatedLinear):
    """
    Use FP32 for higher precision.
    """

    def forward(
        self,
        x: torch.Tensor,
    ) -> torch.Tensor | tuple[torch.Tensor, Parameter | None]:
        assert self.params_dtype == torch.float32
        return super().forward(x.to(torch.float32))