Teams file speed under "non-functional requirements", which is an efficient way of guaranteeing nobody ever works on it. Nothing labelled non-functional has ever won an argument against something labelled a feature.
Budget it like you budget scope
Give latency a number before you start, the same way you give a release a date. A p99 target in the brief is a constraint the design has to satisfy. A general preference for things being fast is something that gets traded away in week three, cheerfully, by someone with a roadmap.
Measure where people actually are
Aggregate averages are very good at hiding the people having the worst time. Your p50 on a fast connection in the same city as your servers tells you almost nothing about the experience quietly driving people away.
Fast is cheaper to keep than to retrofit
Performance recovered late costs several times what performance preserved early would have, because by then the slow decisions are holding up the roof. Same argument as quality, and it fails for the same boring reason: the cost lands in a different quarter than the saving.