This is one of those funny questions, like oversampling, NOS components etc, where you do get people with the kinda "audiophile" attitude that claim they can hear certain limitations of gear. I'm well aware that 2.5ms is well within the ears integration time and equates to the time it takes sound to travel about a foot! However I still have reservations and, since you ask I'll give you the overlong explanation...
I'm a professional gigging violin/fiddle player playing mainly in band context. Professional bands need to be as "tight" as possible and it's really not black and white, sometimes we need all the help we can get. Sound is actually inconveniently slow. We've all observed how surprisingly close you can be to someone, say kicking a football, and see/hear the latency. Many people use this as an excuse that latency in gear is unimportant. I think the opposite as real world and digital latency are all adding up to a total latency.
As with most live musicians, I use more than just 1 effects box, I also like to stand a few meters away from the ear damagingly loud drummer/amps etc, digital desks are the norm now adding yet more latency. It is surprisingly easy to find yourself playing a bit too "lazy". I can hear this clearly in live recordings. Often when "comping" individual notes are moved manually to correct the timing, sometimes this is only a matter of a few ms.
I avoid using digital effects where analog is available but some things need to be digital, poly pitch shift, loopers, I wont let an audience hear my naked piezo PUP - it HAS to be through an FIR loaded with acoustic violin IR.
There are some stomp boxes that have unacceptable latency, EHX POG or Pitchfork being an example. Whatever anyone says I know I can hear the latency of a POG or Digitech Whammy. Having noticed the latency when using a POG into a looper (was finding it hard to get my loop point totally spot on) I measured it! 13ms. I find it very distracting and therefore don't use that sort of gear in performance. (shame as I really want HQ poly pitchshift)
Mostly, these days, stomp box manufacturers are taking this into consideration. My "cheapo Chinese" Nux NSS-5 IR loader claims a latency of 0.7ms (I checked this and actually measured 0.6ms but I'm probably not accurate). My Strymon gear also comes in at about 0.7ms, Source Audio are claiming 0.6ms on their new gear.
So, tightness of a band is not black and white and latency is relevant to tightness. Total latency is a combination of entire system as well as physical distances. Some of these factors can be compromised, others cannot but I feel an effort needs to be made in all areas where possible.
Axoloti is the slowest piece of gear in my current setup. Not only are the converters slow at 48khz some objects also take a little time (envelope follower, convolver or pitch follower for instance). Sometimes you need to route audio backwards or upwards leading to 16 samples each time. What if one Axo is not powerful enough and you have to use 2, the latency doubles!
I've accepted this with the Axoloti, it's not the end of the world. It seems impractical to change on a well developed system and it would take more of Axos limited processing power. However, if we're discussing a new and substantially more poweful system it would be amiss not to take latency into consideration.