Yes, you should solder all of them. If the axoloti doesn't have exactly 40 holes, or you want to use only a section of the GPIO, you can also break it to smaller pieces and solder those separately. Usually there are grooves to make it easy to break at the right spot.
I've heard of a solderless version of this header, that would clamp in, but I've also heard it doesn't always connect nicely. So I recommend the soldering!