Climbing wall and similar types of playground equipment are not well suited for children under age six, because they tend to lack the upper body and hand strength when younger. Their little hands can get sweaty on the plastic knobs common to climbing walls, they lose their grip or foot slips off, and they're bouncing face first down the climbing wall. It may be a good idea to keep it off the playset until all children are age six, by placing a temporary panel in it's place until it's added back.
Many people are unaware that children tend to have a fundamental shift in understanding self protection at around age six, which is normally the age they learn to put out their arms to brace impacts. Until then, younger kids typically land on their face or head, based on playground injury data. This is essential to understanding that current playground safety standards allow for up to 256% higher HIC scores than toddlers are likely to survive, or suffer from traumatic brain injury that can limit their potential for the rest of their lives.
Climbing walls, or really pretty much any climbing apparatus within playgrounds are where most injuries are likely to occur. Younger children still lacking the upper body and hand abilities need other forms of age appropriate playground activity, and it's best if 30 inches or lower, to minimize falls. Remember that younger children tend have fall impacts on their face or head when they fall, and on certain surfaces this could potentially become fatal, so age appropriate play equipment along with proper safety surfacing create a healthy place to play.
Kids normally figure out their own way of doing things with what they have, often using items in ways they were never intended. The same goes for playground equipment. Children can and will do things unexpectedly. Didn't you, or your friends, when you were little? This is why considering whether to retire certain equipment to trade the space for something more current and engaging can be beneficial. While climbing walls are fun, there are a lot of newer options today, and probably more arriving soon enough.