Smeared mortar on brick pavers, as well as brick walls, is a very common problem. Depending upon the type of brick, the job can be simple or a nightmare. It is my hope that you do not have a deeply grooved wire-cut brick, as these have many small crevices that can make cleaning a career choice.
Typically bricklayers will use muriatic acid to dissolve mortar from most brick. This is simply a form of hydrochloric acid. When you purchase this product it often has a skull and crossbones on the label warning you that it is a powerful chemical. But these products can be neutralized. That is something all of us should have discovered in our high-school chemistry class. In fact, the mortar does neutralize the acid as it works.
There are other acids that will react and dissolve the alkaline chemicals in the mortar. Even white vinegar that is in your kitchen will work to some degree. Since you eat vinegar in many foods, you know it is not a toxic acid. The challenge for you is finding an acid somewhere between vinegar and hydrochloric acid that will efficiently remove the mortar buildup. You should consider calling a physical chemistry professor at a local college or high school. You can also visit a building supply company to see if they stock a non-toxic acid that will dissolve the mortar.
The first step in the process is to remove as much of the mortar as possible with a scraper or chisel without scratching the brick face. Clear water flowing over the brick acts as an effective lubricant that will minimize damage to the brick. If you have to tap the scraper with a hammer, do so at a low angle with the face of the scraper nearly parallel with the face of the brick. You are trying to get the mortar film to a thickness less than that of a plastic credit card.
Once the majority of the mortar is off the brick, you then should consider bathing the brick with a diluted acid solution. As mentioned before, muriatic acid is highly effective, and it works on most brick. Usually you mix one part of the acid with ten parts of clean water. Always read the instructions on the acid label and pay particular attention to all safety instructions.
I have found in most cases you should dampen the brick with clear water before you apply the acid solution. Always try this in an out-of-the-way area if you are unsure if the acid will harm the brick. If you use muriatic acid, you should see tiny bubbles forming where the solution contacts the smeared mortar. This tells you that the acid is reacting with the mortar paste and is starting to dissolve it into the liquid solution.
Use a scrub brush to help remove the mortar. Wait five or ten minutes after the acid solution has been applied to the brick before you scrub. Allow the acid to do much of the work for you. It can take multiple applications of acid solution to remove all of the mortar. Rinse the acid solution from the brick with massive amounts of water. The more water you use, the better.
Working with acid solutions is very tricky. You can hurt yourself, your clothes, your brick, the new mortar and vegetation around your home. If you have valuable vegetation and landscaping, you should try to stop any acid solution from getting into the soil.
Be sure to wear all of the protective gear to prevent burns to your skin and eyes. Muriatic acid straight from the bottle is a wicked liquid. The fumes are very toxic, and it can cause serious burns to your skin very quickly. Do not underestimate this chemical.