I'm a pastry chef, and so can answer your question with a bit of authority on the subject!
Using almond meal or coconut flour in baking can be a great experience and taste experiment. However, the reason all of the flour is not replaced (and in fact I would suggest no more than half at maximum) is that wheat flour contains gluten, which is what gives a baked good such as bread the strength to stay risen once the leaveners do their thing. In the case of bread, the yeast rises the dough, and the strong gluten in bread flour helps trap the CO2 within the dough instead of letting it escape.
Hope this helps!