Let's begin with expressionism. I use this term not like an art critic but like a lay person interested in art. Expressionism is an approach to art which does not attempt to reflect any sort of seen reality directly in the photorealistic sense, but to express mood and emotion visually. In the context of fantasy art, it means art which does not seek to produce an image which would persuade you it is really real or could be a real thing in some fantasy world. Rather it means art which evokes the quality of a fantastical thing in pictoral form.
Hence, some examples:
|"Crossing the Vimur River", by Hellanim (https://hellanim.deviantart.com/art/Crossing-the-Vimur-river-348494026)|
|"The Old Saints, VIII" by Valin Mattheis (http://strange-gods.tumblr.com/)|
|"The Ichthyosaur's Pool", an illo Matthew Adams did for Issue #1 of The Peridot|
Next comes realism, or the attempt to create actual pictoral representations of fantastical things in such a way that you could imagine them appearing to the naked eye if you were to actually see them. ("Realism" is a slightly unsatisfactory label because, of course, we are not talking about "real" things; maybe a better term would be "Unnaturalism"?) It does not have to be photorealist; it may be impressionistic in its brush strokes.
|"Lancelot", by John Howe|
|"Scythe" by Jakub Rozalski (https://www.artstation.com/artwork/L3RPP)|
|"The North Watch" by Keith Parkinson|
Finally, there is, I would suggest, a third category, which I refer to as "hyper-realism" (or, perhaps more appropriately for reasons alluded to above, "hyper-unrealism"). This is fantasy art which attempts to produce a version of a fantastical reality which is more vivid and exaggerated than the more prosaic "realism" of the second category, while being faithful enough to an imagined visual reality that it cannot be described as being "expressionistic" (unless by "expressionistic" you mean "evokes a sense of vividness and dynamism").
|"Mirror Men's Judgement" by Rhineville|
|You know where this comes from.|
I have nothing against hyper-realist art particularly, but I would suggest that there is a preponderance of it in RPG products and this trend has become more pronounced over time rather than less so. I can think of two reasons for this: 1) RPG art is strongly influenced by comic and, nowadays, video game art; 2) RPGs are about action and imagination and hyper-realist art is the natural bi-product of the interaction between those two impulses.
Challenge me and prove me wrong, or agree with me and praise me and my taxonomy to the heavens.