Yeah - selling to vendors is most definitely screwing yourself. If you can sell it in the AH, you will most certainly get more. That said, there are a few mats that are just so common, and such a low demand, that they aren't worth listing in the AH (Gel Drops seem to be one).

First suggestion - look up the item in question in the AH, and select "Sort by Buy Price" - this will show you the bottom end of the buyout price range for that item.

But some people don't seem to have discovered how to sort in this fashion, and appear to be buying based on the prices that appear on the first page for the default sort (by time remaining). I've successfully sold a few "overpriced" mats (compared to the buyout range) by setting a short auction time, so they appear at the beginning of the "sort by time remaining" list. This carries some risk, however, as those who *are* aware of the other sorts will quickly realize my goods are overpriced, and I may wind up getting them back, losing the listing fee.

Note that the selling price for a mat isn't always in line with its star level. For instance, last time I was shopping for mats, 3 star "Fine Fabrics" were actually selling for higher prices than 4 star "Miracloth". It all depends on the demand for that particular mat.

Also consider how a mat is used when deciding whether to sell single pieces or in larger quantities. Don't put up auctions for things like Swordstones in quantities of 10, since most people are buying for a specific crafting recipe, which will never require more than 4 of them. For those, selling singles or pairs seems to work out well. For something like Shards, on the other hand, selling in 10's seems to work out quite well, since each type of Shard is used in a lot of different recipes.