Reread the mechanics I suggested carefully, because I think you'll find your concerns already mostly addressed.
I don't want OOO to penalize players simply for high volume of CE market transactions - only for doing so in manipulative ways.

Shill bidding is the situation where a CE seller acts as a buyer to increase the bid price for their own profit. For example:

Current market price of offers to by is 7000
Post 50 offers to buy CE at 7200 as an "anchor", since these won't sell through that quickly.
Quickly begin posting 1-5 offers, each for 10-20 above the previous price, and hope people engage you in bidding wars.
Withdraw the lower offers and set new "anchors" at higher points, keeping the top five "high" offers on the market.
Repeat as long as desired.
Profit from selling to the people who have engaged in a bidding war with you.

This requires large CE and crown pools and high trade volume to be profitable, but for someone that regularly sells large volumes of CE, it's totally feasible, because the price jumps take a while to level out. The shill bidder doesn't want their offers to be the ones accepted, at least not very many of them - the purpose of their bids is to raise the market price, and only to raise the market price.

This suggestion counters that, because the number of proposals you withdraw versus the number you ultimately complete is a good indication of whether a high volume trader is trading in good faith (ie with the actual intent to buy or sell)

Limiting the number, rather than the size of active offers also helps. You'd still be able to list as many offers as you want, just the number of price points you can list at the same time would be limited, so that someone doesn't post 50 separate offers 10 crowns apart.

Lets just simplify this, a lot. Shill bidding to either anchor prices or artificially inflate a bidding war is what's holding the market high right now despite ongoing adjustments by OOO to reduce farming.

Track canceled market offers over a 24 hour window. Every time you cancel an offer, the count increases, every time an offer is successfully completed, the count decreases by a 1 and a half until it goes back to 0 canceled units. Every 25 canceled units (where a unit is 100 CE, ) accumulates a 1% cancellation fee. Cap the fee at 10%

This relies on the fact that legitimate market traders want their trades to succeed, and will only pull a listing if they see it won't be successful, therefore they will have enough completed trades that their cancellation fees will stay 0 or close to it. By contrast, the people abusing the market are making bids that they try to cancel before they succeed - pumping dozens of units of CE onto the market at one time and counting on a bid war to ensue above that price so that they can profit from selling CE at higher prices.

Very few legitimate traders will even see the fee, unless they do something reckless, like listing for 500 units at a time and then cancelling the offer, and even then, the situation will correct itself relatively quickly by either posting offers that are in line with the market or by waiting for the fee to reset.

example of what I'm seeing

Limiting how many prices are posted is only part of it. You need to have a penalty built in for posting bids you don't intend to honor, so that when someone posts hundreds of bids like that, they are going to have to eat the seller fee one way or another.

What's happening is that sellers are placing offers to buy that are acting as anchors, and counting on the fact that people will bid above it to get their bids through quickly. Then they wait for (or create) a bid war, and repeat the process. This keeps the market high, even when OOO corrects the rewards of missions and boss gates to compensate.

At the extremes we're seeing, I predict that if OOO keeps trying to fix this through their standard techniques of nerfing rewards, then the net effect will be to make the game unplayable for anyone other than farmers and very deep pocketed p2p players. Since the micropayment model of the game actually relies on a healthy f2p player base to increase CE consumption and maintain high numbers of players online, this will ultimately lead to both f2p and p2p players leaving en masse.