This is just a Mathematica feature.
I.e., if you use -> (Rule) the right-hand side is evaluated before
applying the rule (and Tr[a.b.GA] just evaluates to 0).
So, just use :> (RuleDelayed):
(Please, to all: study and learn Mathematica. There are screencasts,
Loading FeynCalc from /home/rolfm/HighEnergyPhysics
FeynCalc 8.0.0.beta2 Type ?FeynCalc for help or visit http://www.feyncalc.org/
$PrePrint is set to FeynCalcForm. Use FI and FC to change the display format.
Loading FeynArts, see www.feynarts.de for documentation
FeynArts 3.4 patched for use with FeynCalc
In:= expr = GS[a, b, c, d];
In:= expr /. Dot[x_, y__] :> Tr[Dot[x,y,GA]]
Out= (-4 I) eps[a, b, c, d]
This archive was generated by hypermail 2b29 : 07/26/17-10:40:01 AM Z CEST