It sounds like you have the configurations setup correctly; at least to the level you described. A couple of things to check that I can think of.
1. There are validation settings to suppress the warning to the end user (and even allow the submit to go through). Is it possible your settings are configured this way?
2. If the record submit process is actually being allowed through, I expect it is creating an error on the TP Connect side of things. From your description, I expect this would be throwing a key field violation type exception. This would show up in the log of TP Connect. I would be curious to hear if the log itself indicated why this value might be sneaking through the Unique In Table validation.
3. If you are a licensed user with an active support agreement, you could always create a support ticket and someone on our side could review your project. You would just need to do an export of your TP Desktop project via File-->Export and attach the created .tpe file to the ticket itself. A support ticket can be created from www.tracerplus.com
under the Support menu option.
Thanks, Let me know if this helps