Many API callers will call unlink even for directories and it can mess
up with some wrappers like the encryption wrapper

Signed-off-by: Morris Jobke <hey@morrisjobke.de>