... Well it's a little hard to get the key without Fjord's consent or probably killing him. It is inside of him now.

I do not know what else could keep him down other than divine interference, which is very difficult to do directly since the Divine Gate was put in place to separate the gods from the Material Plane. ... But that doesn't mean their creations were sealed beyond it as well. Hence... this problem.