Don't worry, you still have a ways to go. After the Ice Caves, you still have the Crystal Caverns, the Dark Forest, the Eastern Swamp, the Goblin Cave and the Ruined Temple before reaching the Fortress.
Just how it happens to work out (take it up with the GPG writers, I guess ) Anyways, there are plenty of quests to do in the meantime, so don't worry...