From what I have experienced fishing and diving are extremely unbalanced. Swimming/Diving requires a larger investment in AP, items, and vitality and the mini-game is slower/more difficult. At the same time the rewards for diving are much lower than for fishing. I suppose this could be fixed by simply making the items received via diving much more valuable, but preferably there would be more progression the the diving mini-game/rewards to match the investment of time and money required to train the swimming skill.
I am still a new player so there may be things I've missed entirely. When I starting my first game I decided to focus on diving on a whim and the rewards were so low it almost put me off the game entirely. Then I tried fishing and realized how much easier everything was.