Personally, as an economist, the main difference between prediction and inference is that the former is about "prediction", as it says. That's the be-all and end-all of the point of a predictive model, namely how well does it predict future or out-of-sample outcomes.
Inference on the other hand is widely used in trying to understand what factors contributed to an outcome, and by how much. A classic example in economics is trying to understand what are the factors that drive long-run economic growth. You can see an example here: https://www.imf.org/external/pubs/ft/wp/2013/wp13236.pdf. Another classic example ia what are the factors that predict individual educational attainment?
In these classic cases of statistical inference, you are trying to explain how various factors contribute to an outcome. Yes, I suppose that the explanation should ideally give rise to good future predictions, but the explanation is the more important goal. Consequently, inferential models need to be highly transparent.
If you take the linked example I used above, you want to know what are the key growth drivers, and you often even want to know how to rank them. After all, you often have a limited budget or political capital when making public policies, so you want to know what factors you should focus on. A black-box predictive model like SVM or XGBoost cannot provide the necessary insights for these purposes.