Model-based machine learning techniques for fiber nonlinearity compensation