Hi Itay, actually you are right. The /opt/spark/conf is mounted by the deployment. In order to overcome it, you can submit the spark app with the following config :

--conf spark.metrics.conf=/SOME_FOLDER/metrics.properties.

I hope it helps.