You should be concerned. Ilegal building can and are in fact demolished. Check with the Câmara if you can submit a project for it to get approved. The best solution would be to submit a "Pedido de Informação prévio" to the Câmara.

This is a simple process and assures you that if the Câmara says it's ok that you have one year to submit and get your project approved.

The fact that there are other ilegal buildings doesn´t make that building more legal does it?