Ill try to argue here a bit about a strong form:
If you use unique_ptr and don’t pass ownership you, probably, can use RAII instead. If you pass ownership you can still shoot the foot.

You even wrote how exactly.

The problem is that controlling ownership could become tricky. Its the same as using raw pointers but on more safe level. You cant forget to delete data, you cant copy them, but you still can move ownership and try to access it after that.

Shared_ptr solves this problem but not without the price. There is overhead and you should keep it in mind. When you need a bit more optimization you should use other instruments.

About shared_ptr: i saw a lot of posts demonizing them. Yes – when you start to use them as global data, they become global data. Yes – if you store them in some permanent buffers they will stay in memory. Yes – if you put some mutable object inside and then affect it from different sources, things might become messy. To not avoid such cases – is rookie mistake imo.

And the most important part – using with immutable data.

Plus: they are costly. unique_ptr’s can be entirely optimized-out by the compiler; shared pointer always carries some overhead.

I would re-phrase it as: Use unique_ptr whenever object is owned, or as a return value/parameter whenever ownership is transferred.
Use reference or reference_wrapper whenever borrowing an object.
Use raw pointer ONLY as an optional, non-owning references only if the value can legally be null. If you see something like this in your code:

assert(ptr); ptr->x();

you just wrote a bug. Use reference instead.

I find shared_ptr abuse to be a give-away rookie mistake. They are convenient, powerful, and poorly understood.