I am involved with Ceph and OpenStack as an integrator. I set it up or consult with clients for private cloud deployments. Ceph is my storage of choice for OpenStack and general object-storage needs.
Ceph has simplified my storage integration. I no longer need two or three storage systems, as Ceph can support all my storage needs. I no longer need OpenStack Swift for REST object storage access, I no longer need NFS or GlusterFS for filesystem sharing, and most importantly, I no longer need LVM or DRBD for my virtual machines in OpenStack.
The ability to present Rest API, POSIX filesystem, and block devices from the same distributed object storage back-system (RADOS) is of major value to me.
Ceph lacks a little bit only in performance. It needs to scale a lot and needs very fast and well-orchestrated/configured hardware for best performance. This not a downside though, it is a challenge. Ceph only improves the given hardware.
One to three years.
I have encountered issues with stability when replication factor was not 3, which is the default and recommended value. Go below 3 and problems will arise.
Ceph has no issues with scalability but needs proper planning regarding the hardware.
Technical support from the mailing lists is very good but you always need to get your hands dirty. However, if you pay for a product like Red Hat Ceph Storage, support is a big advantage.
I have used many solutions but not as extensively as Ceph. I switched to Ceph because of its architecture and because it is a one-stop shop when it comes to storage.
Ceph is complex, as with all distributed systems. It has a long learning curve but it pays for itself after that.
If you can afford a product like Red Hat Ceph Storage then go for it. If you cannot, then you need to test Ceph and get your hands dirty.
I have been using Ceph since 2015, in both SOHO and bigger private cloud installations.
Ceph, as a distributed storage solution, is amazing and I can only rate it a 10 out of 10. However, being distributed and complex, Ceph needs engineers with a good understanding of its internals and Linux, for it to shine.
Overall, it is a great product.