What is our primary use case?
We're implementing an assistant using Azure OpenAI. The challenge is grounding OpenAI responses to our specific data.
We can only offer users basic querying, like for documents they're stuck on. It handles the request. It's primarily the question-answering feature.
What is most valuable?
It's very powerful. It allows users to query our documents using natural language and receive answers in the same way. This makes our product information much more accessible than traditional keyword-based search.
It's focused on information retrieval and question-answering, which suits our needs perfectly. It is more like a natural language query tool we leverage.
We use Azure OpenAI alongside Azure Cognitive Search. These are both new services we've deployed. There's a process where we need to ask Microsoft to create private endpoints to link OpenAI to Azure as a connectivity service.
What needs improvement?
Since we don't train the model on our data, it's a struggle to ensure OpenAI answers questions exclusively from our data. During user testing, we found ways to make the system provide answers from outside sources.
As a governance department, accuracy and control are crucial. We're trying to tune the system to stick with our content, but it's an ongoing challenge.
We've been working on fine-tuning prompts and parameters for about four weeks now.
For how long have I used the solution?
I've been using Azure OpenAI as a creative source for the past six months.
What do I think about the scalability of the solution?
We've noticed some issues with scaling. It takes time for the service to adapt when we increase the load. We're still in the pre-production phase, and we're seeing this even during testing.
Also, there's limited capacity in our region (Canada East), which makes it difficult to accommodate the expected load. We've submitted capacity increase requests, but we're not sure if they'll be approved.
The main challenge we've faced is around capacity. Even after running extensive load tests, we don't have sufficient capacity to handle our projected volume.
How are customer service and support?
We have a consultant from Microsoft working with us. They've been very helpful.
However, they're very busy. We could use more of their time if they were available. But they're very competent and helpful. We just wish we could have more access to their expertise.
How would you rate customer service and support?
Which solution did I use previously and why did I switch?
We have an alternative search engine that indexes our document base. We use Azure OpenAI's question-answering feature to query that index, generating answers from relevant documents.
We don't use GPT-4 specifically, nor are we training any models. Our IT group leverages Azure OpenAI for its existing capabilities.
It is our first implementation of this kind.
What other advice do I have?
There are some limitations right now. For our specific use case, where we need a traditional information retrieval system, it's not an ideal fit.
Azure OpenAI is a question-answering system built on top of information retrieval, and that distinction is important for us. Given our use case, I don't think it's well-suited.
Our management team requires accurate and complete results, with precision that matches our existing keyword search tools. It's difficult to evaluate and prove that Azure OpenAI consistently meets that standard.
We're still early in our adoption, so the rating could change as we deploy it to a larger audience.
For now, I would rate the solution a five out of ten.
Disclosure: My company does not have a business relationship with this vendor other than being a customer.