Request Tracing

Request Tracing is a powerful Pro feature that allows you to record the full content of requests and responses made to AI providers. This detailed logging gives you comprehensive visibility into how your users and applications interact with AI models.

Benefits of Request Tracing

Request Tracing provides several key advantages:

• Debugging: Troubleshoot issues by seeing exactly what was sent and received
• Auditing: Maintain a record of all AI interactions for compliance or verification
• Training Data: Build datasets for fine-tuning your own models in the future
• Quality Assurance: Evaluate the quality of responses and identify improvement opportunities
• Compliance: Monitor content for policy violations or inappropriate usage

Enabling Request Tracing

Enabling Tracing for a Single User

You can also enable tracing for a specific user:

1. Navigate to “Users” and select the user
2. Click “Create Policy” in the User Policy section
3. Toggle “Request Tracing” on
4. Set an expiration date if this is a temporary change
5. Click “Create Policy” to save

This is useful when you need to debug issues for a specific user without enabling tracing for everyone.

Viewing Traced Requests

Once tracing is enabled, you can view the recorded data:

1. Navigate to “Usage” in the sidebar
2. Find the request you want to examine
3. Click on the request to open the details
4. View the complete request and response information:
   • Request URL and headers
   • Request body/prompt
   • Response body/completion
   • Timing information
   • Token counts and cost data

Privacy and Security Considerations

When using Request Tracing, be mindful of privacy and security:

• User Consent: Inform users that their interactions with AI may be recorded
• Sensitive Data: Avoid enabling tracing for workflows that process highly sensitive information
• Data Retention: Consider how long you need to retain traced data
• Access Control: Limit access to traced data to authorized personnel

Building Training Datasets

One powerful use of Request Tracing is to build datasets for fine-tuning custom models:

1. Enable tracing for a policy used by typical users
2. Allow usage over time to collect diverse examples
3. Export the data (contact support for assistance)
4. Clean and format the data for fine-tuning
5. Use the dataset to create specialized models that perform better at your specific tasks

This approach can significantly reduce costs in the long run by allowing you to train smaller, more efficient models that achieve similar results for your specific use cases.

Combining with Request Caching

Request Tracing works seamlessly with Request Caching:

1. Enable both features on a policy
2. Monitor which requests are frequently repeated
3. Use these insights to optimize your application design
4. Let caching automatically improve performance for common requests

Next Steps

After enabling Request Tracing:

1. Explore Request Caching to improve performance
2. Learn how to track usage across your project
3. Consider building training datasets based on the traced data