Job Details

Job Description:

• The Responsible AI Scaled Testing Team within Trust & Safety performs pre-launch structured testing for client's AI applications against safety, fairness and neutrality policies and standards.
• It is a global team with Responsible AI domain expertise and diverse backgrounds in operations, strategy, ethics, risk management, product management, and program management.

Responsibilities:

• Lead end-to-end structured pre-launch testing for safety, neutrality, and fairness in client's GenAI products.
• Define applicable standards, execute prompt generation strategies, collaborate with product teams to scrape responses, and oversee prompt/response rating, including providing instructions, clarifying gray areas, and ensuring quality calibrations.
• Conduct in-depth quantitative and qualitative analysis of results.
• Intake and triage new launch submissions, assess requirements, and initiate engagement or schedule start dates.
• Manage ongoing launches through the pre-launch testing process to track progress, working independently or coordinating with stakeholders.
• Align on safety, neutrality, and fairness standards, ensuring consistency across product areas. Translate standards into clear evaluation guidelines.
• Develop and execute prompt generation strategies using LLM-based tools or vendor teams.
• Collaborate with product teams on response scraping, advising on scaled solutions, obtaining model/UI access, and guiding vendor teams.
• Oversee prompt/response rating, providing clear instructions, resolving ambiguities, and maintaining quality calibrations.
• Perform deep-dive analysis, identifying unexpected, interesting, and edge cases, and provide actionable insights for pre- and post-launch mitigation.

Mandatory:

• 4 years of experience in any one of data analytics, Trust & Safety, policy, cybersecurity, or related fields.
• Experience using data to provide solutions and recommendations.
• Excellent communication and presentation skills (written and verbal) and the ability to influence cross-functionally at various levels.
• This role may be exposed to graphic, controversial, and/or upsetting content.
• Experience in business process analysis, operations management, and/or global program management, or leading cross-functional process improvements is a plus.
• Strong understanding of AI systems, machine learning, and their potential risks is a plus.

Education:

• Bachelor's degree or equivalent practical experience.

About US Tech Solutions:

US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com.

US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.