To address these challenges, we designed and built a large language model (LLM) evaluation and deployment pipeline, which would allow Mizzen engineers to create, train, and test their own custom LLMs. We used this interface to create Mizzy, a Mizzen-specific LLM agent capable of sporting both educators and administrators while only referencing Mizzen’s content. 

Using Retool, we first developed the interface that allowed Mizzen engineers to create an LLM agent, prompt the agent with test prompts, review its responses, and provide structured ratings and feedback for improvement. This evaluation process ensured Mizzy could accurately answer questions, reference relevant lessons or content, and demonstrate a clear understanding of Mizzen’s curriculum and mission. The agent itself was built on AWS Bedrock, where we selected and configured a base model fitted with a knowledge base for it to reference materials from. The pipeline also tracked prompt histories, response qualities, and agent versions, allowing us to mark iterations of the agent as deployment ready in the pipeline. We designed this pipeline to be extensible, so engineers could use it to create, evaluate, and test other LLM agents for various other use cases. 

Once the evaluation pipeline was complete, we integrated our agent Mizzy directly into Mizzen’s platform. On the educator-facing side, Mizzy functions as an in-platform chatbot that allows users to ask questions, request modules, or modify/find activities that match specific criteria. On the administrative side, Mizzy automatically generates tags and categories for activities when edited or created. Finally, to ensure safe rollout of these new features into the platform, we built an automated quality assurance testing suite using Playwright to run various integration and end-to-end tests on deployment of new code to the platform, also enabling future engineers to scale AI-driven functionality safely and efficiently.