How do you measure the effectiveness of generative AI training? How to create evaluation sheets and practical assignments
The effectiveness of generative AI training cannot be judged solely by participant satisfaction. We will use tasks that are similar to actual practice and evaluate them, including determining input, creating requests, and checking output. We prepare assignments of the same level of difficulty before and after the training, and check afterwards to see if they were able to use them in actual practice.
Divide the evaluation into three time points
| Time | Things to check | Method |
|---|---|---|
| Before training | Current work and tasks | Performing similar tasks using normal means |
| Immediately after training | Can you reproduce the procedures you learned | Evaluate your practical skills using different fictitious data |
| After trial in practice | Is it possible to continue using it? | Check the implementation record of the target business |
If you use the same questions as they are, your memorization of the questions will be reflected in your grade. We will compare different questions with the same amount of input and difficulty.
Evaluation example using negotiation memo organization
Prepare an assignment that summarizes customer requests, decisions, responsibilities, and deadlines from hypothetical business negotiation notes. Deliberately include items with no deadlines and see if you can notice if guesses are mixed into the AI output.
| Perspective | 0 points | 1 point | 2 points |
|---|---|---|---|
| Input judgment | Using unconfirmed materials | Check if pointed out | Check availability by yourself |
| Request details | Purpose is unknown | Purpose is present but conditions are insufficient | Purpose, conditions, and output format are clear |
| Accuracy | Important errors remain | Can be corrected and completed | Completed correctly by checking with evidence |
| Confirmation action | Adopt as is | Confirm part | Confirm unclear points and basis |
| Reuse | No steps left | Save only instructions | Leave instructions and confirmation steps |
This is an example of grading for training design. Even if your total score is high, if you enter confidential information or make a major error, we will deal with it separately. The total score alone does not determine whether or not it can be used for practical purposes.
Time is measured until completion
This includes input preparation, requests to AI, waiting time, and human verification/correction. Don't compare it to conventional work only based on the time it took to generate. If the instructor helped you along the way, the details of the assistance will also be recorded.
You can use the Evaluation Sheet CSV to record scores for each perspective, time required, and whether support is provided. The scores in the sample entries are fictitious and do not reflect the results of the training.
Items to check after attending the course
Check whether you have had the opportunity to try out the target work, whether you have actually used it, and whether your burden, including confirmation, has changed. If the reason for not being able to use the materials is due to restrictions on the use of materials, organizing the information management may be a better option than additional training.
The difference before and after training is also affected by the amount of work, the experience of the person in charge, and the improvements made at the same time. Do not conclude that all changes are due to training effects; record conditions.
FAQ
Is a survey unnecessary?
This is useful for finding out about parts that are difficult to understand or problems you may have. Satisfaction and practical evaluation are used as separate indicators.
Should I give everyone the same assignment?
Basic input judgments can be standardized, but application to work is tailored to the type of job. We will align the evaluation criteria and adjust the content of the assignments.
Before consulting ZETTAi
Deciding on the work tasks before training and the deliverables you want to check after training will make the evaluation axis clear. Regarding ZETTAi's AI training for corporations, you can consult about evaluation methods that are connected to practical work. The scope of individual support will be confirmed at the time of consultation.
Articles you may also want to read
- How to choose generative AI training for corporations | Curriculum that will stick in practice and comparison items
- How to create dummy data for generative AI training? Exercises similar to actual practice and confirmation before input
- What is the reason why generative AI does not take root within the company? Improvement procedures by usage status
Information on training and introduction support
If you are the president and would like to try automating your work, please see Contents and prices of AI training for presidents. This is a training program in which we listen to your business and requests, create something, try it out, and make improvements.
Contents and prices of AI training for presidentshttps://ceo-claude-code-kennsyuu.onrender.com/
When considering what to automate with Grokbot, you can specify the work you want to delegate from Grokguide usage examples/prompts/how to make.
Grokguide usage examples/prompts/how to makehttps://grokguide.jp/
If you would like to consult with us about building a system tailored to your business, please check Details of AI agent construction support.
Details of AI agent construction supporthttps://zettai-ai-agent-site-render.onrender.com/