Refactor MLLM #529
Refactor MLLM #529
Conversation
| val_dataset = [ | ||
| dict( | ||
| type=MMELLaVADataset, | ||
| data_file='/mnt/petrelfs/huanghaian/code/xtuner/LMUData/MME.tsv', |
There was a problem hiding this comment.
像这种路径，是让用户修改config来设置，还是说像VLMEvalKit一样，内部自动下载
There was a problem hiding this comment.
我觉得不用这么智能吧，直接要他们提前下载好，省的出现一些奇怪问题
| # set log processor | ||
| log_processor = dict(by_epoch=False) | ||
|
|
||
| # ==================== val and test cfg ======================= |
There was a problem hiding this comment.
此部分，是放到上面 PART 3 Dataset & Dataloader 还是单独放在下面？
There was a problem hiding this comment.
我觉得单独放下面，用户用起来更方便？
| print_log('============================================', 'current') | ||
| print_log(score, 'current') | ||
| print_log('============================================', 'current') | ||
| print_log(f'YOrN_eval successfully finished evaluating', 'current') |
There was a problem hiding this comment.
| print_log(f'YOrN_eval successfully finished evaluating', 'current') | |
| print_log('Hallusion successfully finished evaluating', 'current') |
| with pd.ExcelWriter(osp.join(work_dir, self.results_xlsx_path), engine='openpyxl') as writer: | ||
| results_df.to_excel(writer, index=False) | ||
|
|
||
| score = Hallusion_rating(data) |
There was a problem hiding this comment.
把 Hallusion_rating 函数放到本文件？ 因为这个应该只有这里会用到，所以方便查看。
| with pd.ExcelWriter(osp.join(work_dir, self.results_xlsx_path), engine='openpyxl') as writer: | ||
| results_df.to_excel(writer, index=False) | ||
|
|
||
| score = MME_rating(data) |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| results = collect_results(results, len(dataset)) | ||
| self.runner.logger.info('========= Starting the evaluation of a data ===========') | ||
| if is_main_process(): | ||
| metric = dataset.evaluate(results, self.runner.work_dir) |
There was a problem hiding this comment.
self.runner.work_dir 是否考虑携带 iter 信息，类似这里
xtuner/xtuner/engine/hooks/evaluate_chat_hook.py
Lines 93 to 98 in 0b5708c
| print_log('Samples: {}, Accuracy: {:.2f}%'.format(len(pred_list), acc), 'current') | ||
| print_log('============================================', 'current') | ||
| print_log(f'TextVQA successfully finished evaluating', 'current') | ||
| return {'acc': acc} |
There was a problem hiding this comment.
规定必须要返回字典吧，我记得，因为到时候 bset checkpoint 时候会读取
| @@ -96,7 +97,11 @@ def main(): | |||
| runner = RUNNERS.build(cfg) | |||
|
|
|||
| state_dict = guess_load_checkpoint(args.checkpoint) | |||
There was a problem hiding this comment.
checkpoint 现在是一个 optional argument，看起来是之前残留的bug，最好改成 positional argument。
|
Hi @hhaAndroid |
In order to enhance the flexibility and usability of the multimodal model, it is necessary to refactor the existing llava code.