MMTEB Author check

[KennethEnevoldsen]

Hei all contributors, hope you are well,

We plan to submit it to ICLR in the coming days.

I have attached a (non-final) version of the paper here:
MMTEB-3.pdf

Edit: updated version of the cover is here:

This post here is to allow authors to opt-out of authorship and/or allow authors to question ordering/authorship.

There are still a few final things that need to be added (especially template and anonymization), but if you find that there are specific points that need addressing let me know.

Tagging authors for reference:
@akshita-sukhlecha
@rbroc
@jupyterjazz
@bp-high
@Rysias
@dwzhu-pku
@gowitheflow-1998
@x-tabdeveloping
@simon-clematide
@jphme
@rafalposwiata
@manandey
@gentaiscool
@henilp105
@anpalmak2003
@MathieuCiancone
@rasdani
@dipam7
@awinml
@artemsnegirev
@shreeya-dhakal
@Akash190104
@ZhengLiu101
@Art3mis07
@taeminlee
@GabrielSequeira
@jaygala24
@xiamengzhou
@ljvmiranda921
@Samoed
@KennethEnevoldsen
@digantamisra98
@sivareddyg
@mrshu
@Ruqyai
@asparius
@staoxiao
@MariyaTikhonova
@isaac-chung
@violenil
@SaitejaUtpala
@xhluca
@ManuelFay
@wissam-sib
@bjoernpl
@Alenush
@Andrian0s
@ABorghini
@taidnguyen
@cassanof

[KennethEnevoldsen]

The rest of the authors:

@Sakshamrzt
@imenelydiaker
@HLasse
@guenthermi
@MartinBernstorff
@xu3kev
@swj0419
@mmhamdy
@guangyusong
@john-b-yang
@slvnwhrl
@PranjalChitale
@malteos
@davidstap
@vaibhavad
@AlexeyVatolin
@KranthiGV
@ShawonAshraf
@jordiclive
@jankounchained
@Ab1992ao
@crystina-z
@Muennighoff
@hgissbkh
@schmarion
@dokato
@mariyahendriksen
@sted97
@orionw
@tomaarsen

[xhluca]

@KennethEnevoldsen It appears Mila appears twice, as 13 and 52

[HLasse]

and as 15

[isaac-chung]

ServiceNow Research shows up as 16 & 21

[davidstap]

Similar issue for University of Amsterdam; it shows up as 4 and 53.

[imenelydiaker]

Maybe we should standardize affiliations. Either have "Institution, Country" or just "Institution", wdyt @KennethEnevoldsen ?

[KennethEnevoldsen]

Institution is probably the simplest (no need to collect new information) - Will do this

[imenelydiaker]

Btw @KennethEnevoldsen, I removed my Esker affiliation from the points.md file, I'm just keeping INSA Lyon

[SaitejaUtpala]

To find duplicates in the institutions listed, I'll go through the list and note any that appear more than once:

1. Stanford University - appears twice (entries 6 and 40)
2. Mila - Quebec AI Institute - appears three times (entries 13, 15, and 52)
3. ServiceNow Research - appears twice (entries 16 and 21)
4. University of Amsterdam - appears twice (entries 4 and 53)
5. SaluteDevices, Russia - appears twice (entries 46 and 54, though the second entry includes "HSE University")

These are the institutions that appear more than once in the list. All other institutions are listed only once.

i asked claude, here are the duplicates

[isaac-chung]

Some names (including mine) don't have superscripts. Could those authors be "individual contributors"?

[KennethEnevoldsen]

Added

[Sakshamrzt]

Please drop me ***@***.*** <https://github.com/Sakshamrzt>) from the list. Thanks!

…

On Thu, 26 Sept 2024 at 08:54, Isaac Chung ***@***.***> wrote: Some names (including mine) don't have superscripts. Could those authors be "individual contributors"? — Reply to this email directly, view it on GitHub <#1241 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQOBY27WCMUXWORMUZRZ2ALZYQU5FAVCNFSM6AAAAABO5J6MUWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANZWGU2TANA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[KennethEnevoldsen]

Will do @Sakshamrzt, do you still want to be in the table of contributors or should I remove you there as well?

[KranthiGV]

Can we update the affiliation? If so, what's the process?
Should we raise a PR to update points.md - contributor info?

[isaac-chung]

Might just be easier to make a comment here and one of us can make the edits in the Overleaf file directly. What would you like to change it to?

[KennethEnevoldsen]

yes as @isaac-chung says

[isaac-chung]

I'll go through the draft to remove the duplicates.

[KennethEnevoldsen]

I will probably do this in a script (easier to do with the other changes as well)

[isaac-chung]

@KennethEnevoldsen feel free to override my changes as you see fit.

[crystina-z]

My name (Xinyu Zhang) seems to duplicate twice while @thakur-nandan (Nandan Thakur) name is not included in the author list

[KennethEnevoldsen]

Thanks!! Would have hated to miss that. I assume it is:

first name: Nandur
last name: Thakur
Aff: University of Waterloo
id on openreview: ~Nandan_Thakur1

(but do correct me if I am wrong)

[thakur-nandan]

That's correct. Sorry I just checked this.

[KennethEnevoldsen]

I have now corrected the duplicates, missing authors, added "individual contributor" for non-aff authors and removed @Sakshamrzt

(I have added an updated screenshot in the top)

[cassanof]

Hi! Thank you so much for including me in the list of contributors.
Would it be possible to change my affiliation to "Northeastern University and Cursor AI"?

Thanks again!

[imenelydiaker]

Please update this file and open a pull request: https://github.com/embeddings-benchmark/mteb/blob/main/docs%2Fmmteb%2Fpoints.md

[KennethEnevoldsen]

fixed !

[digantamisra98]

@KennethEnevoldsen Can my affiliation please be updated to: Max Planck Institute for Intelligent Systems and ELLIS Institute Tübingen. Thank you!

[imenelydiaker]

Please update this file and open a pull request: https://github.com/embeddings-benchmark/mteb/blob/main/docs%2Fmmteb%2Fpoints.md

[KennethEnevoldsen]

Fixed!

[digantamisra98]

#1247
Done

[dwzhu-pku]

Thank you so much for including me in the list of contributors! @KennethEnevoldsen

[jhrystrom]

Great work, Kenneth! On the screenshot, your email address seems to be missing a letter:))

[KennethEnevoldsen]

fixed ;)

[kwojtasi]

@KennethEnevoldsen Is it possible to be included in the author's list? I have contributed BEIR-PL benchmarch for Polish language for MTEB benchmark.

[KennethEnevoldsen]

Of course @kwojtasi. I went through the PR that would result in 22 points (from dataset submission). I have not counted any bonuses here, but it does justify for an authorship.

Can you (just here) write your author first, last name, affiliation and most importantly your openreview ID.

[kwojtasi]

Thank you!
Providing the details:
Konrad Wojtasik, Wrocław University of Science and Technology, ~Konrad_Wojtasik1

[digantamisra98]

Who's registering the abstract? The deadline is today EOD I believe. @KennethEnevoldsen

[KennethEnevoldsen]

I am - doing it now

[digantamisra98]

Thank you!

[dokato]

Anything we could do to help with addressing reviewers' comments @KennethEnevoldsen ?

[Samoed]

It would be great if you could help close #1475. The reviewers are requesting statistics for the task, and with descriptive stats, these can be calculated automatically.

[isaac-chung]

I think @imenelydiaker's supporting that. Pinging her to see if she needs any additional support.

[KennethEnevoldsen]

Thanks @dokato I believe most points are being addressed. One point which might be ideal is the comparison of MTEB(classic) and MTEB(eng). Where we want to compare the runtime in a comparable manner (currently we are reporting runtime on A100 on one and H100 on the other), a solution to this is the just report the number of characters to embed pr. benchmark.

I could imagine the code could look something like:

bench = mteb.get_benchmark("mteb(eng, beta)")
n_char = 0 

for task in bench.tasks:
  n_char = task.metadata.descriptive_stats["number of characters"]

And then update the abstract intro and sections where we discuss runtime/size

[SaitejaUtpala]

@KennethEnevoldsen Just want to say I love newly added Figure 4 !!

[KennethEnevoldsen]

Great to hear! @x-tabdeveloping is the man behind it, but I agree - it quite reasonably reveals the current language gap even for reasonably mid to high-resource languages.

[manandey]

@KennethEnevoldsen any plans on having the paper on arxiv?

[KennethEnevoldsen]

yes! We plan to push it as soon as the next version of the leaderboard is ready (start of january). Hopefully that works?

[KennethEnevoldsen]

(i believe it is also public here)

[manandey]

Great, thanks!