Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] add support for mixtral #145

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Conversation

tohrnii
Copy link

@tohrnii tohrnii commented Jan 30, 2024

Mixtral WIP

@tohrnii tohrnii force-pushed the mixtral branch 7 times, most recently from 709d2e3 to d54e095 Compare January 31, 2024 19:06
@danielhanchen
Copy link
Contributor

Fantastic and fabulous work @tohrnii!!! Super appreciate it! I will take a look later today!

@tohrnii tohrnii force-pushed the mixtral branch 2 times, most recently from e2d6b62 to ff31b00 Compare February 5, 2024 16:43
@kaykyr
Copy link

kaykyr commented Feb 12, 2024

Any update about this pull request?

@cm2435
Copy link

cm2435 commented Feb 12, 2024

@kaykyr @danielhanchen @tohrnii You guys open to some collaboration on this? I think I just my Phi2 implementation done (big touch wood) so I'm happy to take a look

@tohrnii
Copy link
Author

tohrnii commented Feb 13, 2024

Apologies, I got stuck on something else. I'd love to collaborate @cm2435. If however you are close to completing the implementation, I'm happy to close this PR in favor of yours.

@danielhanchen
Copy link
Contributor

danielhanchen commented Feb 13, 2024

Hey - thanks again on the PR @tohrnii and super appreciate it again :) Ye more than happy to make this happen and collab with you all @kaykyr @cm2435 - I was just a bit bogged down recently on chat templates and making a UI - I will be much more free next week, then we can make Mixtral happen :)

@kaykyr
Copy link

kaykyr commented Feb 14, 2024

For sure! I am trying to fine tune a MoE pretrained if I had progress I will create pull requests guys. I also able to offer my small server (2x RTX 3090 with NVLink + i9 11900HK + 64GB DDR4) for collaborators who wanna run tests with multi-gpu.

@danielhanchen
Copy link
Contributor

@kaykyr Oh thanks for the kind offer!!! I'll take up for that offer later in the month :)

@cm2435
Copy link

cm2435 commented Feb 16, 2024

@kaykyr funny you mention that- I've got almost the exact same setup! I'm going to be very sad when they deprecate the SLI bridge as cuda supported hardware

@ilkersigirci
Copy link

Great work. Is there any estimate about when this will be merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants