Click here to Skip to main content
15,886,963 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.

 
GeneralRe: Wordle 813 Pin
GKP199210-Sep-23 5:34
professionalGKP199210-Sep-23 5:34 
Generalworldle 596 1/6 Pin
jmaida9-Sep-23 10:56
jmaida9-Sep-23 10:56 
Generalmade a trouble in my GitHub repository Pin
Southmountain9-Sep-23 8:08
Southmountain9-Sep-23 8:08 
GeneralRe: made a trouble in my GitHub repository PinPopular
Jon McKee9-Sep-23 8:52
professionalJon McKee9-Sep-23 8:52 
GeneralRe: made a trouble in my GitHub repository Pin
Southmountain9-Sep-23 9:09
Southmountain9-Sep-23 9:09 
GeneralRe: made a trouble in my GitHub repository Pin
jschell11-Sep-23 6:01
jschell11-Sep-23 6:01 
GeneralRe: made a trouble in my GitHub repository Pin
Southmountain12-Sep-23 8:45
Southmountain12-Sep-23 8:45 
GeneralFinetune LLMs via the Finetuning Hub Pin
rsaha79-Sep-23 8:58
rsaha79-Sep-23 8:58 
Hi community, I have been working on benchmarking publicly available LLMs these past couple of weeks. More precisely, I am interested on the finetuning piece since a lot of businesses are starting to entertain the idea of self-hosting LLMs trained on their proprietary data rather than relying on third party APIs.

GitHub repo: https://github.com/georgian-io/LLM-Finetuning-Hub

To this point, I am tracking the following 4 pillars of evaluation that businesses are typically look into: - Performance - Time to train an LLM - Cost to train an LLM - Inference (throughput / latency / cost per token)

For each LLM, my aim is to benchmark them for popular tasks, i.e., classification and summarization. Moreover, I would like to compare them against each other.

So far, I have benchmarked Flan-T5-Large, Falcon-7B and RedPajama and have found them to be very efficient in low-data situations, i.e., when there are very few annotated samples. Llama2-7B/13B and Writer’s Palmyra are in the pipeline.

But there’s so many LLMs out there! In case this work interests you, would be great to join forces.

GitHub repo attached — feedback is always welcome Smile | :)

Happy hacking!

modified 9-Sep-23 15:14pm.

GeneralRe: Finetune LLMs via the Finetuning Hub Pin
Southmountain9-Sep-23 9:27
Southmountain9-Sep-23 9:27 
GeneralRe: Finetune LLMs via the Finetuning Hub Pin
Richard MacCutchan9-Sep-23 21:17
mveRichard MacCutchan9-Sep-23 21:17 
GeneralRe: Finetune LLMs via the Finetuning Hub Pin
BillWoodruff12-Sep-23 3:27
professionalBillWoodruff12-Sep-23 3:27 
JokeI know it's a long shot... PinPopular
Sander Rossel8-Sep-23 23:42
professionalSander Rossel8-Sep-23 23:42 
GeneralRe: I know it's a long shot... Pin
BillWoodruff9-Sep-23 0:05
professionalBillWoodruff9-Sep-23 0:05 
GeneralRe: I know it's a long shot... Pin
Richard MacCutchan9-Sep-23 2:04
mveRichard MacCutchan9-Sep-23 2:04 
GeneralRe: I know it's a long shot... Pin
StarNamer@work9-Sep-23 1:31
professionalStarNamer@work9-Sep-23 1:31 
GeneralRe: I know it's a long shot... Pin
Mike Hankey9-Sep-23 2:36
mveMike Hankey9-Sep-23 2:36 
GeneralRe: I know it's a long shot... Pin
obermd9-Sep-23 5:36
obermd9-Sep-23 5:36 
GeneralRe: I know it's a long shot... Pin
MarkTJohnson9-Sep-23 6:52
professionalMarkTJohnson9-Sep-23 6:52 
GeneralRe: I know it's a long shot... Pin
Brisingr Aerowing9-Sep-23 15:05
professionalBrisingr Aerowing9-Sep-23 15:05 
GeneralRe: I know it's a long shot... Pin
jmaida9-Sep-23 16:03
jmaida9-Sep-23 16:03 
GeneralRe: I know it's a long shot... Pin
jmaida10-Sep-23 12:44
jmaida10-Sep-23 12:44 
GeneralRe: I know it's a long shot... Pin
jschell11-Sep-23 6:06
jschell11-Sep-23 6:06 
GeneralRe: I know it's a long shot... Pin
jschell11-Sep-23 6:03
jschell11-Sep-23 6:03 
GeneralRe: I know it's a long shot... Pin
PIEBALDconsult9-Sep-23 7:50
mvePIEBALDconsult9-Sep-23 7:50 
GeneralRe: I know it's a long shot... Pin
Greg Utas9-Sep-23 9:22
professionalGreg Utas9-Sep-23 9:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.