databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase

i am

who’s gonna run the spark job

wait what

haha

but not the one that crashes prod

like optimize a spark job

but that’s too easy

like print hello world

maybe we need a simpler task

what’s the point of benchmarking if they can’t even pip install

ok but like

some agents can’t even install the dependencies

ugh

now back to benchmarking

ok so donuts are decided

🍩

but only if we get donuts

i’ll do it

ok so who’s free to review

maybe someone from qa

ok so we need a separate reviewer

but i’m also benchmarking

who’s gonna review the prs

like add a new column to a dataframe

ok so let’s try a medium task

like hello world

what’s the point of benchmarking if they can’t even cd

some agents can’t even cd into the repo

ok so pizza’s decided

🍕