During SQL Eval., incorrect username or password was specified. Are we even able to eval SQL?

[vxnuaj]

When trying to compute the final eval using,

python3 evaluate.py --result_dir gpt-4o-mini-test-sql --mode sql

I run across the following:

Traceback (most recent call last):
  File "/root/spider_eval/Spider2/spider2-snow/evaluation_suite/evaluate.py", line 429, in <module>
    evaluate_spider2sql(args)
  File "/root/spider_eval/Spider2/spider2-snow/evaluation_suite/evaluate.py", line 345, in evaluate_spider2sql
    exe_flag, dbms_error_info = get_snowflake_sql_result(pred_sql_query, database_id, True, "temp", f"{id}.csv")  
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spider_eval/Spider2/spider2-snow/evaluation_suite/evaluate.py", line 183, in get_snowflake_sql_result
    conn = snowflake.connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/__init__.py", line 55, in Connect
    return SnowflakeConnection(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/connection.py", line 467, in __init__
    self.connect(**kwargs)
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/connection.py", line 786, in connect
    self.__open_connection()
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/connection.py", line 1116, in __open_connection
    self.authenticate_with_retry(self.auth_class)
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/connection.py", line 1404, in authenticate_with_retry
    self._authenticate(auth_instance)
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/connection.py", line 1432, in _authenticate
    auth.authenticate(
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/auth/_auth.py", line 416, in authenticate
    Error.errorhandler_wrapper(
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 283, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 341, in hand_to_other_handler
    connection.errorhandler(connection, cursor, error_class, error_value)
  File "/root/spider_eval/Spider2/methods/spider-agent-snow/venv/lib/python3.12/site-packages/snowflake/connector/errors.py", line 214, in default_errorhandler
    raise error_class(
snowflake.connector.errors.DatabaseError: 250001 (08001): Failed to connect to DB: RSRSBDK-YDB67606.snowflakecomputing.com:443. Incorrect username or password was specified.

I note that this would be because the evaluation_suite/snowflake_credential.json isn't populated correctly with the provided username and password given to access the snowflake database, but I specifically made sure that it was so.

The same credentials work when running the initial runs with the model as,

python3 run.py --model gpt-4o-mini -s test2

so I'm not sure what the problem is honestly.

I made sure credentials were being loaded properly into snowflake.connector.connect() in evaluation_suite/evaluate.py but even with some modifications,

def load_json(filename):
    with open(filename, 'r') as file:
        data = json.load(file)
    return data

def get_snowflake_sql_result(sql_query, database_id, is_save, save_dir=None, file_name="result.csv"):
    """
    is_save = True, output a 'result.csv'
    if_save = False, output a string
    """
   
    snowflake_credential = load_json('snowflake_credential.json') 
    # snowflake_credential = json.load(open('snowflake_credential.json'))
   
    conn = snowflake.connector.connect(
        **snowflake_credential,
        database=database_id,
    )

creds. aren't recognized and can't access the database for computing final evals based on SQL.

Perhaps you guys don't intend for us to eval this way given your update on the README.md

2024-12-26: Using Spider-Agent to benchmark your LLMs! Considering the widespread attention to the traditional text-to-SQL setting, we now recommend using [spider-agent-lite](https://github.com/xlang-ai/Spider2/tree/main/methods/spider-agent-lite) and [spider-agent-snow](https://github.com/xlang-ai/Spider2/tree/main/methods/spider-agent-snow) to work with [spider2-lite](https://github.com/xlang-ai/Spider2/tree/main/spider2-lite/evaluation_suite) and [spider2-snow](https://github.com/xlang-ai/Spider2/tree/main/spider2-snow/evaluation_suite) for benchmarking your LLMs. The final output should be CSV files, not SQLs.

but only 281 out of the 547 examples sampled from the LLM are rendered as CSV files and the rest are as SQL queries, so it's uncertain how to evaluate the 266 SQL queries if the above doesn't work.

[lfy79001]

Hi, we have fixed this bug