Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for connecting to snowpark container services #100

Open
punitchauhan771 opened this issue Jul 26, 2024 · 8 comments · May be fixed by #103
Open

Add support for connecting to snowpark container services #100

punitchauhan771 opened this issue Jul 26, 2024 · 8 comments · May be fixed by #103

Comments

@punitchauhan771
Copy link

punitchauhan771 commented Jul 26, 2024

I'm facing issues connecting my Django application to a Snowflake warehouse using Snowpark Container Service and OAuth authentication. While the application can connect via the external network using username and password, it requires username when using OAuth within the internal network. I'm using Django-snowflake version 4.1 and would like to establish a connection without external network access.

@timgraham
Copy link
Collaborator

I believe you're referencing this check:

if settings_dict['USER']:
conn_params['user'] = settings_dict['USER']
else:
raise ImproperlyConfigured(self.settings_is_missing % 'USER')

Do you have a suggestion for how to modify the check? I'm not sure we should remove it completely. For example, we have logic not to require a password if any of these options are specified: `'private_key', 'private_key_file', 'authenticator'.

@timgraham
Copy link
Collaborator

@punitchauhan771 Can you clarify your question/request?

@punitchauhan771
Copy link
Author

punitchauhan771 commented Aug 31, 2024

@punitchauhan771 Can you clarify your question/request?

Hi @timgraham

I have an application that i want to host in snowpark container service, so if i am using external communication i.e. egress based AUTH then the container is able to connect to the snowflake warehouse and everything works fine (inside the container), but if i try to use internal network i.e. snowflake_host , token etc it Fails,

also I tried modifying this code base, even if i make the username field optional it still tries to connect to the host URL rather than the proxy URL created by snowflake for the connectivity.

I believe you're referencing this check:

if settings_dict['USER']:
conn_params['user'] = settings_dict['USER']
else:
raise ImproperlyConfigured(self.settings_is_missing % 'USER')

Do you have a suggestion for how to modify the check? I'm not sure we should remove it completely. For example, we have logic not to require a password if any of these options are specified: `'private_key', 'private_key_file', 'authenticator'.

The only challenge in the egress based AUTH is it creates additional latency for each of my request, so I was hoping we might have some way to connect to internal network like how we do in Flask/python:

Flask/python example code:

# Environment variables below will be automatically populated by Snowflake.
SNOWFLAKE_ACCOUNT = os.getenv("SNOWFLAKE_ACCOUNT")
SNOWFLAKE_HOST = os.getenv("SNOWFLAKE_HOST")
SNOWFLAKE_DATABASE = os.getenv("SNOWFLAKE_DATABASE")
SNOWFLAKE_SCHEMA = os.getenv("SNOWFLAKE_SCHEMA")

# Custom environment variables for LOCAL Testing only
SNOWFLAKE_USER = os.getenv("SNOWFLAKE_USER")
SNOWFLAKE_PASSWORD = os.getenv("SNOWFLAKE_PASSWORD")
SNOWFLAKE_ROLE = os.getenv("SNOWFLAKE_ROLE")
SNOWFLAKE_WAREHOUSE = os.getenv("SNOWFLAKE_WAREHOUSE")
DATA_DB = os.getenv("DATA_DB")
DATA_SCHEMA = os.getenv("DATA_SCHEMA")
LLAMA2_MODEL = os.getenv("LLAMA2_MODEL")

def get_login_token():
  """
  Read the login token supplied automatically by Snowflake. These tokens
  are short lived and should always be read right before creating any new connection.
  """
  with open("/snowflake/session/token", "r") as f:
    return f.read()
    
def get_connection_params():
  """
  Construct Snowflake connection params from environment variables.
  """
  # internal connection
  if os.path.exists("/snowflake/session/token"):
    return {
      "account": SNOWFLAKE_ACCOUNT,
      "host": SNOWFLAKE_HOST,
      "authenticator": "oauth",
      "token": get_login_token(),
      "warehouse": SNOWFLAKE_WAREHOUSE,
      "database": SNOWFLAKE_DATABASE,
      "schema": SNOWFLAKE_SCHEMA,
    }
  else:
   # external connection
    return {
      "account": SNOWFLAKE_ACCOUNT,
      "host": SNOWFLAKE_HOST,
      "user": SNOWFLAKE_USER,
      "password": SNOWFLAKE_PASSWORD,
      "role": SNOWFLAKE_ROLE,
      "warehouse": SNOWFLAKE_WAREHOUSE,
      "database": SNOWFLAKE_DATABASE,
      "schema": SNOWFLAKE_SCHEMA
    }

Hope this helps.

@timgraham
Copy link
Collaborator

I tried to replicate the logic in the example code you provided. Can you test it? The example code also omits role when using a token. Is that correct?

diff --git a/django_snowflake/base.py b/django_snowflake/base.py
index e2c4fcd..7e2a8ec 100644
--- a/django_snowflake/base.py
+++ b/django_snowflake/base.py
@@ -99,21 +99,26 @@ class DatabaseWrapper(BaseDatabaseWrapper):
             'interpolate_empty_sequences':  True,
             **settings_dict['OPTIONS'],
         }
+        if use_token := os.path.exists("/snowflake/session/token"):
+            conn_params["token"] = self.get_login_token()
+
         if os.environ.get('RUNNING_DJANGOS_TEST_SUITE') != 'true':
             conn_params['application'] = 'Django_SnowflakeConnector_%s' % __version__
 
         if settings_dict['NAME']:
             conn_params['database'] = self.ops.quote_name(settings_dict['NAME'])
 
-        if settings_dict['USER']:
-            conn_params['user'] = settings_dict['USER']
-        else:
-            raise ImproperlyConfigured(self.settings_is_missing % 'USER')
+        # Omit USER/PASSWORD if using token.
+        if not use_token:
+            if settings_dict['USER']:
+                conn_params['user'] = settings_dict['USER']
+            else:
+                raise ImproperlyConfigured(self.settings_is_missing % 'USER')
 
-        if settings_dict['PASSWORD']:
-            conn_params['password'] = settings_dict['PASSWORD']
-        elif all(x not in conn_params for x in self.password_not_required_options):
-            raise ImproperlyConfigured(self.settings_is_missing % 'PASSWORD')
+            if settings_dict['PASSWORD']:
+                conn_params['password'] = settings_dict['PASSWORD']
+            elif all(x not in conn_params for x in self.password_not_required_options):
+                raise ImproperlyConfigured(self.settings_is_missing % 'PASSWORD')
 
         if settings_dict.get('ACCOUNT'):
             conn_params['account'] = settings_dict['ACCOUNT']
@@ -132,6 +137,15 @@ class DatabaseWrapper(BaseDatabaseWrapper):
 
         return conn_params
 
+    def get_login_token():
+        """
+        Read the login token supplied automatically by Snowflake. These tokens
+        are short lived and should always be read right before creating any new
+        connection.
+        """
+        with open("/snowflake/session/token", "r") as f:
+            return f.read()
+
     @async_unsafe
     def get_new_connection(self, conn_params):
         return Database.connect(**conn_params)

@punitchauhan771
Copy link
Author

punitchauhan771 commented Sep 2, 2024

Hi @timgraham,

I made the above changes, now I am getting this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1414, in _authenticate
    auth.authenticate(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/auth/_auth.py", line 250, in authenticate
    ret = self._rest._post_request(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/network.py", line 739, in _post_request
    ret = self.fetch(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/network.py", line 854, in fetch
    ret = self._request_exec_wrapper(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/network.py", line 971, in _request_exec_wrapper
    raise e
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/network.py", line 896, in _request_exec_wrapper
    return_object = self._request_exec(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/network.py", line 1156, in _request_exec
    raise err
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/network.py", line 1100, in _request_exec
    raise OperationalError(

The above exception (251012: 251012: Login request is retryable. Will be handled by authenticator) was the direct cause of the following exception:
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 282, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 263, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django_snowflake/base.py", line 154, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/__init__.py", line 55, in Connect
    return SnowflakeConnection(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 456, in __init__
    self.connect(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 771, in connect
    self.__open_connection()
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1099, in __open_connection
    self.authenticate_with_retry(self.auth_class)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1386, in authenticate_with_retry
    self._authenticate(auth_instance)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1458, in _authenticate
    raise auth_op from e
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1435, in _authenticate
    auth_instance.handle_timeout(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/auth/by_plugin.py", line 212, in handle_timeout
    raise error

The above exception (250001: 250001: Could not connect to Snowflake backend after 2 attempt(s).Aborting) was the direct cause of the following exception:
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 486, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 448, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
    return await fut
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 490, in thread_handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/contrib/admin/sites.py", line 441, in login
    return LoginView.as_view(**defaults)(request)
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/debug.py", line 92, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 133, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/views.py", line 90, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 152, in post
    if form.is_valid():
  File "/usr/local/lib/python3.11/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.11/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.11/site-packages/django/forms/forms.py", line 438, in full_clean
    self._clean_form()
  File "/usr/local/lib/python3.11/site-packages/django/forms/forms.py", line 459, in _clean_form
    cleaned_data = self.clean()
  File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/forms.py", line 217, in clean
    self.user_cache = authenticate(
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper
    return func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/__init__.py", line 77, in authenticate
    user = backend.authenticate(request, **credentials)
  File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/backends.py", line 46, in authenticate
    user = UserModel._default_manager.get_by_natural_key(username)
  File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/base_user.py", line 46, in get_by_natural_key
    return self.get(**{self.model.USERNAME_FIELD: username})
  File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 646, in get
    num = len(clone)
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 376, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1867, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 87, in __iter__
    results = compiler.execute_sql(
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1396, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 323, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 299, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 281, in ensure_connection
    with self.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 282, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 263, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/django_snowflake/base.py", line 154, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/__init__.py", line 55, in Connect
    return SnowflakeConnection(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 456, in __init__
    self.connect(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 771, in connect
    self.__open_connection()
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1099, in __open_connection
    self.authenticate_with_retry(self.auth_class)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1386, in authenticate_with_retry
    self._authenticate(auth_instance)
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1458, in _authenticate
    raise auth_op from e
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/connection.py", line 1435, in _authenticate
    auth_instance.handle_timeout(
  File "/usr/local/lib/python3.11/site-packages/snowflake/connector/auth/by_plugin.py", line 212, in handle_timeout
    raise error

Exception Type: OperationalError at /admin/login/
Exception Value: Could not connect to Snowflake backend after 2 attempt(s).Aborting

also I noticed that when I am connecting via external network the url/host is account-name.reigon.snowflakecomputing.com
but when I use internal network os.getenv('Host') it becomes : account-name.snowflakecomputing.com,
not sure whether this is causing the issue or not

for additional info:
I am trying to replicate this in django:
link

@timgraham
Copy link
Collaborator

I'm unsure if I can take this any further since I don't have a way to test it. If you have some Python code that works, I can try to blindly adapt it, or else you'll have to debug what I provided and make corrections.

@punitchauhan771
Copy link
Author

Hi @timgraham ,

Does this reference help spcs_python

@punitchauhan771
Copy link
Author

Hi @timgraham ,
I have raised a PR for this, hope this helps

@timgraham timgraham changed the title Making user field optional Add support for connecting to snowpark container services Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants