Pass http_proxy information to xarray.open_zarr

[loicdtx]

I'm trying to access a remote zarr store via my work network which uses a proxy. It seems that the requests go through aoihttp, which does not pick up proxy information from environment variables by default (HTTP_PROXY and HTTPS_PROXY are defined but ignored). I couldn't find any way to provide proxy information via xarray.open_zarr(). (I tried directly passing proxy='http://...' and trust_env=True hoping they would be picked up by kwargs and passed all the way to either session.request() or aiohttp.ClientSession() respectively but that did not work).
Any suggestions?

[nniraj123]

Hey @loicdtx, I ran into the same problem today. Did you find a solution eventually?

[loicdtx]

@nniraj123 Unfortunately, I haven't found a solution to that issue. It seems like addressing this might require significant changes to xarray or zarr. Given that proxies are commonly used in corporate networks, resolving this could be crucial if zarr is to become more widely adopted for online resources.

[keewis]

if you're using fsspec you should be able to use client_kwargs (replace trust_env with proxy if you want to pass the proxy directly):

fs = fsspec.filesystem("http", client_kwargs={"trust_env": True}, asynchronous=True)
store = zarr.storage.FsspecStore(fs)

ds = xr.open_dataset(store, engine="zarr", ...)

(I can't test this myself, so no guarantee this actually works)

Edit: it looks like I forgot to pass the actual url... not sure how you'd do this with zarr-python>=3.0

[keewis]

actually, it appears there's an even easier option: just pass storage_options={"client_kwargs": {"trust_env": True}} to xr.open_dataset:

xr.open_dataset(url, engine="zarr", storage_options={"client_kwargs": {"trust_env": True}}, ...)

(this works for the http filesystem, other filesystem implementations might have different names, e.g. "session_kwargs" instead of "client_kwargs" for gcfs)

[loicdtx]

I confirm this works; thanks a lot @keewis

[nniraj123]

It's still not working for me but glad that it worked for @loicdtx . Thanks.

[keewis]

did you try both {"client_kwargs": {"trust_env": True}} with the environment variables set and {"client_kwargs": {"proxy": "proxy_url"}}? If that doesn't work it might be time to check if bare aiohttp allows you to use the proxy.

[nniraj123]

Yup, both didn't work, but I still found a solution. To pass proxy kwargs to aiohttp client session, one needs to pass session_kwargs (based on this issue: fsspec/gcsfs#392), So, passing {"trust_env":True} as session_kwargs worked for me.