Setup S3 bucket for project

"How do I ... ?"
1

My app ID is 18446. Can someone help me add S3 to the app?
CC @luke.smith

1 Like
2

Hi @ankit, thanks for writing in! I’ll be happy to configure this for you; stand by for confirmation.

3

@ankit, please deploy the first version of your app in order for me to add the S3.

4

@luke.smith ,

Please do check in and deploy code.

5

@dmitrii.k Deployment done successfully. Please do add S3.

cc @luke.smith

6

@dmitrii.k we also need sendgrid credentials for email confirmation. Thank you

7

Hi @luke.smith!

For SendGrid we’re not providing it as an add-on anymore. Please ask the client if they have an account with any e-mail service already, e.g. SendGrid, MailChimp, Mailgun, SendInBlue, etc. and use that if they do. If they don’t already have an account, ask your PM to set it up for them using their provided e-mail address.

Please stand by for S3 confirmation.

8

@ankit, @luke.smith

The following variables have been added to your app’s production environment:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_STORAGE_BUCKET_NAME
AWS_STORAGE_REGION

They are hidden from your CB Dashboard but available to the app.

Please let me know if there is anything else I can help you with. Happy coding!

9

Thanks a lot, Dmitrii

1 Like
10

You are always welcome!

11

Hi Dmitrii, any ideas on how I can access these environment variables as I need to test file upload to s3 bucket in development setting?
Also, as for gaining access to the s3 account is this possible in order to check file uploads are functioning correctly? Thanks

1 Like
12

@luke.smith, S3 is provided only for production and staging environment. We encourage use of standard file storage in local development. Our django template is tested to work with these variables and comes with S3Boto included, so there shouldn’t be any errors.

If you need to check anything you can always post to support.

13

Hi @dmitrii.k ,
Just tried to deploy the application and I’m getting a key error for the AWS production environment variables, are they definitely set, or am I doing something wrong?
Error & traceback:

Step 9/12 : RUN python3 manage.py collectstatic --no-input
—> Running in 0efe59ca628f
e[91mTraceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/environ/environ.py”, line 273, in get_value
value = self.ENVIRON[var]
File “/usr/lib/python3.6/os.py”, line 669, in getitem
raise KeyError(key) from None
KeyError: ‘AWS_ACCESS_KEY_ID’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “manage.py”, line 21, in
main()
File “manage.py”, line 17, in main
execute_from_command_line(sys.argv)
File “/usr/local/lib/python3.6/dist-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/usr/local/lib/python3.6/dist-packages/django/core/management/init.py”, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/usr/local/lib/python3.6/dist-packages/django/core/management/init.py”, line 211, in fetch_command
settings.INSTALLED_APPS
File “/usr/local/lib/python3.6/dist-packages/django/conf/init.py”, line 79, in getattr
self._setup(name)
File “/usr/local/lib/python3.6/dist-packages/django/conf/init.py”, line 66, in _setup
self._wrapped = Settings(settings_module)
File “/usr/local/lib/python3.6/dist-packages/django/conf/init.py”, line 157, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/opt/webapp/ex_curator_18446/settings.py”, line 63, in
AWS_ACCESS_KEY_ID = env.str(“AWS_ACCESS_KEY_ID”)
File “/usr/local/lib/python3.6/dist-packages/environ/environ.py”, line 134, in str
value = self.get_value(var, default=default)
File “/usr/local/lib/python3.6/dist-packages/environ/environ.py”, line 277, in get_value
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the AWS_ACCESS_KEY_ID environment variable
e[0mThe command ‘/bin/sh -c python3 manage.py collectstatic --no-input’ returned a non-zero code: 1

And the relevant part of settings.py:

AWS_ACCESS_KEY_ID = env.str(“AWS_ACCESS_KEY_ID”)
AWS_SECRET_ACCESS_KEY = env.str(“AWS_SECRET_ACCESS_KEY”)
AWS_STORAGE_BUCKET_NAME = env.str(“AWS_STORAGE_BUCKET_NAME”)
AWS_S3_REGION_NAME = env.str(“AWS_STORAGE_REGION”)

14

@dmitrii.k
(sorry, pinging you again as I forgot to tag you in last post)

15

Hi @luke.smith,

S3 is meant to be used for uploaded media, static files should still use the standard file backend. AWS vars are not available during build, so once you re-configure static files to use the file storage it’ll deploy.

16

Hi Dmitrii, thanks again for the response.
I am already using file storage for static files and have not modified the settings.py from the scaffold:

Static files (CSS, JavaScript, Images)
Managing static files (e.g. images, JavaScript, CSS) | Django documentation | Django
STATIC_URL = ‘/static/’
MIDDLEWARE += [‘whitenoise.middleware.WhiteNoiseMiddleware’]
AUTHENTICATION_BACKENDS = (
‘django.contrib.auth.backends.ModelBackend’,
‘allauth.account.auth_backends.AuthenticationBackend’
)
STATIC_ROOT = os.path.join(BASE_DIR, “staticfiles”)
STATICFILES_DIRS = [
os.path.join(BASE_DIR, ‘static’)
]
STATICFILES_STORAGE = ‘whitenoise.storage.CompressedManifestStaticFilesStorage’
STATICFILES_FINDERS = (
‘django.contrib.staticfiles.finders.FileSystemFinder’,
‘django.contrib.staticfiles.finders.AppDirectoriesFinder’,
)

And for uploaded media I have defined another media root:

MEDIA_ROOT = os.path.join(BASE_DIR, ‘media’)
MEDIA_URL = “/media/”

Which, in development I define as a url in urls.py:

if settings.DEBUG:
urlpatterns += [
url(r’^media/(?P.*)$', serve, {
‘document_root’: settings.MEDIA_ROOT,
}),
]

I’m thinking this may be an issue with the scaffold not having the s3 support dependencies installed as looking in the pipfile, s3boto isn’t included:

[[source]]
verify_ssl = true
url = “Simple index”
name = “pypi”
[dev-packages]
[requires]
python_version = “3.7”
[packages]
django = “~=2.2.3”
django-environ = “~=0.4.5”
psycopg2 = “~=2.8.5”
waitress = “~=1.4.3”
whitenoise = “~=5.1.0”
djangorestframework = “~=3.11.0”
django-bootstrap4 = “~=1.1.1”
django-allauth = “~=0.41.0”
django-rest-auth = “~=0.9.5”
django-extensions = “~=2.2.9”
packaging = “*”
drf-yasg = “~=1.17.1”
fcm-django = “~=0.3.4”
pyyaml = “~=5.3.1”

1 Like
17

Update: I changed from env.str() to os.environ.get() and the build succeeds, but files are hosted locally on the server and not served from s3

18

Hi @luke.smith,

Adding S3 Boto is definitely going to be helpful. However, I caught the real reason for the failed build:

You’re not providing the default value, and since this var is absent inside the build environment it’s raising an error. Apologies for not catching it initiially.

19

Hi,
I changed from env.str to os.environ.get() and build succeeded, should I still be using env.str with a default value instead?

Also, is there a link to an updated django scaffold pipfile? Seeing as this app’s missing s3boto it is also likely that it is also missing other s3 dependencies

20

Hi @dmitrii.k,

I changed from env.str to os.environ.get() and build succeeded, should I still be using env.str with a default value instead?

Also, is there a link to an updated django scaffold pipfile? Seeing as this app’s missing s3boto it is also likely that it is also missing other s3 dependencies

1 Like