[BUG] Configuring S3 backend for backups failing

Hello, I guess this is because your options were not parsed as json, but it tries to use the unpacking operator on the string. Maybe you can try using the config file?

Hi, thanks for the reply. Yes it seems like it is not parsing it correctly. I tried doing it in the config file instead, adding:

backup_storage: storages.backends.s3boto3.S3Boto3Storage
backup_options: {access_key: <access key>, secret_key: <secret key>, bucket_name: inventree-backup-backupbucket, default_acl: private}

Then, I added the following to .env:
SETUP_EXTRA_PIP=django-storages[boto3]
Despite this, I got an error message about the Python module storages not being installed. I also tried
SETUP_EXTRA_PIP=django-storages django-storages[boto3]
with the same result.
Error message:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/root/.local/lib/python3.9/site-packages/dbbackup/utils.py", line 120, in wrapper
    func(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/dbbackup/management/commands/dbbackup.py", line 80, in handle
    self.storage = get_storage()
  File "/root/.local/lib/python3.9/site-packages/dbbackup/storage.py", line 33, in get_storage
    return Storage(path, **options)
  File "/root/.local/lib/python3.9/site-packages/dbbackup/storage.py", line 70, in __init__
    self.storageCls = get_storage_class(self._storage_path)
  File "/root/.local/lib/python3.9/site-packages/django/core/files/storage.py", line 373, in get_storage_class
    return import_string(import_path or settings.DEFAULT_FILE_STORAGE)
  File "/root/.local/lib/python3.9/site-packages/django/utils/module_loading.py", line 17, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'storages'

I also tried adding django-storages[boto3]==1.13.2 to plugins.txt, still getting ModuleNotFoundError: No module named 'storages'.

@felixeriksson SETUP_EXTRA_PIP is only working when using the bare metal installer. Where did you read that those can be used with docker?

The plugins are only installed when running the 'invoke update' command or after the server is started up.

The canonical way of adding dependencies to docker images would be to create a custom image that builds upon the original one.

@felixeriksson SETUP_EXTRA_PIP is only working when using the bare metal installer. Where did you read that those can be used with docker?

My bad, I was searching for ways of making Inventree pip install additional libraries, and stumbled across that without realizing it was only for bare metal setups.

The plugins are only installed when running the 'invoke update' command or after the server is started up.

Does that mean that plugins.txt cannot be used to install django-storages[boto3] in a way so that backup with S3 backend would work?

The canonical way of adding dependencies to docker images would be to create a custom image that builds upon the original one.
Would that mean modifying the inventree-worker image to also install django-storages[boto3]? Would you recommend this as an approach to be able to use AWS S3 as backend for the backup system?

I don't have very much experience with Docker yet, so I would have to do some research on how complex that would be, and how it could be integrated in our setup.

To give some context, I'm automating the setup of an EC2 instance (using Cloudformation) that sets up Inventree using Docker, like the instructions specify. The setup of the instance works well, but I also want it to include a solid way of doing backups. My idea is also to make the instance fetch the backup from S3 when being set up, so the instance easily could be taken down and restarted from scratch, while keeping all the data. Maybe an alternative to using django-dbbackup could be to simply backup the database and media files, and exporting the database as JSON, as the Docker setup instructions indicate. But I haven't found any instructions on how to restore such a backup - do you have any hints?

Does that mean that plugins.txt cannot be used to install django-storages[boto3] in a way so that backup with S3 backend would work?

No, because it is not loaded at that moment. The next release will include the option to disable backups outright - that should enable you to disable the function, start-up (thus letting it install) and then activation backups

A bare metal install (Debian/ubuntu with the installer) would probably be more performant on EC2 and enable adding the custom python setup env.

But I haven't found any instructions on how to restore such a backup - do you have any hints?

you can use import-records to import the data and copy the media files to new destination.

Does that mean that plugins.txt cannot be used to install django-storages[boto3] in a way so that backup with S3 backend would work?

No, because it is not loaded at that moment. The next release will include the option to disable backups outright - that should enable you to disable the function, start-up (thus letting it install) and then activation backups

A bare metal install (Debian/ubuntu with the installer) would probably be more performant on EC2 and enable adding the custom python setup env.

But I haven't found any instructions on how to restore such a backup - do you have any hints?

you can use import-records to import the data and copy the media files to new destination.

Thanks a lot! I found the migration instructions that also suggest use of import-records. So in addition to using this tool to import DB data that has previously been exported with export-records, is it only the media folder that needs to be restored? In the directory that is found at the path specified by INVENTREE_EXT_VOLUME, I have the following folders: backup, media, pgdb and static, plus the files config.yaml, plugins.txt and secret_key.txt - would it only be necessary to backup the media folder?

Thanks also for the hint of using a bare metal install. For now, I'll go with Docker since it seems to be working well, and I like the automagic of just using your docker-compose.yml :)

On a side note, in config.yaml (untouched by me) debug: True is specified, despite having the line INVENTREE_DEBUG=False in .env. Is this expected and OK for a production server?

If you only use environment variables to configure your instance you do not need config.yaml or plugins.txt - you will need to either keep the secret_key.txt or set the secret_key with an environment variable. In my scaling deployments, I only use environment variables.

Enviroments variables -> config.yaml -> database -> defaults

Thanks again! Okay, great, I only use environment variables to configure the instance. Then I will just backup the media folder.

Regarding the secret key, I haven't explicitly set that, I've instead let Inventree generate one. Does it need to be backed up to enable migration to a new server instance, or could I just let Inventree generate a new one each time an instance is set up? I haven't completely understood what is encrypted/signed with the secret key.

The key is needed as seed for passwords, API keys etc.

@felixeriksson is this working now? I am not really sure if all problems (especially setting the S3 backend settings) could be resolved

I went the route of creating scripts for backing up and restoring the exported data.json plus the media folder. Since the server I'm running it on is running Amazon Linux on EC2, necessary tools for transfering to/from S3 are already installed by default. So I haven't tried more with django-dbbackup.

Regarding the secret_key.txt, I have not backed it up, but I've tried to tear down my EC2 instance and restart it again, importing the exported database data.json and the media folder. It seems to work fine. Is it necessary to back up the secret key?

@felixeriksson it will be a problem once you use api integrations as those keys are encrypted

@felixeriksson it will be a problem once you use api integrations as those keys are encrypted

Thanks, I see. Can I start backing up the secret key once I would start using the API integrations? Or do I need to keep the original secret key that was generated the first time the server was set up?

The key stays the same, just make sure to note it somewhere secure.

The key stays the same, just make sure to note it somewhere secure.

Hello, thanks for the reply. Perhaps I'm misunderstanding, but when I delete the server, and set it up from scratch again, and reload the backup (media folder plus data.json obtained with export-records), the secret key has changed and is no longer the same. Should the secret_key.txt be restored together with the media folder and data.json? Does it need to be restored before or after the first time inventree-server invoke update is run?

What are the consequences of not backing this up - will all previously generated API tokens be rendered unusable if the server is taken down, setup from scratch and loaded with a backup (i.e. so that the secret key changes)? Can this situation be resolved by requesting new tokens, or could I run into more serious problems?

If you do not have anything that is encrypted in your database it is not a problem. There is the option for plugins to encrypt external (Slack, ServiceNow, ...) API tokens when saving - not having the secret key would be a problem then as they could not be decrypted.
Tokens are for access to the InvenTree API, they can be reissued easily.