django-remote-submission issueshttps://code.ornl.gov/reflectometry/django-remote-submission/-/issues2017-08-16T01:10:34Zhttps://code.ornl.gov/reflectometry/django-remote-submission/-/issues/39Tests fail2017-08-16T01:10:34ZBorreguero Calvo, JoseTests fail*Created by: djmitche*
* Django Remote Submission version: master
* Django version: latest
* Python version: 3.5
* Operating System: Ubuntu 16.04
### Description
Running the tests fails. I'm sure something in my SSH setup is m...*Created by: djmitche*
* Django Remote Submission version: master
* Django version: latest
* Python version: 3.5
* Operating System: Ubuntu 16.04
### Description
Running the tests fails. I'm sure something in my SSH setup is missing, so this is probably going to come down to a doc update as to how to set up what i didn't set up!
### What I Did
I started a standard Ubuntu openssh-server install, with a user 'worker' and a password. No ssh keys are set up (this is in a clean docker container). My `.env` is:
```
TEST_PYTHON_ARGUMENTS=-u
TEST_PYTHON_PATH=/usr/bin/python3
TEST_REMOTE_DIRECTORY=/home/worker
TEST_REMOTE_FILENAME=foo.py
TEST_REMOTE_PASSWORD=foo#123
TEST_REMOTE_USER=worker
TEST_SERVER_HOSTNAME=127.0.0.1
TEST_SERVER_PORT=22
```
Result summary:
```
tests/test_models.py::test_server_string_representation PASSED
tests/test_models.py::test_job_string_representation PASSED
tests/test_models.py::test_log_string_representation PASSED
tests/test_models.py::test_result_string_representation PASSED
tests/test_tasks.py::test_submit_job_normal_usage[True] PASSED
tests/test_tasks.py::test_submit_job_normal_usage[False] PASSED
tests/test_tasks.py::test_submit_job_multiple_streams[True] PASSED
tests/test_tasks.py::test_submit_job_multiple_streams[False] PASSED
tests/test_tasks.py::test_submit_job_failure[True] PASSED
tests/test_tasks.py::test_submit_job_failure[False] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_total[True] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_total[False] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_none[True] PASSED
tests/test_tasks.py::test_submit_job_log_policy_log_none[False] PASSED
tests/test_tasks.py::test_submit_job_timeout[True] PASSED
tests/test_tasks.py::test_submit_job_timeout[False] PASSED
tests/test_tasks.py::test_submit_job_modified_files[True] PASSED
tests/test_tasks.py::test_submit_job_modified_files[False] FAILED
tests/test_tasks.py::test_submit_job_modified_files_positive_pattern[True] PASSED
tests/test_tasks.py::test_submit_job_modified_files_positive_pattern[False] FAILED
tests/test_tasks.py::test_submit_job_modified_files_negative_pattern[True] PASSED
tests/test_tasks.py::test_submit_job_modified_files_negative_pattern[False] FAILED
tests/test_tasks.py::test_submit_job_deploy_key[True] FAILED
tests/test_tasks.py::test_submit_job_deploy_key[False] FAILED
tests/test_tasks.py::test_delete_key_old_way FAILED
tests/test_tasks.py::test_deploy_and_delete_key FAILED
```
And the first exception:
```
========================================================================================================================== FAILURES ===========================================================================================================================
____________________________________________________________________________________________________________ test_submit_job_modified_files[False] ____________________________________________________________________________________________________________
env = Env(server_hostname='127.0.0.1', server_port=22, remote_directory='/home/worker', remote_filename='foo.py', remote_user='worker', remote_password='******', python_path='/usr/bin/python3', python_arguments=['-u']), job = <Job: 1-job-title>
wrapper_cls = <class 'django_remote_submission.remote.RemoteWrapper'>
@pytest.mark.django_db
@pytest.mark.job_program('''\
from __future__ import print_function
import time
import sys
for i in range(5):
with open('{}.txt'.format(i), 'w') as f:
print('line: {}'.format(i), file=f)
time.sleep(0.1)
''')
def test_submit_job_modified_files(env, job, wrapper_cls):
from django_remote_submission.models import Job, Log, Result
from django_remote_submission.tasks import submit_job_to_server, LogPolicy
import re
results = submit_job_to_server(job.pk, env.remote_password,
wrapper_cls=wrapper_cls)
> assert len(results) == 5
E assert 0 == 5
E + where 0 = len({})
tests/test_tasks.py:423: AssertionError
-------------------------------------------------------------------------------------------------------------------- Captured stdout setup --------------------------------------------------------------------------------------------------------------------
DEBUG:environ.environ:Read environment variables from: /django-remote-submission/.env
DEBUG:environ.environ:get 'TEST_SERVER_HOSTNAME' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_SERVER_PORT' casted as '<class 'int'>' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_DIRECTORY' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_FILENAME' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_USER' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_REMOTE_PASSWORD' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_PYTHON_PATH' casted as 'None' with default '<NoValue>'
DEBUG:environ.environ:get 'TEST_PYTHON_ARGUMENTS' casted as '<class 'list'>' with default '<NoValue>'
-------------------------------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------------------------------
INFO:django_remote_submission.remote:Connecting user worker to 127.0.0.1 with password.
DEBUG:paramiko.transport:starting thread (client mode): 0x94d5a0f0
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.2.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_7.2p2)
DEBUG:paramiko.transport:kex algos:['curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group14-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] server encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] client mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: ecdh-sha2-nistp256
DEBUG:paramiko.transport:HostKey agreed: ssh-ed25519
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha2-256
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-ed25519 host key for 127.0.0.1: b'c629794f404e0cb071153c17cd672cc4'
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (password) successful!
DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
DEBUG:paramiko.transport:Received global request "hostkeys-00@openssh.com"
DEBUG:paramiko.transport:Rejecting "hostkeys-00@openssh.com" global request from server.
DEBUG:paramiko.transport:[chan 0] Max packet out: 32768 bytes
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
INFO:paramiko.transport.sftp:[chan 0] Opened sftp connection (server version 3)
DEBUG:paramiko.transport.sftp:[chan 0] stat(b'/home/worker')
DEBUG:paramiko.transport.sftp:[chan 0] normalize(b'/home/worker')
DEBUG:paramiko.transport.sftp:[chan 0] open(b'/home/worker/foo.py', 'wt')
DEBUG:paramiko.transport.sftp:[chan 0] open(b'/home/worker/foo.py', 'wt') -> b'00000000'
DEBUG:paramiko.transport.sftp:[chan 0] close(b'00000000')
exec_command(command='cd /home/worker && /usr/bin/python3 -u foo.py')
DEBUG:paramiko.transport:[chan 1] Max packet in: 32768 bytes
DEBUG:paramiko.transport:[chan 1] Max packet out: 32768 bytes
DEBUG:paramiko.transport:Secsh channel 1 opened.
DEBUG:paramiko.transport:[chan 1] Sesch channel 1 request ok
DEBUG:paramiko.transport:[chan 1] EOF received (1)
DEBUG:paramiko.transport.sftp:[chan 0] listdir(b'/home/worker/.')
DEBUG:paramiko.transport:[chan 1] EOF sent (1)
<SFTPAttributes: [ size=34 uid=1000 gid=1000 mode=0o40700 atime=1502843554 mtime=1502843554 ]>
<SFTPAttributes: [ size=189 uid=1000 gid=1000 mode=0o100664 atime=1502845059 mtime=1502845058 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844437 mtime=1502844433 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844437 mtime=1502844434 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844437 mtime=1502844435 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844438 mtime=1502844436 ]>
<SFTPAttributes: [ size=7 uid=1000 gid=1000 mode=0o100664 atime=1502844438 mtime=1502844437 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
<SFTPAttributes: [ size=8 uid=0 gid=0 mode=0o100644 atime=1502845057 mtime=1502845057 ]>
INFO:paramiko.transport.sftp:[chan 0] sftp session closed.
DEBUG:paramiko.transport:[chan 0] EOF sent (0)
-------------------------------------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------------------------------------
INFO :: Connecting user worker to 127.0.0.1 with password.
```https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/33No github tags for recent versions2017-08-16T15:27:19ZBorreguero Calvo, JoseNo github tags for recent versions*Created by: djmitche*
I see releases up to 0.11.1 on pypi, but the tags on this repo don't include versions up that high. Tags are helpful in trying to figure out what has changed when upgrading.*Created by: djmitche*
I see releases up to 0.11.1 on pypi, but the tags on this repo don't include versions up that high. Tags are helpful in trying to figure out what has changed when upgrading.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/28First connection2017-02-13T21:18:51ZLeal, RicardoFirst connectionGive last 10 jobs in the socket connection.,Give last 10 jobs in the socket connection.,https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/27Job - Log2017-02-13T20:19:02ZLeal, RicardoJob - LogShall we get Job - Log as `OneToOneField` ?
Shall we get Job - Log as `OneToOneField` ?
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/26Socket loses connection?2017-02-13T21:25:52ZLeal, RicardoSocket loses connection?
Better using this:
`ReconnectingWebSocket`
https://github.com/joewalnes/reconnecting-websocket
Better using this:
`ReconnectingWebSocket`
https://github.com/joewalnes/reconnecting-websocket
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/25Add modified date to socket data2017-02-13T21:59:23ZLeal, RicardoAdd modified date to socket data1. Add `modified` date to the code below.
2. Update the example to when the Job is modified change the table line and not add another one.
3. Sort by `modified` date.
`django_remote_submission/signals.py`
```python
'te...1. Add `modified` date to the code below.
2. Update the example to when the Job is modified change the table line and not add another one.
3. Sort by `modified` date.
`django_remote_submission/signals.py`
```python
'text': json.dumps({
'job_id': instance.id,
'title': instance.title,
'status': instance.status,
}),
```https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/24Add websocket to show live logs2017-02-13T22:55:38ZBorreguero Calvo, JoseAdd websocket to show live logs*Created by: player1537*
* Django Remote Submission version: 0.7.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
Basically the same as #11 but there should be a websocket that shows the lat...*Created by: player1537*
* Django Remote Submission version: 0.7.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
Basically the same as #11 but there should be a websocket that shows the latest logs from the running tasks.
Like in #23, this should add a new `post_save` signal to `signals.py` that listens for each new `Log` object, and then writes to a `Group('job-log-{}'.format(log.job.id))`.
There will also need to be another example page to show how to use this websocket, but it will be about the same as the current one for job statuses.
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/22Add page in example application to show downloadable results2017-02-13T20:29:05ZBorreguero Calvo, JoseAdd page in example application to show downloadable results*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
There should be a page in the example application that, for each job, shows ...*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
There should be a page in the example application that, for each job, shows the results from that job and the links to download them with.
Per [this StackOverflow post](http://stackoverflow.com/questions/11871104/django-generate-download-link), this should be as easy as passing along the `result_set` to the view for the `job-detail` view, and then having an `<a href="{{ result.url }}">Download</a>` in a table or list.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/21Look into way to synchronize setup.py and requirements.txt2017-08-17T13:30:46ZBorreguero Calvo, JoseLook into way to synchronize setup.py and requirements.txt*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
While making changes to fix #11, I realized that it's very easy for the requ...*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
While making changes to fix #11, I realized that it's very easy for the requirements that `setup.py` lists to become out of sync with the ones in `requirements.txt`. I noticed that we've actually been missing Django Rest Framework in `setup.py` for a while.
I know some other projects use some code to parse the requirements file into `install_requires` to keep them in sync. I'm not sure the tradeoffs, though, so I'll need to look into that some.
I'm also not clear what the best method is to have an optional dependency in `setup.py`. For example, the library doesn't require Celery (because it can use some fallback behavior), but it does support Celery.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/19Tests should automatically create the remote directory if it doesn't exist2017-08-18T15:14:56ZBorreguero Calvo, JoseTests should automatically create the remote directory if it doesn't exist*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: N/A
* Python version: N/A
* Operating System: N/A
### Description
The tests should automatically create the `TEST_REMOTE_DIRECTORY` if it does...*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: N/A
* Python version: N/A
* Operating System: N/A
### Description
The tests should automatically create the `TEST_REMOTE_DIRECTORY` if it doesn't already exist. I think we had code that did this already, but it either got lost somewhere or is part of the code for the users of the library rather than in the library itself.
Additionally, the documentation for `TEST_REMOTE_DIRECTORY` in `.env.base` will need to be updated once this is finished.
### What I Did
```
$ git clone <this repo>
$ cp .env.base .env
$ vi .env
$ cat .env
...
# The remote directory to store scripts in. Ideally this directory will already
# exist, though if it doesn't then the tests should create it.
#
# Also note that if this directory is used by other things, it could make the
# tests fail. For example, if you use /tmp/ as the remote directory and a root
# process writes something to /tmp/ while a test is running, the test will
# likely error because it doesn't have read permissions on that file.
#
# Type: String
# Example(s):
# TEST_REMOTE_DIRECTORY=/tmp/django-remote-submission/
#
TEST_REMOTE_DIRECTORY=/tmp/does-not-exist/
...
$ python3 -m virtualenv venv
$ source venv/bin/activate
(venv)$ python3 -m pip install -r requirements_test.txt
(venv)$ make test
```
The tests will currently fail because `/tmp/does-not-exist/` does not exist and isn't automatically created.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/16Improve comments in .env.base file2017-01-27T22:58:48ZBorreguero Calvo, JoseImprove comments in .env.base file*Created by: player1537*
### Description
The base environment file should be made more descriptive, so that it includes better descriptions of the purpose of each variable as well as a default or typical value.
Additionally, it ne...*Created by: player1537*
### Description
The base environment file should be made more descriptive, so that it includes better descriptions of the purpose of each variable as well as a default or typical value.
Additionally, it needs to include the `TEST_INTERPRETER_ARGUMENTS` variable and a description of how to use the comma separated parts of the list.
Lastly, it probably shouldn't use `*_INTERPRETER_*` and instead `*_PYTHON_*` since it's only expecting a Python interpreter, which also means the name isn't needed.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/15DOI for this repo2017-01-26T15:17:16ZLeal, RicardoDOI for this repoHere the info:
https://guides.github.com/activities/citable-code/
LOW PRIORITY
Here the info:
https://guides.github.com/activities/citable-code/
LOW PRIORITY
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/14Remote Live logs after '\n' and not on buffer_size2017-08-16T12:44:08ZLeal, RicardoRemote Live logs after '\n' and not on buffer_sizeWe have:
https://github.com/ornl-ndav/django-remote-submission/blob/master/django_remote_submission/remote.py#L149
```python
channel.exec_command(command)
while True:
current_time = now()
...We have:
https://github.com/ornl-ndav/django-remote-submission/blob/master/django_remote_submission/remote.py#L149
```python
channel.exec_command(command)
while True:
current_time = now()
if channel.recv_ready():
output = channel.recv(1024).decode('utf-8')
stdout_handler(current_time, output)
if channel.recv_stderr_ready():
output = channel.recv_stderr(1024).decode('utf-8')
stderr_handler(current_time, output)
```
i.e., logs are flushed after 1024 bytes. Can we flush them after `\n`?
See `paramiko-expect` tail function:
https://github.com/fgimian/paramiko-expect/blob/943630ac499284e6441854a9c4ae1e04301bfdd9/paramiko_expect.py#L213
`buffer = self.channel.recv(1)` and search for `\n`.
Is it going to be too slow?
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/11Web page to show Job list and status2017-02-10T22:56:51ZLeal, RicardoWeb page to show Job list and statusCan we create a HTML snipped based on this:
https://github.com/VincentTide/django-channels-celery-example
The idea is to include the snippet in the any desired webpage and list all jobs.
Edit:
This should list all the jobs and re...Can we create a HTML snipped based on this:
https://github.com/VincentTide/django-channels-celery-example
The idea is to include the snippet in the any desired webpage and list all jobs.
Edit:
This should list all the jobs and respective status, and if clicked should give the Live Log is activated.
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/10Change celery task function return values2017-01-27T21:43:42ZLeal, RicardoChange celery task function return valuesThe model `Task results` in the admin is not very useful...
The `submit_job_to_server` returns a list of Models (`Result`). Models are not serialisable to JSON by default... It fails...
So, I'm using in django settings:
```
CELER...The model `Task results` in the admin is not very useful...
The `submit_job_to_server` returns a list of Models (`Result`). Models are not serialisable to JSON by default... It fails...
So, I'm using in django settings:
```
CELERY_ACCEPT_CONTENT = ['pickle'] # also experimented with "json" alone and ['json','pickle'] as well
CELERY_EVENT_SERIALIZER = 'pickle'
CELERY_TASK_SERIALIZER = 'pickle'
CELERY_RESULT_SERIALIZER = 'pickle'
```
However this renders the `Task results` useless.
Do you have better ideas?
Otherwise I'd suggest in the `submit_job_to_server` to return something else that could be readable... a list of files for example...
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/9Update doc for make release (upload to pypi)2017-01-28T18:51:06ZLeal, RicardoUpdate doc for make release (upload to pypi)Where is the user defined when uploading to pypi `make release`?
Our user is `ornl.ndv`.
We need some small doc about how to release:
- username
- version change
Where is the user defined when uploading to pypi `make release`?
Our user is `ornl.ndv`.
We need some small doc about how to release:
- username
- version change
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/8Result files saved in MEDIA with random name2017-01-20T13:13:29ZLeal, RicardoResult files saved in MEDIA with random nameSo the files FTPed to local are stored in the `settings.MEDIA_ROOT`.
I guess the problem is that everyone has access to `settings.MEDIA_URL` and guess the file names...
Can we add a random string to the file?
E.g.: `job1/results.txt...So the files FTPed to local are stored in the `settings.MEDIA_ROOT`.
I guess the problem is that everyone has access to `settings.MEDIA_URL` and guess the file names...
Can we add a random string to the file?
E.g.: `job1/results.txt` would be `job1/RANDOMSTUFF/results.txt`
Leal, RicardoLeal, Ricardohttps://code.ornl.gov/reflectometry/django-remote-submission/-/issues/7Implememt delete_key in the Wrapper2017-01-27T22:16:26ZLeal, RicardoImplememt delete_key in the WrapperIn the the wrapper, close needs an option to remove the key. Eg.
```python
def close(self, remove_key=False):
"""Close any open connections and clear their attributes."""
if remove_key:
#TODO
...In the the wrapper, close needs an option to remove the key. Eg.
```python
def close(self, remove_key=False):
"""Close any open connections and clear their attributes."""
if remove_key:
#TODO
remove_key_if_it_exists(self.client, self.public_key_filename):
```
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/1Task List2017-01-18T20:04:41ZLeal, RicardoTask List# TODOs
- [x] 1. Remote file list (given a folder) and transfer (given a file name).
- [x] 2. Results table:
- When a task is executed, it would be nice to have some sort of parameter like this:
```python
store_results = N...# TODOs
- [x] 1. Remote file list (given a folder) and transfer (given a file name).
- [x] 2. Results table:
- When a task is executed, it would be nice to have some sort of parameter like this:
```python
store_results = None # Don't store results
store_results = ['*.txt'] # Store as results all *.txt files
store_results = ['filename1', 'filename2'] # Store as results files ( a result entry per file)
store_results = ["*", "!*.nxs"] # All files except *.nxs
```
- [x] 3. Examples with the password caching using something default in Django. Ideas:
- `SESSION_ENGINE = "django.contrib.sessions.backends.cache"`. It uses memory sessions (rather than the DB) for all sessions. The problem is when the memory runs out the sessions disappear..... The server would eventually purge the cache and make the user put on their passwords again...
https://docs.djangoproject.com/en/1.10/topics/http/sessions/#using-cached-sessions
> Session data will be stored directly in your cache. However, session data may not be persistent: cached data can be evicted if the cache fills up or if the cache server is restarted.
- [x] 4. `Log` model needs a flag to tag the log either as `stdout` or `stderr`.
- [x] 5. Choose the interpreter in `submit_task`. E.g. a parameter like `interpreter='/usr/bin/python3` or `interpreter='/usr/bin/bash`.
- [ ] 6. Minimize dependencies (Let's keep just celery and redis for now).
- [x] 7. `submit_task` needs a `timeout`. We have to see how to kill the process remotely and interrupt the celery task...
- [ ] 8. Ideas (not urgent!):
- Using `django-channels` to have a real update of the tasks. See example: https://github.com/VincentTide/django-channels-celery-example
- Use `django-celery-results` to see in the admin the celery results tables.
Leal, RicardoLeal, Ricardo