django-remote-submission issueshttps://code.ornl.gov/reflectometry/django-remote-submission/-/issues2017-08-16T15:27:19Zhttps://code.ornl.gov/reflectometry/django-remote-submission/-/issues/33No github tags for recent versions2017-08-16T15:27:19ZBorreguero Calvo, JoseNo github tags for recent versions*Created by: djmitche*
I see releases up to 0.11.1 on pypi, but the tags on this repo don't include versions up that high. Tags are helpful in trying to figure out what has changed when upgrading.*Created by: djmitche*
I see releases up to 0.11.1 on pypi, but the tags on this repo don't include versions up that high. Tags are helpful in trying to figure out what has changed when upgrading.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/27Job - Log2017-02-13T20:19:02ZLeal, RicardoJob - LogShall we get Job - Log as `OneToOneField` ?
Shall we get Job - Log as `OneToOneField` ?
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/28First connection2017-02-13T21:18:51ZLeal, RicardoFirst connectionGive last 10 jobs in the socket connection.,Give last 10 jobs in the socket connection.,https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/26Socket loses connection?2017-02-13T21:25:52ZLeal, RicardoSocket loses connection?
Better using this:
`ReconnectingWebSocket`
https://github.com/joewalnes/reconnecting-websocket
Better using this:
`ReconnectingWebSocket`
https://github.com/joewalnes/reconnecting-websocket
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/25Add modified date to socket data2017-02-13T21:59:23ZLeal, RicardoAdd modified date to socket data1. Add `modified` date to the code below.
2. Update the example to when the Job is modified change the table line and not add another one.
3. Sort by `modified` date.
`django_remote_submission/signals.py`
```python
'te...1. Add `modified` date to the code below.
2. Update the example to when the Job is modified change the table line and not add another one.
3. Sort by `modified` date.
`django_remote_submission/signals.py`
```python
'text': json.dumps({
'job_id': instance.id,
'title': instance.title,
'status': instance.status,
}),
```https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/24Add websocket to show live logs2017-02-13T22:55:38ZBorreguero Calvo, JoseAdd websocket to show live logs*Created by: player1537*
* Django Remote Submission version: 0.7.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
Basically the same as #11 but there should be a websocket that shows the lat...*Created by: player1537*
* Django Remote Submission version: 0.7.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
Basically the same as #11 but there should be a websocket that shows the latest logs from the running tasks.
Like in #23, this should add a new `post_save` signal to `signals.py` that listens for each new `Log` object, and then writes to a `Group('job-log-{}'.format(log.job.id))`.
There will also need to be another example page to show how to use this websocket, but it will be about the same as the current one for job statuses.
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/22Add page in example application to show downloadable results2017-02-13T20:29:05ZBorreguero Calvo, JoseAdd page in example application to show downloadable results*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
There should be a page in the example application that, for each job, shows ...*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
There should be a page in the example application that, for each job, shows the results from that job and the links to download them with.
Per [this StackOverflow post](http://stackoverflow.com/questions/11871104/django-generate-download-link), this should be as easy as passing along the `result_set` to the view for the `job-detail` view, and then having an `<a href="{{ result.url }}">Download</a>` in a table or list.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/21Look into way to synchronize setup.py and requirements.txt2017-08-17T13:30:46ZBorreguero Calvo, JoseLook into way to synchronize setup.py and requirements.txt*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
While making changes to fix #11, I realized that it's very easy for the requ...*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: n/a
* Python version: n/a
* Operating System: n/a
### Description
While making changes to fix #11, I realized that it's very easy for the requirements that `setup.py` lists to become out of sync with the ones in `requirements.txt`. I noticed that we've actually been missing Django Rest Framework in `setup.py` for a while.
I know some other projects use some code to parse the requirements file into `install_requires` to keep them in sync. I'm not sure the tradeoffs, though, so I'll need to look into that some.
I'm also not clear what the best method is to have an optional dependency in `setup.py`. For example, the library doesn't require Celery (because it can use some fallback behavior), but it does support Celery.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/16Improve comments in .env.base file2017-01-27T22:58:48ZBorreguero Calvo, JoseImprove comments in .env.base file*Created by: player1537*
### Description
The base environment file should be made more descriptive, so that it includes better descriptions of the purpose of each variable as well as a default or typical value.
Additionally, it ne...*Created by: player1537*
### Description
The base environment file should be made more descriptive, so that it includes better descriptions of the purpose of each variable as well as a default or typical value.
Additionally, it needs to include the `TEST_INTERPRETER_ARGUMENTS` variable and a description of how to use the comma separated parts of the list.
Lastly, it probably shouldn't use `*_INTERPRETER_*` and instead `*_PYTHON_*` since it's only expecting a Python interpreter, which also means the name isn't needed.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/19Tests should automatically create the remote directory if it doesn't exist2017-08-18T15:14:56ZBorreguero Calvo, JoseTests should automatically create the remote directory if it doesn't exist*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: N/A
* Python version: N/A
* Operating System: N/A
### Description
The tests should automatically create the `TEST_REMOTE_DIRECTORY` if it does...*Created by: player1537*
* Django Remote Submission version: 0.6.0
* Django version: N/A
* Python version: N/A
* Operating System: N/A
### Description
The tests should automatically create the `TEST_REMOTE_DIRECTORY` if it doesn't already exist. I think we had code that did this already, but it either got lost somewhere or is part of the code for the users of the library rather than in the library itself.
Additionally, the documentation for `TEST_REMOTE_DIRECTORY` in `.env.base` will need to be updated once this is finished.
### What I Did
```
$ git clone <this repo>
$ cp .env.base .env
$ vi .env
$ cat .env
...
# The remote directory to store scripts in. Ideally this directory will already
# exist, though if it doesn't then the tests should create it.
#
# Also note that if this directory is used by other things, it could make the
# tests fail. For example, if you use /tmp/ as the remote directory and a root
# process writes something to /tmp/ while a test is running, the test will
# likely error because it doesn't have read permissions on that file.
#
# Type: String
# Example(s):
# TEST_REMOTE_DIRECTORY=/tmp/django-remote-submission/
#
TEST_REMOTE_DIRECTORY=/tmp/does-not-exist/
...
$ python3 -m virtualenv venv
$ source venv/bin/activate
(venv)$ python3 -m pip install -r requirements_test.txt
(venv)$ make test
```
The tests will currently fail because `/tmp/does-not-exist/` does not exist and isn't automatically created.https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/14Remote Live logs after '\n' and not on buffer_size2017-08-16T12:44:08ZLeal, RicardoRemote Live logs after '\n' and not on buffer_sizeWe have:
https://github.com/ornl-ndav/django-remote-submission/blob/master/django_remote_submission/remote.py#L149
```python
channel.exec_command(command)
while True:
current_time = now()
...We have:
https://github.com/ornl-ndav/django-remote-submission/blob/master/django_remote_submission/remote.py#L149
```python
channel.exec_command(command)
while True:
current_time = now()
if channel.recv_ready():
output = channel.recv(1024).decode('utf-8')
stdout_handler(current_time, output)
if channel.recv_stderr_ready():
output = channel.recv_stderr(1024).decode('utf-8')
stderr_handler(current_time, output)
```
i.e., logs are flushed after 1024 bytes. Can we flush them after `\n`?
See `paramiko-expect` tail function:
https://github.com/fgimian/paramiko-expect/blob/943630ac499284e6441854a9c4ae1e04301bfdd9/paramiko_expect.py#L213
`buffer = self.channel.recv(1)` and search for `\n`.
Is it going to be too slow?
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/11Web page to show Job list and status2017-02-10T22:56:51ZLeal, RicardoWeb page to show Job list and statusCan we create a HTML snipped based on this:
https://github.com/VincentTide/django-channels-celery-example
The idea is to include the snippet in the any desired webpage and list all jobs.
Edit:
This should list all the jobs and re...Can we create a HTML snipped based on this:
https://github.com/VincentTide/django-channels-celery-example
The idea is to include the snippet in the any desired webpage and list all jobs.
Edit:
This should list all the jobs and respective status, and if clicked should give the Live Log is activated.
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/10Change celery task function return values2017-01-27T21:43:42ZLeal, RicardoChange celery task function return valuesThe model `Task results` in the admin is not very useful...
The `submit_job_to_server` returns a list of Models (`Result`). Models are not serialisable to JSON by default... It fails...
So, I'm using in django settings:
```
CELER...The model `Task results` in the admin is not very useful...
The `submit_job_to_server` returns a list of Models (`Result`). Models are not serialisable to JSON by default... It fails...
So, I'm using in django settings:
```
CELERY_ACCEPT_CONTENT = ['pickle'] # also experimented with "json" alone and ['json','pickle'] as well
CELERY_EVENT_SERIALIZER = 'pickle'
CELERY_TASK_SERIALIZER = 'pickle'
CELERY_RESULT_SERIALIZER = 'pickle'
```
However this renders the `Task results` useless.
Do you have better ideas?
Otherwise I'd suggest in the `submit_job_to_server` to return something else that could be readable... a list of files for example...
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/9Update doc for make release (upload to pypi)2017-01-28T18:51:06ZLeal, RicardoUpdate doc for make release (upload to pypi)Where is the user defined when uploading to pypi `make release`?
Our user is `ornl.ndv`.
We need some small doc about how to release:
- username
- version change
Where is the user defined when uploading to pypi `make release`?
Our user is `ornl.ndv`.
We need some small doc about how to release:
- username
- version change
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/8Result files saved in MEDIA with random name2017-01-20T13:13:29ZLeal, RicardoResult files saved in MEDIA with random nameSo the files FTPed to local are stored in the `settings.MEDIA_ROOT`.
I guess the problem is that everyone has access to `settings.MEDIA_URL` and guess the file names...
Can we add a random string to the file?
E.g.: `job1/results.txt...So the files FTPed to local are stored in the `settings.MEDIA_ROOT`.
I guess the problem is that everyone has access to `settings.MEDIA_URL` and guess the file names...
Can we add a random string to the file?
E.g.: `job1/results.txt` would be `job1/RANDOMSTUFF/results.txt`
Leal, RicardoLeal, Ricardohttps://code.ornl.gov/reflectometry/django-remote-submission/-/issues/7Implememt delete_key in the Wrapper2017-01-27T22:16:26ZLeal, RicardoImplememt delete_key in the WrapperIn the the wrapper, close needs an option to remove the key. Eg.
```python
def close(self, remove_key=False):
"""Close any open connections and clear their attributes."""
if remove_key:
#TODO
...In the the wrapper, close needs an option to remove the key. Eg.
```python
def close(self, remove_key=False):
"""Close any open connections and clear their attributes."""
if remove_key:
#TODO
remove_key_if_it_exists(self.client, self.public_key_filename):
```
https://code.ornl.gov/reflectometry/django-remote-submission/-/issues/1Task List2017-01-18T20:04:41ZLeal, RicardoTask List# TODOs
- [x] 1. Remote file list (given a folder) and transfer (given a file name).
- [x] 2. Results table:
- When a task is executed, it would be nice to have some sort of parameter like this:
```python
store_results = N...# TODOs
- [x] 1. Remote file list (given a folder) and transfer (given a file name).
- [x] 2. Results table:
- When a task is executed, it would be nice to have some sort of parameter like this:
```python
store_results = None # Don't store results
store_results = ['*.txt'] # Store as results all *.txt files
store_results = ['filename1', 'filename2'] # Store as results files ( a result entry per file)
store_results = ["*", "!*.nxs"] # All files except *.nxs
```
- [x] 3. Examples with the password caching using something default in Django. Ideas:
- `SESSION_ENGINE = "django.contrib.sessions.backends.cache"`. It uses memory sessions (rather than the DB) for all sessions. The problem is when the memory runs out the sessions disappear..... The server would eventually purge the cache and make the user put on their passwords again...
https://docs.djangoproject.com/en/1.10/topics/http/sessions/#using-cached-sessions
> Session data will be stored directly in your cache. However, session data may not be persistent: cached data can be evicted if the cache fills up or if the cache server is restarted.
- [x] 4. `Log` model needs a flag to tag the log either as `stdout` or `stderr`.
- [x] 5. Choose the interpreter in `submit_task`. E.g. a parameter like `interpreter='/usr/bin/python3` or `interpreter='/usr/bin/bash`.
- [ ] 6. Minimize dependencies (Let's keep just celery and redis for now).
- [x] 7. `submit_task` needs a `timeout`. We have to see how to kill the process remotely and interrupt the celery task...
- [ ] 8. Ideas (not urgent!):
- Using `django-channels` to have a real update of the tasks. See example: https://github.com/VincentTide/django-channels-celery-example
- Use `django-celery-results` to see in the admin the celery results tables.
Leal, RicardoLeal, Ricardo