Merge pull request #40 from nova-model/1-add-job-status-details (6748ad8a) · Commits · NDIP / Nova Packages / nova-galaxy

CHANGELOG.md

+6 −20

Original line number	Diff line number	Diff line
		### Nova Galaxy 0.10.2
		## Nova Galaxy 0.12.0 (in progress)


		### Nova Galaxy 0.11.0
		- Added detailed job status (thanks to Sergey Yakubov) [Pull request 40](https://github.com/nova-model/nova-galaxy/pull/40)
		- Added ability to mark datasets as remote files, and Nova-Galaxy will attempt to ingress them when running tools (thanks to Gregory Cage). [Merge Request 25](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/25)
		- Added ability to mark datasets as remote files, and Nova-Galaxy will attempt to ingress them when running tools (thanks to Gregory Cage). [Merge Request 25](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/25)
		- Datasets can now be linked to existing datasets when uploaded as tool parameters using force_upload parameter. This saves users from having to upload a dataset multiple times if not necessary (thanks to Gregory Cage). [Merge Request 25](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/25)

		### Nova Galaxy 0.10.1
		- Dependency update (thanks to Sergey Yakubov). [Commit](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/commit/1f532dbbd5c6603c7e358101c0b3830fb2b36f5a)

		### Nova Galaxy 0.10.0
		- Added ToolRunner class to facilitate an event driven running of tools (thanks to Sergey Yakubov). [Merge Request 24](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/24)

		### Nova Galaxy 0.9.1
		- Added `get_full_status` method to tool in order to get detailed messages mostly for error states (thanks to Gregory Cage). [Merge Request 23](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/23)

		### Nova Galaxy 0.9.0
		@@ -18,13 +19,8 @@
		- Made tool status thread safe (thanks to Sergey Yakubov and Gregory Cage). [Merge Request 22](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/22)
		- If canceling or stopping jobs in the uploading data state, will stop the uploading when able (thanks to Sergey Yakubov and Gregory Cage). [Merge Request 22](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/22)
		- Misc backend code cleanup (thanks to Sergey Yakubov and Gregory Cage). [Merge Request 22](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/22)

		### Nova Galaxy, 0.8.2
		- Now returns file type automatically if available (thanks to Gregory Cage). [Merge Request 21](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/21)
		- Returns file content as bytes instead of string (thanks to Gregory Cage). [Merge Request 21](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/21)


		### Nova Galaxy, 0.8.1
		- Can now fetch specific stdout and stderr positions and length (thanks to Gregory Cage). [Merge Request 19](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/19)

		### Nova Galaxy, 0.8.0
		@@ -32,20 +28,12 @@
		- `Connections.connect()` can now be used with or without the `with` keyword. Consequently, stores can also be created outside a `with` block. `Connection.close()` performs the clean up that exiting the `with` block provides (thanks to Gregory Cage). [Merge Request 18](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/18)
		- Data stores can be cleaned up manually (thanks to Gregory Cage). [Merge Request 18](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/18)
		- Can now wait for the result of a running tool (thanks to Gregory Cage). [Merge Request 18](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/18)

		### Nova Galaxy, 0.7.4
		- Allow users to choose to check URL when calling get_url() from a Tool (thanks to Gregory Cage). [Merge Request 17](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/17)
		- Return more detailed information when getting the content of a DatasetCollection (thanks to Gregory Cage). [Merge Request 17](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/17)
		- Data stores are now persisted by default. A new mark_for_cleanup method has been provided to clean up data stores after usage. The persist method's behavior remains unchanged (thanks to Gregory Cage). [Merge Request 17](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/17)

		### Nova Galaxy, 0.7.3
		- Allow Dataset content to be set manually in memory rather than only loading from a file or downloading from Galaxy (thanks to Gregory Cage). [Merge Request 16](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/16)
		- Add file type (extensions) to Datasets (thanks to Gregory Cage). [Merge Request 16](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/16)

		### Nova Galaxy, 0.7.2
		- Add more states to Work State enum (thanks to Gregory Cage). [Merge Request 15](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/15)

		### Nova Galaxy, 0.7.1
		- Speeds ups recovering tools from data stores. (thanks to Gregory Cage). [Merge Request 14](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/14)

		### Nova Galaxy, 0.7.0
		@@ -53,8 +41,6 @@
		- Added a lot more user documentation (thanks to Gregory Cage). [Merge Request 13](https://code.ornl.gov/ndip/public-packages/nova-galaxy/-/merge_requests/13)
		- Changed the Workstate enum to have string values (much more useful when trying to serialize the value) (thanks to Gregory Cage). [Merge Request 13](https://code.ornl.gov/ndip/public-packages/ndip-galaxy/-/merge_requests/13)
		- Changes Nova class name to Connection, and NovaConnection to ConnectionHelper (thanks to Gregory Cage). [Merge Request 13](https://code.ornl.gov/ndip/public-packages/ndip-galaxy/-/merge_requests/13)

		### Nova Galaxy, 0.6.1
		- Fix dictionary bug with data stores (thanks to Gregory Cage). [Merge Request 12](https://code.ornl.gov/ndip/public-packages/ndip-galaxy/-/merge_requests/12)

		### Nova Galaxy, 0.6.0

pyproject.toml

+1 −1

Original line number	Diff line number	Diff line
		[tool.poetry]
		name = "nova-galaxy"
		version = "0.10.2"
		version = "0.11.0"
		description = "Utilties for accessing the ORNL Galaxy instance"
		authors = ["Greg Watson <watsongr@ornl.gov>", "Gregory Cage <cagege@ornl.gov>", "Sergey Yakubov <yakubovs@ornl.gov>"]
		readme = "README.md"

src/nova/galaxy/job.py

+40 −8

Original line number	Diff line number	Diff line
		"""Internal job related classes and functions."""

		import time
		from datetime import datetime, timezone
		from threading import Lock, Thread
		from typing import TYPE_CHECKING, Dict, Optional
		from typing import TYPE_CHECKING, Any, Dict, Optional

		from bioblend import galaxy
		from bioblend.galaxy.datasets import DatasetClient
		@@ -18,12 +19,42 @@ from .parameters import Parameters
		REGISTER_NEUTRON_DATA_TOOL = "neutrons_register"


		def get_job_details(state: WorkState, job: Dict[str, Any]) -> Dict[str, Any]:
		if "job_messages" not in job or not job["job_messages"]:
		return {}
		for message in job["job_messages"]:
		if "status_details" in message:
		match message["status_details"]["status_details_source"]:
		case "slurm":
		return get_slurm_status(state, message["status_details"])
		return {}


		def to_local_time(input: str) -> str:
		utc_time = datetime.fromisoformat(input).replace(tzinfo=timezone.utc)
		local_time = utc_time.astimezone()
		return local_time.strftime("%Y-%m-%d %H:%M:%S")


		def get_slurm_status(state: WorkState, status: Dict[str, Any]) -> Dict[str, Any]:
		if state == WorkState.RUNNING:
		short_details = (
		f"Slurm Job Details: Id:{status['JobId']}, Job State:{status['JobState']}, Run Time:{status['RunTime']}"
		)
		else:
		short_details = (
		f"Slurm Job Details: Id:{status['JobId']}, Job State:{status['JobState']},"
		f" Last Evaluation:{to_local_time(status['LastSchedEval'])}"
		)
		return {"message": short_details, "original_dict": status}


		class JobStatus:
		"""Internal structure to hold job status info."""

		def __init__(self) -> None:
		self.lock = Lock()
		self._details = ""
		self._details: Dict[str, Any] = {}
		self._state = WorkState.NOT_STARTED

		@property
		@@ -37,12 +68,12 @@ class JobStatus:
		self._state = value

		@property
		def details(self) -> str:
		def details(self) -> Dict[str, Any]:
		with self.lock:
		return self._details

		@details.setter
		def details(self, value: str) -> None:
		def details(self, value: Dict[str, Any]) -> None:
		with self.lock:
		self._details = value

		@@ -72,7 +103,7 @@ class Job:
		self.status.state = WorkState.CANCELED
		return
		self.status.state = WorkState.ERROR
		self.status.details = str(e)
		self.status.details = {"message": str(e)}
		return

		self.status.state = WorkState.FINISHED
		@@ -225,7 +256,7 @@ class Job:
		if response:
		return True
		else:
		self.status.details = "could not stop job"
		self.status.details = {"message": "could not stop job"}
		return False

		def cancel(self) -> bool:
		@@ -247,15 +278,16 @@ class Job:

		def get_state(self) -> JobStatus:
		"""Returns current state of job."""
		if self.status.state == WorkState.QUEUED:
		if self.status.state == WorkState.QUEUED or self.status.state == WorkState.RUNNING:
		try:
		job = self.galaxy_instance.jobs.show_job(self.id)
		job = self.galaxy_instance.jobs.show_job(self.id, full_details=True)
		if job["state"] == "running":
		self.status.state = WorkState.RUNNING
		elif job["state"] == "error":
		self.status.state = WorkState.ERROR
		elif job["state"] == "deleted":
		self.status.state = WorkState.DELETED
		self.status.details = get_job_details(self.status.state, job)
		except Exception:
		pass
		return self.status

src/nova/galaxy/tool_runner.py

+6 −5

Original line number	Diff line number	Diff line
		@@ -120,14 +120,14 @@ class ToolRunner:

		if tool_state == WorkState.ERROR:
		if tool_status.details:
		self.current_outputs.stderr = f"{tool_status.details}\n{self.current_outputs.stderr}"
		self.current_outputs.stderr = f"{tool_status.details['message']}\n{self.current_outputs.stderr}"

		async def _monitor_run(self) -> None:
		while True:
		try:
		if self.nova_tool or self.error:
		status = self._get_job_status()
		if self.current_status.state != status.state:
		if self.current_status.state != status.state or self.current_status.details != status.details:
		self.current_status = status
		await self._send_status_change_signal()
		if job_stopped(self.current_status.state):
		@@ -139,7 +139,8 @@ class ToolRunner:

		async def _send_status_change_signal(self) -> None:
		if self.current_status.state == WorkState.ERROR:
		await self.error_message_signal.send_async(self.sender_id, error_message=self.current_status.details)
		error_message = self.current_status.details.get("message", "")
		await self.error_message_signal.send_async(self.sender_id, error_message=error_message)
		await self.progress_signal.send_async(
		self.sender_id, state=self.current_status.state, details=self.current_status.details
		)
		@@ -148,11 +149,11 @@ class ToolRunner:
		if self.nova_tool:
		status = copy(self.nova_tool.get_full_status())
		if status.state == WorkState.ERROR:
		status.details = "Error running NDIP tool. Please see tool outputs for more information."
		status.details = {"message": "Error running NDIP tool. Please see tool outputs for more information."}
		else:
		status = JobStatus()
		status.state = WorkState.ERROR
		status.details = self.error
		status.details = {"message": self.error}
		return status

		def _run_in_background(self) -> None: