Commit a331b49d authored by Cage, Marshall Andrew's avatar Cage, Marshall Andrew
Browse files

Fix first run bugs

After testing on a different machine, a few new bugs reared their
heads. These have been squashed for now.

- Reorganise project
- Allow email fetch without filters (only if mailbox is explicit)
- Handle incorrect login credential crash
- Update document dump description
- Clean up certain imports
- Fix empty contents section bug when opening a profile
- Add static method to emit profileChanged signal
- migrate to new repo (but I didn't want to keep revision history,
    hence the copy/paste)

Date:   Tue Jul 26 11:48:55 2016 -0400

Fix divide by zero bug in

- Fix wrong email body being searched in

Date:   Tue Jul 26 10:53:27 2016 -0400

Update documentation

In preparation for impending release, the documentation has been
updated to describe installation procedure. It also explains how to
create new packages and plugins.

- Add PreferenceManager for persistant module and plugin level prefs
- Add prefernce save/load
- Fix package search path to look in the executable's directory, not
 the cwd
- Add preferences for default package (email stuff)

Date:   Thu Jul 14 12:24:32 2016 -0400

Add profile save status

I beefed up saving profiles. I removed 'Report' and made the
Profile the main data container. Open, Save and New all now open/
save/create Profiles. Shortcut keys work as expected (^+s, ^+o,
^+n, ^+shift+s.) The title of the window now reflects the Profile
name (instead of the Report title) and the save state. I added a
new signal to the ARCTool class. When the first instance is
created, the Class's signal is replaced with the instance's signal.
Whenever the save state of the profile is changed, the main ARCTool
gets notified via this signal. Modules with QLineEdits, QTextEdits,
QCheckBoxes, QSpinBoxes, QDateTimeEdits and some others will
automatically notify the tool that the profile has changed. But,
for custom widgets or certain signals, the signal will need to be
mannually emitted.

- Add 'Save Profile As...' action
- Remove 'Open' and 'Save' which were for Reports
- Fix tableChanged signal in emailfilter to reflect individual
  filter changes
- Add Document Module to insert documents via pandoc.
- Add docx exporting
- Swap odf (QT) export with odt (pandoc)
- Fix Section heading style on export (add <h1/> tags)

Date:   Tue Jul 12 16:54:47 2016 -0400

Fix certain formattings

Date:   Tue Jul 12 16:15:58 2016 -0400

Add PreferenceManager

PreferenceManager adds additional persistance for more concrete
settings that might span modules. Preferences are stored in
wherever QT's AppData folder is located. Preferences can be set
for packages as a whole or individual modules in the preferenceDict
of either.

- Format code to PEP 8 (mostly)
- Fix OR in email filters
- Add highlight non existant filter labels
- Add mailbox select
- Add IMAP server preference
- Add IMAP port preference
- Add SSL preference
- Add statusBar messages
- Add IMAP error handling
- Add strip email replies
- Add strip email headers from inline
- Add strip email's duplicate text
- Add strip email's extra whitespace
- Add strip email formatting
- Add email delimiter
- Fix minor bugs all over the place

Date:   Fri Jul 1 17:06:40 2016 -0400

Tweak NLP to better recognise lines and paragraphs

- Tweak email representations (images)
- Add other export options

Date:   Thu Jun 30 16:36:16 2016 -0400

Finish first pass of NLP

Basic email scraping is there but it's somewhat lacking. Output
formatting works fine, but text detection could use work. Next I
should add wc matching (making sure we have as many words in the
selection as the samples.) When there is uncertainty in the match
the output is marked in red (in the document) and tagged with the
(un?)certainty percentage.

- Add context checking to emailsrapebasic Module
- Add contextChanged signal to ARCTool
- Fix email rendering in emailscrapebasic (images and such)
- Disable group/selectionBox when appropriatae
- Tweak emailsrapebasic.ui
- Tweak Date Context

Date:   Wed Jun 29 16:59:25 2016 -0400

Add Glob and GlobItem for NLP

If anyone sees this, keep in mind I'm self teaching NLP thru
intuition and I'm not a statistician, so...

Date:   Tue Jun 28 16:59:27 2016 -0400

Add Group assignment

Add Group selection
Begin NLP in email filters
Fix filter label bug (\w+)

Date:   Mon Jun 27 16:54:13 2016 -0400

Connect filters to IMAP Fetch

Add basic email loading in 'mailBrowser' widget. HTML/plain text
Add LoginDialog for taking usernames/passwords

Date:   Fri Jun 24 17:41:34 2016 -0400

Add more regex

Date:   Thu Jun 23 16:51:41 2016 -0400

Add EmailFilterTable

Began the EmailFilterTable class which allows the user to create
and label IMAPv4 filters for search. Much of the logic checking is
implemented in the validator for the logic LineEdit.

Add emailscrapebasic module
Add basic documentation structure (sphinx)
Adjust margins in TextEdit (

Date:   Fri Jun 17 16:47:33 2016 -0400

Tweak modules, reorganise files

Date:   Fri Jun 17 14:17:19 2016 -0400

Add Text module in Default package

The Text module is just a simple rich text editor. It tracks its
state through an html string stored as an extra. It also makes use
of a seperate python file to implement its ui. This proves that
modules should be able to consist of multiple files.

-Add extra icons (need to be renamed)
-Add entry recall for the List module
-Add font recall for the Cover Page module
-Add 'extras' for modules (can store non UI state data)
-Add template Module in packages/Default/

Date:   Thu Jun 16 16:54:10 2016 -0400

Add more persistence

Added persistence for contexts and whether or not titles are drawn.

-Fix load settings from profile (mostly)
-Fix minor save profile errors
-Change to QTextDocument to preserve formatting
-Add export to pdf
-Remove export to docx
-Add ORNL package with cover page module
-Add list module to default package

Date:   Tue Jun 14 14:01:26 2016 -0400

Setup proper value packing for module persistence

Date:   Mon Jun 13 11:45:32 2016 -0400

Add basic Document export to to .docx

A Document can be exported to a .docx. The content of the document
is retrieved from the Content objects in a given Section. To update
the Contents in a Section, the Section's updateConent calls its
Module's storeOptions and then its Module's generate. It's on the
module developer to make sure his module's storeOptions or generate
performs the steps necessary to produce the desired output.
storeOptions is also (will be, rather) called when the Profile is
saved (so that the module state can be restored later.) Therefore,
storeOptions should only do things like grabbing information from
ui elements (textboxes, etc.) and calculations or data fetch should
be left in generate.

- Add Document class to hold data before export
- Add .docx export
- Add Content classes to abstract document content
- Fix moduleSelectDialog title
- Add generateReport and exportReport to MainWindow

Date:   Fri Jun 10 17:00:11 2016 -0400

Fix open profile to load module (but not options)

Date:   Fri Jun 10 14:55:29 2016 -0400

Change Packages to use python module utilities

In order to more easily/properly load python code, Packages were
restructured to behave as python modules, and each Module is a sub-

Date:   Thu Jun 9 20:27:39 2016 -0400

Add Profile open support

Openning the profile correctly reconstructs the section list, but
doesn't load module information, as that stuff isn't even stored

- Add .ui for test modules
- Fix enabled moduleToolsWidget after deleting last section
- Fix minor save profile formmating issues

Date:   Thu Jun 9 17:09:43 2016 -0400

Add Module Select dialog
- Remove module combobox
- Flesh out Module class
- Add Package class

Date:   Thu Jun 9 10:00:26 2016 -0400

Add default Contexts
- Add Date Context
- Add Time Context
- Add 'None' Context
- Add Context widget swapping
- Fix splitter expansion and section tools width
- Create *.ui build script
- Fix .gitignore

Date:   Wed Jun 8 14:38:37 2016 -0400

Add Profile saving
- Add Profile saving
- Add XMLEscape function
- Add Profile, Section and Module serialisation
- Add 'Profile:' and 'Report:' labels
- Add .gitignore

Date:   Wed Jun 8 10:38:59 2016 -0400

Create repo
__all__ = ['arcclasses', 'arcdocument', 'arcgui', 'arcpreferences']
__name__ = 'arc'
\ No newline at end of file
from contextlib import suppress
from .num2word import num2word
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import QDateTime, QDate, QTime, QObject, QEvent
from types import MethodType
from arctool import ARCTool
class Profile():
def __init__(self,path=None):
if path:
print("not ready yet")
return = "New Profile"
# Section Titles map to indicies in "sections"
self.sectionTitles = {}
self.sections = []
self.context = None
self.listWidget = None
def __len__(self):
return len(self.sections)
# Assign the QListWidget to keep track of
def setListWidget(self,widget):
self.listWidget = widget
lambda: ARCTool.signalProfileChanged()
# Update section mappings in "sectionTitles"
def setSectionTitle(self,oldTitle,title):
if oldTitle == title: return
index_ = self.sectionTitles[oldTitle]
# Assure no duplicates
if title in self.sectionTitles:
i = 1
dup = ''
while title+dup in self.sectionTitles:
dup = ' ' + num2word(i).capitalize()
i += 1
title += dup
self.sectionTitles[title] = index_
if self.listWidget is not None:
def getSectionCount(self):
return len(self.sections)
def getSections(self,copy=False):
return copy[:] if copy else copy
def getCurrentSection(self):
if len(self.sections) == 0:
return None
return self.sections[
def removeSection(self):
title = self.listWidget.currentItem().text()
index_ = self.listWidget.currentIndex().row()
for s in self.sectionTitles:
self.sectionTitles[s] -= 1 if self.sectionTitles[s] > index_ else 0
def moveSectionUp(self):
item = self.listWidget.currentItem()
row = self.listWidget.currentIndex().row()
self.listWidget.insertItem(row - 1,self.listWidget.takeItem(row))
return True
def moveSectionDown(self):
item = self.listWidget.currentItem()
row = self.listWidget.currentIndex().row()
self.listWidget.insertItem(row + 1,self.listWidget.takeItem(row))
return True
def moveSectionTo(self,src,dst):
self.sections.insert(self.sections.pop(src),dst + 1)
def addSection(self,title="Untitled Section"):
# Assure no duplicates
if title in self.sectionTitles:
i = 1
dup = ''
while title+dup in self.sectionTitles:
dup = ' ' + num2word(i).capitalize()
i += 1
title += dup
# Map section title to index in "sections"
self.sectionTitles[title] = len(self.sections)
# Add section to widget
if self.listWidget is not None:
return self.sections[-1]
def getName(self):
def setName(self,name): = name
def getTOC(self):
toc = ''
sections = []
for i in range(len(self.sections)):
title = self.listWidget.item(i).text()
toc += "\t<section title=\"%s\" pos=%d/>\n" %(title,i)
toc = "<contents>\n%s</contents>\n" %(toc)
return (toc, sections)
class Section():
def __init__(self,title="Untitled Section"):
self.title = title
self.plugin = None
self.pluginName = "None"
self.content = None
self.showTitle = True
def setTitle(self,title):
self.title = title
def getTitle(self):
return self.title
def setShowTitle(self,show):
self.showTitle = show
def updateContent(self):
if self.plugin is None:
self.content = self.plugin.generate()
def getContent(self):
return self.content
def hasPlugin(self):
return (self.plugin is not None)
def setPlugin(self,plugin):
self.plugin = plugin
self.pluginName = self.plugin.getName()
return self.plugin
def serialise(self):
if self.plugin is None:
return "<section title=\"%s\" plugin=\"None\" showTitle=%d/>\n"\
%(self.title,1 if self.showTitle else 0)
return ("<section title=\"%s\" plugin=\"%s\" showTitle=%d>\n\t%s</sec"\
"tion>\n") %(self.title, self.plugin.__name__,
1 if self.showTitle else 0, self.plugin.serialise())
class Package():
def __init__(self,path):
self.path = path
import os, importlib.machinery
self.plugins = []
self.pluginNames = [] = None
self.dependencies = []
self.sources = []
loader = importlib.machinery.SourceFileLoader(
package = loader.load_module()
self.pluginNames = package.__all__ =
self.version = package.version
self.authors = package.authors
self.preferenceDict = package.preferenceDict
def __lt__(self,a):
return <
def addPlugin(self,plugin):
def setVersion(self,version):
self.version = [int(n) for n in version.split('.')]
def setName(self,name): = name
def getName(self):
def addAuthor(self,author):
def setDate(self,date): = date
def addDependency(self,dependancy):
def addSource(self,source):
def getPluginNames(self):
return self.pluginNames[:]
def newPlugin(self,name,noUi=False):
import os, importlib.machinery#, re
path = os.path.join(self.path, name)
loader = importlib.machinery.SourceFileLoader(
name + "MOD",path + ".py"
# try:
plugin = loader.load_module()
m = plugin.Plugin(self)
if not noUi:
m.loadWidget(path + ".ui")
if m.setupUi():
return m
# except:
# print("Unable to load plugin %s" %path)
# return None
class Plugin():
def __init__(self,name="None",package=None): = name
self.package = package
self.netReq = False
self.subCount = 0
self.contexts = []
self.version = (0,0,0)
self.widget = None #set from .ui file?
self.authors = []
self.description = "No Description"
self.options = {} # ui_name : (value, property_name)
self.extras = {} # key : "value"
self.preferenceDict = None
def setName(self,name): = name
def setVersion(self,version):
self.version = [int(n) for n in version.split('.')]
def addAuthor(self,author):
def addContext(self,context):
def setDescription(self,description):
self.description = description
def getName(self):
def getVersion(self):
return "%d.%d.%d" %(self.version[0],self.version[1],self.version[2])
def getAuthors(self):
return ' '.join(self.authors)
def getDescription(self):
return self.description
def getPreferenceDict(self):
return self.preferenceDict or {}
def loadWidget(self,path):
from PyQt5 import uic
self.widget = uic.loadUi(path)
# Attach signals to notify the Tool that the profile has changed.
# If you want to use custom widgets that don't use these
# signals, you'll have to hook this up yourself.
sig = ARCTool.signalProfileChanged()
for c in self.widget.__dict__.values():
with suppress(Exception):
with suppress(Exception):
with suppress(Exception):
with suppress(Exception):
with suppress(Exception):
with suppress(Exception):
with suppress(Exception):
def loadCode(self,path):
import importlib
def hasWidget(self):
return self.widget is not None
def getWidget(self):
#should return QWidget
return self.widget
# Generate formatted text
def generate(self):
print("No overridden generate method")
def setOption(self,key,value):
self.options[key] = value
def setExtra(self,key,value):
# There are some type issues here
self.extras[key] = value
# print("set %s to %s" %(key, value))
# Update user interface elements from options
def update(self):
if self.widget:
for k in self.options:
qw = self.widget.findChild(QObject,k)
if qw:
qw.setProperty(self.options[k][1], self.options[k][0])
# Store the current options
def serialise(self):
# what if the plugin didn't load? should check somewhere
return "<plugin name=\"%s\" package=\"%s\">\n\t%s%s</plugin>\n" \
%(self.__name__, self.package.getName(),PackOptions(self.options),
class Context():
def __init__(self,name="None"): = name
self.callback = lambda x : True
self.ui = None
self.widget = None
def inContext(self,content):
return self.callback(content)
def hasWidget(self):
return self.widget is not None
def getName(self):
def setOptions(self,opts):
if self.widget:
for k in opts:
qw = self.widget.findChild(QObject,k)
if qw:
class DateContext(Context):
def __init__(self):
super(DateContext, self).__init__("Date")
dateTime = QDateTime.currentDateTime()
self.begin =
self.end =
self.hasBegin = False
self.hasEnd = False
# ISO dates are YYYY-MM-DD
from ui.datecontextwidget import Ui_DateContextWidget
self.ui = Ui_DateContextWidget()
self.widget = QtWidgets.QWidget()
def getWidget(self):
return self.widget
def setBegin(self,date):
self.begin = date
def setEnd(self,date):
self.end = date
def updateBegin(self):
state = self.ui.checkBegin.isChecked()
self.hasBegin = state
self.begin =
if state:
def updateEnd(self):
state = self.ui.checkEnd.isChecked()
self.hasEnd = state
self.end =
if state:
def callback(self,content):
# If the content got this far, we'd better hope it meshes with this
# context. Content in this Context should implement "getDate"
# if content.getDate is None
valid = True
date = content.getDate()
if self.hasBegin:
valid = date >= self.begin and valid
if self.hasEnd:
valid = date <= self.end and valid
return valid
# Access
def getBegin(self,format=None):
if self.hasBegin:
return self.begin if not format else self.begin.toString(format)
def getEnd(self,format=None):
if self.hasEnd:
return self.end if not format else self.end.toString(format)
def getMonth(self,format):
cd = QDate.currentDate()
return {
"name" : cd.toString("MMMM"),
"long" : cd.toString("MMMM"),
"abreviation" : cd.toString("MMM"),
"short" : cd.toString("MMM"),
"number" : cd.toString("M"),
def getYear(self,format):
cd = QDate.currentDate()
return {
"long" : cd.toString("yyyy"),
"short" : cd.toString("yy"),
def getDay(self,format):
cd = QDate.currentDate()
return {
"name" : cd.toString("dddd"),
"long" : cd.toString("dddd"),
"abreviation" : cd.toString("ddd"),
"short" : cd.toString("ddd"),
"number" : cd.toString("d"),
def toString(self,format):
return QDate.currentDate().toString(format)
class TimeContext(Context):
def __init__(self):
super(TimeContext, self).__init__("Time")
dateTime = QDateTime.currentDateTime()
self.begin = dateTime.time()
self.end = dateTime.time()
from ui.timecontextwidget import Ui_TimeContextWidget
self.ui = Ui_TimeContextWidget()
self.widget = QtWidgets.QWidget()
def getWidget(self):
return self.widget
def setBegin(self,time):
self.begin = time
def setEnd(self,time):
self.end = time
def updateBegin(self):
state = self.ui.checkBegin.isChecked()
self.hasBegin = state
if state:
def updateEnd(self):
state = self.ui.checkEnd.isChecked()
self.hasEnd = state
if state:
def callback(self,content):
#If the content got this far, we'd better hope it meshes with this context
# Content in this Context should implement "getTime"
# if content.getTime is None
valid = True
time = content.getTime()
if self.hasBegin:
valid = time >= self.begin and valid
if self.hasEnd:
valid = time <= self.end and valid
return valid
def PackOptions(opts):
pack = ''
for k in opts:
pack += "<opt key=\"%s\" value=\"%s\" property=\"%s\" type=\"%s\"/>\n"\
return pack
def packExtras(extras):
pack = ''
for k in extras:
pack += "<extra key=\"%s\" value=\"%s\"/>\n"\
return pack
\ No newline at end of file
from PyQt5.QtGui import *
from PyQt5.QtCore import *
from PyQt5.QtWidgets import *
from PyQt5.QtPrintSupport import *
PANDOC = False
import pypandoc
except ImportError:
class Document(QTextDocument):