Python Src

src/

Contains python scripts for the app.

src/admin.py

Handles admin related page requests and tasks.

Function	Description
class	Defines admin page view (list and detail), layout and field grouping. Seehere.
apache(), aws(), ga(), git()	Handles page requests of /admin/apache/, /admin/aws/, /admin/ga/, /admin/git/.
get_dash()	Retrieves analytics data for requests /admin/dash/(apache
get_stat()	Retrieves statistics for requests /admin/stat/(ver
backup_form()	Get backup related status from cron jobs.
admin_cmd()	Kicks off call_command() for management commands from requests /admin/cmd/(upload
refresh_stat()	Kicks off call_command() for management command from requests /admin/stat/(sys
	versions with flag 1 to override IS_VERSION in config/bot.conf;
	cache with flag 3 or 15 or 30 specified;
	backup statistics by calling get_backup_stat() from src/console.py.
backup(), bot()	Handles page requests of /admin/backup/ and /admin/bot/ pages. Also takes POST request to set configurations.
export()	Handles page request of /admin/export/. Also takes POST request to execute publication export to file.
dir(), man(), ref()	Handles page requests of /admin/dir/, /admin/man/ and /admin/ref/ for django-filemanager and documentation pages.

src/auth.py

Handles admin auto-login when using WebAuth, and EMAIL settings.

Function	Description
USER_GROUP	Parses config/group.conf and is used by group_dash() in src/admin.py.
AutomaticAdminLoginMiddleware	Automatically login daslab user for people listed in GROUP.ADMIN() when using WebAuth in production.
ExceptionUserInfoMiddleware	Mirrors user sunet_id and email into request.META, which is useful for tracking down who encountered a problem from Apache log.

src/console.py

Handles admin analytics console/dashboard data requests, error emails and Slack messages.

Function	Description
send_notify_slack(), send_error_slack(), send_notify_emails()	Uses Slacker API or smtp email to send messages to admin about error reports. Used by cron jobs since the Django send_mail() function fails randomly.
find_slack_id()	Finds match of group Member Slack ID and sunet_id. It matches the first name of the target user to records in Slack, and infers sunet_id by stripping the email address of matching Slack ID.
get_backup_stat()	Reads sizes of folders in data/ and sizes of files in backup/, and a list of TGZ files stored in Google Drive. Text information is saved to cache/stat_backup.json.
refresh_settings()	Reloads config/*.conf files into current running environment. It writes to settings._wrapped to overwrite current Django app setting.
get_sys_crontab(), set_sys_crontab()	GET/SET cron jobs. GET reads from system crontab -l instead of config/cron.conf to reflect the current setting.
get_backup_form(), set_backup_form()	GET/SET backup settings. GET reads from system get_sys_crontab(); SET writes to config/cron.conf.
get_bot_form(), set_bot_form()	GET/SET Bot settings. Reads and writes to config/bot.conf.
restyle_apache()	Parses Apache status page (/server-status/) into a JSON object. Uses urllib2 for credential supply to /server-status/.
aws_stats()	Handles /admin/aws/ data query. Uses gviz-api.py and boto. Calls aws_result() through aws_call() for data formatting. qs=init returns JSON, while others return Google Visualization ToJSONResponse() object. Query format:
	qs : Query string; accepts init (table of basic specs on top of page), latency, request, 23xx (HTTP response 2xx and 3xx combined), 45xx (HTTP response 4xx and 5xx combined), host, status, cpu, credit, and volbytes.
	sp : Period; accepts 7d (7 days) or 48h (48 hours).
	tqx : Required by Google Visualization API. It should has the value like reqId:%d.
ga_stats()	Handles /admin/ga/ data query. Uses gviz-api.py. Uses requests instead of Google Analytics OAuth API for simplicity. qs=init returns JSON, while others return Google Visualization ToJSONResponse() object. Makes 3 attempts if connection fails. Query format:
	qs : Query string; accepts init (table of number ratings), chart (session plots), pie (categorical pie plots), and geo (world map of visitors).
	sp : Period or type; qs=chart accepts 24h, 7d, 1m (1 month), or 3m (3 months), while qs=pie accepts session, user, browser, or pageview.
	tqx : Same as in aws_stats().
git_stats()	Handles /admin/git/ data query. Uses gviz-api.py and PyGithub. qs=init returns JSON, while others return Google Visualization ToJSONResponse() object. Makes 3 attempts if connection fails. Query format:
	qs : Query string; accepts init (table of contributors data), num (repository statistics, e.g. number of branches), c (commits calendar), ad (addition/deletion timeline), and au (author contribution pie plot).
	tqx : Same as in aws_stats().
export_citation()	Handles publication export. Uses pandoc for creating DOCX files from HTML.

src/dash.py

Handles group analytics of servers/services data requests.

Function	Description
cache_aws(), dash_aws()	Saves to or loads from cache/ for AWS CloudWatch data. Mechanism is similar to aws_stats() in src/console.py. Query format:
	qs: Query string; accepts init (GET list of all EC2, ELB, and EBS and their basic information), lat (latency, ELB only), req (request, ELB only), net (network I/O, EC2 only), cpu (cpu credit, EC2 only), and disk (disk I/O, EBS only).
	id: Instance ID. Used for qs!=init.
	tp: Instance type. Used for qs!=init. Accepts ec2, elb, or ebs.
cache_ga(), dash_ga()	Saves to or loads from cache/ for Google Analytics data. Mechanism is similar to ga_stats() in src/console.py. Makes 3 attempts if connection fails. Query format:
	qs: Query string; accepts init (GET list of all Google Analytics tracker IDs and their basic information), sessions, and percentNewSessions.
	id: Instance ID. Used for qs!=init.
cache_git(), dash_git()	Saves to or loads from cache/ for GitHub data. Mechanism is similar to git_stats() in src/console.py. Makes 3 attempts if connection fails. Query format:
	qs: Query string; accepts init (GET list of all GitHub repositories and their table of contributors), num (repository statistics, e.g. number of branches), c (commits calendar), and ad (addition/deletion timeline).
	repo: Repository name. Used for qs!=init. Contains name only, excluding organization.
	org: Organization name. Used for qs!=init.
cache_slack(), dash_slack()	Saves to or loads from cache/ for Slack data. Uses Slacker. Query format:
	qs: Query string; accepts users (list of members and basic information), home (list of members presence status), channels, and files.
cache_dropbox(), dash_dropbox()	Saves to or loads from cache/ for Dropbox data. Uses dropbox client. Query format:
	qs: Query string; accepts sizes (account quota and usage), folders (list of folders sizes and shares), and history (recent activity plot).
cache_schedule(), dash_schedule(), cache_duty(), dash_duty()	Saves to or loads from cache/ for Schedule Spreadsheet and Duty Spreadsheet from Google Drive. Parses the data into JSON format. Makes 3 attempts if connection fails. Checks [THIS WEEK] in the CSV file to locate the row of current week.
cache_cal(), dash_cal()	Saves to or loads from cache/ for Group Calendar. Uses iCalendar to parse the ICS file into JSON format and colors events by type. Makes 3 attempts if connection fails.
dash_ssl()	Uses openssl to retrieve server SSL Certificate expiration date.

src/env.py

Handles *.conf files and HTML template path.

Function	Description
SYS_PATH	Collection of HTML template path, data folder path, /group/ sitemap, and color rotation. It inherits from Singleton. Group app HTML path are replaced wth actual filename for use (GROUP_PATH and in group_pages() in src/views.py).
reload_conf()	Loads config/*.conf files and save as runtime environment variables for the Django app.
error400(), error401(), error403(), error404(), error500(), error503()	Handlers for custom HTTP error response pages. Use ?status=false to suppress the HTTP error code (returns 200 instead).

src/models.py

Handles MySQL database table schema and in-page form models.

Function	Description
models.Model	Database table schema. See here.
	Public site : News, Member, and Publication.
	Group site : FlashSlide, JournalClub, EternaYoutube, RotationStudent, Presentation, DefensePoster, and SlackMessage.
forms.Form	Front-end in-page forms. See here.
	Admin site : BackupForm.
	Group site : UploadForm, ContactForm, and EmailForm.
	Login : LoginForm, PasswordForm.
email_form(), debug_flag(), ga_tracker()	Global context processors for email admin form to front-end pages, convey DEBUG flag and Google Analytics tracker ID.
js_ver()	Global context processors for JavaScript vendor library versions. The versions information is used to load libraries from CDN.

src/settings.py

Handles basic app settings (defined by Django). See here. Loading order is to avoid import loop.

• X_FRAME_OPTIONS is not set to DENY due to /admin/dir/ uses <iframe> of /admin/browse/.

src/urls.py

Handles url routing (defined by Django). Uses AdminSitePlus to override Admin. See django-adminplus here.

• Routes /site_media/ to media/, and /site_data/ to data/; to prevent direct access to files from path.
• Redirect old and synonym urls to current ones with 301. Exposes custom error handler pages, as well as robots.txt as /robots.txt.
• Based on IS_MAINTENANCE in config/t47_dev.py, the server can run in either normal mode or downtime mode. In downtime mode, all requests are responded with HTTP code 503.

src/user.py

Handles requests for user-related activity. Each handler is routed from src/urls.py, and takes PATH.HTML_PATH to locate HTML templates.

Function	Description
user_sunetid()	Retrieves user's sunet_id from HTTP header WEBAUTH_USER.
user_login(), user_password(), user_logout()	Pages related to account. Only used in test server due to WebAuth. In production, the only user in system is daslab.
user_contact(), user_email(), user_upload()	Handles user AJAX POST requests. Returns JSON response.
browse()	django-filemanger view handler, channeled as <iframe> on /admin/dir/.

src/views.py

Handles requests for non-admin pages. Each handler is routed from src/urls.py, and mostly takes PATH.HTML_PATH to locate HTML templates.

Function	Description
index(), pages()	Handles public site requests. Prepares list of all objects from appropriate table in MySQL, with trimming of images or pdf links. For Publications, the start of each year is marked; and the watershed of "previous work".
group_index(), group_pages()	Handles /group/* requests. No longer uses @login_required due to WebAuth gating. Database objects list is same as above.
group_dash()	Group server/service page data query relaying, handles requests from /group/dash/(aws
ping_test()	Returns an empty 200 response. Used by AWS ELB Health Check.
test()	Test handler for test server. Not accessible in production.
get_staff()	Returns admin email and current user; Used by 403 error page (/error/403/).

src/wsgi.py

Handles uwsgi connections. Do not change, unless syntax changes in newer Django releases. Called by Apache mod_wsgi dispatcher.